Data-Augmentation Equivariance Probe¶
Empirical probe — instantiates Symmetry-Commuting Transformation Design
Feeds randomly transformed inputs sampled across the valid transformation range and measures the statistical distribution of how far outputs drift from the correspondingly transformed baseline.
Some equivariance holds only approximately, and the honest question is not "does it commute?" but "how far off, and how often?" A Data-Augmentation Equivariance Probe answers empirically. It draws transformations at random from a declared valid range — the same augmentation distribution a model is trained or stress-tested with — applies each to held-out inputs, runs the map, and measures the residual between F(g_input(x)) and g_output(F(x)) as a distribution across the sample. Its output is not a verdict but a spread: a mean equivariance error, a tail, a worst-case. The defining trait is that it is statistical and range-based — it characterizes approximate equivariance over a sampled continuum of transformations rather than asserting exact agreement on hand-picked pairs.
Example¶
A team runs a land-cover segmentation model (F) on satellite imagery. Aerial scenes have no canonical "up," so a well-behaved model should be rotation-equivariant: rotate the input tile and the segmentation should rotate the same way. But convolutional models are only approximately so, and the team needs to know how approximately.
The probe defines the valid range — rotations across the full 0–360 degrees, since orientation is genuinely arbitrary for overhead imagery — and samples hundreds of random angles. For each, it rotates a held-out tile, segments it, and compares against the segmentation of the original tile rotated by the same angle, scoring the overlap. The result is a distribution: agreement is high near multiples of 90 degrees and dips at oblique angles, with a heavy tail around 45 degrees where interpolation artifacts bite hardest. That profile — "median overlap high, but a fifth of oblique rotations fall below an acceptable overlap" — is the deliverable. It tells the team the model is approximately rotation-equivariant, quantifies where it is weakest, and sets a realistic tolerance for what downstream users can expect, rather than pretending exact equivariance the architecture cannot deliver.
How it works¶
- Declare the valid transformation range. Fix the distribution to sample from — which rotations, scales, or shifts count as legitimate, and their limits. Sampling outside the meaningful range would manufacture failures that don't matter.
- Sample and apply the paired action. Draw transformations at random; for each, transform the input and separately apply the corresponding output action to the baseline result.
- Measure the residual as a distribution. Score the gap for every sample and aggregate — mean, variance, quantiles, worst case — rather than collapsing to pass/fail.
- Report a tolerance band, not a verdict. Summarize the approximate equivariance as an empirical band with its tail, so consumers inherit a calibrated expectation.
Tuning parameters¶
- Sampling density and distribution — how many transformations and drawn from what law (uniform over the range, or concentrated where failure is suspected). Denser sampling sharpens the tail estimate but costs compute.
- Range limits — how wide the valid transformation range is set. Wider ranges test robustness more aggressively but risk including transformations no real input will ever see.
- Residual metric — what "drift" means (overlap score, pixel error, feature distance) and how the band is summarized. The choice changes which failures are visible.
- Aggregation focus — whether the headline is the mean or a high quantile. Mean flatters; a tail quantile governs the worst experience a consumer will hit.
When it helps, and when it misleads¶
Its strength is honesty about maps that are equivariant in spirit but not to the digit — it replaces a false binary with a calibrated band, exposes where in the transformation range the map is weakest, and gives downstream users a number they can plan around. It is the diagnostic complement to data augmentation, the training practice of adding transformed copies of examples to encourage a model toward the invariances one wants.[n1]
Its failure mode is the seductive average. A healthy-looking mean can hide a fat tail of severe failures at particular transformations, and a probe that samples too sparsely — or only near the easy angles — will miss them entirely. The classic misuse is treating training-time augmentation as a guarantee of equivariance and skipping the probe, or reporting only the mean drift and burying the tail. The guarding discipline is to sample densely enough to resolve the tail, report a high quantile alongside the mean, and treat the band as a measured property that must be re-probed whenever the model or the input distribution shifts.
How it implements the components¶
Data-Augmentation Equivariance Probe fills the statistical, approximate slice of the archetype:
input_output_action_pair— each sampled draw applies ag_inputand the matchingg_outputto construct the two sides whose residual is measured.boundary_of_valid_transformations— the declared sampling range is the boundary: it fixes which transformations are legitimate to test and where the range stops.approximation_tolerance_band— the empirical residual distribution, summarized as a band with its tail, is the probe's headline output.
It renders no deterministic pass/fail verdict on curated pairs — that exact, case-by-case equivariance_test_oracle belongs to Transformation-Pair Test Suite; this probe measures a spread where the suite asserts an equality.
Related¶
- Instantiates: Symmetry-Commuting Transformation Design — this probe supplies the approximate, statistical reading of the commutation relation.
- Consumes: Equivariance Tolerance Matrix supplies the declared exactness class the measured band is judged against.
- Sibling mechanisms: Commutative Diagram Review · Transformation-Pair Test Suite · Coordinate-Frame Consistency Check · Permutation Equivariance Audit · Schema and Label Relabeling Harness · Equivariance Tolerance Matrix · Symmetry Exception Register
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Data-Augmentation Equivariance Probe operates as a bounded trial, probe, simulation, or rehearsal that generates evidence from performance because it feeds randomly transformed inputs sampled across the valid transformation range and measures the statistical distribution of how far outputs drift from the correspondingly transformed baseline.
Independent corroboration: The frozen evidence defines Data-Augmentation Equivariance Probe as 'Feeds randomly transformed inputs sampled across the valid transformation range and measures the statistical distribution of how far outputs drift from the correspondingly transformed baseline', so its operative form is Experiment, Test & Rehearsal.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Data Science & Analytics
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Specialized
Rationale: Machine-learning evaluation cohered empirical probes that sample valid transformations, compare transformed outputs on held-out cases, and report the distribution of approximate equivariance error.
Related originating lineages:
- Mathematics — Group actions and equivariance supplied the commuting-transformation criterion against which the empirical residual is defined.
- Statistics & Experimental Design — Sampling and uncertainty analysis supplied range-based estimation of mean, tail, and worst-case residual behavior.
Review resolution: Machine-learning evaluation supplies the empirical augmentation probe, while mathematical equivariance defines the criterion and statistics supplies uncertainty summaries; the method remains specialized to transformation-aware models.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Data augmentation adds transformed copies of training examples (rotations, flips, crops) to push a model toward desired invariances and equivariances. It encourages the property statistically but does not guarantee it — which is exactly why an empirical probe is needed to measure how well the encouragement took. ↩