Transformation-Pair Test Suite¶
Test harness — instantiates Symmetry-Commuting Transformation Design
Turns the commutation claim into repeatable, executable tests that compare the output of a transformed input against the correspondingly transformed baseline output, case by case.
A drawn contract is a promise; a Transformation-Pair Test Suite is the machine that keeps it honest. For each transformation in scope it holds a curated pair of inputs — x and g_input(x) — runs both through the real map F, and applies the paired output action g_output to the baseline, then asserts that F(g_input(x)) matches g_output(F(x)). The defining move is that it compares two computed outputs against each other rather than either against a hand-written gold answer: the expected value is manufactured on the fly by transforming the baseline output, which is exactly what lets it test maps whose correct answer nobody can write down in advance. It is deterministic and case-by-case — a named, versioned assertion per transformation — which is what separates it from a statistical probe.
Example¶
A radiology team maintains a model (F) that outputs a lesion segmentation mask from a chest scan. They want confidence that flipping or rotating a scan flips or rotates the mask correspondingly rather than smearing it. The suite holds a set of curated scans; for each it applies a 90-degree rotation (g_input) to the image, runs the model, and separately rotates the original mask by the same 90 degrees (g_output). The assertion is that the two masks agree within the pixel tolerance the team declared.
The first run fails on a specific family of cases: near the image border, rotating-then-segmenting and segmenting-then-rotating disagree because the model was trained to ignore a fixed margin, so the margin doesn't rotate with the anatomy. The suite pins the failure to border lesions under rotation — a reproducible, named test rather than a vague "sometimes wrong." It also includes an interpretation check: the mask consumer (a volume-measurement step) reads the rotated mask and confirms the measured lesion volume is unchanged, since a rigid rotation should move the mask but not its area.
How it works¶
- Materialize paired cases. For each transformation, store
x, its transformg_input(x), and the output actiong_outputneeded to build the expected result. Include ordinary cases, hand-picked counterexamples, and edge cases. - Run both paths through the real map. Compute
F(g_input(x))andg_output(F(x))using the actual system, not a stub — the point is to catch whatFreally does. - Assert agreement through an oracle. The comparison, with its pass/fail rule, is the test oracle. Because the "expected" side is itself computed, this is a relational assertion, not an equality against a fixed key.
- Check the consumer, not just the value. Where a downstream step interprets the output, run it on the transformed result and confirm the interpretation survives the transformation.
Tuning parameters¶
- Case-set composition — how many ordinary vs adversarial vs boundary cases the suite carries. Adversarial cases catch more but cost curation and can make a healthy map look brittle.
- Transformation composition depth — whether you test single transforms or compositions (rotate and flip). Composed tests catch interaction bugs but multiply the case count.
- Failure localization — whether a failing assertion reports just pass/fail or the pixel/field-level residual. Richer reports speed debugging but slow the suite.
- Refresh trigger — whether the suite runs on every model change or on a schedule. Per-change runs catch regressions immediately; scheduled runs are cheaper on large maps.
When it helps, and when it misleads¶
Its strength is that it converts an abstract contract into regression-proof evidence: once a border-lesion failure is a named test, it cannot silently return. It is the natural home for metamorphic testing — the practice of checking a program through relations between related inputs and outputs when no single correct output can be specified, which is precisely the situation equivariance creates.[n1]
Its failure mode is coverage illusion. A suite only tests the transformations and cases someone thought to write; a map can pass every stored pair and still fail on a transformation not in the set or a case not curated, so a green suite reads as "verified" when it means "not yet falsified on these." The classic misuse is padding the suite with easy, central cases that always pass, which inflates confidence without probing the boundary. The guarding discipline is to keep the case set adversarial and to treat the tolerance and the set of transformations tested as external inputs — declared elsewhere and consumed here — so the suite cannot quietly loosen its own standards to stay green.
How it implements the components¶
Transformation-Pair Test Suite fills the execution-and-evidence side of the archetype:
input_output_action_pair— each stored case operationalizes ag_input/g_outputpair by actually applying both actions to build the two sides of the comparison.equivariance_test_oracle— the pass/fail comparison of the two computed paths is the oracle; its relational form is the mechanism's signature.downstream_interpretation_check— the suite runs a consuming step on the transformed output and asserts the interpretation is preserved, not just the raw value.
It does not decide how close is close enough or which transformations must be covered — the approximation_tolerance_band and the sampling of boundary_of_valid_transformations are declared and probed by Data-Augmentation Equivariance Probe and the tolerance matrix; this suite is deterministic where the probe is statistical.
Related¶
- Instantiates: Symmetry-Commuting Transformation Design — this suite is the evidence engine that validates the commutation relation.
- Consumes: Commutative Diagram Review supplies the commutation contract the suite turns into cases; Equivariance Tolerance Matrix supplies the pass/fail tolerances.
- Sibling mechanisms: Commutative Diagram Review · Coordinate-Frame Consistency Check · Permutation Equivariance Audit · Data-Augmentation Equivariance Probe · Schema and Label Relabeling Harness · Equivariance Tolerance Matrix · Symmetry Exception Register
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Transformation-Pair Test Suite operates as an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation because it turns the commutation claim into repeatable, executable tests that compare the output of a transformed input against the correspondingly transformed baseline output, case by case.
Independent corroboration: The frozen evidence defines Transformation-Pair Test Suite as 'Turns the commutation claim into repeatable, executable tests that compare the output of a transformed input against the correspondingly transformed baseline output, case by case', so its operative form is Experiment, Test & Rehearsal.
Nearest alternative: Assessment, Review & Assurance — Transformation-Pair Test Suite includes features of a bounded evaluation of existing evidence or work that produces a finding or disposition, but its defining operation is an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Convergent development
Present-day reach: Universal
Rationale: Chen et al., Metamorphic Testing: A New Approach for Generating Next Test Cases tests expected relations between paired source and transformed executions when a direct test oracle is unavailable. This directly supports computer science as the best-evidenced historical home of the operation—Turns the commutation claim into repeatable, executable tests that compare the output of a transformed input against the correspondingly transformed baseline output, case by case.—while the alternates record adjacent lineages rather than mere domains of later use.
Related originating lineages:
- Engineering & Design — Engineering design, reliability, and systems-safety practice supplies a parallel or contributing lineage for the mechanism's defining operation: turns the commutation claim into repeatable, executable tests that compare the output of a transformed input against the correspondingly transformed baseline output, case by case.
- Mathematics — Mathematical modeling, proof, and abstract-structure practice supplies a parallel or contributing lineage for the mechanism's defining operation: turns the commutation claim into repeatable, executable tests that compare the output of a transformed input against the correspondingly transformed baseline output, case by case.
Review resolution: The blind reviewers disagree on primary lineage (mathematics versus computer_science). The defining operation is: Turns the commutation claim into repeatable, executable tests that compare the output of a transformed input against the correspondingly transformed baseline output, case by case. The researched Chen et al., Metamorphic Testing: A New Approach for Generating Next Test Cases tests expected relations between paired source and transformed executions when a direct test oracle is unavailable. That is mechanism-specific evidence for computer science as the historical origin. Mathematics remains represented among the uncapped alternates where it contributes a genuine formative practice, but broad deployment or governance of the operation is not by itself evidence that the mechanism originated there. origin_mode=convergent records lineage; domain_reach=universal separately records later applicability.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; medium confidence.
Sources consulted:
Notes¶
[n1] Metamorphic testing validates software through metamorphic relations — required relationships between the outputs of related inputs — rather than against a fixed expected output. Equivariance, F(g_input(x)) ≈ g_output(F(x)), is a metamorphic relation, which is why this suite fits programs (models, renderers, solvers) that have no oracle for any single input. ↩