Order-Effect Check¶
Perturbation test — instantiates Framing Effect Audit
Reorders the same questions, options, or evidence to test whether sequence alone — not content — moves the conclusion far enough to matter.
Order-Effect Check treats sequence as the frame variable. It takes a fixed set of items — survey questions, menu options, pieces of evidence, competitors in a lineup — and re-runs the judgment under deliberately different orderings, holding the content of every item identical. Its defining move is that only the arrangement changes, so any shift in the outcome is attributable to position (primacy, recency, anchoring by whatever came first) rather than to what was said. Because the items themselves are unchanged across orderings, the check carries an explicit record that the content is constant and only sequence differs, and it asks whether the position-driven shift is large enough to matter.
Example¶
A national culinary competition scores twelve chefs, each plating the same dish for a panel over one long afternoon. The organizers suspect that when a chef presents is quietly shaping their score. Order-Effect Check re-derives the ranking under alternative running orders — reversing the sequence, and counterbalancing so no chef is always early or late — while every dish, every judge, and every scoring rubric stays exactly the same. Comparing the orderings surfaces a recency pull: chefs plating in the last third score consistently higher, and one mid-pack chef would have medaled had they cooked later.[n1] The check records that nothing about the dishes changed — only the slot — and measures the average position swing against the threshold the panel set for a fair result. Because the swing clears it, the organizers adopt counterbalanced slots and judge palate-cleanser breaks, treating running order as a frame to be neutralized rather than an incidental logistic.
How it works¶
- Fix the items, vary the sequence. Hold every item's content constant and generate alternative orderings — reversed, rotated, or counterbalanced — as the alternate presentations.
- Re-run the judgment and register the shift. Observe how the outcome (ranking, chosen option, rating) moves across orderings, informally comparing the runs to locate position-driven change.
- Record constancy and test materiality. Note explicitly that content is held identical and only order differs, then check whether the position swing crosses the bar for a consequential shift.
The distinguishing discipline is that it perturbs sequence and certifies that nothing but sequence moved — it does not randomly assign different people to different frames to estimate an effect size, and it does not break results out by audience segment.
Tuning parameters¶
- Ordering scheme — simple reversal, full counterbalancing, or random rotation. Fuller schemes isolate position more cleanly but multiply the runs required.
- Perturbation grain — whether you reorder whole blocks or individual items. Fine grain catches item-level anchoring; coarse grain catches primacy/recency across sections.
- Materiality threshold — how large a position swing counts as consequential. Strict bars matter where a ranking decides awards or funding; loose bars tolerate small drifts in low-stakes ordering.
- Constancy strictness — how carefully you verify that only order changed. Strict checking prevents mistaking a content edit for an order effect; lax checking risks a false attribution.
When it helps, and when it misleads¶
Its strength is catching a bias that hides in plain sight — position — in any process where items are judged in a row: survey batteries, evaluation packets, ballots, competition lineups. Because content is held identical, a shift is hard to dismiss as anything but sequence.
It can mislead if the "reordering" also changes content (a reworded question, a different exemplar) so that a content effect is misread as an order effect; the equivalence record is the guard. It can also over-fire on trivial drifts in low-stakes sequences. The discipline that keeps it honest is to verify strict item constancy across orderings and to gate on a materiality threshold so only consequential position swings drive a change.
How it implements the components¶
Order-Effect Check realizes the sequence-perturbation side of the archetype:
alternate_presentation_set— the reordered runs are the alternate presentations, differing only in the sequence of otherwise-identical items.frame_effect_threshold— it sizes the position-driven swing and tests it against a bar for a consequential shift.equivalence_or_difference_record— it records that item content is held constant and only order differs, guarding against reading a content change as an order effect.
It does not assign respondents to frames and estimate a between-groups effect with a response_comparison_design, nor break the result out with an audience_segment_map — random assignment and segmented estimation are Survey Frame Split Sample. The separation is that an order-effect check re-runs the *same judges over the same content in a new sequence, whereas a split sample sends different people to different frames to estimate the effect.*
Related¶
- Instantiates: Framing Effect Audit — Order-Effect Check is the audit's probe for sequence as a hidden frame variable.
- Sibling mechanisms: Survey Frame Split Sample · Baseline Reference Swap · Blinded Frame Review · Dashboard Framing Review · Gain/Loss Frame Comparison · Presentation Sensitivity Table · Visual Framing Audit · Wording Test
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Order-Effect Check operates as an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation because it reorders the same questions, options, or evidence to test whether sequence alone — not content — moves the conclusion far enough to matter.
Independent corroboration: The frozen evidence defines Order-Effect Check as 'Reorders the same questions, options, or evidence to test whether sequence alone — not content — moves the conclusion far enough to matter', so its operative form is Experiment, Test & Rehearsal.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Psychology
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Order-Effect Check is most directly rooted in psychology and behavioral science's experimental study of judgment, learning, motivation, and behavior. The lineage fits its defining practice: Reorders the same questions, options, or evidence to test whether sequence alone — not content — moves the conclusion far enough to matter.
Related originating lineages:
- Statistics & Experimental Design — Order-Effect Check also draws materially on experimental design and statistics' methods for comparison, uncertainty, sampling, sensitivity, and inferential validation, which shaped this mechanism rather than merely adopting it as an application.
Review outcome: Independent reviewer agreement; high confidence.
Notes¶
[n1] The serial-position effect — items early (primacy) or late (recency) in a sequence are weighted differently from those in the middle, so the order in which options or evidence are encountered can shift a judgment even when the items themselves are unchanged. ↩