Survey Frame Split Sample¶
Experimental design — instantiates Framing Effect Audit
Randomly assigns respondents to alternate framings of the same question so the response difference between frames can be estimated cleanly, segment by segment.
Survey Frame Split Sample is a between-subjects design: the respondent pool is randomly divided, each arm sees a different framing of the same underlying question, and the difference in responses is estimated across arms. Its defining move is random assignment of different people to different frames — because no respondent sees more than one framing, the frame effect is measured as a clean between-groups contrast rather than inferred from a single person reacting to variants. Randomization also makes segment-level estimates trustworthy, so the design can report not just an average frame effect but how it differs across audiences.
Example¶
A polling organization wants to know whether public support for a new public-transit program depends on how the funding is described. It fields one questionnaire in four randomly assigned arms: the identical program is called a "transit tax," a "transit investment," a "transit benefit," and a "transit fee." Because respondents are assigned at random, the arms are statistically equivalent going in, so a gap in stated support between "tax" and "investment" is attributable to the label, not to who happened to answer.[1] The split also lets the pollster estimate the effect by segment: the "tax" penalty is large among one partisan group and negligible among another. The finding is not "support is 58%" but "support ranges from 49% to 64% depending on the word, and the sensitivity is concentrated in one segment" — which tells the client that a single headline number would have masqueraded as stable preference.
How it works¶
- Prepare the framed arms. Take the same underlying question and produce the alternate framings to be tested, one per arm, differing only in the frame variable of interest.
- Randomize assignment. Allocate respondents to arms at random so the groups are equivalent in expectation and the between-arm difference isolates the frame.
- Estimate the contrast, overall and by segment. Compare responses across arms to estimate the frame effect, and break the estimate out by the audience segments defined in advance.
The distinguishing discipline is that it assigns different people to different frames and estimates a between-groups effect — a measurement design, not a re-run of one judgment under reordering, and not a ledger of decisions.
Tuning parameters¶
- Number of arms — how many framings you field. More arms map the frame space but split the sample and shrink per-arm precision.
- Allocation balance — equal arms or weighting toward the contrasts that matter. Balanced is simplest; weighted buys precision where it counts.
- Segmentation depth — how finely you pre-define audience segments for the breakout. Finer segments reveal heterogeneous effects but demand larger samples to estimate them.
- Sample size per arm — how much statistical power each contrast gets. Bigger arms detect smaller frame effects but cost fielding budget.
When it helps, and when it misleads¶
Its strength is a clean causal estimate of a frame effect and its spread across audiences — the rigorous option when a survey, ballot, or message will be treated as evidence of real preference and the stakes justify fielding an experiment.
It can mislead when the arms are not truly equivalent (a "fee" version that also changes the amount is a content difference, not a framing) or when segments are defined after seeing results to manufacture a story. Underpowered arms also invite reading noise as a frame effect. The discipline that keeps it honest is to pre-register the frame variable and segments, confirm the arms differ only in framing, and report uncertainty rather than a point gap. This differs from a Wording Test, which vets a single question's phrasing for a buried premise before it is sent; the split sample instead measures how alternate framings move responses across a randomized sample.
How it implements the components¶
Survey Frame Split Sample realizes the between-subjects measurement side of the archetype:
alternate_presentation_set— the framed arms are the alternate presentations, one per randomized group, differing only in the frame variable.response_comparison_design— random assignment plus between-arm contrast is the formal design that estimates how responses change across frames.audience_segment_map— randomization lets it break the frame effect out by pre-defined audience segments, reporting heterogeneity rather than only an average.
It does not set the materiality bar for whether an estimated gap matters (frame_effect_threshold) or produce the held-constant/differs ledger that certifies the arms are truly equivalent (equivalence_or_difference_record) — sizing the shift against a threshold and recording same-content equivalence are Order-Effect Check. The separation is that a split sample sends *different people to different frames to estimate an effect, whereas an order-effect check re-runs the same judges over the same content in a new sequence.*
Related¶
- Instantiates: Framing Effect Audit — Survey Frame Split Sample is the audit's rigorous between-subjects estimate of a frame effect.
- Sibling mechanisms: Order-Effect Check · Baseline Reference Swap · Blinded Frame Review · Dashboard Framing Review · Gain/Loss Frame Comparison · Presentation Sensitivity Table · Visual Framing Audit · Wording Test
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Survey Frame Split Sample is defined in the frozen evidence as: Randomly assigns respondents to alternate framings of the same question so the response difference between frames can be estimated cleanly, segment by segment. Its operative deployed or enacted form is therefore Experiment, Test & Rehearsal.
Nearest alternative: Analysis, Modeling & Optimization — Analysis, Modeling & Optimization can support this mechanism, but the evidence centers the concrete operation described above rather than the alternative family's defining operation.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Splitting a sampling frame into randomized survey forms is an experimental survey-design method.
Related originating lineages:
- Communication & Media Studies — Communication and media research supplies a parallel or contributing lineage for the mechanism's defining operation: randomly assigns respondents to alternate framings of the same question so the response difference between frames can be estimated cleanly, segment by segment.
- Data Science & Analytics — Data science, analytics, and operational monitoring supplies a parallel or contributing lineage for the mechanism's defining operation: randomly assigns respondents to alternate framings of the same question so the response difference between frames can be estimated cleanly, segment by segment.
- Mathematics — Mathematical modeling, proof, and abstract-structure practice supplies a parallel or contributing lineage for the mechanism's defining operation: randomly assigns respondents to alternate framings of the same question so the response difference between frames can be estimated cleanly, segment by segment.
- Psychology — Question wording and order effects motivate split-ballot tests.
Review resolution: The blind reviewers agree that statistics_experimental_design is the primary origin and differ only on alternate origin disagreement, domain reach disagreement, encyclopedia synthesis disagreement. I preserve every independently explained alternate from both records rather than imposing a numeric cap. I retain single_lineage because the combined evidence shows one traceable formative lineage. The broader reach of multi_domain records portability separately from historical provenance; encyclopedia_synthesis=true preserves the affirmative synthesis judgment where either reviewer identified one.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
References¶
[1] Schuman, H., & Presser, S. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. Academic Press (1981). Uses random split-ballot assignment to isolate question-wording effects from differences in who answered. registry ↩