Deviant Case Selection Protocol¶
Protocol — instantiates Deviant Case Analysis
Selects a case that violates an explicitly stated pattern while documenting the universe of comparison and selection rationale.
An exception is only informative if you can say what it is an exception to. The Deviant Case Selection Protocol is the front-end discipline that makes case choice principled rather than opportunistic: before any case is called deviant, it fixes the expected pattern in writing, defines the universe of cases that pattern is supposed to cover, and states the explicit criterion by which a case qualifies as a violation. Its defining idea is that a case is deviant against a documented background — the protocol's whole purpose is to prevent an analyst from lifting a striking case out of context and treating an undefined comparison as if it were an anomaly.
Example¶
A school district runs a value-added model that predicts each school's test-score growth from its intake demographics. One elementary school beats its predicted band by a wide, sustained margin, and staff are already circulating a hopeful story about its principal. The protocol slows the leap. It states the expected pattern — the model's predicted growth band for a school with that intake. It defines the universe — all district elementary schools, the same three-year window, the same model version, excluding any school with fewer than two years of data. It sets the deviation criterion — a standardized residual beyond roughly two standard deviations, sustained across at least two years rather than a single lucky cohort. Applied, the focal school qualifies; a second apparent outlier is excluded because it has only one year of data and fails the universe rule. The output is not a hunch but a selection dossier that says exactly which case is deviant, against what, and why the others are not.
How it works¶
- State the expected pattern. Write down the theory, rule, benchmark, or model prediction the case appears to violate, in falsifiable terms.
- Define the universe and boundary. Fix inclusion criteria, comparison period, and units of analysis, so "deviant" is measured against a bounded population rather than an implied one.
- Specify the deviation criterion. Name the type of deviation that qualifies — outcome mismatch, extreme residual, anomalous classification, surprising success or failure — and the threshold it must clear.
- Apply and document. Record why each case is included or excluded, so the selection can be audited rather than trusted.
Tuning parameters¶
- Universe breadth — a narrow, tightly-comparable pool versus a wide one. Narrow yields a cleaner notion of deviation but fewer eligible comparison cases downstream.
- Deviation threshold — how extreme a case must be to qualify. Strict thresholds admit fewer, more clearly anomalous cases and reduce false alarms.
- Selection-on-outcome guard — whether ordinary confirming cases are documented alongside the deviant. Sampling only the extremes is the deepest bias risk here.
- Documentation depth — a one-line rationale versus a full inclusion/exclusion log; deeper is auditable but heavier.
When it helps, and when it misleads¶
Its strength is that it fixes the frame before the case is chosen, which is what keeps deviant-case work from collapsing into anecdote: a documented universe is the difference between "this school is special" and "this school deviates from a stated prediction over a defined population." Its central hazard is selection on the dependent variable — choosing cases by their outcome and then generalizing from them without any variation on that outcome, which can manufacture a pattern that is not there.[n1] The classic misuse is gerrymandering the universe after the fact until a favored case looks anomalous. The guarding discipline is to freeze the pattern, universe, and criterion before scanning the cases, in the spirit of pre-registration, so the selection cannot be quietly bent to the conclusion.
How it implements the components¶
The protocol realizes the archetype's front-end — the components that must be pinned down before a deviation can even be named:
expected_pattern_frame— it states, in falsifiable terms, the theory, rule, or prediction the focal case violates.case_universe_and_selection_boundary— it defines the pool, inclusion criteria, period, and units against which deviation is measured.deviation_criterion— it specifies the type and threshold that make a case count as deviant.
It does not reconstruct the case's internal sequence (within_case_process_trace — the Within-Case Process-Tracing Memo), enumerate candidate causes (rival_explanation_inventory — the Omitted Variable Probe), or record the resulting change (theory_revision_register — the Boundary Condition Revision Workshop); it only supplies the well-defined case they act on.
Related¶
- Instantiates: Deviant Case Analysis — supplies the archetype's principled case-selection front end.
- Sibling mechanisms: Deviation Residual Table · Matched Case Pairing Matrix · Within-Case Process-Tracing Memo · Omitted Variable Probe · Boundary Condition Revision Workshop · Follow-Up Case Sampling Plan · Rival Explanation Elimination Table
Editorial Notes¶
Form Classification¶
Form family: Decision, Gate & Allocation
Rationale: Deviant Case Selection Protocol operates as a case-specific gate, selection, routing, prioritization, or resource disposition because it selects a case that violates an explicitly stated pattern while documenting the universe of comparison and selection rationale.
Independent corroboration: The frozen evidence defines Deviant Case Selection Protocol as 'Selects a case that violates an explicitly stated pattern while documenting the universe of comparison and selection rationale', so its operative form is Decision, Gate & Allocation.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Political Science
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Comparative political methodology most directly formalized pre-specified case universes, deviation criteria, and safeguards against selecting on the dependent variable when choosing anomalous cases for inference.
Related originating lineages:
- Ethnography & Qualitative Methods — Qualitative deviant-case analysis supplied intensive follow-up of observations that contradict an expected pattern.
- Statistics & Experimental Design — Statistical sampling and selection-bias theory supplied the inferential warning against choosing cases because their outcomes are already known.
Review resolution: Political science is primary because Geddes and KKV made explicit case-selection rules and selection-on-outcome warnings central to comparative research design. Qualitative deviant-case analysis and statistical selection-bias logic are independently formative components.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
- Barbara Geddes: How the Cases You Choose Affect the Answers You Get
- Gary King: Designing Social Inquiry
Notes¶
This protocol deliberately does select on the outcome — the point is to study a case because it deviated. That is legitimate precisely because the universe and criterion are documented first: the recorded background is what distinguishes principled deviant-case selection from cherry-picking. Strip the documentation and the same act becomes the bias the protocol exists to prevent.
[n1] Selection on the dependent variable — the methodological hazard (discussed at length by King, Keohane & Verba and by Barbara Geddes) of choosing cases according to the outcome you want to explain, which can bias or void causal inference when there is no variation on that outcome. Documenting the full universe is the standard corrective. ↩