Skip to content

Deviant Case Follow-Up Protocol

Follow-up protocol — instantiates Structured Comparative Case Design

Governs what to do with a case that breaks the cross-case pattern — re-investigate it before deciding whether it is error, omission, or a genuine limit on the theory.

Deviant Case Follow-Up Protocol is triggered by an anomaly: a case the established cross-case pattern predicts should behave one way and that stubbornly behaves the other. Rather than let the analyst quietly explain it away or drop it, the protocol governs a disciplined re-investigation and forces the anomaly to be classified — is it a data error, a sign of an omitted variable, or a real boundary the theory has just hit? What makes it THIS mechanism is that it is reactive and diagnostic: it does not build patterns or add cases to confirm them; it exists to squeeze a specific outlier for what it reveals about the theory.

Example

A retail chain has a working model: store revenue tracks footfall, local income, and format, and it predicts new-store performance well across 200 locations. One store defies it — the model puts it near the bottom, yet it is among the strongest in the chain. The protocol takes over. First it re-checks the data (is the footfall sensor miscounting, is the catchment mis-drawn?) — the numbers hold. Then it sends someone to look. The visit surfaces an omitted variable: the store sits atop a commuter rail interchange whose weekday flow never appears in the residential footfall figure. The deviant is reclassified from "error" to "omitted factor," the model gains a transit-adjacency term, and the whole disposition — trigger, checks, field finding, resolution — is written into the audit trail so the next anomaly is judged by the same rule.

How it works

The protocol is a decision procedure with a pre-committed disposition rule. A case crossing a residual threshold is flagged; a graded follow-up runs (desk re-check of the data, then, if it survives, a deeper field investigation); and the anomaly is assigned one of three fates — measurement error to be corrected, an omitted variable to be added, or a genuine scope limit that bounds the theory. Its distinguishing feature is that every step, and especially the final disposition, is documented, so anomalies cannot be silently discarded to keep a tidy result.

Tuning parameters

  • Trigger threshold — how far a case must depart from prediction to be flagged. A loose threshold chases noise; a tight one lets informative anomalies slip through.
  • Follow-up depth — desk re-check only, or escalation to fieldwork. Deeper follow-up resolves more anomalies but costs real investigation.
  • Disposition rule — the pre-set logic for choosing among error, omission, and scope limit, and how much evidence each verdict requires.
  • Anomaly breadth — whether one deviant is pursued in isolation or a few similar deviants are sampled together to tell a one-off from a pattern.

When it helps, and when it misleads

Its strength is that anomalies are where theories improve — a disciplined follow-up turns an embarrassing outlier into an omitted variable or an honest scope condition, both of which strengthen the account. Its twin failure modes pull in opposite directions: with enough ingenuity any anomaly can be rescued by an ad-hoc patch, and with enough impatience any anomaly can be dismissed as "bad data."[1] The classic misuse is the quiet delete — dropping the deviant case so the pattern stays clean and never recording that it happened. The discipline that guards against both is to fix the disposition rule before seeing the anomaly and to document every deviant and its resolution, so the theory is revised in the open.

How it implements the components

  • negative_and_deviant_case_rule — its core: the rule defining what counts as a deviant or negative case and how each must be handled rather than ignored.
  • audit_trail_and_follow_up_sampling_plan — the documented re-investigation, plus any additional similar cases sampled to test whether the anomaly is a one-off.

It does not produce the cross-case pattern that reveals the residual in the first place — that is Cross-Case Evidence Matrix Tool; it does not run the systematic confirm-and-extend sampling of Replication Case Sampling Cycle; and it does not state the resulting scope of the theory — that falls to Sensitivity to Case-Set Analysis and the replication cycle.

  • Instantiates: Structured Comparative Case Design — the protocol is how the design learns from the cases that break its emerging pattern.
  • Consumes: Cross-Case Evidence Matrix Tool surfaces the residual that flags a case as deviant.
  • Sibling mechanisms: Replication Case Sampling Cycle · Sensitivity to Case-Set Analysis · Most-Similar Systems Design · Most-Different Systems Design · Matched Case Pairing Protocol · Measurement Equivalence Audit · Within-Case Process Tracing · Rival Explanation Elimination Table · Case Selection Bias Audit · Case Universe Sampling Frame · Comparative Case Review Panel · Comparative Historical Timeline · Configurational Comparison Truth Table · Counterfactual Contrast Memo · Cross-Case Evidence Matrix Tool

Notes

Keep it distinct from its two learning-side siblings. Deviant follow-up is reactive — it chases a specific anomaly that broke the pattern; Replication Case Sampling Cycle is proactive — it adds new cases to confirm and bound a finding; and Sensitivity to Case-Set Analysis perturbs the existing set to test fragility. A single odd case is this protocol's business; the shape of the whole case set is theirs.

References

[1] Negative (or deviant) case analysis — the practice of deliberately seeking and investigating cases that contradict an emerging explanation, using them to refine, bound, or overturn it rather than to discard. Its value depends on treating the anomaly as evidence, not as an exception to be waved away. ↩