Extreme-Selection Risk Flag¶
Screening — instantiates Regression-to-the-Mean Guardrail
Marks an evaluation as triggered by extreme selection, so its before-after story is treated as regression-suspect before any effect is credited.
Extreme-Selection Risk Flag is the tripwire at the front door. Before anyone measures, compares, or models, it asks one question: did this case enter attention because a noisy measurement was unusually high or low? If yes, it stamps the evaluation "regression-suspect" and names the specific claim or decision that stamp protects. It is deliberately lightweight — it produces neither a baseline, nor a comparison, nor a benchmark. Its whole value is arriving early, before an extreme-before/ordinary-after sequence has been narrated as cause and effect. The flag does not decide whether an effect is real; it only guarantees that the question gets asked and routes the case to the heavier mechanisms that can answer it.
Example¶
A SaaS company runs a "customer-save" team. Each month, accounts whose product usage drops below a threshold are auto-flagged as churn-risk and handed to the save team, which calls them. Leadership wants to credit the save team with a rising retention rate. The Extreme-Selection Risk Flag is the intake step that stops the celebration long enough to check the plumbing. It records that accounts were selected because usage hit an unusually low month — a single noisy reading — and that many low-usage accounts would have drifted back toward their own normal simply because the bad month was atypical. It names the decision at stake: whether to double the save team's headcount next quarter on the strength of the retention number.
The output is not a verdict. It is a one-line annotation on the analysis: "selection conditioned on an extreme low; expect partial reversion; headcount decision on hold pending controlled comparison." That annotation is what sends the case downstream — to a comparator and a benchmark — instead of straight to the budget meeting.
How it works¶
The flag is a short, fixed intake check applied before outcomes are reviewed:
- Was selection conditioned on an extreme? Record the trigger explicitly — the threshold, the ranking window (top or bottom k), the crisis entry, or the retry-after-a-bad-result. If selection was random or on a stable trait, the flag stays down.
- Who was not selected? Naming the cases just inside and just outside the cutoff makes the selection rule auditable and reveals how extreme the cutoff really was.
- Which claim or decision does this feed? Attach the downstream consequence — a promotion, a budget, a program continuation — so the flag's weight scales with the stakes.
The result is a boolean plus annotation that routes the case, not a number. A raised flag is an instruction: do not credit raw change until a comparator or benchmark has spoken.
Tuning parameters¶
- Trigger sensitivity — how extreme selection must be before the flag rises. A hair-trigger flags everything (and courts fatalism); a dull one lets confounded wins through. Set it by how noisy the selecting measure is.
- Scope of "extreme" — whether only hard thresholds count, or also soft rankings, discretionary attention, and repeat-testing. Wider scope catches more, at more overhead.
- Ownership and routing — who raises the flag and where a raised flag sends the case. A flag that routes nowhere is decorative.
- Stakes weighting — how much the downstream decision's reversibility and consequence escalate the response.
When it helps, and when it misleads¶
Its strength is speed and cheapness: it catches the bias at the only moment it is easy to catch — before the story hardens. It is the mechanism that keeps a team from mistaking "we intervened on the worst cases and they improved" for proof, a pattern vivid in the flight-instructor illusion where praising good landings seemed to hurt and berating bad ones seemed to help, purely from reversion.[1]
Its failure mode is the opposite excess. A flag treated as a verdict tips into fatalism — every improvement waved away as "just regression," which is the archetype's overcorrection-to-null trap. The classic misuse is exactly that: reading a raised flag as evidence of no effect. The guarding discipline is to remember the flag is a router, not a ruling. It says this needs a comparator, never this is noise; a genuine effect and a raised flag routinely coexist, and only the downstream evidence resolves which.
How it implements the components¶
extreme_selection_and_trigger_rule— its core act: documenting the threshold, ranking window, discretion, and unselected cases that define why this case is in review.causal_claim_and_decision_scope— it names the improvement being claimed and the decision a false attribution would distort, sizing its own urgency.
It stops there. It does not measure the case's typical level (pre_intervention_repeated_baseline) or its reliability (signal_reliability_and_noise_decomposition) — that is Multi-Baseline Measurement Protocol — and it does not quantify how much reversion to expect (expected_reversion_benchmark), which is Reliability-Based Reversion Simulation.
Related¶
- Instantiates: Regression-to-the-Mean Guardrail — the flag is the guardrail's earliest step, the one that must fire before results are seen.
- Sibling mechanisms: Multi-Baseline Measurement Protocol · Matched Extreme-Case Comparator · Randomized or Staggered Assignment · Reliability-Based Reversion Simulation · Shrinkage-Aware Expectation · Controlled Before–After Contrast · Interrupted Series with Pretrend Check · Placebo Time, Outcome, or Threshold Check · Attribution-Claim Review Gate
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: A fixed intake check evaluates whether selection was conditioned on an extreme and produces a regression-suspect finding with cutoff, comparators, and downstream stakes recorded.
Nearest alternative: Decision, Gate & Allocation — The boolean can route a case, but its defining output is the risk finding about how the evaluation sample was selected.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Flagging before-after comparisons selected on extreme values directly operationalizes statistical regression to the mean.
Related originating lineages:
- Psychology — Psychological intervention research materially popularized regression artifacts in selected extreme groups.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Independent reviewer agreement; high confidence.
References¶
[1] Kahneman, D. Thinking, Fast and Slow. Farrar, Straus and Giroux (2011). In the flight-instructor example, praised unusually good landings regressed downward and criticized unusually bad landings regressed upward; the pattern was reversion, not an intervention effect. registry ↩