Confirmatory Follow-Up¶
Staged validation — instantiates Multiple-Testing Discipline
Turns one promising exploratory lead into a single pre-specified confirmatory test, so a pattern found by searching must earn its status on fresh ground.
A Confirmatory Follow-Up is the staged move that lets broad exploration stay creative while its winners still have to prove themselves. Exploration is allowed to generate as many leads as it likes; a follow-up then takes one promising lead, freezes a single hypothesis and analysis around it, and runs a dedicated confirmatory test whose only job is to accept or reject that one claim. Its defining idea — what separates it from every sibling — is that it is the same program's own next stage: a narrow, pre-specified test aimed at a specific lead the team already found, collapsing many exploratory degrees of freedom down to one pre-committed comparison. The exploratory phase can look anywhere; the confirmatory phase may look at only what was named in advance.
Example¶
A regional government runs a large jobs-training program and, in an exploratory pass over the pilot data, notices that participants under 25 seem to gain far more than the average — an eye-catching subgroup effect across the many age bands, regions, and outcomes the analysts inspected. Treated as a finished discovery, "the program works best for the young" could redirect millions. But it was the most attractive of dozens of subgroup slices, exactly the kind of pattern that appears by chance when you cut the data enough ways. So the agency runs a confirmatory follow-up: it pre-registers a single hypothesis — "under-25 participants show a larger employment gain than older participants" — with one primary outcome and one analysis, and tests it in a fresh cohort recruited for the purpose. Nothing about that follow-up is exploratory; there is one bar and one look. If the young-worker advantage reappears at the pre-set threshold, the claim is relabeled confirmed and can guide policy; if it fades, it stays a lead. The follow-up is what converts a suggestive slice into evidence — or honestly retires it.
How it works¶
- Select one lead. From the exploratory phase, pick the single claim worth the cost of confirmation; the rest stay labeled exploratory.
- Freeze the test. Pre-specify the hypothesis, primary outcome, sample, and analysis so no post-hoc flexibility survives into the confirmatory stage.
- Collect fresh evidence. Run the pre-committed test on new data gathered for it, so the confirmation is not scored on the same observations that suggested the lead.
- Relabel on the result. A pass promotes the claim's status; a fail returns it to the exploratory pool. There is exactly one comparison, so its p-value means what it says.
Tuning parameters¶
- Selection stringency — how strong a lead must look before it earns a costly follow-up; higher stringency wastes fewer confirmations but risks shelving real effects.
- Confirmatory bar — the pre-set threshold the fresh test must clear; stricter bars cut false confirmations at the cost of missing modest true effects.
- Scope of the frozen hypothesis — narrow (one subgroup, one outcome) versus slightly broader; narrower is cleaner but answers less.
- Fresh-evidence source — a brand-new cohort, a later time window, or a pre-sealed portion of the same program; newer and more independent is stronger but slower.
When it helps, and when it misleads¶
Its strength is that it preserves the value of wide search — leads are welcome — while refusing to let any of them cross into "confirmed" without a clean, single-shot test. It directly counters exploratory label laundering, where a finding is called exploratory in the fine print but presented as established in the decision.[n1] By staging discovery and confirmation, it keeps the confirmatory p-value interpretable.
Its failure mode is the winner's curse: the lead was selected because it looked biggest, so even a genuine effect is usually smaller than the exploratory pass implied, and a follow-up powered for the inflated estimate can under-detect it and wrongly retire a real signal.[n2] The classic misuse is contamination — letting the confirmatory analysis quietly acquire new degrees of freedom (an added covariate here, a shifted outcome there) until it is just a second exploration wearing a confirmatory label. The guarding discipline is to lock the follow-up's single hypothesis before it runs and power it for a deflated, not the observed, effect size.
How it implements the components¶
confirmation_requirement— it defines and executes exactly the fresh, single test a selected lead must survive before it may guide action.exploratory_confirmatory_boundary— it is the operational wall between the search phase (leads) and the pre-specified confirmatory stage (claims), enforced by freezing the hypothesis.result_status_label— the follow-up's outcome flips the lead's status between exploratory and confirmed, which is the promotion the mechanism exists to grant.
It does not hand the test to an independent team through an independent_replication_path; that generalizability check belongs to Replication Study, whereas a confirmatory follow-up is the original program's own targeted next stage.
Related¶
- Instantiates: Multiple-Testing Discipline — supplies the staged discovery-then-confirmation form of multiplicity control.
- Consumes: Metric Hierarchy and Multiverse Analysis Report surface the exploratory leads a follow-up is built to confirm.
- Sibling mechanisms: Alpha-Spending Plan · Bonferroni-Like Correction · False Discovery Rate Control · Claim Registry · Holdout Validation · Metric Hierarchy · Multiverse Analysis Report · Preregistration · Replication Study
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Turns one promising exploratory lead into a single pre-specified confirmatory test, so a pattern found by searching must earn its status on fresh ground, making its operative form a bounded trial, probe, simulation, or adversarial exercise that generates evidence from performance.
Independent corroboration: The frozen evidence defines Confirmatory Follow-Up as 'Turns one promising exploratory lead into a single pre-specified confirmatory test, so a pattern found by searching must earn its status on fresh ground', so its operative form is Experiment, Test & Rehearsal.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Experimental design established independent, prespecified confirmatory testing after exploratory selection to control false discovery and effect inflation.
Review resolution: Both reviewers agree on statistics_experimental_design as primary. Reading the source mechanism confirms that its defining operation belongs to that lineage; the final record retains no additional lineage only where it materially formed the mechanism and keeps present-day application breadth separate from provenance.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Exploratory label laundering is presenting a finding as merely exploratory in the methods text while treating it as confirmed in the decision, headline, or recommendation — the failure a confirmatory follow-up prevents by forcing a separate, pre-specified test before promotion. ↩
[n2] The winner's curse in this setting is the tendency for an effect selected because it looked largest to be overestimated, so it typically shrinks when re-tested; follow-ups powered for the observed (inflated) size are therefore prone to miss real but smaller effects. ↩