Relevance and Alternative Explanation Check¶
Checklist — instantiates Evidentiary Trace Warranting
Tests whether a trace actually discriminates among hypotheses or is also expected under alternatives.
A Relevance and Alternative Explanation Check is a short diagnostic run against a single trace to answer one question: does this observation actually discriminate between the hypothesis you favor and its rivals, or would you have seen the same thing anyway? Its defining trait is that it treats relevance as diagnosticity — a trace only counts for a claim to the extent it is less expected under the alternatives. A trace that is equally consistent with every hypothesis is, evidentially, worth nothing, however striking it looks. The check is an act you perform, not a list you keep: you take one trace, line up the live hypotheses, and ask under which of them the trace is surprising. It is the mechanism that catches the archetype's most common error — a correlation or a vivid data point being read as support when it fits the rival story just as comfortably.
Example¶
A product analyst notices that users who adopt the new collaboration feature churn at half the rate of those who don't, and is about to report the feature as a retention driver. The check intervenes before that claim ships. The trace is the churn gap. The favored hypothesis is "the feature causes retention." The check forces the rival hypotheses onto the table and asks whether the same gap is expected under each.
It is. Under "engaged users adopt new features and stay" — reverse causation and a common cause (baseline engagement) — the identical churn gap is fully expected: committed users try everything and also churn less, with the feature doing no work. The trace, in other words, does not discriminate between "feature retains users" and "already-retained users adopt the feature." The check's verdict is that the correlation is real but non-diagnostic as it stands, and it names what would discriminate: a comparison holding baseline engagement fixed, or a staged rollout where adoption is not self-selected. The analyst reframes the finding as suggestive-not-causal and specifies the test that would earn the stronger claim. This is the everyday face of confounding.[n1]
How it works¶
The check is a fixed short sequence applied to one trace:
- State the claim and the live rivals. List the hypotheses actually in contention, including the boring ones (chance, selection, reverse causation, a common cause).
- Ask the discrimination question. For each rival, would this trace be roughly as expected as it is under the favored hypothesis? If yes for a rival, the trace does not separate them.
- Grade relevance by diagnosticity. A trace expected under only the favored hypothesis is strongly relevant; one expected under several is weakly or non-relevant, regardless of how salient it is.
- Name the discriminating test. Where the trace fails to separate, specify what observation would — the check's constructive output.
It is deliberately a test, not a repository: it consumes a list of rival hypotheses and returns a relevance verdict plus, where needed, the experiment that would make a non-diagnostic trace diagnostic.
Tuning parameters¶
- Rival-set breadth — how many alternatives you seriously entertain. Wider sets catch more spurious support but slow the check and can manufacture doubt about everything.
- Discrimination bar — how much less expected a trace must be under rivals to count as relevant. A strict bar demotes weak evidence honestly but may reject usable signal.
- Boring-alternative discipline — whether chance, selection, and reverse causation are always included. Forcing them in guards against exciting-story bias at some cost in speed.
- Depth of the "what would discriminate" step — whether the check merely flags non-diagnosticity or designs the separating test. Designing it is more useful but heavier.
- Trigger scope — every trace, or only load-bearing ones. Checking everything is thorough but expensive.
When it helps, and when it misleads¶
Its strength is that it is the sharpest available cure for treating correlation as evidence: by asking "expected under the alternatives?" it exposes non-diagnostic traces that look persuasive, and its constructive step turns a dead end into a designed comparison. It is the discipline that keeps relevance from collapsing into salience.
Its failure mode is that the check is only as good as the rival set — a trace can pass because the one alternative that actually explains it never made the list, so the check can certify a false "diagnostic" verdict built on an incomplete field of hypotheses. The classic misuse is running it as a one-sided ritual: entertaining only rivals you can easily dismiss, which manufactures apparent diagnosticity. The guarding discipline is to source the rival hypotheses adversarially — ideally from someone invested in a different answer — rather than generating them from the same mind that already prefers the conclusion.
How it implements the components¶
relevance_rule— the check is the relevance rule in operation: it defines a trace's relevance as its power to discriminate the claim from its rivals, not its mere consistency with the claim.defeater_and_alternative_hypothesis_map— it exercises the alternative-hypothesis map as a live test, asking under which rivals the trace would equally be expected, and thereby locating the trace's non-diagnostic alternatives.
It does not maintain the standing catalogue of counterevidence and absence claims (negative_evidence_detector — that's Defeater Register, which stores the alternatives this check merely tests against) nor assign the surviving relation a strength rung (scoped_evidential_weight — that's Evidence Strength Ladder); the check rules a trace diagnostic or not and stops.
Related¶
- Instantiates: Evidentiary Trace Warranting — the check enforces the archetype's relevance-and-discrimination requirement on each trace.
- Consumes: Defeater Register — draws its list of rival hypotheses and alternative explanations from the register's catalogue, then tests the trace against them.
- Sibling mechanisms: Admissibility or Relevance Gate · Claim-Evidence-Reasoning Card · Defeater Register · Evidence Relation Matrix · Evidence Strength Ladder · Evidence Update Review · Trace-to-Claim Diagram
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Relevance and Alternative Explanation Check operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it tests whether a trace actually discriminates among hypotheses or is also expected under alternatives.
Independent corroboration: The frozen evidence defines Relevance and Alternative Explanation Check as 'Tests whether a trace actually discriminates among hypotheses or is also expected under alternatives', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Convergent development
Present-day reach: Universal
Rationale: Strong inference and experimental design demand evidence that discriminates among competing hypotheses rather than merely fitting a favored explanation; epistemology supplies the general account of evidential relevance.
Related originating lineages:
- Philosophy — philosophy contributes epistemic relevance, equality, explanation, and moral responsibility to the mechanism’s formative or independently convergent form; that contribution does not displace the primary statistics_experimental_design lineage.
Review resolution: The blind reviewers disagreed on primary lineage; authoritative research supports statistics_experimental_design over the competing primary. Strong inference and experimental design demand evidence that discriminates among competing hypotheses rather than merely fitting a favored explanation; epistemology supplies the general account of evidential relevance. The cited Multiple competing hypotheses and strong inference provides direct evidence for that defining form. Alternates are retained only where they contributed an independent formative tradition, while domain_reach=universal records later transfer separately from historical origin.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
Notes¶
[n1] The Analysis of Competing Hypotheses (developed by former CIA analyst Richards Heuer) formalizes exactly this move: evidence is scored not by how well it fits the leading hypothesis but by its diagnosticity — its ability to distinguish among all the hypotheses at once — so that observations consistent with everything are correctly recognized as carrying no discriminating weight. ↩