Discriminating Test Matrix¶
Analytic grid — instantiates Alternative Hypothesis Generation
A grid crossing rival hypotheses against pieces of evidence, scoring each cell for consistency and each row for reliability — so effort goes to the observations that actually separate the rivals rather than to evidence that fits them all.
A Discriminating Test Matrix is a grid whose columns are the rival hypotheses and whose rows are the individual pieces of evidence, with each cell scored for whether that evidence is consistent or inconsistent with that hypothesis. Its distinguishing move among its siblings is that it neither generates rivals nor eliminates them — it maps diagnosticity. Read across a row, the matrix reveals the crucial and counter-intuitive fact: a piece of evidence consistent with every column carries no discriminating power, however much it feels like confirmation, while a piece consistent with just one column is decisive. By also grading how reliable each evidence row is, it keeps a diagnostic-but-shaky observation from being over-trusted. The artifact re-aims attention from "how much supports my hypothesis" to "what separates the hypotheses."
Example¶
At 2:00 pm a web service's p99 latency triples. The on-call engineer's first hunch is "the afternoon deploy broke something," but instead of chasing it, she builds a matrix. Columns: bad deploy, database lock, cache-eviction storm, upstream-provider outage, traffic spike. Rows: the deploy timestamp, the slow-query log, the cache hit-rate graph, the provider status page, the request-rate graph. She marks each cell consistent or inconsistent. Request rate was flat — inconsistent with a traffic spike, which drops out. The deploy shipped three hours earlier — weakly consistent at best. The cache hit-rate cratered exactly at 2:00 pm and is consistent with only the cache-eviction column. That single row is where the diagnostic power lives. She also grades the rows: the provider's status page is self-reported and lagging (low reliability), the cache metric is first-party and real-time (high) — so the decisive cell also rests on the most trustworthy evidence.
How it works¶
- Lay out the grid — rival hypotheses as columns, each independent piece of evidence as a row.
- Score every cell for consistency between that evidence and that hypothesis.
- Read down each row, not across each column. An item consistent with all hypotheses is diagnostically worthless even when it feels like proof; an item consistent with one or two is where to spend effort. This inversion — evaluating evidence by how well it discriminates rather than how well it confirms — is the matrix's core.
- Grade each evidence row's reliability, so a highly diagnostic cell that rests on flimsy evidence is flagged rather than trusted blindly.
Tuning parameters¶
- Cell scale — binary consistent/inconsistent versus graded (strongly / weakly / neutral). Finer scales carry more signal but invite false precision about judgments that are still subjective.
- Evidence independence — how strictly rows are kept non-redundant. Correlated rows double-count the same fact and manufacture fake diagnosticity.
- Diagnosticity threshold — how discriminating a row must be before the team acts on it. A strict bar avoids chasing weak separators.
- Evidence-quality weighting — whether low-graded rows are down-weighted or excluded outright.
- Refresh cadence — scored once versus re-scored as new evidence arrives and columns are added or dropped.
When it helps, and when it misleads¶
Its strength is that it kills the natural habit of piling up confirming-but-non-diagnostic evidence and points effort at the one observation that actually separates the rivals — the central discipline of Analysis of Competing Hypotheses.[1] It is at its best when several plausible explanations coexist and the question is which test to run next.
Its failure mode is that a completed matrix looks objective while every cell is a judgment call. Score the cells carelessly, or frame an ambiguous observation to fit the favored column, and the grid launders subjective reasoning into confident-looking analysis. The classic misuse is treating a filled matrix as proof rather than a structuring aid — or, worse, quietly omitting the one hypothesis column that would have won, so the exercise is rigged from the start. The discipline that guards against this is independent scoring, always carrying an "other / not yet imagined" column, and using the evidence-quality grades so a decisive-looking cell built on a weak source is not mistaken for a settled result.
How it implements the components¶
alternative_hypothesis_set— the columns are the rival set the matrix operates over (though it takes that set as input rather than producing it).discriminating_evidence_map— the scored grid is precisely the map of which evidence separates which rivals; this is the matrix's reason to exist.evidence_quality_grade— each evidence row is rated for reliability, so diagnosticity is never confused with solidity.
It does not generate the rivals in the first place (focal_claim_or_leading_hypothesis handling and set generation) — that is Why Else Could This Be True? Prompt or Differential Diagnosis List — nor does it apply the rule-in / rule-out that acts on the results (rival_retention_or_elimination_rule), which the differential owns.
Related¶
- Instantiates: Alternative Hypothesis Generation — it supplies the discrimination map the rest of the process narrows against.
- Consumes: a rival-generating sibling — Why Else Could This Be True? Prompt or Differential Diagnosis List — for the hypothesis columns it maps.
- Sibling mechanisms: Differential Diagnosis List · Why Else Could This Be True? Prompt · Counter-Narrative Probe · Base-Rate Alternative Prompt · Red-Team Rival Explanation Review
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: The mechanism computes how consistently each observation fits rival hypotheses and how diagnostically valuable and reliable each evidence row is.
Nearest alternative: Representation, Specification & Plan — The grid presents the comparison, but its operative contribution is the scored discrimination among hypotheses rather than the matrix layout alone.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Security Studies & Intelligence Analysis
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Intelligence analysis cohered Analysis of Competing Hypotheses as a matrix judging evidence by its diagnosticity across rivals and emphasizing inconsistency over confirmation.
Related originating lineages:
- Philosophy — Abductive and falsificationist epistemology supplied the norm of comparing live rivals with potential defeaters.
- Statistics & Experimental Design — Experimental-design traditions formalized discriminating observations and evidence reliability.
Review resolution: Both current reviews place discriminating_test_matrix primarily in security_intelligence; the reconciled classification retains only lineages that materially shaped the mechanism and keeps breadth of origin separate from reach.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
The most useful thing the matrix does is also the least intuitive: it demotes the evidence people find most persuasive. A fact that fits your favored hypothesis and every rival feels like strong support and contributes nothing to the decision. Teams that skip the matrix tend to accumulate exactly this kind of comfortable, non-diagnostic evidence and mistake its volume for weight.
References¶
[1] Analysis of Competing Hypotheses (ACH), set out in Richards Heuer's Psychology of Intelligence Analysis (1999), is the canonical real-world instance of this mechanism. Its central tenets are that evidence should be judged by its diagnosticity across all hypotheses, that evidence consistent with every hypothesis has no diagnostic value, and that analysts should work to disconfirm rather than confirm. registry ↩