Skip to content

Counterexample Surface Scan

Discovery method — instantiates Salience-Significance Decoupling

Deliberately hunts the disconfirming cases a vivid story leaves unshown, so the counterexamples get weighed too.

A vivid story persuades partly because its counterexamples are quiet. Counterexample Surface Scan goes looking for those quiet cases on purpose. Given a salient narrative — "this move always pays off," "this failure mode is everywhere" — it runs a directed search for the instances that contradict it and that stayed invisible precisely because they were undramatic: the times the pattern didn't hold, the successes that never got told, the failures that were never post-mortemed. Its defining move is searching against the grain of attention — actively probing where salience is low but relevance may be high — so the missing disconfirmers are pulled into view and weighed alongside the vivid confirmers, rather than lost to the fact that nobody thinks to mention them.

Example

A hiring panel is captivated by one candidate's story: single-handedly rescued a collapsing system over a brutal weekend. It is concrete, dramatic, and doing most of the work in the room. The scan intervenes not by arguing about that story but by hunting its counterexamples. It asks two directed questions: which past hires told a similar heroic-rescue story — and how did they actually work out? And which strong performers on the current slate have no dramatic story at all, only a quiet record?

Surfacing the first set turns up the last three "hero" hires, two of whom struggled once the crisis adrenaline was gone. Surfacing the second turns up two steady candidates whose files never pop. Neither set was hidden; both were simply un-vivid, and so absent from the conversation. Once they are on the table, the heroic anecdote is one data point among several instead of the whole picture — which is all the scan is trying to make happen.

How it works

  • State the salient claim explicitly. Name the narrative the vivid case is being used to support, so the search has a proposition to contradict rather than a mood to argue with.
  • Probe against the salience gradient. Deliberately look where attention isn't — the undramatic, the unreported, the "boring" cases — because that is exactly where disconfirming evidence hides.
  • Check what selection removed. Ask why the counter-cases are missing: were they never recorded, filtered by a process, or simply too quiet to recall? The reason tells you how much absence to correct for.
  • Return the counter-set for weighing. Hand the surfaced disconfirmers back to be judged on merits — the scan finds them; it does not decide the outcome.

Tuning parameters

  • Search direction — whether you hunt confirming-pattern cases that failed, or non-vivid cases that succeeded, or both. Each exposes a different blind spot; both together are strongest but costlier.
  • Effort ceiling — a five-minute "name three exceptions" prompt versus a funded records pull. More effort surfaces deeper counterexamples but spends time you may not have.
  • Missingness suspicion — how hard you push on why a counter-case is absent. High suspicion catches systematic filtering; too high turns every silence into a conspiracy.
  • Stopping rule — how many counterexamples, or how much search, is enough before the vivid case is safely contextualized rather than exhaustively litigated.

When it helps, and when it misleads

Its strength is that it directly counters confirmation bias by manufacturing the disconfirming evidence attention would never volunteer — the successes that were never publicized, the failures that never became a story.[1] It is cheap, portable, and works as a solo checklist move or a group prompt, and it is especially potent against a single anecdote doing more persuasive work than its weight warrants.

It misleads in two ways. It can manufacture false balance — dredging up weak or irrelevant counterexamples to blunt a vivid case that happens to be genuinely representative, treating "I found an exception" as if it refuted a rate. And it is easily run backwards: pointed only at conclusions you dislike, it becomes a tool for endless doubt rather than honest search. The discipline is to hold the search symmetric — hunt counterexamples to the comfortable story as hard as to the uncomfortable one — and to hand the counter-set to a weighting step rather than letting a single exception carry the day.

How it implements the components

This scan realizes the go-find-the-missing-disconfirmers side of the archetype — the qualitative hunt, not the quantitative baseline:

  • counter_salience_probe — it is this probe: a directed search aimed against the salience gradient, at the low-attention region where counterexamples sit.
  • missingness_and_selection_check — it asks what disconfirming material is absent and why, distinguishing "never happened" from "never recorded" from "never noticed."

It surfaces the missing cases but does not quantify the baseline they imply — Base-Rate Visibility Panel does that — nor set the criteria that decide which counter-cases actually matter, which Evidence Weighting Rubric supplies.

Notes

The scan is a finder, not a judge. It deliberately stops at surfacing the counter-cases and checking why they were missing; deciding how much each one is worth belongs to Evidence Weighting Rubric. Keeping the two separate is what stops a single surfaced exception from simply replacing one vivid anecdote with another.

References

[1] Survivorship bias — reasoning from the visible survivors while the failures are silently absent, as in Abraham Wald's WWII analysis of returning aircraft, where the un-returned planes carried the decisive information. Directing search at the missing, non-vivid cases is the corrective the scan operationalizes. ↩