Counterexample Search¶
Actively search for cases that would break a proposed rule, pattern, or generalization before treating it as reliable.
The Diagnostic Story¶
Symptom: A rule is being trusted because the cases that fit it are loud and the cases that break it were never sought. Confirming anecdotes pile up with no record of where the claim has been challenged; inconvenient exceptions get waved off as noise; and the rule keeps working in ordinary cases while quietly failing at the boundary. The cost of being wrong is real, but supporting evidence alone is carrying the decision.
Pivot: Turn negative evidence into a designed target: state the rule and its assumed scope, predefine what would count as a counterexample, and go hunting in the spaces most likely to break it.
Resolution: Overbroad rules get narrower, qualified, and more reliable; hidden boundary failures surface before decisions ride on them; and the claim comes away with explicit limits of applicability. The rule stays falsifiable, and candidate exceptions are judged for relevance rather than reflexively kept or dismissed.
Reach for this when you hear…¶
[ML review] “It hits 98% on every test we have run: has anyone actually tried to find the case where it falls over?”
[clinical] “This sign always means it in our experience, but have we looked at the patients where it didn't?”
[strategy] “Every example on this slide supports the thesis; what would we expect to see if it were wrong?”
When This Archetype Applies¶
Partial catalog groundingSome structural conditions are represented by existing abstractions, but no sufficient condition set is fully represented.
Diagnostic problem
A generalization, rule, diagnosis, model pattern, or strategic claim is accepted because fitting cases are salient while breaking cases are absent, hidden, inconvenient, or never deliberately sought.
What this problem means
The structural problem is one-sided visibility. Supporting examples are salient because they were noticed, collected, rewarded, or easy to explain. Breaking cases may be rare, embarrassing, hidden at edges, excluded by sampling, or dismissed as noise. As a result, a rule can become trusted before anyone knows where it stops applying.
Counterexample Search treats that gap as a design problem. Instead of asking whether the rule has support, it asks what would count as an in-scope violation and where such a violation would most likely be found.
Show the applicability expression
Applicability expression4 distinct conditions
groundedpartly groundedopen
4 conditions, all required.
4Required in every casenumbered 1–4
These hold no matter which pattern applies.
Example-based generalization · grounded · 3 illustrations, not alternatives
A claim is being generalized from visible examples, successful cases, or repeated observations.
It is especially valuable when the claim uses broad language, when failure at the boundary would be costly, when a group is attached to the claim, or when rare cases matter more than average cases. The narrower requirement in this condition set is: A claim is being generalized from visible examples, successful cases, or repeated observations.
Overbroad rule language · open
The rule contains broad language such as always, never, usually, all, safe, reliable, sufficient, or necessary.
It is especially valuable when the claim uses broad language, when failure at the boundary would be costly, when a group is attached to the claim, or when rare cases matter more than average cases. The narrower requirement in this condition set is: The rule contains broad language such as always, never, usually, all, safe, reliable, sufficient, or necessary.
Plausible edge violations · open
Known edge conditions, subgroups, time shifts, adversarial conditions, or exceptional contexts might violate the rule.
This is a load-bearing situation condition in the diagnostic expression. The condition is: Known edge conditions, subgroups, time shifts, adversarial conditions, or exceptional contexts might violate the rule. If it does not hold, this particular condition set is incomplete.
Iterative discordant testing · open
A current account is deliberately exposed to discordant cases and then falsified, narrowed, revised, or explicitly rescoped in an iterative loop.
This condition preserves a load-bearing part of the diagnostic problem that was not captured by a source-condition atom. It remains explicit because omitting it would weaken the sufficient condition set.
Other requirements and context (3)
Why these sit outside the expression
Supporting context — it may accompany or help interpret the situation, but it is not a load-bearing condition in a sufficient diagnostic set.
Goal — a goal states an intended outcome or evaluation criterion, not a pre-existing situation that independently summons the archetype.
Supporting contextThe decision cost of being wrong is significant enough that supporting evidence alone is insufficient.
Supporting contextA group has incentives, habits, or confirmation bias that make disconfirming cases socially or cognitively easy to ignore.
It is especially valuable when the claim uses broad language, when failure at the boundary would be costly, when a group is attached to the claim, or when rare cases matter more than average cases. In this archetype, the relevant contextual consideration is: A group has incentives, habits, or confirmation bias that make disconfirming cases socially or cognitively easy to ignore. It helps interpret the situation or strengthens the practical case for examining the archetype.
GoalA pattern-detection, diagnostic, policy, or design process needs a deliberately negative test before adoption.
Typical triggers include a diagnostic explanation that fits most signs, a policy rule that seems fair in standard cases, a software rule that passes normal inputs, a strategic thesis based on successful examples, or a research generalization that has not been checked against negative cases. In this archetype, the relevant goal is: A pattern-detection, diagnostic, policy, or design process needs a deliberately negative test before adoption. It supplies a criterion for evaluating what the intervention should accomplish or preserve.
Coverage
1 of 4 conditions grounded · 3 open.
Mechanisms / Implementations¶
- Falsification checks: translate a claim into a form that can be challenged.
- Exception searches: deliberately look for cases that violate a rule.
- Edge-case testing: stresses thresholds and boundary conditions.
- Red-team reviews: assign challengers to seek disconfirming cases.
- Adversarial example generation: Constructs hard inputs deliberately engineered to make a rule fail, then keeps only the ones that stay realistic enough to matter in the real operating scope.
- Proof by counterexample: Refutes an over-broad universal claim — that some method handles an entire class — by exhibiting one well-formed instance on which it demonstrably fails.
- Negative case analysis: Studies the cases that do not fit a theory and uses them to revise its boundary and confidence, rather than defending the theory or throwing it out.
- Boundary condition matrices: organize where the rule has and has not been challenged; they are artifacts that support the archetype.
- Boundary Condition Matrix: Lays a rule's operating dimensions on a grid and marks each cell tested-pass, tested-fail, or untested, so the coverage gaps become as visible as the found failures.
- Exception Search: Hunts the histories, subgroups, and edge conditions where a rule is most likely to have already broken, and captures the violating cases it finds.
- Falsification Check: Restates a confident claim as an explicit rule with a bounded scope and a pre-committed breaking criterion, so later evidence can actually refute it.
- Red-Team Review
Related Abstractions¶
Abstractions this archetype builds on — directly (a source ingredient) or as a related pattern. Links follow the typed catalog namespace.
Built directly on (4)
- Deductive Reasoning: General to specific conclusions.
- Hypothesis Testing (Null vs. Alternative): Null vs alternative evaluation.
- Inductive Reasoning: Specific to general inference.
- Uncertainty: Incomplete knowledge.
Also references 6 related abstractions
- Black Swan (High-Impact, Low-Probability Events): High-impact unexpected events.
- Boundary: Defines system limits.
- Confirmation Bias: Favor confirming evidence.
- Counterfactual Reasoning: Hypothetical alternatives.
- Robustness: Maintain functionality under stress.
- Sampling (Representativeness): Representative subset selection.
Variants¶
Narrower or domain-specific specializations that share this archetype's core structure. Recognized variants are established; candidate variants are provisional.
Edge-Case Counterexample Search · risk or failure variant · recognized
Searches boundary conditions where a rule is most likely to fail rather than sampling ordinary cases.
Universal Claim Counterexample · subtype · recognized
Uses a valid single breaking case to invalidate or qualify a universal rule.
Diagnostic Exception Search · domain variant · recognized
Looks for cases, symptoms, or observations that would rule out or narrow a favored diagnosis or pattern match.
Adversarial Counterexample Generation · implementation variant · recognized
Generates deliberately difficult cases to expose where a rule, classifier, plan, or system fails.
Editorial Notes¶
Problem Classification¶
Classification: Uncertainty, Evidence & Inference Failure → Explanatory Hypothesis, Pattern & Case Reasoning
Problem kernel: generalization is accepted without searching for breaking cases
Rationale: Fitting examples are salient while boundary and deviant cases remain absent, allowing a favored pattern to appear universal.
Independent corroboration: The earliest necessary condition in the frozen evidence is: A generalization, rule, diagnosis, model pattern, or strategic claim is accepted because fitting cases are salient while breaking cases are absent, hidden, inconvenient, or never deliberately sought. That is a explanatory hypothesis pattern and case reasoning problem because Partial or recurring observations are forced into a favored explanation, analogy, pattern, or universal before alternatives and boundary cases are tested.
Review outcome: Independent reviewer agreement; high confidence.