Skip to content

Counterexample Search

Actively search for cases that would break a proposed rule, pattern, or generalization before treating it as reliable.

The Diagnostic Story

Symptom: A rule is being trusted because the cases that fit it are loud and the cases that break it were never sought. Confirming anecdotes pile up with no record of where the claim has been challenged; inconvenient exceptions get waved off as noise; and the rule keeps working in ordinary cases while quietly failing at the boundary. The cost of being wrong is real, but supporting evidence alone is carrying the decision.

Pivot: Turn negative evidence into a designed target: state the rule and its assumed scope, predefine what would count as a counterexample, and go hunting in the spaces most likely to break it.

Resolution: Overbroad rules get narrower, qualified, and more reliable; hidden boundary failures surface before decisions ride on them; and the claim comes away with explicit limits of applicability. The rule stays falsifiable, and candidate exceptions are judged for relevance rather than reflexively kept or dismissed.

Reach for this when you hear…

[ML review] “It hits 98% on every test we have run: has anyone actually tried to find the case where it falls over?”

[clinical] “This sign always means it in our experience, but have we looked at the patients where it didn't?”

[strategy] “Every example on this slide supports the thesis; what would we expect to see if it were wrong?”

Mechanisms / Implementations

  • Falsification checks: translate a claim into a form that can be challenged.
  • Exception searches: deliberately look for cases that violate a rule.
  • Edge-case testing: stresses thresholds and boundary conditions.
  • Red-team reviews: assign challengers to seek disconfirming cases.
  • Adversarial example generation: constructs hard cases designed to expose failure.
  • Proof by counterexample: Refutes an over-broad universal claim — that some method handles an entire class — by exhibiting one well-formed instance on which it demonstrably fails.
  • Negative case analysis: studies non-fitting cases so a theory, diagnosis, or explanation can be revised rather than merely defended.
  • Boundary condition matrices: organize where the rule has and has not been challenged; they are artifacts that support the archetype.
  • Boundary Condition Matrix
  • Exception Search

Abstractions this archetype builds on — directly (a source ingredient) or as a related pattern. Links follow the typed catalog namespace.

Built directly on (4)

Also references 6 related abstractions

Variants

Narrower or domain-specific specializations that share this archetype's core structure. Recognized variants are established; candidate variants are provisional.

Edge-Case Counterexample Search · risk or failure variant · recognized

Searches boundary conditions where a rule is most likely to fail rather than sampling ordinary cases.

Universal Claim Counterexample · subtype · recognized

Uses a valid single breaking case to invalidate or qualify a universal rule.

Diagnostic Exception Search · domain variant · recognized

Looks for cases, symptoms, or observations that would rule out or narrow a favored diagnosis or pattern match.

Adversarial Counterexample Generation · implementation variant · recognized

Generates deliberately difficult cases to expose where a rule, classifier, plan, or system fails.