Exception Search¶
Search procedure — instantiates Counterexample Search
Hunts the histories, subgroups, and edge conditions where a rule is most likely to have already broken, and captures the violating cases it finds.
Given a rule that has been made testable, someone still has to go looking — and looking in the right places, not the well-lit ones. Exception Search is that hunt. Its defining trait is that it mines cases that have already occurred: it designs a search space out of real histories, logs, subgroups, and boundary conditions where a violation is most likely and where ordinary confirmation is least informative, then captures each candidate it finds with enough context to be adjudicated later. It does not fabricate hard cases, it does not pre-state the criterion, and it does not judge whether a found case truly refutes the rule. It finds and it records; that is the whole job.
Example¶
An airline operations team holds a working rule: "our current de-icing procedure prevents wing-surface ice under all winter departure conditions." Exception Search designs the hunt against reality rather than the claim. It pulls ten winters of cold-weather incident and maintenance logs; filters to departures where the fluid's holdover time was exceeded; targets regional airports known for freezing drizzle (a boundary condition ordinary flights never see); and interviews crews about near-misses that never became formal reports.
It surfaces three logged cases where ice was observed on the wing after de-icing, all under freezing drizzle with long taxi times. Each is captured as a counterexample record — airport, precipitation type, holdover interval, outcome, and why it may challenge the rule. The value is in where it looked: normal clear-cold departures always pass and prove nothing, so the search spent its budget exactly where the rule was most likely to have already cracked. It hands the three candidates on without yet declaring them refutations — that verdict is a separate step.
How it works¶
- Invert salience. Deliberately avoid the cases that got noticed and rewarded; aim at the rare, the filtered-out, the historically embarrassing, and the boundary.
- Enumerate breaking spaces. Edge conditions, subgroups, time shifts, and cases that prior sampling quietly excluded — the places a violation would hide.
- Sample and capture. Log each candidate with enough surrounding context that a later relevance test can adjudicate it.
- Keep search separate from judgment. The output is a pile of dated candidates, not verdicts — confirming a candidate is a downstream check.
Tuning parameters¶
- Search intensity — how wide and deep to look. More coverage finds more but costs time; scale it to how badly a boundary failure would hurt.
- Space targeting — aim hard at suspected weak spots, or sweep broadly. Targeting is efficient but can miss the surprise you did not anticipate.
- Capture richness — how much context to record per candidate. Richer records make later relevance tests possible but slow the hunt.
- History window — how far back to mine. A longer window surfaces rarer cases but mixes in stale operating regimes.
When it helps, and when it misleads¶
Its strength is that it is the concrete engine that turns "we ought to check for exceptions" into found cases, attacking one-sided visibility head-on by spending effort where confirming cases are absent.
Its failure mode is narrow search disguised as rigor: a shallow look that finds nothing and is then read as evidence the rule is universal. The classic misuse is searching only the spaces you already believe are safe and then declaring victory — the survivorship trap, where the informative cases are exactly the ones that never made it into your sample.[n1] The guarding discipline is that absence of counterexamples licenses confidence only in proportion to how thoroughly you searched — which is why Exception Search feeds a coverage map rather than a verdict.
How it implements the components¶
counterexample_search_space— designs the targeted set of histories, subgroups, and edge conditions where violations are most likely to be found.counterexample_record— captures each found candidate with its context and the reason it may challenge the rule.
It mines cases that already exist; it does not construct novel breaking cases, nor run the relevance_test and adversarial_reviewer_role that govern manufactured ones — that is Adversarial Example Generation's lane, its nearest twin.
Related¶
- Instantiates: Counterexample Search — it is the search step that supplies real found candidates.
- Consumes: Falsification Check supplies the stated rule and falsification condition that tell the search what a violation looks like.
- Sibling mechanisms: Falsification Check · Adversarial Example Generation · Boundary Condition Matrix · Negative Case Analysis · Proof by Counterexample
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Exception Search operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it hunts the histories, subgroups, and edge conditions where a rule is most likely to have already broken, and captures the violating cases it finds.
Independent corroboration: The frozen evidence defines Exception Search as 'Hunts the histories, subgroups, and edge conditions where a rule is most likely to have already broken, and captures the violating cases it finds', so its operative form is Assessment, Review & Assurance.
Nearest alternative: Protocol, Workflow & Routine — The search uses an ordered sampling and capture procedure, but its purpose is a bounded investigation of existing histories and edge cases.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Philosophy
Origin pattern: Convergent development
Present-day reach: Universal
Rationale: Actively seeking cases that violate a general rule is the classic philosophical and scientific method of counterexample testing.
Related originating lineages:
- Statistics & Experimental Design — Adversarial subgroup and edge-case sampling materially operationalizes the search in empirical settings. Adversarial sampling of subgroups and edge conditions materially shapes the systematic search procedure.
Review resolution: Both reviewers agree that philosophy is primary. I retain statistics_experimental_design only as formative origin lineages; convergent is appropriate because the same operational pattern arose through parallel professional lineages. Reach is universal because the structure is portable across essentially any domain with the stated problem, an applicability judgment kept separate from provenance. Encyclopedia synthesis is true because the exact generalized artifact is an encyclopedia-authored combination or refinement. No unresolved historical ambiguity remains after reconciling the secondary fields.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Survivorship bias — reasoning only from the cases that remained visible. Abraham Wald's WWII analysis of returning aircraft is the canonical illustration: the armor belonged where the surviving planes showed no damage, because hits there downed the planes that never came back. Exception Search is built to look where the missing cases would be. ↩