Seeded Defect Calibration Exercise¶
Calibration exercise — instantiates Parallel Independent Inspection Design
Plants known defects into the inspection stream to measure each inspector's catch rate and calibrate how much the process is really finding.
Seeded Defect Calibration Exercise plants a set of known, deliberately-created defects — a calibration set, a gold standard — into the inspection stream, then measures how many each inspector catches. Its primary product is a measurement of the inspectors and the process, not a corrected artifact: the planted defects are a ruler laid against detection. Because their true count is known in advance, they let you compute a catch rate that real defects — whose count is unknown — never can. Its defining move is to inject known defects to measure sensitivity, rather than compare real passes against each other.
Example¶
A trust-and-safety team wants to know how good its human moderators really are at catching a banned content class. Into the daily review queue they salt a calibration set of known items — pre-labeled by a gold-standard panel to span defect classes (clear violations, edge cases, benign look-alikes) at graded severities, and made indistinguishable from real traffic. Over a week, moderator M1 catches roughly 90% of the planted clear violations but only about 40% of the planted edge cases; M2 is more even across both but slower.
Those measured catch rates calibrate each moderator and reveal that edge cases are the process's weak class — steering training and marking where a second independent pass would pay off. The real queue keeps flowing the whole time; the planted items are the yardstick, not the work.
How it works¶
Known defects are created to span the defect taxonomy at graded severities and injected indistinguishably from real items. As inspection runs, the exercise measures per-inspector and process-wide catch rate — planted defects found over planted defects seeded — broken out by defect class so weak classes surface. That catch rate can be extrapolated to a rough estimate of real detection, and the seeds are retired and refreshed on a cadence so inspectors cannot simply memorize them.
Tuning parameters¶
- Number and realism of seeds — more and more-realistic seeds sharpen the estimate but cost more to create and risk confusing real handling.
- Taxonomy and severity spread — how widely seeds cover defect classes and severities, trading per-class resolution against effort concentrated on the classes that matter.
- Extrapolation stance — whether catch rate is used only to compare inspectors or is projected onto real defects; projection is powerful but leans on a strong assumption.
- Refresh cadence — how often the seed set is rotated; frequent rotation prevents memorization but raises maintenance overhead.
When it helps, and when it misleads¶
Its strength is a direct, per-inspector, per-class sensitivity measurement that real defects cannot provide, because a real defect's existence is only known once found.
Its failures come from the seeds' fidelity. Seeded defects that are easier or more gameable than real ones flatter the catch rate, so the process looks better than it is. And extrapolating catch rate to real defects assumes the planted and the real are equally detectable — usually false, since the genuinely dangerous real defect is often the one no seed resembles.[n1] The guarding discipline is to match seed difficulty to real defects, rotate seeds so they aren't memorized, and treat any extrapolation to real-defect counts as indicative rather than exact.
How it implements the components¶
calibration_set— the planted known defects are the calibration set, the gold standard the whole mechanism turns on.defect_taxonomy_and_severity_scale— seeds are spread across the taxonomy at graded severities so catch rate is measured per class rather than only in aggregate.marginal_yield_and_saturation_metric— the measured catch rate feeds a saturation read: a low catch rate on a class says that class is far from saturated and needs more inspection.
It plants known defects rather than comparing two real independent passes, so it does not implement overlap_coverage_plan or independence_protocol — inferring residual defects from pass overlap is Capture-Recapture Defect Estimation. It also runs no reconciliation_and_adjudication_loop over real findings; that workflow is Finding Reconciliation Board.
Related¶
- Instantiates: Parallel Independent Inspection Design — calibrates inspectors and process sensitivity with a planted gold standard.
- Sibling mechanisms: Blind Document Proofing Passes · Capture-Recapture Defect Estimation · Dual or Triple Diagnostic Read · Finding Reconciliation Board · Independent Checklist Variant Rounds · Independent Security Review Lenses · Multi-Inspector Manufacturing Sort · Overlap Heatmap · Parallel Code Review Round
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Seeded Defect Calibration Exercise operates as an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation because it plants known defects into the inspection stream to measure each inspector's catch rate and calibrate how much the process is really finding.
Independent corroboration: The frozen evidence defines Seeded Defect Calibration Exercise as 'Plants known defects into the inspection stream to measure each inspector's catch rate and calibrate how much the process is really finding', so its operative form is Experiment, Test & Rehearsal.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Planting known defects and measuring which are found is an experimental calibration of detection sensitivity and reviewer performance. NIST evaluates bug finders against seeded faults and NASA studies inspection defect detection; audit and education contribute the review exercise format.
Related originating lineages:
- Accounting & Auditing — accounting_auditing contributes controls, independent assurance, traceability, and exception review to this mechanism's defining operation—Plants known defects into the inspection stream to measure each inspector's catch rate and calibrate how much the process is really finding—without displacing the selected primary historical lineage.
- Data Science & Analytics — Data science, analytics, and operational monitoring supplies a parallel or contributing lineage for the mechanism's defining operation: plants known defects into the inspection stream to measure each inspector's catch rate and calibrate how much the process is really finding.
- Education & Pedagogy — education_pedagogy contributes structured learning, calibration exercises, facilitation, and assessment to this mechanism's defining operation—Plants known defects into the inspection stream to measure each inspector's catch rate and calibrate how much the process is really finding—without displacing the selected primary historical lineage.
- Engineering & Design — Manufacturing inspection materially applies seeded defects.
- Mathematics — Mathematical modeling, proof, and abstract-structure practice supplies a parallel or contributing lineage for the mechanism's defining operation: plants known defects into the inspection stream to measure each inspector's catch rate and calibrate how much the process is really finding.
Review resolution: The blind reviewers disagree on primary lineage (accounting_auditing versus statistics_experimental_design). Authoritative or primary research supports statistics_experimental_design as the best historical origin: Planting known defects and measuring which are found is an experimental calibration of detection sensitivity and reviewer performance. NIST evaluates bug finders against seeded faults and NASA studies inspection defect detection; audit and education contribute the review exercise format. The cited NIST, Evaluating Bug Finders with Seeded Faults; NASA, Software Inspection and Defect Detection directly supports the mechanism's defining operation. All independently supported contributing domains are retained without an arbitrary cap. origin_mode=cross_disciplinary_synthesis records the lineage relationship, while domain_reach=multi_domain records later applicability separately from provenance.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
Notes¶
Both this exercise and Capture-Recapture Defect Estimation end at a statement about how much is being found, but from opposite directions: capture-recapture plants nothing and infers the hidden count from the overlap of two real passes, whereas this exercise plants a known set and measures the catch rate directly. When real defects and plausible seeds are hard to tell apart, they corroborate; when they diverge, the gap is itself a finding.
[n1] Error seeding (sometimes called bebugging) is the classic technique of injecting a known number of defects and estimating detection from the fraction of seeds found. Its standing caveat is the fidelity assumption: the estimate is only trustworthy to the extent the seeded defects are as hard to catch as the real ones. ↩