Sample Audit of Exclusions¶

A quality-control review — instantiates Search Space Pruning

Re-examines a representative sample of what was pruned — not what was kept — to catch false negatives, bias, and drift before a filter quietly discards the answers that mattered.

Sample Audit of Exclusions is the mechanism that watches the other siblings. Every pruning method produces two piles — kept and cut — and almost all quality checks look only at the kept pile. This one deliberately reaches into the discard pile, pulls a representative sample of what was excluded, and re-evaluates it fully to answer the question a pruning system cannot ask about itself: did we throw away something we should have kept? Its defining focus is the false negative — the good candidate silently deleted — which is invisible precisely because it left the process. By sampling exclusions, holding out pruned material for a proper look, and logging what the deeper review finds, it turns a filter's blind spot into a measured error rate, and it feeds that finding back as a trigger to fix the criteria.

Example¶

A social platform auto-removes posts flagged as policy-violating, moving through millions a day. Reviewing the removals for correctness catches over-blocking, but says nothing about the pile that was kept down — and nothing about good content wrongly taken down at scale. A Sample Audit of Exclusions instead draws a representative sample of removed posts each week and has trained reviewers re-judge them from scratch.

The audit finds that satire and reclaimed-slur posts are removed at several times the rate of a control — a false-negative pattern (good content wrongly excluded) that the removal logs alone never surfaced, because nobody was looking at what left. The finding is logged against the specific rule, and because it crosses a threshold, it triggers a criteria update: the classifier's satire handling is revised. The audit did not do the pruning; it measured what the pruning was quietly getting wrong.

How it works¶

Sample the discards, representatively — draw from the excluded pile (stratified by rule, segment, or time), not from survivors, so the sample reflects real exclusions.
Re-evaluate at full depth — judge each sampled item properly, as if it had never been cut, to reveal genuine false negatives.
Log against the responsible rule — record what was wrongly excluded, by which rule, under what assumptions, extending the pruning audit trail.
Trigger criteria updates — when the measured false-negative or bias rate crosses a threshold, escalate a fix to the filter's owner.

The distinctive commitment is looking where no one else does — at what left the system — because that is the only place a false negative can be seen.

Tuning parameters¶

Sample size and stratification — how many exclusions, drawn how. Larger, well-stratified samples detect rarer or segment-specific false negatives at more review cost.
Audit cadence — one-off, periodic, or continuous. Continuous auditing catches drift fast; periodic is cheaper but lets errors accumulate between passes.
Depth of re-evaluation — a quick second look versus a full independent adjudication; deeper review finds subtler misses but costs more per item.
Escalation thresholds — what false-negative or disparity rate triggers a criteria change versus a note; too high and real problems are tolerated, too low and the filter thrashes.
Blinding — whether auditors see the original exclusion reason; blinding reduces confirmation of the first decision.

When it helps, and when it misleads¶

Its strength is making a filter's invisible errors visible and measurable. It is the standing guard against premature closure — the reason a pruning system can be trusted to remove aggressively, because someone is checking the discard pile for the option that mattered, and turning "we probably didn't lose anything" into a number.

It misleads when the sample is drawn only from survivors or from a convenient slice — auditing what you kept tells you nothing about what you lost, a survivorship blind spot that makes a broken filter look healthy.^[1] A too-small or non-representative sample misses exactly the rare, segment-specific false negatives that matter most, and an audit with no teeth — findings logged but never triggering a criteria fix — becomes theater. The disciplines are to sample the excluded pile representatively, size it to detect the errors you actually fear, and wire findings to a real update trigger and owner.

How it implements the components¶

false_negative_review — this is its core purpose: systematically detecting valuable or safety-critical candidates that were wrongly excluded.
representative_sample_holdout — it holds out and re-examines a representative sample from pruned regions, the only view into what a filter is losing.
pruning_audit_trail — it consumes and extends the record of what was pruned by which rule, attaching audited outcomes to those entries.
criteria_update_trigger — a measured error or bias rate above threshold is exactly the event that requires the pruning rules to be revisited.

It does no pruning itself and defines no filter; it is the review layer over siblings like Constraint Filtering, Negative Keyword Filter, and Red-Flag Screen, and it does not own the criteria — that accountability sits with Eligibility Screening's owner or the filter's operator.

Instantiates: Search Space Pruning — Sample Audit of Exclusions is the archetype's safeguard against premature closure, operating on any sibling's discard pile.
Consumes: the exclusion output and pruning audit trail of whatever filter it audits.
Sibling mechanisms: Red-Flag Screen · Eligibility Screening · Branch and Bound · Beam Search · Constraint Filtering · Decision Tree Pruning · Dominated-Option Removal · Negative Keyword Filter · Safety or Compliance Exclusion · Shortlisting · Triage Filter

References¶

[1] Survivorship bias is drawing conclusions only from the cases that made it through, ignoring those that didn't. A filter audited solely on what it kept is survivorship bias by construction — the false negatives, having been removed, are exactly the cases excluded from the review. ↩