Cascade Error Audit¶
Test or assessment — instantiates Cascaded Hierarchical Recognition
Analyzes false positives, false negatives, delays, and reviewer disagreement by stage.
When a recognition cascade misfires, the reflex is to ask "was the classifier wrong?" — but that charges the whole system for a mistake that almost always belongs to one stage. Cascade Error Audit is the periodic assessment that decomposes the cascade's errors by the stage that produced them: which coarse gate discarded a true case, which fine test confused two close candidates, which threshold sat too tight, which handoff sat too long in a queue, and where two reviewers looked at the same case and disagreed. Its defining move is attribution — every false positive, false negative, delay, and disagreement is charged to the stage where it originated, so recalibration changes the one gate at fault instead of nudging the whole pipeline and hoping.
Example¶
A payments company screens card transactions through a cascade: a cheap velocity gate, then a rules layer, then a fine machine-learning scorer, then a human review desk. Chargebacks keep slipping through, and the instinct is to "retrain the model." Instead the risk team runs a quarterly Cascade Error Audit. They draw a sample of transactions — including a heavy oversample of the ones the cascade rejected — and attach ground truth from settled chargeback and dispute outcomes. Charging each miss to its stage, they find that most missed fraud never reached the model at all: the first velocity gate, tuned months ago to cut review volume, was silently dropping a class of low-value-but-fraudulent transactions. The fine scorer everyone wanted to retrain was fine. The audit's output is a per-stage table — false-positive rate, false-negative rate, median latency, and reviewer-disagreement rate for each stage — and it points the fix at the velocity gate's threshold, not the model.
How it works¶
- Sample and ground-truth. Pull a case sample, deliberately oversampling rejected cases, and attach true labels from the most credible downstream source.
- Attribute each error to a stage. Using the logged trail, locate the stage where a true case was dropped, a false case survived, or a case stalled.
- Compute stage-conditional metrics. For each stage: false-positive rate, false-negative rate, latency, and inter-reviewer disagreement.
- Feed recalibration. Hand the per-stage findings to whoever tunes thresholds and feature order, with the single worst-offending stage flagged.
Tuning parameters¶
- Sampling scheme — uniform versus rejected-case oversampling; oversampling rejects is the only way coarse-gate false negatives become visible at all.
- Ground-truth source — settled outcomes, expert re-adjudication, or consensus; each trades cost against how trustworthy the labels are.
- Attribution granularity — per-stage versus per-feature; finer attribution localizes the fix but costs more analyst time.
- Disagreement metric — raw agreement versus a chance-corrected statistic; the latter is fairer when one class dominates.
- Audit cadence — how often the audit runs; too rare and drift accumulates, too frequent and each run is noise.
When it helps, and when it misleads¶
Its strength is converting "the system is off" into a stage-addressable fix, and it is the standard cure for cascade opacity — the failure where only the final label is known and no one can say which gate erred.
Its central trap is that a stage can look healthy in isolation while the cascade as a whole degrades: fixing each stage to its own local metric can raise the aggregate false-negative rate, a stage-wise version of Simpson's paradox.[n1] The classic misuse is auditing only the cases the cascade acted on — the surviving stream — so the true cost, the rare cases quietly rejected at stage one, never enters the sample. The discipline is to always evaluate any stage change against end-to-end outcomes and to oversample rejected cases so coarse-gate misses cannot hide.
How it implements the components¶
feedback_calibration_loop— the audit's per-stage error table is the recalibration signal that drives threshold and feature-order changes; it closes the loop between observed harm and cascade tuning.false_negative_guardrail— the rejected-case-sampling facet: by deliberately re-examining cases the cascade threw away, it is how coarse-gate false negatives surface for correction.recognition_cost_budget— it measures per-stage latency and review load, charging delay to the stage that caused it so the cost side is audited alongside accuracy.
It measures thresholds but does not set them (stage_threshold_set is multi_stage_classifier_pipeline) and it does not itself carry the working candidate set (candidate_set_carrier is drilldown_recognition_interface); the audit reads the trail, it does not run the cascade.
Related¶
- Instantiates: Cascaded Hierarchical Recognition — the recalibration assessment that keeps the cascade honest.
- Consumes: Stage Transition Log — the per-case trail the audit reads to attribute errors to a stage.
- Sibling mechanisms: Coarse Screening Checklist · Drilldown Recognition Interface · Expert Review Checkpoint · Feature Pyramid or Hierarchical Model · Multi-Stage Classifier Pipeline · Salience Map or Attention Heatmap · Stage Transition Log · Triage Queue with Escalation Rules
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Analyzes false positives, false negatives, delays, and reviewer disagreement by stage, making its operative form a bounded evaluation of existing evidence or work that produces a finding or disposition.
Independent corroboration: The frozen evidence defines Cascade Error Audit as 'Analyzes false positives, false negatives, delays, and reviewer disagreement by stage', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Data Science & Analytics
Origin pattern: Convergent development
Present-day reach: Specialized
Rationale: Machine-learning evaluation established stage-wise accounting of false positives, false negatives, delays, and disagreement in cascades.
Related originating lineages:
- Statistics & Experimental Design — Diagnostic error analysis supplies sampling, attribution, and end-to-end validation against local-metric paradoxes.
Review resolution: Data science is primary because the method audits errors at each stage of a classifier-and-review pipeline, including false positives, false negatives, latency, disagreement, and rejected cases. Statistical error analysis is a genuine contributing lineage; organizational review is an implementation setting rather than a co-origin.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
Notes¶
[n1] Simpson's paradox — a trend that appears within each subgroup can reverse when the subgroups are combined. In a cascade, each stage's local metrics can all look acceptable while the end-to-end false-negative rate worsens, which is why stage changes must always be checked against aggregate outcomes. ↩