Skip to content

Precision / Recall Tradeoff Review

Statistical assessment — instantiates Adaptive Threshold Recalibration

Picks a threshold by weighing false-alarm burden against missed cases when positives are rare and the team that must act is finite.

Precision / Recall Tradeoff Review chooses an operating point under two conditions that dominate real deployments: positives are rare, and the people who must act on each flagged case are few. That framing is what distinguishes it from the frontier-surveying view of its ROC sibling. Under low prevalence, precision — the share of flags that are real — collapses even at a good threshold, so the true question is not abstract accuracy but where a finite response team should spend its attention. The review weighs the burden of chasing false alarms against the cost of a missed case, at the base rate that actually holds, and lands on a threshold the responders can sustain.

Example

A security team triages user-reported phishing. Genuinely malicious reports are ≈2% of the flow, and three analysts can work only so many a day. Lowering the auto-escalate threshold raises recall — more real attacks caught — but at a 2% base rate, most newly flagged items are false alarms, so precision craters and the queue balloons past what three people can clear; real attacks then wait in a backlog. The review plots precision against recall at the real 2% base rate, overlays the analysts' daily capacity, and weighs a missed attack against a wasted investigation. It settles on a threshold that keeps the queue inside capacity while an F-beta weighting nudges toward recall — because here a missed attack costs far more than a wasted hour.[1]

How it works

The defining discipline is that everything is evaluated at the operational base rate, not on a balanced test set — because precision is prevalence-dependent and a threshold that looks fine at 50% positives can be unusable at 2%. The review pairs the precision/recall curve with an explicit capacity constraint (how many positives can actually be worked) and an explicit consequence weighting (miss cost versus false-alarm cost, expressed as an F-beta or a cost ratio). The output is a single operating point justified by all three: prevalence, capacity, and cost asymmetry.

Tuning parameters

  • Operating point — precision-favoring or recall-favoring; where on the curve to sit given the cost of each error.
  • F-beta weighting — how much more recall matters than precision (or vice versa); the knob that encodes the cost asymmetry.
  • Capacity constraint — the responder throughput the flagged volume must fit inside; often the true binding limit.
  • Base-rate assumption — the prevalence the curve is computed at; the most consequential and most often mis-set input.
  • Queue prioritization — whether flagged cases are worked in score order, so scarce capacity hits the strongest signals first.

When it helps, and when it misleads

Its strength is that it is the right tool exactly when positives are rare and responders are the bottleneck — the regime where naïve accuracy metrics lie and ROC alone doesn't tell you where to stand. Its classic misuse is optimizing precision and recall on a balanced benchmark and then deploying that threshold into a low-prevalence world, where the flood of false positives buries the team — the base-rate fallacy in operational form. The discipline is to compute the tradeoff at the true base rate and against real capacity, and to revisit both as prevalence drifts.

How it implements the components

  • false_positive_false_negative_review — it makes the false-alarm-burden-versus-missed-case tradeoff the explicit object of choice.
  • consequence_weighting_policy — it encodes the asymmetric cost of a wasted investigation versus a missed case, via an F-beta or cost ratio.
  • threshold_performance_metric — it reports precision, recall, and F-beta computed at the operational base rate.

It does not display the full prevalence-free geometry of operating points (monitored_variable_or_score swept across thresholds) — Receiver Operating Characteristic Review does — nor does it check the score is calibrated (drift_detection_signal), which Calibration Curve Review owns.

References

[1] The base-rate fallacy is the tendency to judge a test's usefulness from its sensitivity and specificity while ignoring how rare positives are. When prevalence is low, even a specific test produces mostly false positives, which is why precision must be evaluated at the operational base rate. ↩