Skip to content

Base-Rate and Precision Dashboard

Monitoring dashboard — instantiates Signal Value Preservation

Tracks how often a signal fires, how often a firing is actually correct, and how the two move over time, so quiet erosion of precision becomes visible before receivers give up on the channel.

The Base-Rate and Precision Dashboard measures whether a signal still separates real cases from background: it reports firing frequency, precision (the share of firings that correspond to the genuine condition), and the event base rate, and it tracks how all three move over time. The one idea that makes it this mechanism is that it watches the signal's own correctness, not the receiver's behavior — it is the instrument that catches devaluation early, when the channel is still firing confidently but meaning less per firing. It sits upstream of every human judgment about the channel, turning "the alerts feel noisy lately" into a measured precision trend read against the base rate that makes a rare-event signal hard in the first place.

Example

A bank's card-fraud system texts customers to confirm suspicious transactions, and analysts have a nagging sense that too many blocks are hitting legitimate purchases. The dashboard makes the suspicion concrete: firing rate is up roughly 40% quarter over quarter, but precision — blocked transactions that were truly fraud — has slid from about one in four to about one in nine, against a fraud base rate well under 1% of transactions. A ledger records each false alarm (a real customer blocked) and each known miss (fraud that slipped through). The picture is unmistakable: the model is firing more and meaning less. That reading is the trigger to retune before customers learn to ignore texts from their own bank — the point at which the channel would be lost regardless of its accuracy.

How it works

Every firing is logged with its eventual outcome — confirmed true, false alarm, or miss — alongside the underlying event base rate, and the dashboard computes precision, firing rate, and their trends per signal type. The false-alarm-and-miss ledger keeps each error attributable rather than aggregated away, and precision is always displayed against the base rate, so a signal for a rare condition is not flattered by a hit rate that ignores how uncommon the condition is.

Tuning parameters

  • Outcome-labeling method — how a firing is confirmed true or false: adjudication, delayed ground truth, or sampling. Cheaper labeling is faster but noisier, and some outcomes are only knowable long after the firing.
  • Aggregation window — short windows catch fast erosion but jump around; long windows are stable but slow to raise the alarm.
  • Precision-versus-recall emphasis — how heavily the view weights false alarms against misses; a miss-averse domain (safety) reads the same numbers very differently from a false-alarm-averse one (customer friction).
  • Base-rate stratification — whether precision is shown pooled or split by segment and condition, exposing a bad slice that a single blended number would hide.

When it helps, and when it misleads

Its strength is that it converts a vague sense of noise into an objective precision trend against base rate — the earliest measurable sign that a signal is devaluing. Its failure modes come from the base rate itself: the base rate fallacy cuts both ways, so a signal can look "accurate" while being mostly false alarms whenever the condition is rare, and misses are systematically undercounted because true cases that never fired are invisible to the ledger. The dashboard is also easily run backwards — a pet detection defended by choosing a flattering window or aggregation. The discipline that keeps it honest is to fix the window in advance, stratify precision by base rate, and invest in discovering misses, not just counting false alarms.[^base_rate]

How it implements the components

  • signal_base_rate_monitor — its core: it tracks firing frequency and the underlying event base rate, so a rare-condition signal's inherent difficulty is always on screen.
  • false_alarm_and_miss_ledger — it logs every false alarm and known miss, making precision computable and each error attributable to a firing.

It does not measure whether receivers still act on firings (habituation_and_adaptation_monitor) — that's Receiver Response Dashboard — nor adjudicate and assign individual false positives (issuer_accountability_hook) — that's False-Positive Review Board.