Proximity Signal Backtest¶
Validation procedure — instantiates Counterfactual Proximity Signal Calibration
Checks against history whether past near-miss proximity signals actually foreshadowed later harm, learning, or improvement, and recalibrates the signal that did not.
A Proximity Signal Backtest turns the question inward: does the near-miss signal we have been emitting actually predict anything? It takes a body of historical proximity scores and checks them against what subsequently happened — did the cases flagged "very close" go on to produce more real harm, more useful learning, or more improvement than the cases flagged "distant"? Its defining move is retrospective validation across many cases, not investigation of one: it is measuring the signal's track record, then adjusting the scoring rule to fit the evidence. If close calls scored high but nothing downstream ever correlated with the score, the backtest says so and forces recalibration. This is the mechanism that keeps proximity scoring from being a self-satisfied ritual with no demonstrated predictive value.
Example¶
An air navigation service provider has years of recorded loss-of-separation events — instances where two aircraft came closer than the required spacing but did not collide — each carrying a proximity score at the time (closest approach in nautical miles and seconds to conflict). A Proximity Signal Backtest asks whether those scores earned their keep. It lines the historical scores up against later outcomes: subsequent incidents in the same sectors, procedural changes that followed, controller reports. Suppose it finds that horizontal-distance scores barely separated the sectors that later had problems from those that did not, while time-to-conflict scores did — the sectors with the tightest timing margins really were the ones that produced later trouble. The backtest's conclusion is a recalibration: down-weight raw horizontal distance, up-weight closing time in the proximity metric. The signal is not thrown out; it is retuned to the axis history shows actually predicts.
How it works¶
The procedure needs a historical corpus of proximity scores paired with realizable later outcomes. It defines the proximity metric under test — the exact scoring rule whose predictive value is in question — and then runs the calibration feedback loop: compare each past score to the outcome that followed, measure how well the scores discriminated (did higher proximity actually track higher later harm/learning), and feed the discrepancy back as an adjustment to the metric's dimensions and weights. Guarding against fooling itself, it separates the data used to tune from the data used to check, so a metric cannot be graded on the very cases that shaped it. The output is an updated, evidence-fitted scoring rule plus an honest statement of how much predictive power it actually has.
Tuning parameters¶
- Outcome horizon — how far forward "did it predict?" looks. Short horizons give fast feedback but miss slow-burning consequences; long horizons are more meaningful but muddied by intervening changes.
- Discrimination metric — how predictive skill is scored (rank correlation, hit rate on the top band, calibration curve). Different metrics reward different virtues of the signal.
- Train/test split — how strictly tuning data is held apart from validation data. Stricter separation guards against overfitting but needs more history.
- Recalibration aggressiveness — how hard weights are moved toward the latest evidence. Aggressive updates chase noise; timid updates ossify a stale metric.
When it helps, and when it misleads¶
Its strength is accountability: it is the only sibling that can tell you the proximity score has been wrong, and it converts an intuition-driven metric into one that has survived contact with its own history. Its failure mode is overfitting the past — with enough dimensions and enough tuning, a metric can be made to "predict" historical outcomes perfectly by memorizing noise, a form of data-snooping bias that collapses the moment it meets new cases.[n1] It is also vulnerable to survivorship gaps: near-misses that were never scored, or whose later outcomes went unrecorded, quietly bias the check. The guarding discipline is strict out-of-sample validation, preferring a simpler metric that generalizes over a complex one that fits, and reporting the honest predictive power rather than the flattering in-sample fit.
How it implements the components¶
calibration_feedback_loop— it is the loop: compare historical proximity scores to later outcomes and feed the gap back as an adjustment to the scoring rule.proximity_metric_or_ordering— the metric under test is its object of study; the backtest re-weights and reshapes the proximity dimensions to fit the evidence.
It does NOT implement factual_outcome_anchor or learning_update_boundary — investigating a single almost-event and bounding its follow-up actions is Close-Call Review Protocol's job; this procedure validates the signal across the whole history rather than working one incident.
Related¶
- Instantiates: Counterfactual Proximity Signal Calibration — supplies the recalibration step that keeps the archetype's proximity signal empirically honest.
- Consumes: Near-Miss Distance Scorecard supplies the historical proximity scores whose predictive value is tested.
- Sibling mechanisms: Close-Call Review Protocol · Almost-Reward Annotation · Counterfactual Plausibility Filter · Counterfactual Value-Delta Table · Near-Miss Distance Scorecard · Near-Miss Response Tier · Regret-Weighted Decision Log · Salience Overweighting Check · Threshold Band Map
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Proximity Signal Backtest operates by joins past proximity scores to outcomes and computes discrimination, calibration, and a revised scoring rule. That concrete deployed or enacted form is Analysis, Modeling & Optimization under the frozen taxonomy.
Nearest alternative: Assessment, Review & Assurance — Although Assessment, Review & Assurance can support this mechanism, the frozen evidence makes its operative form the act that joins past proximity scores to outcomes and computes discrimination, calibration, and a revised scoring rule; the alternative is therefore secondary rather than defining.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Testing historical predictive performance and recalibrating a signal is a statistical validation procedure.
Related originating lineages:
- Data Science & Analytics — Time-indexed feature backtesting and recalibration supplied the operational workflow.
- Disaster Management & Risk Reduction — Near-miss and precursor indicators are materially developed in hazard and safety learning.
- Security Studies & Intelligence Analysis — Warning-indicator analysis supplies an independent lineage for threat proximity signals.
Review resolution: Both blind reviewers agree on statistics_experimental_design as the primary origin. Explicit reconciliation resolves reported_ambiguity, alternate_origin_disagreement. The merged alternate lineages retain only domains the reviewers identified as materially formative; domain_reach=multi_domain records later applicability separately from origin breadth.
Attribution caveat: The specific proximity-signal formulation is an encyclopedia synthesis over standard backtesting.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; medium confidence.
Notes¶
[n1] Data-snooping bias (also called look-ahead or overfitting bias in backtesting) is the inflation of apparent predictive skill that comes from tuning and evaluating a rule on the same historical data. It is why a proximity metric must be validated on cases it was not fitted to before its predictive power is believed. ↩