Distance-Choice Sensitivity Analysis¶
Sensitivity analysis — instantiates Metric-Space Specification and Validation
Perturbs the distance function and measures how much the resulting neighborhoods and decisions move, exposing conclusions that depend on an arbitrary metric choice.
A distance number always looks definite, but the definiteness can be an illusion of one particular formula. Distance-Choice Sensitivity Analysis deliberately swaps the distance function for plausible alternatives — Euclidean for Manhattan, cosine for correlation, one feature weighting for another — and watches how far the answers move. Its defining idea is that a metric-space conclusion is only trustworthy if it survives reasonable variation in the rule that produced it: if the ranked neighbors, the cluster boundaries, or the final decisions reshuffle every time the formula changes, then the result was an artifact of representation, not a fact about the domain. It does not ask whether the metric is correct against ground truth; it asks whether the conclusion is stable across the space of defensible metrics.
Example¶
A lending team builds a k-nearest-neighbor model that flags a new loan application as high-risk by finding the most similar past applicants and reading off their default rates. Before trusting it, an analyst runs a sensitivity sweep. The baseline uses Euclidean distance over standardized features; the analyst re-runs the same applicants under Manhattan distance, under a version that down-weights income, and under a cosine similarity that ignores magnitude. For most applicants the neighbor set barely changes and the risk flag holds. But for a cluster of thin-file applicants, the "five nearest" turn over almost completely between Euclidean and cosine — and the risk flag flips with them.
The output is not a better metric but a map of fragility: the model's verdict is robust for established borrowers and unreliable for thin-file cases, where the choice of formula silently decides the answer. That finding reroutes those borderline applications to manual review instead of letting an arbitrary formula make the call, and it tells the team exactly which subpopulation any future metric change must be re-tested against.
How it works¶
- Assemble a slate of plausible metrics, not arbitrary ones — variants a reasonable practitioner might have picked first: alternative \(L_p\) norms, weighted versus unweighted features, magnitude-sensitive versus magnitude-free scores.
- Hold everything else fixed — the same domain, the same items, the same downstream rule — so that only the distance function varies. The point is to isolate the metric as the source of any movement.
- Re-run the downstream artifact (the neighbor list, the clustering, the routing decision) under each variant and record the output.
- Quantify the disagreement, not just note it: rank-correlation between neighbor lists, Jaccard overlap of top-k sets, the fraction of items whose decision flips. Stability is a measured quantity, reported per-subgroup because fragility is rarely uniform.
Tuning parameters¶
- Perturbation breadth — how wide a family of alternative metrics you sweep. A narrow sweep (two norms) is cheap but can miss fragility; a broad sweep (norms × weightings × normalizations) is thorough but combinatorial.
- Disagreement statistic — top-k overlap, rank correlation, or decision-flip rate. Overlap is forgiving; decision-flip rate is what the stakeholder actually feels. Choose the one closest to the action.
- Granularity of reporting — a single global stability score versus per-subgroup breakdowns. Global scores hide exactly the pockets (like thin-file borrowers) where the metric decides everything.
- Stability tolerance — how much movement you are willing to call "robust." A strict tolerance sends more cases to human review; a loose one trusts the baseline metric more.
When it helps, and when it misleads¶
Its strength is deflating false certainty: it turns "the nearest neighbors say X" into "the nearest neighbors say X and that holds across the metrics we'd have plausibly chosen," or else names the cases where they don't. It is especially valuable in high dimensions, where distances tend to concentrate and many formulas produce near-identical, near-meaningless separations[n1] — sensitivity analysis is how you discover that your neighborhoods are noise before you ship them.
Its failure mode is mistaking agreement for correctness. Several metrics can concur and all be wrong in the same direction — if every variant shares a mis-scaled feature, they will agree on a distorted neighborhood and the sweep will bless it. Sensitivity analysis measures robustness, never validity; a stable-but-meaningless metric passes cleanly. The classic misuse is running a token two-metric sweep, declaring the result "robust," and treating that as evidence the metric is right. The guarding discipline is to pair the sweep with an external check of semantic fit and to report stability per-subgroup, so that a reassuring global average never buries a subpopulation the metric is quietly governing by coin-flip.
How it implements the components¶
distance_function_candidate— it treats the distance rule as the variable under test, generating and running a slate of candidate functions rather than committing to one.invariance_requirement_set— its core deliverable is a statement of which conclusions must remain invariant under metric change; a decision that flips across plausible metrics fails the invariance requirement.nonmetric_score_boundary— when a similarity score's rankings prove wildly unstable, the sweep is often what exposes that it was never behaving like a metric, marking the boundary where metric guarantees stop applying.
It does not fix a neighborhood_threshold_policy or score neighbors against a labeled gold set — that accuracy-against-truth work belongs to Nearest-Neighbor Benchmark, its nearest twin; this mechanism measures stability under perturbation, not correctness. Nor does it convene experts to define proximity_semantics_contract — that is Domain Expert Calibration Panel.
Related¶
- Instantiates: Metric-Space Specification and Validation — it supplies the robustness evidence the archetype requires before a metric is trusted downstream.
- Consumes: Feature Scaling and Normalization Pipeline — the normalized representation is one of the choices the sweep perturbs.
- Sibling mechanisms: Nearest-Neighbor Benchmark · Domain Expert Calibration Panel · Distance Threshold Review · Metric Axiom Test Suite · Pairwise Distance Matrix · Graph Shortest-Path Metric · Triangle-Inequality Counterexample Search
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Distance-Choice Sensitivity Analysis operates as a computation, comparison, model, or analytic representation used to infer, estimate, or choose because it perturbs the distance function and measures how much the resulting neighborhoods and decisions move, exposing conclusions that depend on an arbitrary metric choice.
Independent corroboration: The frozen evidence defines Distance-Choice Sensitivity Analysis as 'Perturbs the distance function and measures how much the resulting neighborhoods and decisions move, exposing conclusions that depend on an arbitrary metric choice', so its operative form is Analysis, Modeling & Optimization.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Data Science & Analytics
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Specialized
Rationale: Machine-learning and data-analysis practice cohered perturbing distance functions and feature weights to test whether neighborhoods, clusters, and decisions are representation-robust.
Related originating lineages:
- Mathematics — Metric-space theory supplies the admissible distance functions and invariance concepts being varied.
- Statistics & Experimental Design — Sensitivity and robustness analysis supplies the discipline of carrying plausible specifications through to conclusions.
Review resolution: Both current reviews place distance_choice_sensitivity_analysis primarily in data_science; the reconciled classification retains only lineages that materially shaped the mechanism and keeps breadth of origin separate from reach.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] The curse of dimensionality (a term coined by Richard Bellman): as the number of dimensions grows, the distances between all pairs of points tend to converge, so the ratio between the nearest and farthest neighbor approaches one and "nearest" loses discriminating power. This is precisely the regime where a sensitivity sweep is most diagnostic. ↩