Skip to content

Outlier, Range, and Transformation Sensitivity Review

Sensitivity review — instantiates Correlation Structure Characterization

Re-computes the association with and without outliers, across restricted and full ranges, and under raw versus transformed scales, to see how much of it survives those choices.

A single reported coefficient conceals every choice that produced it — which points were kept, what slice of the variables was observed, what scale they were measured on. Outlier, Range, and Transformation Sensitivity Review is the structured re-run that perturbs exactly those three choices and reports how far the coefficient moves. Its defining question is not what is the correlation but how much of it is real — whether the association is a robust property of the relationship or an artifact of a handful of influential points, a truncated sample, or a lucky scale. Where a rolling dashboard tests whether a correlation holds over time, this review tests whether it holds over specification: same data, different defensible analytic choices, and a spread of answers instead of one.

Example

A firm has a screening test for a role and finds it correlates only weakly with later job performance — about 0.15 — and is on the verge of scrapping it. The review re-runs the association across the specification choices. The decisive one is range: performance data exist only for people who were hired, and hiring selected the high-scorers, so the observed test-score range is severely truncated. Estimated over the fuller applicant range rather than the restricted hired range, the association is materially stronger — the weak coefficient was an artifact of range restriction, a well-known phenomenon, not evidence that the test is useless.[1] The same grid also shows the coefficient is stable to dropping one high-leverage employee and barely changes under a log transform — so it is the sampling frame, not outliers or scale, that was doing the damage.

How it works

The review runs a small, pre-declared grid of re-computations and lays the results side by side. Along the outlier axis it deletes or down-weights influential points identified by a leverage measure and refits. Along the range axis it compares the restricted observed range against the relevant fuller population, applying a correction where one is warranted. Along the transformation axis it recomputes under raw, log, and rank scales. Each cell reports the resulting coefficient, so fragility appears as an explicit spread rather than being assumed away — and every exclusion and transformation applied is written down, so the reported number is reproducible rather than a lucky path.

Tuning parameters

  • Outlier rule — which influence metric (leverage, Cook's distance), what cutoff, and whether to winsorize, down-weight, or delete. Aggressive trimming stabilizes the coefficient but can amputate the real signal.
  • Range definition — what counts as the relevant "full" population and which correction (if any) is applied for restriction. Get the target population wrong and the correction misleads as badly as the truncation did.
  • Transformation set — which scales are tested (log, rank, Box-Cox). Broader sets catch scale-artifacts but widen the researcher's latitude.
  • Reporting stance — whether to publish a single robust estimate or the whole sensitivity spread. The spread is more honest; the point estimate is more usable.
  • Refit versus reweight — delete-and-refit against a robust estimator. Refitting is transparent; a robust estimator is smoother but hides which points mattered.

When it helps, and when it misleads

Its strength is that it exposes correlations that live or die on a few points, catches the attenuation and inflation that range restriction quietly causes, and prevents a conclusion that is really an artifact of a scale choice. A coefficient that survives the whole grid can be reported with far more confidence than one that only appears in a single favorable cell.

Its danger is that the very knobs it turns are researcher degrees of freedom: run the same review backwards and it becomes a search for the specification that yields the wanted answer — the garden of forking paths, where enough defensible choices guarantee a publishable coefficient. Dropping "outliers" can quietly delete the effect itself. The classic misuse is reporting only the specification that maximizes (or conveniently minimizes) the correlation. The discipline that guards against this is to pre-specify the perturbation grid and report the entire spread — with every exclusion and transform recorded — rather than the cell that flatters the story.

How it implements the components

Outlier, Range, and Transformation Sensitivity Review realizes the robustness side of the archetype — establishing whether the association is stable to defensible analytic choices:

  • stability_check — the review is the specification-stability test: does the coefficient survive perturbations of points, range, and scale.
  • sampling_frame — range restriction is a sampling-frame effect; the review re-frames the estimate to the relevant population and records which frame each estimate used.
  • provenance_record — every exclusion and transformation applied is logged as processing provenance, so the reported coefficient is reproducible and its path auditable.

It does NOT test stability over time or regime (correlation_drift_monitor, temporal_validity_window — the Rolling Correlation Dashboard) nor guard against chance across many pairs (multiplicity_control_record — Permutation Null and Multiplicity Check); nor does it select the base measure (dependence_measure_policy — Dependence-Measure Selection Matrix).

References

[1] Range restriction is the attenuation of a correlation that occurs when the observed sample spans only part of a variable's full range — classically, when selection on one variable (here, being hired) truncates it. Thorndike's correction formulas are the standard adjustment; the point for this review is that the same relationship can look weak or strong depending purely on the range observed. ↩