Skip to content

Permutation Null and Multiplicity Check

Statistical method — instantiates Correlation Structure Characterization

Builds a chance baseline by shuffling the pairing and corrects for how many correlations were examined, so the largest coefficient in a big matrix isn't mistaken for a real one.

Scan enough variable pairs and some will correlate strongly for no reason at all — the maximum of many chance coefficients is large by construction. Permutation Null and Multiplicity Check is the two-part guard against being fooled by that. First it permutes — repeatedly shuffles one variable to break the real pairing — building an empirical null: the distribution of correlations you would see if the variables were truly independent, so an observed coefficient is judged against chance rather than against zero. Second it corrects for the number of pairs examined, because a threshold that is stringent for one test is trivially cleared by the best of half a million. Its defining purpose is to defend against spurious correlation from search — both single-test luck and the multiple-comparisons inflation of trawling a whole matrix. Where a bootstrap interval quantifies the uncertainty of one estimate, this method asks whether an association clears a chance baseline at all, across many.

Example

A bioinformatician computes pairwise correlations across roughly a thousand gene-expression variables — on the order of half a million pairs. Dozens exceed |r| = 0.5, and it is tempting to read each as a discovered co-regulation. The permutation null shows what magnitudes arise under independence at this sample size: with that many pairs, coefficients of that size appear by chance in abundance. A false-discovery-rate correction then sets the surviving threshold so that only pairs beyond the chance envelope — a small, defensible shortlist — are reported as real. The check turns a wall of impressive-looking coefficients into the handful that a chance baseline cannot explain away.

How it works

The method manufactures its null from the data itself rather than assuming normality: permuting one variable many times yields the correlations expected under independence, valid even when the data are non-normal. On top of that it applies a multiplicity correction — family-wise (Bonferroni, Holm) or false-discovery-rate (Benjamini–Hochberg) — tied explicitly to the size of the family of tests performed. Crucially, both the family definition and the correction are recorded, so significance cannot be quietly re-earned by shrinking the family after the results are in. The output is a filtered set of associations, each having cleared a chance baseline appropriate to how many were examined.

Tuning parameters

  • Number of permutations — how many shuffles build the null. More permutations sharpen the tail estimate (needed for small p-values) at a compute cost.
  • Correction method — family-wise (Bonferroni/Holm) versus false-discovery-rate (Benjamini–Hochberg). The former controls any false positive and sacrifices power; the latter tolerates a controlled fraction of them and keeps more discoveries.[1]
  • Family definition — which set of tests counts as "the family" being corrected over. This is the highest-leverage and most-abused dial: enlarge it and everything fails, shrink it and everything passes.
  • Sidedness — one- versus two-sided null, depending on whether the sign of the association was predicted in advance.
  • Significance / FDR level — the q or α at which a pair survives. Looser levels surface more candidates and more false ones.

When it helps, and when it misleads

Its strength is that it kills the spurious correlations that large scans manufacture, uses a distribution-free null that holds when parametric assumptions fail, and — most valuably — makes the number of tests an explicit, auditable input rather than a hidden inflator of significance.

Its failure modes cluster around honesty and assumptions. The correction is only truthful if the family is declared truthfully — redefining the family after the fact re-inflates significance, which is precisely the classic misuse: run the check, then quietly redraw the family boundary until the favored pair passes. Permutation assumes exchangeability, which time-series or segment structure violates — shuffle autocorrelated data naively and the null is invalid, understating how often chance produces large coefficients. And "significant" still means only beyond chance, not large, stable, or causal. The discipline that keeps it honest is to pre-declare the family and the correction, use a block permutation that preserves autocorrelation for time series, and report survivors with their effect sizes rather than significance alone.

How it implements the components

Permutation Null and Multiplicity Check realizes the chance-and-multiplicity side of the archetype — separating real association from the patterns that search invents:

  • multiplicity_control_record — the declared family size and the applied correction are its core record; the method exists to make multiplicity explicit rather than silent.
  • evidence_and_source_basis — the empirical null is the evidential basis for claiming an association is more than chance, and it is derived from the data rather than an assumed distribution.
  • audit_trail — permutation count, random seed, family definition, and correction are logged so the significance verdict is reproducible.

It does NOT quantify the uncertainty band of a single estimate (uncertainty_model — the Bootstrap Association Interval) nor detect nonlinear form (correlation_or_dependence_profile, relation_strength_indicator — Nonlinear Dependence Screen); and the causal reading of a survivor stays gated by causal_claim_guard / causal_claim_scope (the Causal-Claim Labeling Template).

Notes

Clearing this check licenses exactly one claim — that the association is unlikely to be chance. It says nothing about whether the association is large (an effect size), stable (the sensitivity review and rolling dashboard), or causal (the labeling template). Each of those is a separate guard, and passing this one does not stand in for the others.

References

[1] The Benjamini–Hochberg procedure controls the false discovery rate — the expected proportion of false positives among the rejected hypotheses — rather than the probability of any false positive at all. On a half-million-pair scan it is far less conservative than Bonferroni and so preserves genuine discoveries that family-wise control would discard, which is why it is the usual choice for wide correlation screens. ↩