Permutation Tests¶
Resampling test — instantiates Assumption-Light Inference
Builds an exact null by reshuffling the labels the hypothesis says are exchangeable, replacing a distributional assumption with a randomization one.
Permutation Tests answer "could this difference be chance?" by literally simulating chance. If the null hypothesis says a grouping label carries no information, then the observed data are just one of the many equally likely arrangements you would get by shuffling the labels — so you compute the test statistic on the real labeling, then recompute it on thousands of random relabelings, and read where the real value falls in that shuffled distribution. Its defining move is to swap one assumption for another, honestly: it discards the parametric distributional assumption entirely, but in its place it requires exchangeability — that under the null the labels really are interchangeable. That substitution is the whole mechanism, and stating the exchangeability condition as the result's interpretation limit is inseparable from it.
Example¶
An agronomist runs a field trial: 12 plots, half randomly assigned a new fertilizer, half a control, with yield measured at harvest. The treated plots average higher, but with only 12 plots and soil that varies across the field, no parametric distribution for yields is credible. Because the treatment was randomly assigned, though, the null has a precise meaning: if fertilizer did nothing, which plots got labeled "treated" was arbitrary. So the agronomist computes the mean difference, then re-assigns the treated/control labels across the 12 plots in every way the randomization allowed, recomputing the difference each time.
The observed difference sits in the top 3% of that permutation distribution. The p-value is therefore exact and rests on nothing but the coin-flips the experiment itself performed. The report states the limit plainly: the conclusion is valid because treatment was randomized, and it would collapse if the plots were not exchangeable under the null — for instance if the treated plots shared a wetter corner of the field. The randomization that created the data is exactly what licenses the inference.
How it works¶
- Name the exchangeable unit. Identify what the null says is interchangeable — group labels, pairing, time order — because that is what will be shuffled and what the validity rests on.
- Fix the permutation plan. Choose the statistic, the set of admissible relabelings (all permutations, or those the randomization design permits), and how many to sample.
- Build the null by shuffling. Recompute the statistic across relabelings to form the exact reference distribution the null implies, then locate the observed statistic within it.
- State the exchangeability limit. Report the p-value together with the condition that earns it — the labels being exchangeable under the null — since that assumption, not a distribution, is now doing the work.
Tuning parameters¶
- Permutation scope — exhaustive (all relabelings) versus a random Monte-Carlo subset. Exhaustive is exact but only feasible for small samples; sampling is scalable with a small simulation error.
- Test statistic — mean difference, rank sum, trimmed statistic, anything. The choice tunes what alternative the test is sensitive to without changing the null machinery.
- Exchangeability structure — free permutation versus restricted (within blocks, respecting pairing or clustering). Restricting to the design's real structure is what keeps the exactness honest under dependence.
- Resample count — how many random permutations. More sharpen the p-value's resolution near the decision threshold at a computational cost.
When it helps, and when it misleads¶
Its strength is exactness with minimal assumptions: when data come from a genuine randomization, a permutation test gives a valid p-value with no distributional assumption at all — the inference is guaranteed by the experiment's own design. This is the logic of R. A. Fisher's randomization test, the Lady Tasting Tea, where the physical act of random assignment is what makes the probability statement true.[n1]
Its failure mode is treating exchangeability as free. If the labels are not exchangeable under the null — because of confounding, clustering, time trends, or unequal variances masquerading as a location shift — the shuffled distribution is the wrong reference and the exact-looking p-value is exactly wrong. The classic misuse is applying a permutation test to observational data as if reshuffling recreated a randomized experiment; it does not, because nothing was ever randomized. Permutation also tests a sharp null (no effect for anyone) that can reject for reasons other than the shift of interest. The guarding discipline is to permute only within the structure the design actually supports and to carry the exchangeability condition as the stated limit on every result.
How it implements the components¶
resampling_or_permutation_plan— the explicit relabeling scheme (exchangeable unit, admissible permutations, count, statistic) is the mechanism's core artifact and the source of its reference distribution.assumption_light_method_choice— trading a distributional assumption for a randomization/exchangeability one is a deliberate assumption-light selection made against a named parametric threat.interpretation_limit— the exchangeability condition travels with the p-value as the boundary that earns it; the result is valid only within that stated assumption.
It builds a null by shuffling labels; it does not resample the data with replacement to gauge an estimate's stability, nor does it define the rank rule a statistic might use. It does NOT implement sensitivity_comparison — that belongs to Bootstrap-Like Checks, the resampling sibling that measures stability rather than testing a null — nor rank_or_distribution_free_rule, which is Rank-Based Methods.
Related¶
- Instantiates: Assumption-Light Inference — supplies an exact test whose validity rests on randomization rather than distribution.
- Sibling mechanisms: Bootstrap-Like Checks · Nonparametric Tests · Rank-Based Methods · Model Comparison Table · Robust Statistics · Median-Based Summaries · Assumption Audit Checklist · Diagnostic Plot Review
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Permutation Tests operates as an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution because it builds an exact null by reshuffling the labels the hypothesis says are exchangeable, replacing a distributional assumption with a randomization one.
Independent corroboration: The frozen evidence defines Permutation Tests as 'Builds an exact null by reshuffling the labels the hypothesis says are exchangeable, replacing a distributional assumption with a randomization one', so its operative form is Analysis, Modeling & Optimization.
Nearest alternative: Experiment, Test & Rehearsal — Permutation Tests includes features of an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation, but its defining operation is an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Permutation Tests is rooted in experimental design and statistics: Fisher's randomization-test tradition derives exact null probabilities from exchangeable assignments.
Review outcome: Independent reviewer agreement; high confidence.
Notes¶
[n1] R. A. Fisher's randomization test — illustrated by the Lady Tasting Tea, in which a taster claims to tell whether milk or tea was poured first — grounds the p-value in the physical randomization of the experiment: under the null, every assignment of labels was equally likely, so the observed arrangement's rarity among all possible ones is an exact probability requiring no distributional model. ↩