Hypothesis Testing (Null vs. Alternative)¶

Version: v2 · 2026-08-30 · History
Prime #: 434
Domain group: Formal Sciences
Origin domain: Experimental Design & Statistics
Aliases: Null Hypothesis Significance Testing, NHST, Significance Testing, Frequentist Hypothesis Testing
Related primes: Statistical Significance (p-Value), Statistical Power, Type I & Type II Errors, Confidence Intervals, Randomization, Sampling (Representativeness), Effect Size, Bayesian Updating, Multiple Comparisons Correction, Reproducibility & Replicability

Core Idea¶

Hypothesis Testing frames an inquiry around a "null hypothesis" (often positing no effect or difference) and an "alternative hypothesis" (the effect or difference is real), using data to decide whether to reject or fail to reject the null.

How would you explain it like I'm…

Picking a Rule Before Peeking

Imagine you say a coin is fair. Before you flip it, you decide: if it lands on heads way too many times, you'll stop believing it's fair. So you flip a bunch and count. If heads shows up too much, you change your mind. You picked your rule before you peeked, so you can't trick yourself.

Testing Two Rival Guesses

Hypothesis testing is a way to check an idea using data. First you write two guesses: a boring one (called the null, like 'this new medicine does nothing extra') and an interesting one (the alternative, like 'it actually helps'). Before looking at the results, you set a rule for how surprising the data must be to make you reject the boring guess. Then you collect data and follow your rule. Setting the rule first stops you from cheating by changing it after you peek.

Null vs. Alternative Hypothesis Testing

Hypothesis testing is a formal way scientists decide whether evidence is strong enough to overturn a default claim. You write the null hypothesis (usually 'no effect') and an alternative ('there is an effect'). Then you pick, in advance, how unlikely the data would have to be under the null before you reject it; this cutoff is the significance level, often 5%. After running the study, you compute a p-value: the probability of seeing data this extreme if the null were true. If the p-value is below your cutoff, you reject the null. Locking in the rules ahead of time prevents cherry-picking and keeps the long-run false-alarm rate controlled.

Hypothesis testing is a decision framework for handling uncertainty in samples. You start with a null hypothesis (H0), typically asserting 'no effect' or some baseline parameter value, and an alternative hypothesis (H1) asserting H0 is wrong in a specified way. Before collecting data, you pick a test statistic (a number computed from data whose distribution under H0 is known) and a significance level alpha (the long-run probability of rejecting H0 when it is actually true, called a Type I error). You then gather data, compute the test statistic, and compare it to a critical threshold; equivalently, you compute a p-value (the probability of data at least as extreme as observed if H0 were true) and reject H0 when p is below alpha. The modern framework fuses Fisher's evidential p-value with Neyman-Pearson decision rules. Common pitfalls: misreading the p-value as 'probability H0 is true,' treating alpha=0.05 as principled rather than conventional, and publication bias toward significant results.

Tensions in Practice¶

Pre-specified tests in tension with discovery

Broad Use¶

Pharmacological Trials: Null: "Drug A and placebo have no difference in recovery rates." Alternative: "Drug A improves recovery more than placebo."
Psychology Experiments: Null: "Average response time is the same in both conditions." Alternative: "Condition 2 yields faster responses."
Manufacturing Quality: Null: "New assembly method doesn't affect defect rate," vs. Alternative: "It lowers defects."
Marketing A/B Tests: Null: "Email subject lines have the same open rates," vs. Alternative: "Subject line B yields higher open rates."

Clarity¶

Hypothesis testing sets up a structured approach: gather evidence about a proposed difference/effect, weigh it against chance variation, then conclude whether data justifies rejecting the null.

Manages Complexity¶

Formal testing (with significance levels, p-values, confidence intervals) distills messy real-world data into a yes/no inference about an effect, though it must be used carefully to avoid misinterpretation.

Abstract Reasoning¶

Highlights that system changes or differences can be systematically probed by adopting a baseline "no-effect" stance and seeing if evidence strongly contradicts it—a concept applied well beyond classical stats.

Knowledge Transfer¶

Engineering Trials: Evaluate whether a new design truly increases structural strength beyond random fluctuations.
HR Policy Changes: Hypothesize that flexible hours lower turnover, test if turnover truly differs or if random variation might explain the difference.

Example¶

A diet study: Null says "No difference in average weight loss between new diet and standard diet." If the difference in mean weight loss is large and improbable under random variation, researchers reject the null.

Relationships to Other Abstractions¶

Current abstraction Hypothesis Testing (Null vs. Alternative) Prime

Parents (2) — outgoing typed relations

Hypothesis Testing (Null vs. Alternative) is a kind of Statistical Inference Prime

Hypothesis testing is a specialization of statistical inference that frames the inferential question as a pre-specified decision between two complementary hypotheses.
Hypothesis Testing (Null vs. Alternative) is a kind of Verification Prime

Hypothesis testing is a specific kind of verification, checking sample evidence against a pre-specified null with controlled error rates.

Children (22) — abstractions linked here by typed relations

Anderson–Darling test Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)

The procedure literally compares sample evidence with a null distribution through a rejection rule.
Augmented Dickey–Fuller Test Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)

Augmented Dickey–Fuller Test is a strict kind of Hypothesis Testing (Null vs. Alternative): it tests a unit-root null against stationarity alternatives using an augmented regression.
Brown–Forsythe test Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)

It remains its own entry because its identity is fixed by the groups and observations, independence assumptions, group medians, absolute-deviation transformation, sample sizes, transformed group and grand means, F statistic and degrees of freedom, p-value or critical rule and robustness limits.
Chow Test Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)

Hypothesis Testing (Null vs.
Complex segregation analysis Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)

CSA compares explicit inheritance hypotheses against pedigree evidence.

▸ Show 17 more

F-test of equality of variances Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)
What makes it its own entry: The test is not robust to modest nonnormality and should not be treated as a routine general test of dispersion equality; alternative robust tests answer related questions.
Generalized p-value Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)
It remains its own entry because its identity is fixed by the statistical model and hypothesis, data and sufficient summaries, nuisance parameters, generalized test variable, observed value, null distribution or simulation law, tail direction and exactness claim.
Information matrix test Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)
Information Matrix Test instantiates Hypothesis Testing (Null vs.
KPSS test Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)
It remains its own entry because its identity is fixed by the observed series and sample period, level- or trend-stationary null, unit-root alternative, deterministic regression, residual partial sums, long-run variance kernel and bandwidth, statistic, critical values and interpretation with complementary tests.
Null distribution Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)
Null Distribution instantiates Hypothesis Testing (Null vs.
One- and Two-Tailed Tests Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)
One- and Two-Tailed Tests is a strict kind of Hypothesis Testing (Null vs. Alternative): they allocate null-rejection probability according to one- or two-direction alternatives.
Peres–Horodecki criterion Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)
The criterion tests separability against entanglement through a matrix transformation.
Randomness Test Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)
Hypothesis Testing (Null vs.
Sargan–Hansen Test Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)
Hypothesis Testing (Null vs.
Sign test Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)
The sign test literally compares a calibrated null with a directional or two-sided alternative and uses a statistic to decide whether evidence crosses a rejection rule; sign reduction and binomial calibration supply the specialization.
Student's t-Test Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)
the procedure compares a specified mean null with one- or two-sided alternatives.
Universal Hypothesis Testing Domain-specific is a kind of Hypothesis Testing (Null vs. Alternative)
consistency and exponents describe the large-sample limit.
HARKing (Hypothesizing After the Results are Known) Domain-specific includes Hypothesis Testing (Null vs. Alternative)
The HARKing practice contains a hypothesis test whose apparent prespecification and nominal error rate are the objects being falsified.
Jeffreys-Lindley Paradox Domain-specific includes Hypothesis Testing (Null vs. Alternative)
The Jeffreys-Lindley construction contains a frequentist point-null hypothesis test whose tail-area verdict supplies one side of the disagreement.
Null Ritual Domain-specific includes Hypothesis Testing (Null vs. Alternative)
The Null Ritual contains a mechanically executed null-hypothesis test whose inferential furniture has been stripped away while its verdict remains authoritative.
Statistical Significance (p-Value) Prime presupposes Hypothesis Testing (Null vs. Alternative)
Statistical significance presupposes hypothesis testing because the p-value is read as evidence-against only within a pre-specified null/alternative testing frame.
Type I & Type II Errors Prime presupposes Hypothesis Testing (Null vs. Alternative)
Type I and Type II errors presuppose hypothesis testing because they are precisely the two ways its reject/retain decision can be wrong.

Hierarchy paths (4) — routes to 4 parentless roots

Hypothesis Testing (Null vs. Alternative) → is a kind of Statistical Inference → is a kind of Inductive Reasoning

Show alternative paths (3)

Not to Be Confused With¶

Hypothesis Testing (Null vs. Alternative) is not Prediction because hypothesis testing specifies a comparison structure between competing claims evaluated through prespecified thresholds, while prediction is the structured claim about future states; testing decides between hypotheses given data, prediction projects future outcomes.
Hypothesis Testing (Null vs. Alternative) is not Forecasting because hypothesis testing is binary or bounded decision-making (reject/fail-to-reject at α level), while forecasting projects quantitative trajectories; testing answers "is this effect real?", forecasting answers "what will happen when?"
Hypothesis Testing (Null vs. Alternative) is not Optimization because hypothesis testing specifies a threshold-based accept/reject decision, while optimization searches a decision space for the best candidate; testing judges hypothesis viability, optimization searches for superiority.