Statistical tests, P values, confidence intervals, and power

Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337-350.

Type

Journal article

Intellectual base

Primary research

Year

2016

DOI

10.1007/s10654-016-0149-3

Link

https://doi.org/10.1007/s10654-016-0149-3

Cited from

statistics_experimental_design

Cited by

7 citations across 7 artifacts.

Each citation links to the sentence it supports in the citing article.

Primes

• Confidence Intervals
  • Immature practice reports CIs mechanically (CI just because it is required) without assessing model adequacy, misinterprets the realized interval as having 1−α probability of containing the parameter, or treats CIs as superfluous decoration alongside the "real" analysis (p-value or point estimate alone)

    This sourceCatalogs and corrects common misinterpretations, including the error of treating a realized interval as having 1-alpha probability of containing the parameter.

• Distributional Assumption
  • This clarity distinguishes between defensible assumptions (supported by prior domain knowledge, physical principles, or extensive historical data) and convenient assumptions (chosen because they are mathematically tractable or computationally fast)—a distinction Greenland et al. (2016) operationalize in their guide to misinterpretation of statistical tests and confidence intervals.
• Effect Size

  • This sourceReal and authoritative, but it documents 25 misinterpretations of P/CI/power — it does NOT cover Type M/S errors. Demoted from marker 072 to bibliography-only after re-sourcing that marker to gelman-carlin-2014; linked.

• Hypothesis Testing (Null vs. Alternative)
  • Without the frame, people treat statistical evidence as binary (significant/not), conflate statistical with practical significance, misinterpret p-values as posterior probabilities, and select hypotheses after seeing data
• Statistical Power
  • Without the frame, people run studies whose sample sizes are determined by convenience or convention without regard for what effects they can detect, interpret non-significant results as evidence of no effect regardless of achieved power, and conflate "observed power" with independent diagnostic information
• Statistical Significance (p-Value)
  • It is a continuous summary of sample evidence's incompatibility with the null, ranging from 0 (data impossible under null) to 1 (data exactly typical under null)
• Type I & Type II Errors
  • The tension is between the simplicity of individual-test error control and the reality that most studies effectively conduct many tests, requiring multiplicity-adjusted reasoning

Verification

This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.

Registry ID ref:e91360457d43 · see in the full table