Beyond power calculations¶
Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing type S (sign) and type M (magnitude) errors. Perspectives on Psychological Science, 9(6), 641-651.
Cited by¶
6 citations across 6 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Effect Size
- Gelman and Carlin's 2014 extension into "Type M" (magnitude) and "Type S" (sign) errors showed mathematically that underpowered studies reaching statistical significance tend to dramatically exaggerate the true effect size (a "winner's curse" phenomenon)
This sourceIntroduces Type M (exaggeration) and Type S (sign) errors and shows underpowered significant studies exaggerate true effect size — the correct source for marker 072. Replaces the mis-keyed greenland-2016 (which is about P-value misinterpretation, not Type M/S).
Supported in partVerified against the publisher's abstract
Gelman & Carlin's own abstract names Type S (sign) and Type M (magnitude/exaggeration ratio) errors and says significant results in noisy small samples can mislead; it does not itself assert the 'dramatic exaggeration' finding.
“(b) the factor by which the magnitude of an effect might be overestimated ( Type M [ magnitude] error or exaggeration ratio) are estimated.”
- Gelman and Carlin's 2014 extension into "Type M" (magnitude) and "Type S" (sign) errors showed mathematically that underpowered studies reaching statistical significance tend to dramatically exaggerate the true effect size (a "winner's curse" phenomenon)
- Regression to the Mean
- T4 — Name: RTM in Replication and Winner's-Curse Contexts. RTM is a fundamental contributor to the replication crisis: original studies published because they reached statistical significance over-select on noise-inflated effect estimates; replications find smaller effects even when the true effect is real
This sourceGelman-Carlin extending error-rate concepts to effect-size estimation sign and magnitude errors.
Supported in partVerified against the publisher's abstract
Gelman & Carlin's abstract backs the mechanism that significance selection in noisy small samples yields overestimated effect magnitudes, but it never names regression to the mean, the replication crisis, or the winner's curse.
“In noisy, small-sample settings, statistically significant results can often be misleading. To help researchers address this problem in the context of their own studies, we recommend design calculations in which (a) the probability of an estimate being in the wrong direction ( Type S [ sign] error) and (b) the factor by which the magnitude of an effect might be overestimated ( Type M [ magnitude] error or …”
- T4 — Name: RTM in Replication and Winner's-Curse Contexts. RTM is a fundamental contributor to the replication crisis: original studies published because they reached statistical significance over-select on noise-inflated effect estimates; replications find smaller effects even when the true effect is real
- Statistical Power
This sourceGelman-Carlin extending error-rate concepts to effect-size estimation sign and magnitude errors.
- Type I & Type II Errors
This sourceGelman-Carlin extending error-rate concepts to effect-size estimation sign and magnitude errors.
Domain-specific¶
- Type M Error
- Gelman and Carlin (2014) formalized the exaggeration ratio as the key diagnostic: the ratio of the expected significant estimate to the true effect, computed prospectively from the design before data are collected
This sourceThe exaggeration ratio as the expected absolute value of a statistically significant estimate divided by the true effect, computed in a design analysis from an externally assumed effect size. The retrodesign() function, which computes power, Type S error rate and the exaggeration ratio from an assumed true effect and a study's standard error.
Supported in partVerified against the source
- Gelman and Carlin (2014) formalized the exaggeration ratio as the key diagnostic: the ratio of the expected significant estimate to the true effect, computed prospectively from the design before data are collected
- Type S Error
- In extreme low-power settings — a study powered at 6%, where the true effect is only about a quarter of a standard error — the probability of a sign-correct result given significance can fall to roughly 0.76, meaning roughly one in four significant results points the wrong way
This sourceWorks an example with 6% power (a 2-point true effect against a standard error of 8.1) giving a 24% Type S error rate, about one significant result in four with the wrong sign. Introduces design calculations that estimate a study's Type S (wrong-sign) and Type M (exaggeration) errors. Gives an extreme illustration with 6% power, where the assumed 2-point true effect is about a quarter of the 8.1-point standard error. Computes a 24% Type S error rate and a 9.7-fold expected exaggeration for the 6%-power design.
SupportedVerified against the source
- In extreme low-power settings — a study powered at 6%, where the true effect is only about a quarter of a standard error — the probability of a sign-correct result given significance can fall to roughly 0.76, meaning roughly one in four significant results points the wrong way
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 4 of 6 citations: 1 supported, 3 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:f0339a8a117e · see in the full table