Type S Error

Quantify the risk that a statistically significant estimate points the wrong way by computing, before data collection, the probability that a two-sided significance filter is cleared from the opposite tail when the true effect is near zero relative to noise.

Core Idea

A Type S (sign) error is the failure mode in which a statistically significant estimate has the wrong sign. When a study's power is low relative to the true effect, the significance threshold is cleared almost only by large draws, and when the true effect is near zero relative to sampling noise both tails reach significance with comparable probability. Gelman and Carlin's prospective diagnostic, Pr(sign correct | significant), asks: given that I will only act on a significant result, how likely is its direction to be right?

Scope of Application

Requires a two-sided significance filter, a p-value, and a power calculation — the significance-testing sciences with true effects near zero relative to noise.

• Behavioural-science replication — priming and embodied-cognition reversals against high-powered replications.
• Health-policy A/B tests — underpowered studies reporting the opposite direction of the truth.
• Genome-wide association studies — top discovery-stage SNP hits flipping sign at replication.
• Election-poll subgroups — small-cell swings reversing direction across consecutive surveys.
• Exploratory contrasts — interaction terms and subgroup comparisons read off noisy cells.

Clarity

The construct punctures the comforting fallback "at least the direction is right," showing this is exactly the assumption that fails under low power. It separates two questions the classical frame fuses — "is there an effect?" versus "is the direction of a significant result trustworthy?" — and converts an apparently safe positive claim into one with a computable failure probability available at design time.

Manages Complexity

Scattered direction-of-effect anxieties across interactions, subgroups, and discovery-stage hits collapse onto one mechanism and one prospective scalar. What the analyst tracks reduces to a single ratio — true effect size relative to sampling noise — because that alone sets how the two tails contribute significant draws. A near-zero-versus-far-from-zero fork then reads the verdict off, prescribing more power rather than better magnitude measurement.

Abstract Reasoning

The defining move conditions the sign on the filter: not "is the direction right?" but "given I only report significance, how often does it point wrong?" A boundary-drawing move locates the regime where the "at least the sign is right" comfort fails, fixing the remedy by diagnosis. A retrospective move reads replication sign-reversals as materialized Type S risk, not scandal — computed jointly with its magnitude sibling, Type M.

Knowledge Transfer

Within the significance-testing sciences the whole apparatus transfers literally — the two-tail geometry, the prospective Pr(wrong sign), the power-not-precision remedy — because the substrate of a noisy estimator near zero under a two-sided filter is held fixed. Beyond that scaffolding the portable content is the parent winner_s_curse (within the selection_bias family): selection on a noisy estimator near zero can flip the sign. Type S operationalises that flip for the frequentist significance filter.

Relationships to Other Abstractions

Current abstraction Type S Error Domain-specific

Parents (4) — more general patterns this builds on

• Type S Error is a kind of Selection on Noisy Estimates Prime

  Type S is the near-zero two-sided-threshold species in which the selected estimate can enter from the tail opposite the true effect and reverse its sign.

• Type S Error is part of Effect Size Prime

  Type S Error contains an estimated and true Effect Size whose signed directions are compared after significance selection.

• Type S Error is part of Statistical Power Prime

  Type S Error contains Statistical Power because the effect-to-noise detection regime determines whether the opposite tail remains reachable.

• Type S Error is part of Statistical Significance (p-Value) Prime

  Type S Error contains a two-sided statistical-significance gate whose opposite-tail admissions can carry the wrong sign.

Hierarchy paths (23) — routes to 8 parentless roots

• Type S Error → Selection on Noisy Estimates → Selection Bias → Bias

Show alternative paths (22)

Neighborhood in Abstraction Space

Type S Error sits in a sparse region of the domain-specific corpus (79^th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Statistical Bias & Sampling Artifacts (6 abstractions)

Nearest neighbors

• Type M Error — 0.89
• Underfitting — 0.82
• Jeffreys-Lindley Paradox — 0.82
• Attenuation Bias — 0.82
• Funnel Plot Asymmetry — 0.81

Computed from structural-signature embeddings · 2026-07-12