Metric Hierarchy¶
Priority structure — instantiates Multiple-Testing Discipline
Ranks metrics into primary, secondary, and exploratory tiers before the data land, so a disappointing primary can't be quietly swapped for a flattering secondary.
A Metric Hierarchy disciplines multiplicity by ranking the claims in advance rather than adjusting a threshold or reserving data. Before any results exist, every metric an analysis will report is assigned a tier — one designated primary claim that alone decides success, a small set of secondary claims read only in a fixed order, and a wider pool of exploratory metrics that can generate leads but never verdicts. The defining idea, unique among its siblings, is this pre-committed ordering of importance: the hierarchy pins which single result the decision hangs on, so that when the primary disappoints, a team cannot go shopping through the secondaries and crown whichever one happens to look good. It does not compute a corrected p-value or run a follow-up test; it structures the claims so that metric shopping has nowhere to hide.
Example¶
A product team is testing a redesigned checkout flow in an A/B experiment. The redesign could touch dozens of measurable things — conversion rate, average order value, time-on-page, cart-abandonment, support tickets, scroll depth. If they launch, look at everything, and declare victory on whichever metric moved, they will "win" almost every experiment by noise alone. So before the test starts they fix a hierarchy: the primary metric is checkout conversion rate — the experiment succeeds or fails on that alone. A short ranked list of secondary metrics (order value, then abandonment) may be reported only if the primary passes, in that order, each as supporting evidence. Everything else — scroll depth, session time, per-segment breakdowns — is explicitly exploratory: it may raise questions for the next experiment but can never justify shipping. When the results come in, conversion is flat. Under the hierarchy that is a failed test, full stop; a promising bump in average order value stays a labeled lead, not a rescue. The pre-set ranking is what stops a null primary from being laundered into a launch.
How it works¶
- Enumerate every metric in advance. List all metrics the analysis could report, before data exist, so none can be added post hoc.
- Assign tiers and an order. Name exactly one primary, rank the secondaries, and mark the rest exploratory; the ranking is fixed and recorded.
- Gate reading by rank. Confirmatory attention flows in the fixed sequence — the primary decides the verdict; secondaries are interpreted only in order and only if earlier claims hold, a gatekeeping logic that concentrates the confirmatory budget on what was declared to matter.[n1]
- Label by tier, not by outcome. Every reported result inherits its tier's status — decisive, supporting, or exploratory lead — regardless of how attractive it turned out to be.
Tuning parameters¶
- Primary breadth — one metric versus a small composite; a single primary is cleanest but may miss a multi-dimensional benefit, while a composite is richer but harder to interpret.
- Number of secondaries — how many ranked supporting claims are allowed; more give a fuller picture but stretch attention and invite re-ranking pressure.
- Ordering rule — fixed-sequence (each gated on the previous) versus parallel-with-allocation; sequences are simple and powerful when the order reflects real priority, but rigid if the wrong metric was ranked first.
- Exploratory openness — how freely the bottom tier may be mined for leads; broad exploration is valuable precisely because it cannot promote itself to a verdict.
When it helps, and when it misleads¶
Its strength is that it defeats metric shopping structurally and cheaply: by naming the decisive metric before the data arrive, it removes the analyst's freedom to choose the winner afterward, and it keeps a rich field of exploratory metrics useful without letting any of them masquerade as confirmation. It is the right tool when the main risk is a team quietly re-anchoring on whatever moved.
Its failure mode is that the hierarchy is only as honest as the moment it was fixed. Outcome switching — changing which metric is called "primary" after seeing results — voids the entire protection, and it is easy to do informally in a slide deck if the original ranking was never recorded.[n2] The hierarchy also cannot rescue a badly chosen primary: if the truly important effect lives in a metric ranked secondary, a rigid sequence can under-credit a real benefit. And ranking alone corrects nothing statistically — a primary read across many segments still needs its own multiplicity handling. The guarding discipline is to timestamp the hierarchy before data collection and to treat any later re-ranking as a new, exploratory analysis.
How it implements the components¶
claim_hierarchy— the tiered, ordered ranking of metrics is this mechanism; establishing which claim outranks which is its entire function.exploratory_confirmatory_boundary— the line between the primary/secondary tiers (confirmatory) and the exploratory pool operationalizes the exploratory–confirmatory split at the level of metrics.result_status_label— each metric's tier stamps its result as decisive, supporting, or exploratory lead, fixing its interpretive status ahead of the outcome.
It ranks and gates but does not itself run the confirmatory re-test a demoted metric would need — that confirmation_requirement belongs to Confirmatory Follow-Up — nor does it compute a corrected threshold, which is the multiplicity_adjustment_rule of Bonferroni-Like Correction.
Related¶
- Instantiates: Multiple-Testing Discipline — supplies the priority-ranking form of multiplicity control.
- Sibling mechanisms: Alpha-Spending Plan · Bonferroni-Like Correction · False Discovery Rate Control · Claim Registry · Confirmatory Follow-Up · Holdout Validation · Multiverse Analysis Report · Preregistration · Replication Study
Editorial Notes¶
Form Classification¶
Form family: Rule, Policy & Commitment
Rationale: Metric Hierarchy operates as a standing rule, threshold, contractual commitment, or policy constraint governing future conduct because it ranks metrics into primary, secondary, and exploratory tiers before the data land, so a disappointing primary can't be quietly swapped for a flattering secondary.
Independent corroboration: The frozen evidence defines Metric Hierarchy as 'Ranks metrics into primary, secondary, and exploratory tiers before the data land, so a disappointing primary can't be quietly swapped for a flattering secondary', so its operative form is Rule, Policy & Commitment.
Nearest alternative: Decision, Gate & Allocation — The hierarchy prioritizes metrics, but it does so as a standing precommitted gatekeeping rule fixed before results arrive.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Predeclaring primary, secondary, and exploratory outcomes is a statistical multiple-testing discipline.
Related originating lineages:
- Medicine & Healthcare — Clinical trials institutionalized endpoint hierarchies to prevent outcome switching.
- Organizational & Management Science — For Metric Hierarchy, coordination, governance, strategy, process design, and management-control practice materially shaped the mechanism's characteristic form.
Review resolution: Both independent reviews place the primary provenance in statistics_experimental_design. The queued differences (alternate_origin_disagreement) concern secondary metadata, not primary lineage. The final retains medicine_healthcare, organizational_management only where a reviewer supplied a formative-lineage rationale; downstream use or broad applicability by itself is not treated as origin. origin_mode=cross_disciplinary_synthesis because the supplied rationales identify formative contributions that are composed in the mechanism's present form. domain_reach=multi_domain records established application breadth separately from provenance. confidence=high preserves the more cautious evidence assessment. encyclopedia_synthesis=false records whether either reviewer identified deliberate corpus-level composition.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] In confirmatory trials, gatekeeping (hierarchical / fixed-sequence testing) allocates the error budget across ranked endpoints so that a secondary claim is tested only after the higher-ranked one succeeds — the standard way regulators let a study report multiple endpoints without inflating the false-positive rate. ↩
[n2] Outcome switching is the practice of reporting a different primary outcome than the one originally designated, typically after results are known; comparisons of registered plans against publications are how it is detected, which is why recording the hierarchy before data collection matters. ↩