Skip to content

Case Universe Sampling Frame

Design protocol — instantiates Structured Comparative Case Design

Fixes the population of cases the study could have chosen — the boundary, the unit, and the eligibility rule — before any case is picked.

Cherry-picking is hardest when the pool is fixed before anyone knows which cases would flatter the hypothesis. The Case Universe Sampling Frame is the front-end protocol that specifies, ahead of selection, the full set of cases eligible for the study: the causal question and its scope, the boundary of the case universe (what is in, what is out, and why), and the unit and level of analysis (is a "case" a country, a hospital, an incident, a fiscal year?). Its distinguishing role is that it sits above selection — it defines the frame from which cases are later drawn, so choosing a dozen becomes a visible choice from a stated population rather than a convenience grab whose alternatives no one can see.

Example

Education researchers want to know whether a new phonics curriculum lifted Grade-2 reading. Before picking a single school, the sampling-frame protocol pins down three things. The causal question: the effect of adopting the curriculum on Grade-2 reading in the first year. The universe boundary: all public elementary schools in the state eligible to adopt in 2023–24 — excluding private schools, schools already using it, and K-only sites, each exclusion with a reason. The unit: the school-year, because a school that adopted mid-year splits into a pre- and post-case.

This yields an explicit frame of, say, ≈1,200 eligible school-years. Only now does selection happen — and because the frame is on paper, a later reader can see exactly which cases were in play and judge whether the chosen sample over-represents, say, well-funded districts. Every downstream step inherits this boundary and unit.

How it works

  • State the causal question and its scope conditions — the bounds of population, setting, and time within which an answer could even apply.
  • Draw the universe boundary with explicit inclusion and exclusion criteria and their reasons — the core anti-cherry-picking move.
  • Fix the unit and level of analysis, resolving cases that straddle it (mergers, mid-period switches, nested units).
  • Publish the frame so that selection is auditable against it rather than asserted after the fact.

Tuning parameters

  • Boundary breadth — how wide the universe is drawn; wider supports broader claims but strains comparability across the pool.
  • Inclusion-criterion strictness — how tight the eligibility rule is; tighter frames are cleaner but shrink the pool and the reach of any conclusion.
  • Unit granularity — country versus region versus year; finer units multiply cases and control but can fracture a coherent case into pieces.
  • Frame explicitness — an enumerated list versus a stated rule; lists are fully auditable, rules scale to large universes.

When it helps, and when it misleads

Its strength is that it is the single most effective guard against cherry-picking, because it fixes the opportunity set before anyone knows which cases help — and it simultaneously sets the outer limit on how far the conclusions may travel. Its failure modes mirror that power: a frame quietly gerrymandered to fit a pet set of cases just launders the bias behind a rigorous-looking boundary, and an over-narrow universe caps generalisation while looking careful. The classic misuse is redrawing the boundary after peeking at outcomes. The discipline that guards against it is to derive the boundary from the causal question and its scope conditions, not from the cases you hope to study, and to freeze it before selection begins.[1]

How it implements the components

  • causal_question_and_scope — it states the question and the scope conditions that make a case relevant at all.
  • case_universe_boundary — its core artifact: the explicit in/out boundary of the eligible population, with reasons.
  • unit_and_level_of_analysis — it fixes what counts as one case, and at what level, so the frame is countable.

It does not check whether the cases actually drawn from this frame were selected without bias (that's Case Selection Bias Audit), nor profile how comparable those cases are to one another (that's most-similar and most-different systems design); it defines the frame they are drawn from.

References

[1] Scope conditions — the stated bounds of population, setting, and time within which a causal claim is asserted to hold. Fixing them up front both disciplines case selection and marks the outer edge of legitimate generalisation. ↩