Scope Creep

one_liner: A perimeter ratchets outward through small additions each judged against a drifting current baseline rather than the original charter, with no owner of the trajectory.

core_idea:
A bounded undertaking expands beyond its original perimeter not through
a single deliberate decision but through a sequence of small additions,
each individually defensible against the *current* boundary, none
accountable for the cumulative trajectory. The baseline against which
"small" is measured drifts with every accepted increment, so the
perimeter ratchets outward and rarely retracts. The undertaking ends up
far from where it began, yet no single step can be identified as the
wrong one — because each step was reasonable against the state that
immediately preceded it.

The pattern requires three structural conditions, and is absent when any
is missing. First, a *definable initial scope* — a perimeter that
existed before the additions began. Second, an *evaluation procedure
that judges each proposed addition against the current state rather than
the original state*, so that the reference frame moves with the work.
Third, *asymmetric costs for assent versus refusal*: saying yes requires
only accommodating a small request, while saying no requires defending
the boundary, an effortful and often unrewarded act. Together these
produce a *moving reference frame whose motion has no single author*.
The load-bearing content is precisely this — not that projects grow,
but that they grow through *baseline-relative approval* under
*asymmetric refusal cost*, which makes the growth invisible to the
participants while it happens, since at each step they are evaluating a
defensible local increment rather than the indefensible cumulative
trajectory. The prime also separates two outcomes that look alike from
outside: deliberate re-scoping, where someone explicitly owns the new
perimeter, versus drift, where the new perimeter has no owner at all —
and the intervention space differs for each.

structural_signature:
*the bounded initial perimeter* — *the stream of individually-defensible increments* — *the baseline-relative evaluation procedure* — *the drifting reference frame* — *the asymmetric assent-versus-refusal cost* — *the absent trajectory owner* — *the outward-ratcheting cumulative boundary*

The pattern is present when each of the following holds:

- **A definable initial scope.** A perimeter existed before the
  additions began — an original charter, mission, statute, or
  commitment against which "creep" is even meaningful.
- **A stream of increments.** Growth arrives as a sequence of small
  proposed additions, not a single deliberate decision.
- **Baseline-relative evaluation.** Each increment is judged against
  the *current* state rather than the original one, so the reference
  frame moves with the work and every step looks locally defensible.
- **A drifting reference frame.** Because the baseline shifts with each
  accepted increment, "small" is measured against an ever-moving
  anchor, and the cumulative trajectory is never the unit of judgment.
- **Asymmetric assent cost.** Saying yes only accommodates a small
  request; saying no requires effortful, often unrewarded boundary
  defense — biasing the procedure toward accumulation.
- **An absent trajectory owner.** No single party is accountable for
  the perimeter's total motion, so the drift has no author even though
  each step has a decider.

These compose into an outward-ratcheting boundary that is invisible to
participants while it happens — they evaluate defensible local
increments, never the indefensible whole. The pattern is distinguished
from deliberate re-scoping (where someone explicitly owns the new
perimeter), and the failure lives in the evaluation procedure and its
baseline, not in the merits of any addition.

what_it_is_not:
- **Not `lock_in`.** Lock-in is a committed *state* hard to leave. Scope creep
  is a *process* of incremental perimeter expansion judged against a drifting
  baseline. One names a destination; the other names a direction of motion
  (see `lock_in`).
- **Not `parkinsons_law`.** Parkinson's Law is an activity expanding to *fill
  a fixed container* with the requirement set held constant. Scope creep is
  the requirement set itself *growing* through added increments. One consumes
  slack within fixed scope; the other enlarges scope (see `parkinsons_law`).
- **Not deliberate re-scoping.** When an accountable party explicitly owns a
  new perimeter, that is authorized re-scoping, not creep. Scope creep's
  defining feature is the *absent trajectory owner* — growth reached by
  summing increments no one authored as a whole.
- **Not `ratchet_effect` in general.** Scope creep is a *specific* ratchet —
  driven by baseline-relative evaluation plus asymmetric refusal cost. The
  generic ratchet needs only a lock; scope creep names the particular lock
  (drifting reference frame) and the particular driver (cheap assent).
- **Not legitimate adaptation.** A stream of increments can be healthy
  responsiveness to new information. The pathology is specifically *unauthored
  drift* — increments judged only against yesterday's state, never re-owned
  against the charter.
- **Common misclassification.** Hunting post-hoc for "the bad decision" that
  expanded the scope. By construction every individual increment was
  defensible against its local baseline; no single indefensible step exists.
  The tell: if you cannot name one indefensible step, the failure is
  structural — the evaluation procedure — not any single decider.

broad_use:
The same baseline-drift-plus-asymmetric-cost structure recurs across
substrates that share little but the presence of an authorizing
procedure. In **software and product development**, features are added
one at a time, each cheap against the current release, until the project
ships late or never — the familiar feature creep and gold-plating.<!-- FACT-D71-251 -->[^brooks-1975] In
**military and humanitarian operations**, a mission acquires policing
duties, then nation-building, then counterinsurgency, each step
justified by yesterday's state — the canonical mission creep.<!-- FACT-D71-252 -->[^allard-2002] In
**regulatory systems**, a statute targeting one harm is reinterpreted to
cover adjacent cases, precedent expanding its reach far beyond its
drafters' intent.<!-- FACT-D71-253 -->[^kerr-2010] In **personal commitments**, a one-time favor grows
into a standing obligation, and a volunteer role quietly absorbs new
responsibilities.<!-- FACT-D71-254 -->[^freedman-fraser-1966] In **scientific projects**, a paper's question expands
as reviewers ask for one more analysis, and a thesis swells past
completability.<!-- FACT-D71-255 -->[^duke-beck-2014] And in **meetings**, agendas accept one more topic until
they outrun their time slot. In each, the drifting object differs — a
feature list, a war's aims, a statute's reach, a personal duty, a
research question — but the mechanism is identical: incremental additions
judged against a moving baseline with no one accountable for the
perimeter's total motion.

clarity:
The prime names a failure mode that participants typically cannot see
while it is happening, because at each step they evaluate a small
increment against a local baseline. It makes visible the *trajectory*
that the local view hides — the boundary as a moving reference frame
whose motion has no single author — so that the question "have we
drifted?" becomes askable in a way that "is this addition reasonable?"
never permits. It also separates two failure modes that look alike from
outside: deliberate re-scoping, in which a party explicitly owns the new
perimeter, versus drift, in which the new perimeter has no owner. The
clarifying force is to relocate attention from the merits of each
addition — which by hypothesis all look reasonable — to the *evaluation
procedure and its baseline*, which is where the failure actually lives.

manages_complexity:
The prime compresses a long sequence of micro-decisions into a single
shape: a ratcheting boundary. This lets participants reason about the
trajectory rather than about each increment in isolation, and it lets
distant actors — a steering committee, a court, a budget board — audit
*the boundary's movement* as a first-class object even when they cannot
evaluate the merits of each individual step. By bundling the responsible
mechanism — local-baseline evaluation plus asymmetric assent costs — into
a recognizable signature, the prime makes the same diagnosis applicable
whether the drifting object is a feature list, a war's aims, or a
statute's reach. The compression also tells the analyst *where to look*:
not at the additions, which are individually unimpeachable, but at the
reference frame against which they were approved, collapsing a
bewildering history of reasonable decisions into one analyzable defect.

abstract_reasoning:
Once recognized, scope creep licenses several abstract moves. *Anchor to
the original baseline*: re-evaluate any proposed addition against the
*first* state, not the current one, neutralizing the moving reference.
*Periodically re-charter*: schedule moments where the whole perimeter is
redrawn from scratch, breaking the ratchet. *Impose symmetric assent
costs*: require every addition to displace something already inside,
forcing the trade-off the incremental view conceals. *Assign a boundary
owner*: designate a single party whose job is to defend the perimeter,
supplying the missing accountability for the trajectory. These moves are
substrate-neutral because they act on the structural conditions
themselves — the baseline, the symmetry of assent, the ownership of the
perimeter — rather than on any domain's particular content. The reasoning
concerns a *bounded undertaking under incremental authorization with a
drifting reference frame*, a configuration indifferent to whether the
undertaking is an engineering project, a military operation, or a
regulatory regime.

knowledge_transfer:
The pattern carries a diagnostic and a remediation kit that work across
the domain map. If a reasoner can identify a definable initial scope, an
incremental-approval procedure, and asymmetric costs of refusal versus
assent, they can predict drift and prescribe the interventions without
knowing the substrate — and the fix that works in one domain suggests the
analogous fix in another. The "displace-something" rule appears as
work-in-progress limits in software, zero-base budgeting in finance, and
sunset clauses in law;<!-- FACT-D71-256 -->[^pyzdek-keller-2018] the "anchor to original baseline" move is
counterfactual reasoning applied to one's own decision history; and the
"periodic re-charter" move is the scheduled zero-base audit common to all
three.<!-- FACT-D71-257 -->[^pyhrr-1973] Critically, the prime tells the practitioner *where* to intervene:
not on the merits of individual additions, which all look reasonable, but
on the evaluation procedure and its baseline.

The structural roles map across substrates. The *bounded initial
perimeter* is the original charter, mission, statute, or commitment; the
*stream of increments* is the feature requests, mission tasks,
reinterpretations, or favors; the *drifting baseline* is the current
state that each increment is judged against; the *asymmetric assent cost*
is the cheapness of yes against the effort of defending the boundary; and
the *absent trajectory owner* is the missing accountability for the
perimeter's cumulative motion. A program manager imposing WIP limits, a
budget board mandating zero-base review, and a legislature attaching a
sunset clause are performing the same structural act:<!-- FACT-D71-258 -->[^pyhrr-1973] reinstalling a
fixed reference frame, a symmetric assent cost, or an owner for the
perimeter that the drift had eroded. The diagnostic — *is each addition
judged against the original charter or a drifting baseline, and who owns
the trajectory?* — travels unchanged across software, military operations,
regulation, personal commitments, and science. Because the remediation
kit is identical across these media — anchor, re-charter, displace, assign
an owner — a practitioner who has arrested drift in one domain can import
the whole repertoire into a domain that knows the pattern only by a
domain-flavored name like feature creep or mission creep.<!-- FACT-D71-259 -->[^gao-zbb-sunset-1977]

formal_abstract:
Model a software release as a sequence of feature-approval decisions. The
*bounded initial perimeter* is the chartered scope: ship features A, B, C by a
fixed date. The *stream of increments* is a series of requests — add D, then E,
then F — each arriving after the previous was accepted. The structural defect is
the *baseline-relative evaluation procedure*: each request is judged against the
*current* committed scope, not the original charter. When D is proposed, the
question asked is "is adding D to {A,B,C} a small, reasonable increment?" — and it
is. But once D is in, E is judged against {A,B,C,D}, and so the *reference frame
drifts* with every acceptance. The *asymmetric assent cost* compounds this:
saying yes to a sponsor's request costs a moment; saying no requires effortful,
politically unrewarded boundary defense. With no *trajectory owner* accountable
for the cumulative motion, the perimeter *ratchets outward* — the release ends up
at {A,...,K}, ships late, yet no single addition can be named as the wrong one,
because each was defensible against its local baseline. The diagnosis the prime
licenses is to stop auditing the merits of individual features and instead
inspect the evaluation procedure: the fix is to *anchor to the original baseline*
(judge every request against {A,B,C}), *impose symmetric assent cost* via a
work-in-progress limit (every addition must displace something already inside),
and *assign a boundary owner* whose job is to defend the perimeter.

**Mapped back:** the chartered scope is the initial perimeter, the feature
requests are the increments, current-scope evaluation is the drifting baseline,
and the missing release-owner is the absent trajectory owner — the outward
ratchet worked end-to-end.

applied_industry:
Two real substrates carry the identical mechanism. First, military mission creep:
a deployment chartered for a narrow objective — secure an airfield — acquires
policing duties (small, against the current state), then humanitarian logistics,
then nation-building, then counterinsurgency.<!-- FACT-D71-260 -->[^allard-2002] The *initial perimeter* is the
original mandate; each *increment* is justified against yesterday's posture, never
against the founding charter; the *asymmetric cost* is that expanding the mission
answers an immediate need while refusing it means visibly abandoning a problem on
the ground; and no single actor *owns the trajectory* of total mission scope. The
result is a commitment far from the original aim that no one deliberately
authorized. The intervention is the same kit: re-charter periodically against the
original mandate, and assign an owner accountable for the mission boundary.
Second, regulatory drift: a statute drafted to address one harm is reinterpreted
through a sequence of precedents, each extending its reach to an adjacent case
that resembles the last, until the statute covers conduct its drafters never
contemplated.<!-- FACT-D71-261 -->[^kerr-2010] The *drifting baseline* is the accumulated case law against which
each new extension looks small; the *absent owner* is the lack of any body
accountable for the statute's total expansion. The remediation — anchor
interpretation to original legislative intent, sunset provisions that force
periodic re-charter — is structurally identical to the work-in-progress limit
in software, confirming the same prime underlies both.<!-- FACT-D71-262 -->[^pyhrr-1973]

**Mapped back:** the mandate and the statute are the initial perimeters; mission
tasks and reinterpreting precedents are the increments; yesterday's posture and
accumulated case law are the drifting baselines; and the missing mission-owner and
oversight body are the absent trajectory owners — the ratchet recurring across
operations and law with one remediation kit.


Screening

type: prime

slug: screening

one_liner: Inducing self-revelation.

status: draft

core_idea:
*Screening* names the abstraction that (1) when an *uninformed* party must interact with agents whose types they cannot directly observe, (2) the uninformed party can design a *menu* — a structured set of terms (price, quality, coverage level, deductible, contract duration) — (3) such that agents of different types find it in their own interest to *self-select* into different options, (4) with the chosen option revealing the agent's type and allowing the uninformed party to respond differentially — overcoming adverse selection without requiring direct verification. The term itself was introduced by Stiglitz (1975)[^stiglitz-1975] in the context of education and labor-market sorting.<!-- FACT-D44-046 -->[^stiglitz-1975]

Screening is the *response*; adverse selection is the *problem* it responds to. An adverse-selection-plagued uniform market (a single product for a heterogeneous population) is the baseline against which a screening menu is evaluated.

structural_signature:
Screening has a tightly specified anatomy, inverse to signaling, presented in canonical form by Laffont and Martimort (2002)[^laffont-martimort-2002] in their adverse-selection chapter:

- **An uninformed designer** (employer, insurer, lender, buyer, regulator) who cannot directly observe agent types.<!-- FACT-D44-047 -->[^laffont-martimort-2002]
- **Informed agents** with heterogeneous types (high/low risk, high/low ability, high/low quality, high/low willingness to pay) known to themselves.
- **A menu of options** — different combinations of terms (price, quality, coverage level, deductible, workload, compensation, warranty, contract duration) offered by the designer.
- **Type-differentiated preferences** — agents of different types have systematically different preferences across the menu, so each type's utility-maximizing choice differs.
- **Incentive-compatibility constraints** — the menu is designed so each type's preferred option is the one the designer wants them to take, with no type preferring to "cheat" by selecting a different type's option.
- **Individual-rationality constraints** — each type must prefer their designated option to the outside option of not participating at all.
- **A revelation outcome** — the agent's chosen option reveals enough about their type for the designer to respond appropriately.

The design problem is to construct a menu that satisfies incentive compatibility and individual rationality for all types while maximizing the designer's objective (revenue, efficiency, risk-pool quality).

what_it_is_not:
Screening is not *signaling*. Signaling is sender-initiated costly communication by the informed party; screening is receiver-initiated menu design by the uninformed party. The two address the same information-asymmetry problem from opposite directions and are routinely presented together as a tight pair, with Spence (1973)[^spence-1973] supplying the canonical sender-side counterpart to the screening literature.<!-- FACT-D44-048 -->[^spence-1973]

It is not *direct verification* or *monitoring*. Screening operates through *incentive-compatible menu design*; verification relies on direct inspection, audit, or testing. In many real contexts both are combined — insurers screen through deductible menus *and* verify through medical exams — but the abstractions are conceptually distinct.

It is not *customer segmentation*. Marketing segmentation offers different products to different groups based on the seller's assessment of their characteristics; screening uses *self-selection* by the informed party as the type-revealing mechanism. A seller offering different SKUs to rural vs. urban customers based on demographic data is segmenting; a seller offering a menu and letting customers choose is screening.

It is not always *welfare-improving*. Screening often involves deliberate *downward distortion* of the low-type's option — worse than what would be offered under full information — to reduce the information rents the high-type would otherwise extract. Screening is a second-best response to information asymmetry, not a first-best allocation.

It is not limited to *binary* type structures. The theory has been developed for continuous type distributions (Mirrlees 1971[^mirrlees-1971] for optimal income taxation; Mussa-Rosen 1978[^mussa-rosen-1978] for quality discrimination; Maskin-Riley 1984[^maskin-riley-1984] for quantity discrimination), with rich equilibrium structures including *bunching* of types at certain menu points.

broad_use:
In **insurance**, structuring deductibles, copayments, and coverage tiers lets insurers offer multiple policies so high-risk clients select high-coverage-low-deductible plans while low-risk clients select low-coverage-high-deductible plans. Rothschild and Stiglitz (1976)[^rothschild-stiglitz-1976] formalized this.<!-- FACT-D44-054 -->[^rothschild-stiglitz-1976]

In **labor-market hiring**, multi-stage interviews, coding tests, case-study work products, take-home assignments, and graduated trial-period arrangements screen applicants beyond what résumés and credentials alone reveal.

In **credit markets**, loan menus offering various combinations of interest rate, collateral, reporting requirements, and covenants induce borrower self-selection — Stiglitz and Weiss (1981)[^stiglitz-weiss-1981] showed how interest-rate menus combined with credit rationing serve as a screening device.<!-- FACT-D44-049 -->[^stiglitz-weiss-1981] Low-risk borrowers accept strict covenants and lower rates; high-risk borrowers accept higher rates for looser covenants — or the high-risk ones self-select out entirely.

In **education**, application processes with essays, interviews, portfolios, and recommendation letters screen for applicant characteristics — commitment, fit, ability — that test scores alone do not reveal.

In **software pricing and SaaS product design**, tiered plans (basic/pro/enterprise) induce customers to self-select into the tier matching their willingness to pay and feature needs — a close cousin to second-degree price discrimination.

In **regulatory policy**, optional compliance pathways (self-certification vs. third-party audit vs. full agency inspection) let firms select the compliance level matching their risk profile and operational capability, with the chosen pathway itself conveying information.

In **academic admissions and grant allocation**, qualifying exams, essays, research proposals, and interviews screen applicants whose underlying characteristics are difficult to observe directly.

In **cooperative and association membership**, application processes, probationary periods, and participation requirements screen for commitment and alignment in ways that simple dues-paying does not.

clarity:
The abstraction clarifies *why* many real-world contracts, menus, and application processes look more complex than they "need" to be — the baseline against which screening is evaluated is the lemons-style market collapse described by Akerlof (1970)[^akerlof-1970], in which uniform pricing under information asymmetry drives the market to unravel.<!-- FACT-D44-050 -->[^akerlof-1970] Insurance with multiple coverage levels, loans with covenants, software with tiered feature sets, and universities with detailed admission procedures make more sense once the screening structure is recognized: the complexity is not bureaucratic accretion but the mechanism through which the uninformed party overcomes adverse selection.

The clarity extends to policy design. Means-tested social programs with deliberately awkward application processes are often criticized as bureaucratic; screening theory suggests some of the friction is a deliberate screening device — the public-economics term of art is an *ordeal mechanism* (Nichols and Zeckhauser 1982[^nichols-zeckhauser-1982]) — that imposes a cost only genuinely needy applicants will bear, improving targeting through self-selection. Kleven and Kopczuk (2011)[^kleven-kopczuk-2011] formalized the resulting trade-off: more rigorous screening improves classification but raises application complexity, induces incomplete take-up, and creates a margin between Type I exclusion and Type II inclusion errors. The stronger distributional critique — that the same frictions fall disproportionately on less-resourced participants — is developed in the *administrative burden* literature (Moynihan, Herd, and Harvey 2015[^moynihan-herd-harvey-2015]; Herd and Moynihan 2025[^herd-moynihan-2025]) and in Sunstein's *sludge audits* framework[^sunstein-2022]. Empirical work on welfare take-up (Currie 2006[^currie-2006]; Bhargava and Manoli 2015[^bhargava-manoli-2015]) shows that simplification and default enrollment materially raise participation, while field evidence on self-targeting cautions that the incidence of ordeals is not theoretically fixed (Alatas et al. 2016[^alatas-et-al-2016]). The current policy-design recommendation is not "never screen" but manage the trade-off explicitly: prefer administratively verifiable eligibility and default enrollment when possible, reduce learning and compliance costs, and test whether a given friction improves targeting net of exclusion costs.

manages_complexity:
Without screening, the uninformed party in markets with heterogeneous types faces a bad choice: ignore the heterogeneity and offer a uniform option (producing adverse selection), or attempt direct type verification for every applicant (often impractical or prohibitively costly). Screening collapses the complexity by *delegating type-revelation to the informed party* — the agents reveal their type through their choices, and the menu is designed to make that revelation incentive-compatible. Bolton and Dewatripont (2005)[^bolton-dewatripont-2005] develop this delegation logic in their canonical contract-theory treatment.<!-- FACT-D44-051 -->[^bolton-dewatripont-2005]

This delegation scales remarkably well. An insurer with millions of policyholders need not individually assess each one's risk profile; policyholders self-sort through deductible and premium choices. A university with thousands of applicants need not interview each one thoroughly; the application essay, test scores, and extracurricular record induce self-selection. A cloud-computing provider with hundreds of thousands of enterprise customers need not negotiate each contract individually; the tier menu lets customers choose their own scale.

Screening has informational limits. It can reveal type only along the dimensions the menu distinguishes; types differing on unobserved dimensions still pool. And screening does not solve moral hazard — once types have selected their option, they may still misbehave — requiring additional monitoring or incentive-compatible features within each option.

abstract_reasoning:
Consider the canonical two-type screening problem, presented in standard textbook form by Mas-Colell, Whinston, and Green (1995)[^mas-colell-whinston-green-1995] in their adverse-selection chapter.<!-- FACT-D44-052 -->[^mas-colell-whinston-green-1995] Let types be \(\theta \in \{\theta_L, \theta_H\}\) with probabilities \(p_L, p_H\). The principal offers a menu \(\{(q_L, t_L), (q_H, t_H)\}\) (quality-quantity \(q\), payment \(t\)). The principal maximizes expected profit \(\sum_i p_i (t_i - c(q_i))\) subject to:

- **Individual rationality (IR)** for each type: \(u_i(q_i, t_i, \theta_i) \geq 0\).
- **Incentive compatibility (IC)** for each type: \(u_i(q_i, t_i, \theta_i) \geq u_i(q_j, t_j, \theta_i)\) for all \(j \neq i\).

Under standard assumptions (*single-crossing*, type-ordered preferences) the optimal menu exhibits a characteristic pattern: the high-type's option is at the efficient quality/quantity; the low-type's option is *distorted downward* relative to the full-information efficient level. This downward distortion is the cost the principal pays to reduce information rents the high type would otherwise extract by threatening to select the low-type option. The high type receives a positive *information rent*; the low type receives zero rent at their option.

The pattern generalizes with remarkable consistency. In *optimal income taxation* (Mirrlees 1971), the government screens high- from low-ability workers through progressive tax schedules. In *optimal insurance contracts* (Rothschild-Stiglitz 1976), the insurer screens high- from low-risk clients through coverage-deductible menus. In *optimal quality discrimination* (Mussa-Rosen 1978), the monopolist screens high- from low-valuation customers with the low-quality option distorted downward. All three share the structural skeleton.

The mathematics yields useful design anchors. The *revelation principle* applies: without loss of generality the principal can restrict attention to *direct menus* in which each type truthfully reports their type and the menu assigns the designated option. The *no-distortion-at-the-top* result — the highest type's option is at the efficient level — appears with substantial generality.

knowledge_transfer:
Structural role mappings, drawing on Salanié's (2005)[^salanie-2005] cross-domain treatment of screening mechanisms:

- **Uninformed designer** ↔ insurer / employer / lender / platform / monopolist / tax authority
- **Informed agents** ↔ policyholders / applicants / borrowers / customers / taxpayers
- **Menu options** ↔ policy tiers / interview stages / loan products / pricing plans / tax brackets
- **Type-differentiated preferences** ↔ single-crossing / type-ordered utilities
- **Incentive compatibility** ↔ "each type's chosen option is the one the designer wants them to take"
- **Individual rationality** ↔ participation constraint / outside-option threshold
- **Downward distortion of low-type option** ↔ information-rent minimization cost
- **No distortion at the top** ↔ highest type always gets the efficient allocation

Where the structure transfers: in **online lending platforms**, loan menus with various interest-rate / collateral / reporting combinations induce borrower self-selection; riskier borrowers accept higher rates for looser covenants, lower-risk borrowers accept covenants for lower rates, and very-high-risk borrowers are screened out entirely by credit-score floors.<!-- FACT-D44-053 -->[^salanie-2005] In **retail loyalty programs**, premium tiers with higher fees and higher benefits induce high-engagement customers to self-select, revealing willingness-to-spend and enabling marketing customization. In **software subscription pricing**, usage-based vs. seat-based vs. enterprise-unlimited tiers induce customers to self-select based on expected usage patterns and organizational scale.

The transfer extends to less commercial domains. In **health insurance**, high-deductible health plans paired with HSAs attract healthier enrollees who expect low medical spending, while lower-deductible plans attract enrollees anticipating more care; ACA exchange design, risk-adjustment mechanisms, and metal-level tiers extensively address the resulting screening and adverse-selection dynamics. In **academic journal submission**, journal tiers with different prestige levels, review-time expectations, and article-processing fees induce authors to submit to the tier they believe their work can reach — a soft screening mechanism. In **cooperative membership**, tiered membership classes with different governance rights, patronage-rebate structures, and participation requirements induce members to self-select into the class matching their engagement level. In **technical hiring**, take-home coding challenges and multi-hour on-sites screen not only for ability but for commitment — low-commitment applicants drop out before completing the screens, revealing themselves as poor fits for high-investment roles.

prose:
_(Pass B — pending; see tracker/v2_upgrade.md.)_

### Catalog cross-references (v1, batches 02–29)

Solution archetypes that reference this prime (per `solution_archetypes/_catalog/catalog_v1_batches_02_29.yaml`):

**As source prime (3):** `adverse_selection_filtering`, `hidden_type_screening`, `progressive_narrowing`

**As related prime (3):** `adaptive_threshold_recalibration`, `credible_signaling`, `payoff_restructuring`

notes:
Pass B will articulate the *tight-pair* relationship with *Signaling* and the closer-than-it-seems relationship with *Mechanism Design* and *Incentive Compatibility*. Screening is a specific mechanism-design problem — menu design under adverse selection — and the incentive-compatibility constraints in screening are the same type imposed by mechanism design generally. Pass B should present screening as the complement of signaling, the receiver-side response to the same adverse-selection problem.

Pass B should address *ethical and fairness* considerations in screening design. The Clarity section frames the trade-off through the established *ordeal mechanism* / *administrative burden* / *sludge* literatures (Nichols-Zeckhauser 1982; Kleven-Kopczuk 2011; Moynihan, Herd, and Harvey 2015; Herd-Moynihan 2025; Sunstein 2022); Pass B should deepen the treatment by working through concrete screening contexts — means-tested welfare with onerous application processes, health insurance with deductibles exceeding household savings, and academic admissions with high application friction — and by presenting the policy-design recommendations (administrative verification of eligibility where feasible, default enrollment, routine sludge audits, and explicit evaluation of whether a screening friction improves targeting net of exclusion costs) as concrete design patterns rather than abstract caveats.

Pass B should also address *dynamic screening* (Courty-Li 2000[^courty-li-2000]; Baron-Besanko 1984[^baron-besanko-1984]) where menus evolve and agents may reveal type progressively, and *continuous-type screening* with its bunching and ironing phenomena (Mussa-Rosen 1978; Maskin-Riley 1984).

Review flags: `tight_pair_with_signaling`. The two abstractions must be presented together. The origin is unambiguously economics-finance (Rothschild-Stiglitz, Mirrlees, Mussa-Rosen, Maskin-Riley); operations research is a secondary domain where menu-design optimization and selection-procedure algorithms are studied; psychology_behavioral_sciences is relevant because real-world screening must account for bounded rationality, choice architecture effects, and non-standard preferences that depart from the classical model.


Script

slug: script

one_liner: A stored representation of a familiar situation as a temporally and causally ordered sequence of events with role-slots, props, and conditions, against which encounters are interpreted and deviations detected as breaches.

core_idea:
A script is a stored representation of a familiar situation as an ordered sequence of events, including the roles that typically fill each step, the props or resources involved, the typical preconditions and termination conditions, and the standard variations. Where a schema is a generic structured knowledge representation — a slot-and-filler bundle that may or may not be temporally ordered — a script is specifically the *temporally and causally ordered* schema: this happens, then this, then this, with each step setting up the preconditions for the next, and with each role available to be filled by different actors on different occasions.<!-- FACT-D71-276 -->[^schank-abelson-1977] The structural commitment is that some recurring situations are stored not as facts about the world but as procedures with slots, and that future encounters with a matching situation are interpreted by binding the current particulars to the script's roles and predicting the next step from the script's sequence.

Once a script is active, it does powerful inferential work. It lets the agent skip unmentioned-but-expected steps without surprise, fill in missing detail by default, predict what is about to happen, and detect deviations as anomalies that need explanation. The same mechanism that makes routine situations effortless also makes deviations conspicuous: *script breach* — the predictive failure when observation does not match the script's expected next step — is the diagnostic signature of the pattern, and it is what distinguishes script-based interpretation from mere recall. The structural skeleton has a recurring set of parts: a triggering situation that activates a stored script; an ordered sequence of steps the script predicts; roles, the slots filled by particular actors on each performance; props, the resources the script presupposes; preconditions and termination conditions that bracket the script's applicability; default fillers that supply expected values for unstated particulars; the breach that fires when observation deviates from prediction; and composition, by which larger procedures are built from nested and sequenced scripts. The four commitments that mark the type — temporal ordering, role-binding, slot-filling with defaults, and breach-diagnosis — travel together, and it is their conjunction that distinguishes a script from its near-neighbors, none of which carries all four.

structural_signature:
*a triggering situation that activates a stored representation* — *a temporally and causally ordered sequence of steps* — *role-slots filled by particular actors on each performance* — *props and preconditions/termination conditions bracketing applicability* — *default fillers supplying unstated particulars* — *a breach that fires when observation deviates from the predicted next step* — *composition by nesting and sequencing of sub-scripts*

The pattern is present when each of the following holds:

- **A triggering situation.** A recurring situation whose recognition activates a stored procedural representation rather than a bare fact.
- **An ordered sequence.** Steps in temporal and causal order, each setting up the preconditions of the next — the commitment that distinguishes a script from an unordered schema.
- **Role-slots.** Generic slots in each step, bound to particular actors on each performance, so the same script runs with different fillers.
- **Props and bracketing conditions.** Presupposed resources, plus preconditions and termination conditions that bound where the script applies.
- **Default fillers.** Expected values for unstated particulars, letting the agent skip unmentioned-but-expected steps and fill in missing detail without surprise.
- **A breach operation.** A continuously-running comparison of observation against the predicted next step; a mismatch fires as an anomaly needing explanation. This breach is the diagnostic payload that distinguishes script-interpretation from mere recall.
- **A composition invariant.** Larger procedures are built by nesting and sequencing smaller scripts, covering a vast situation space with a modest composable library.

These four commitments — temporal ordering, role-binding, slot-filling with defaults, and breach-diagnosis — travel together; their conjunction is what marks the type apart from its near-neighbors.

what_it_is_not:
- **Not a schema.** See `schema`. A schema is the *generic* structured-knowledge bundle — slots and fillers — that *may or may not be temporally ordered*. A script is the *specifically temporally-and-causally ordered* schema. The added commitments (sequence, breach-on-next-step) are what make a script more than a schema.
- **Not sequencing.** See `sequencing`. Sequencing is the bare ordering of items. A script adds *role-slots, default fillers, bracketing conditions, and breach-diagnosis* on top of order; an ordered list with no slots to bind and no prediction to violate is sequencing, not a script.
- **Not an algorithm.** See `algorithm`. An algorithm is a *precise, deterministic procedure guaranteeing a result*. A script is a *defeasible expectation* about how a familiar situation typically unfolds, tolerant of variation and informative precisely through its breaches; it predicts rather than computes.
- **Not a ritual.** See `ritual`. A ritual is a *socially-prescribed, often symbolically-charged* performance whose correctness is normative. A script is a *descriptive cognitive representation* used to interpret and predict; a ritual may be one kind of situation a script represents, but the script is the interpretive structure, not the prescription.
- **Not a mental model.** See `mental_model`. A mental model represents *how a system works* (causal structure). A script represents *what happens next in a situation* (temporal procedure). One supports counterfactual reasoning about mechanism; the other supports prediction of the next step.
- **Common misclassification.** Calling any stored knowledge of a routine a "script." The catch: check for all four commitments — temporal ordering, role-binding, defaults, and breach-diagnosis. Knowledge that is unordered (schema), has no slots (sequencing), is deterministic (algorithm), or is prescriptive (ritual) is missing one of the conjuncts that define the type.

broad_use:
- **Cognitive science.** The canonical restaurant script — enter, be seated, read the menu, order, eat, pay, leave — and the broader claim that much of everyday cognition is script-driven rather than reasoned;<!-- FACT-D71-277 -->[^schank-abelson-1977] story comprehension and default inference in language understanding run on script-like representations.
- **Service design and UX.** Customer journey maps are scripts: step-ordered sequences with role-slots (customer, agent, system), preconditions (account state, prior step completion), and variations (express versus full path, error branches).<!-- FACT-D71-278 -->[^shostack-1984]
- **Software and automation.** Scripts in the literal sense — shell scripts, test scripts, CI/CD pipelines, playbooks — are ordered sequences of operations on parameterized objects; behavior trees in robotics and game AI are the same structure with branching.<!-- FACT-D71-279 -->[^colledanchise-ogren-2018]
- **Law and procedure.** Legal proceedings, voting procedures, parliamentary rules, surgical protocols, and aviation checklists are scripts with required ordering, role-slots, and termination conditions — the script form is what makes "due process" meaningful.<!-- FACT-D71-280 -->[^robert-2020]
- **Social interaction and ritual.** Greeting routines, dinner-party scripts, weddings, and funerals; cultural fluency is partly script fluency — knowing what comes next in the situations one is in.
- **Robotics and embodied AI.** Action sequences, manipulation scripts, and task-and-motion plans — ordered procedures with parameter slots and precondition checks.<!-- FACT-D71-281 -->[^ghallab-etal-2004]
- **Pedagogy.** Lesson plans, lab protocols, and training curricula are scripts with role-slots, ordered steps, and termination conditions.

clarity: Naming the script representation separates three things that talk of routine behavior routinely conflates: knowledge of what is *true* (a schema or fact), knowledge of what to *do* (a procedure or script), and knowledge of how to *do it* (a skill or embodied competence). A waiter has all three about restaurant service, and failing to distinguish them is much of what makes "tacit knowledge" seem mysterious — the procedural layer, the script, gets folded into either the factual layer or the skill layer and thereby disappears as an object of analysis. Pulling the script out as its own representational type makes the procedural knowledge inspectable: one can ask what the steps are, what fills each role, what the preconditions are, independently of the facts the agent knows and the motor skills they have. The frame also corrects a common error: treating *deviation from script* as a flaw, when it is frequently the most informative thing in the situation. Because the script generates a prediction at each step, a mismatch between prediction and observation is a signal, not noise — anomaly detection in long-running systems often reduces precisely to script-breach detection. The clarifying move is to recognize the breach as the script's payload rather than as a failure of the script: the value of having a script is not only that it makes the routine effortless but that it makes the non-routine conspicuous, and naming the breach as the diagnostic signature makes that value explicit and usable.

manages_complexity: A script compresses a recurring situation's interpretive demands to a few diagnostic operations: identify the script the situation triggers, bind the current particulars to the script's roles, locate the current step in the sequence, predict the next step, and compare the actual to the predicted. Most situations the agent encounters require only the first four operations — recognition, binding, location, prediction — and generate no anomaly, so they are handled with almost no deliberative effort. Only the few situations that produce a mismatch at the fifth operation demand attention. This is a radical reduction in cognitive load for routine encounters, because the overwhelming majority of moments in a familiar situation are disposed of by the script without any reasoning at all, and the agent's scarce attention is spent only where the script's prediction fails. The compression is what makes fluent functioning in a complex social and physical world possible: an agent who had to reason explicitly about every step of every familiar situation would be paralyzed, whereas an agent equipped with scripts treats the routine as effectively free and reserves deliberation for the genuinely novel. The same structure also manages complexity through composition. Larger procedures need not be represented as monolithic sequences; they are built by nesting and sequencing smaller scripts — the restaurant script nested inside a date-night script nested inside a meeting-the-parents script — so that a vast space of complex situations is covered by a modest library of composable scripts rather than an unmanageable enumeration of whole scenarios.

abstract_reasoning: Script-based reasoning supports inference about *what was not explicitly mentioned* in a story or report: the reader fills in the unstated steps from the script, so that a narrative naming only a few events is understood as the whole sequence. It supports *prediction of the next likely event* given a recognized situation, which is the basis of the fluency with which agents anticipate what is about to happen in familiar settings. It supports *anomaly detection* via script-breach, turning the script's prediction into a continuously running check against observation. It supports *role-based substitution*: the same script run with different actors yields predictable variations, so that knowing the script lets an agent reason about how the situation will differ when the roles are filled differently. And it supports *script composition*, the inference that a large unfamiliar procedure can often be decomposed into nested and sequenced scripts the agent already has, so that novel situations are made tractable by recognizing their familiar sub-scripts. Each of these inferences is stated in terms of the script's ordered-steps-with-role-slots structure rather than in terms of any particular content, which is why they hold across substrates: the inference that unstated steps can be filled in from the script applies identically to a story about a restaurant, a customer-support transcript missing intermediate steps, and a partially-logged software pipeline, because in each case a recognized script supplies the defaults the record omits.

knowledge_transfer:
The script formalism transfers across substrates because its four commitments — temporal ordering, role-binding, slot-filling with defaults, and breach-diagnosis — are stated in structural terms that carry from cognition into software, services, law, and robotics with little translation. The cognitive-science formalism ports into service design as customer-journey-mapping: the role-slots and precondition-postcondition machinery of the original restaurant-style scripts directly populate service-blueprint methodology, so that a service is designed by authoring and instrumenting a script. Cognitive scripts port into automation, where literal software scripts and behavior trees inherit the slot-binding and sequencing logic, and test scripts borrow the precondition/action/postcondition structure wholesale.<!-- FACT-D71-282 -->[^meszaros-2007] The legally-rigorous ordering-with-role-slots pattern ports from procedural law into medicine, where surgical timeouts and procedural protocols adopt it — the same script discipline that makes a contract enforceable makes a checklist effective, because both depend on the right steps occurring in the right order with the right roles filled.<!-- FACT-D71-283 -->[^gawande-2009] And the pattern ports from robotics into embodied AI, where hierarchical task networks and behavior trees carry the script structure to autonomous agents, with precondition-checking and recovery-on-failure inheriting the script's anomaly-handling directly.<!-- FACT-D71-284 -->[^colledanchise-ogren-2018]

The role mappings that make these transfers reliable are direct. The triggering situation maps to the recognized restaurant, the start of a support interaction, the filing of a case, the initiation of a pipeline, the beginning of a surgical procedure. The ordered steps map to the seating-ordering-eating-paying sequence, the journey-map stages, the arraignment steps, the build-test-deploy stages, the pre-op checklist items. The roles map to diner and server, customer and agent and system, judge and counsel, the parameterized objects a pipeline operates on, surgeon and circulating nurse. The breach maps to the menu handed at the door rather than the table, the support flow that skips a required verification, the procedure that omits a step, the pipeline stage that fails its postcondition. Because the structure and its breach-diagnostic are shared, a practitioner who has internalized scripts in one substrate recognizes them in another: a surgeon running a pre-op checklist, a pilot running pre-flight, a court running an arraignment, and a CI pipeline running build-test-deploy all instantiate the same structural pattern of ordered steps, role-slots, anomaly detection, and recovery branches. What transfers is the recognition that a recurring situation can be represented as a temporally ordered, role-bound, default-filling procedure whose predictive breaches are its most informative output — and the design discipline that follows, of authoring the steps, naming the roles, specifying the preconditions and termination conditions, and instrumenting the breach so that deviation becomes a detectable signal rather than a silent failure.

formal_abstract:
Schank and Abelson's restaurant script is the canonical worked instance, and it exhibits all four commitments that mark the type.<!-- FACT-D71-285 -->[^schank-abelson-1977] The *triggering situation* is recognizing "I am at a restaurant," which activates a stored procedure rather than a bare fact. The *ordered sequence* is the causally chained steps — enter, be seated, read the menu, order, eat, pay, leave — where each sets up the next: ordering presupposes having read the menu, paying presupposes having eaten. The *role-slots* are generic positions bound to particulars on each performance: the *customer* slot is filled by me tonight and by you tomorrow, the *server*, *cook*, and *cashier* slots by whoever is working. The *props and bracketing conditions* are the presupposed resources (tables, menus, food, money) plus preconditions (the restaurant is open) and termination (the bill is settled and you exit). The *default fillers* do the inferential heavy lifting: told only "Mary went to a restaurant and later complained the steak was tough," a comprehender infers — without being told — that Mary was seated, ordered, and was served, because the script supplies those steps by default.<!-- FACT-D71-286 -->[^bower-etal-1979] This is exactly the *abstract-reasoning* payload: the script lets a narrative naming a few events be understood as the whole sequence. The *breach operation* is the diagnostic signature: if the script predicts "server brings food" but the story says the server brought a bill before any food arrived, the comprehender registers an anomaly demanding explanation — and that breach, not the smooth recall, is the script's most informative output. The *composition invariant* shows in nesting: the restaurant script slots inside a larger date-night script, so a vast situation space is covered by a small composable library.

**Mapped back:** Recognizing the restaurant is the trigger, enter-order-eat-pay is the ordered sequence, customer and server are role-slots, the inferred-but-unstated "she ordered" is a default filler, and the bill-before-food anomaly is the breach — the four commitments traveling together in one schema.

applied_industry:
A CI/CD pipeline is a script in the literal software sense, and instrumenting it through the frame turns silent failures into detectable breaches. The *triggering situation* is a code push or merge, which activates the stored pipeline definition. The *ordered sequence* is the causally chained stages — checkout, build, unit-test, integration-test, deploy-to-staging, deploy-to-production — each whose postcondition is the next stage's precondition:<!-- FACT-D71-287 -->[^humble-farley-2010] you cannot deploy an artifact that did not build, cannot run integration tests against a staging environment that was never provisioned. The *role-slots* are the parameterized objects the pipeline operates on: the *commit* slot bound to this particular SHA, the *environment* slot to staging or production, the *artifact* slot to the built image. The *props and bracketing conditions* are the presupposed resources (build agents, credentials, a container registry) plus preconditions (the branch is mergeable) and termination (a green deploy or an aborted run). The *default fillers* are the pipeline's defaults — the standard build flags, the default environment — supplied for unstated particulars. The *breach operation* is precisely where the engineering value lives: each stage's postcondition is a continuously-checked prediction, and a stage that fails its postcondition (tests red, deploy health-check failing) fires as an anomaly that halts the run and triggers a recovery branch (rollback, alert). The frame's design discipline maps directly onto good pipeline practice — author the stages, name the parameterized roles, specify each stage's pre- and postconditions, and *instrument the breach* so a deviation surfaces as a failed check rather than a silently-shipped bug. The identical structure governs a surgical pre-op checklist (ordered steps, surgeon and circulating-nurse roles, the breach firing when a step is skipped) and an aviation pre-flight, where the script discipline is what makes the omission of a step a detectable signal.<!-- FACT-D71-288 -->[^gawande-2009]

**Mapped back:** The code push is the trigger, build-test-deploy is the ordered sequence, the commit SHA and target environment are role-slots, the standard build flags are default fillers, and a stage failing its postcondition is the breach that the pipeline is instrumented to catch — the cognitive script formalism realized as automation.


Search and Retrieval

one_liner: Locate and extract information.

core_idea: Search and Retrieval is the process of *locating, identifying, and retrieving relevant information, resources, or objects from a larger dataset, environment, or memory system*, often optimizing for speed, accuracy, and efficiency. The essential commitment is that given a query or information need, a system must navigate a search space (continuous or discrete, structured or unstructured) to discover items matching specified criteria, balancing exhaustiveness against computational cost<!-- FACT-D31-046 -->[^salton-1983]. Every search-and-retrieval system faces trade-offs between precision (false positives excluded), recall (false negatives excluded), and query latency, and must determine both what is relevant and how efficiently to locate it.

structural_signature:
- *The query or information need specification and formalization* <!-- FACT-D31-047 -->[^salton-1983]
- *The search space representation and traversal strategy (linear scan, tree index, hash index, graph traversal)* <!-- FACT-D31-048 -->[^learned-indexes-2018]
- *The relevance model and matching criterion (exact, fuzzy, semantic, ranked)* <!-- FACT-D31-049 -->[^bm25-1994]
- *The indexing and pre-computation structures enabling sub-linear retrieval* <!-- FACT-D31-050 -->[^learned-indexes-2018]
- *The ranking and ordering function when multiple matches exist* <!-- FACT-D31-051 -->[^pagerank-1998]
- *The recall-precision-latency trade-off tuning at deployment time* <!-- FACT-D31-052 -->[^trec-eval]

what_it_is_not:
- **Not identical to sorting.** Sorting orders a complete dataset; search retrieves a subset matching a criterion. A search may use sorting as a sub-operation but is not defined by it. You can search without sorting (hash lookup), and sort without searching (produce all elements in order).

- **Not mere pattern matching.** Pattern matching tests a single string or item against a pattern; search applies that test across many candidates and aggregates results. Pattern matching is a sub-operation; search is the larger orchestration.

- **Not equivalent to filtering.** Filtering removes items that do not meet criteria; search adds the dimension of *how to locate candidates efficiently* rather than inspecting all items. A filter applied naively to all data is inefficient search.

- **Not a single algorithm.** Linear search, binary search, hash-table lookup, B-tree traversal, inverted-index lookup, graph search, semantic vector similarity all solve search problems with different assumptions about data structure and query semantics.

- **Not retrieval alone.** Retrieval is the act of obtaining the item once found; search is the process of locating it. The two are often composed: search to identify, then retrieve to obtain.

- **Common misclassification:** Confusing "search" with "full-text search" (a specific instantiation), or assuming all search problems are solved by the same method regardless of data structure, index availability, or latency constraints.

broad_use: Search and Retrieval appears across nearly every domain where information or resources must be located at scale. In operating systems, file system lookup uses inode tables and B-trees to traverse directory hierarchies; process scheduling queues are indexed by priority or deadline to efficiently select the next runnable task; memory management systems use page tables and TLBs to translate virtual addresses to physical ones. In databases, query optimization engines select among multiple index strategies (B-tree range scans, hash-join lookups, index-intersection algorithms) based on cardinality statistics and cost models; execution engines iterate through result cursors one row at a time to avoid materializing entire result sets. In search engines, inverted indexes map each term to lists of documents, with compression and skip lists enabling million-document retrieval in milliseconds; PageRank and other ranking signals order results by authority and relevance; distributed index sharding (documents partitioned across machines) enables parallel retrieval. In information retrieval systems, the Salton-McGill vector space model and BM25 probabilistic framework provide mathematically grounded relevance models; TREC (Text REtrieval Conference) benchmarks evaluate precision-recall trade-offs across systems. In machine learning, nearest-neighbor search in high-dimensional vector spaces (embeddings) enables recommendation and similarity tasks; locality-sensitive hashing (LSH) provides approximate retrieval in polynomial rather than exponential time. In memory systems, CPU caches implement set-associative lookup with LRU replacement; virtual memory uses multi-level page tables and TLBs for fast address translation. In cognitive psychology, memory retrieval is triggered by associative cues; priming effects show that recent or frequent memories surface faster. In biology, bacterial chemotaxis searches for nutrient gradients via tumble-and-run algorithms; predators forage by balancing search cost against capture success.

clarity: Search and Retrieval clarifies that information access is not "free" — it requires a strategy. The construct separates the *specification* of what is wanted (query, information need) from the *mechanism* by which it is found (algorithm, index), making explicit the role of pre-computation (indexing, caching) in trading storage for retrieval speed. It forces recognition that relevance is defined rather than objective; the same query can return different results depending on the relevance model used<!-- FACT-D31-053 -->[^bm25-1994]. It shows that efficiency is not intrinsic to data but depends on access patterns and index design. The clarity also reveals why naive linear search (checking every item) fails at scale, motivating the design of indexes, caches, and approximate-retrieval methods. By making the query explicit, the construct enables reasoning about what constitutes a good answer: is precision critical (false positives are costly) or recall (false negatives are costly)? Does latency matter (interactive queries) or throughput (batch processing)? Can the system afford the storage overhead of sophisticated indexes, or must it operate on minimal resources?

manages_complexity: The construct manages complexity by decomposing the search problem into layers: *query parsing* (what are we looking for?), *index maintenance* (how do we organize for rapid access?), *search execution* (how do we traverse the index?), and *ranking* (how do we order results?). Pre-computed indexes (inverted indexes, B-trees, hash tables, semantic vector embeddings) decouple query time from data size, moving expensive work to index build-time rather than query-time<!-- FACT-D31-054 -->[^learned-indexes-2018]. This enables scaling from megabyte datasets to exabyte-scale search engines. Hierarchical and approximate retrieval methods (bounds-based pruning, locality-sensitive hashing, hierarchical navigable small-world graphs) reduce search-space exploration. The structural clarity also supports caching and memoization strategies — retrieving frequently-searched items from fast-path caches rather than full search. Finally, the framework enables reasoning about trade-offs: adding indexes speeds retrieval but increases storage and update cost; relaxing precision (accepting approximate matches) speeds retrieval and reduces index size. Query optimization decisions (which index to use, whether to parallelize) become explicit, analyzable, and tunable.

abstract_reasoning: Search-and-retrieval reasoning proceeds by identifying the search space and its properties (size, structure, dimensionality, growth rate), specifying the relevance criterion or similarity metric (what does "relevant" mean for this domain?), selecting an index structure and retrieval algorithm suited to the workload (lookup-heavy vs. range-heavy vs. nearest-neighbor), and tuning the balance between recall, precision, and latency. It supports systems design (query optimizer decisions, cache eviction policies, hot-data identification), organizational decisions (what information to catalog and how, what metadata to maintain), cognitive strategies (which cues trigger memory retrieval most reliably), and biological fitness (time spent foraging vs. energy obtained)<!-- FACT-D31-055 -->[^salton-1983]. Engineers ask: given expected query patterns (uniform random access vs. skewed popularity), what index structure minimizes latency while staying within memory constraints? Given a fixed computational budget, should we invest in preprocessing (index building) or runtime execution? If data is streaming and freshness matters, can we update indexes incrementally?

knowledge_transfer: A software engineer's search-and-retrieval reasoning (query specification, index design, ranking, cache strategy) transfers across database query optimization, search-engine implementation, cache design, and memory-hierarchy tuning. The structural core is the insight that access patterns matter and can be optimized via pre-computation; what varies is the data structure (hash, tree, vector embedding, graph, bloom filter) and the relevance metric (exact equality, fuzzy string similarity, vector distance, semantic relevance). The same diagnostic framework — is the index appropriate for the query pattern, is precision/recall balanced correctly, is latency acceptable — applies to database indexes, full-text search, nearest-neighbor lookup, CPU caches, TLBs, and cognitive-memory cues. An engineer optimizing a web search engine uses the same principles as a neuroscientist studying memory recall.

formal_abstract:
The vector space model (Salton & McGill, 1983)<!-- FACT-D31-056 -->[^salton-1983] represents documents and queries as vectors in a high-dimensional term space, where each dimension is a term (word) and the value is the term's frequency or importance (TF-IDF weighting). A query is likewise vectorized, and relevance is computed as the cosine similarity between the query vector and document vectors, with retrieval returning documents ranked by similarity. This formalism decouples the representation (vectors) from the retrieval method (cosine distance), enabling systematic study of ranking functions. Modern systems extend to semantic embeddings, where dense vectors capture meaning rather than term occurrence, enabling approximate nearest-neighbor search via locality-sensitive hashing or learned-index methods.

**Mapped back:** This instantiates the structural signature directly — query formalization (vector), search-space representation (high-dimensional space), relevance model (cosine similarity), ranking (similarity scores sorted), and efficiency (vector indexing structures).

applied_industry:
A web search engine indexes billions of documents using an inverted index: a mapping from each word to the list of documents containing it, with positions and frequency metadata. When a user queries "machine learning algorithms," the engine retrieves the posting lists for each term, intersects them to find documents containing all terms, applies ranking signals (PageRank, click history, recency, domain authority), and returns the top K results. The system uses distributed indexes (documents sharded across many machines), caching (popular queries cached with pre-computed result ranks), and approximate retrieval (early termination when confidence in top-K is high). Kubernetes' service discovery indexes services by name and labels, enabling efficient lookup of IP addresses for load balancing. Elasticsearch provides full-text search over inverted indexes with text analysis pipelines, filtering, faceting, and relevance tuning, powering analytics and log search systems.

**Mapped back:** These show search-and-retrieval as the unifying principle behind modern information systems, instantiated via indexing, ranking, and efficiency optimization in production environments.

title: Precision vs Recall vs Query Latency

body:
Exhaustive search retrieves all relevant items (high recall) but is slow. Approximate or early-termination retrieval is fast (low latency) but misses some results (lower recall). Tightening relevance criteria improves precision (fewer false positives) but may reduce recall. No single point is optimal across all use cases<!-- FACT-D31-057 -->[^trec-eval].

- **T2: Index Maintenance Cost vs Retrieval Speed.**
  Sophisticated indexes (B-trees, learned indexes, semantic embeddings) enable rapid retrieval but require expensive rebuild or incremental maintenance when data changes. Write-heavy systems struggle with index staleness; stale indexes reduce retrieval quality, but rebuilding is expensive. Real-time systems must balance: commit-log-based indexes trade some retrieval speed for fresh writes; batch indexing sacrifices freshness for efficiency.

- **T3: Index Size and Memory Pressure.**
  Larger indexes improve retrieval speed and recall but consume storage and RAM. Distributed indexes solve this by sharding but introduce network latency and coordination complexity. Systems with limited memory must prune indexes or use approximate methods, trading accuracy for size.

- **T4: Query Complexity vs Expressiveness.**
  Simple queries (keyword search, exact match) are fast to execute but inexpressive. Complex queries (boolean operators, faceted search, semantic constraints) are expressive but slow and difficult to optimize. The system must decide what query semantics to expose.

- **T5: Centralized vs Distributed Search.**
  A single search index is simple to manage and guarantees consistency but becomes a bottleneck at scale. Distributed indexes parallelize retrieval but introduce consistency, freshness, and coordination complexity<!-- FACT-D31-058 -->[^distributed-search]. Geographic replication adds further trade-offs: local queries are fast but must handle cache-invalidation when indices diverge.

- **T6: Semantic Relevance and Human Satisfaction.**
  Ranking by relevance-score (BM25, cosine similarity, learned-to-rank models) is objective and reproducible but may not match human judgments. Incorporating user feedback (clickthrough, dwell time) improves ranking but introduces position bias (users click more on top results) and feedback loops<!-- FACT-D31-059 -->[^ranking-feedback]. The system must balance algorithmic objectivity with human-grounded quality signals.

prose:
_(Pass B — deferred)_

### Catalog cross-references (v1, batches 02–29)

Solution archetypes that reference this prime (per `solution_archetypes/_catalog/catalog_v1_batches_02_29.yaml`):

**As source prime (4):** `coarse_to_fine_search`, `index_based_retrieval`, `progressive_narrowing`, `strategic_caching`

**As related prime (4):** `constraint_formulation`, `hidden_path_discovery`, `traceability_linking`, `transaction_cost_reduction`

notes: Search and Retrieval is held at High confidence. Foundational abstraction in computer science (databases, search engines, operating systems), cognitive science (memory retrieval), and biology (foraging). The vector space model and inverted indexes are canonical formalisms with decades of empirical validation. Modern instantiations (Elasticsearch, vector databases, learned indexes, semantic embeddings) build on classical information retrieval, demonstrating lasting architectural relevance.


Second Law of Thermodynamics

one_liner: Entropy increases.

origin_domain: physics

core_idea:
The second law of thermodynamics is the empirical and theoretical principle that establishes a time-asymmetric direction for physical processes: in an isolated macroscopic system, entropy does not decrease with time, so that heat flows spontaneously from hot to cold (but not the reverse), no cyclic heat engine can convert heat entirely to work, and macroscopic irreversibility is a systematic feature of the world. The essential commitment is that the microscopic laws of physics are time-symmetric but the macroscopic world is not — a macroscopic arrow of time emerges statistically from the vastly-greater number of microstates corresponding to high-entropy macrostates compared to low-entropy ones — and that this asymmetry is not merely a curiosity but the foundation of thermodynamic engineering (heat engines, refrigerators, heat pumps, all with efficiency bounded by Carnot's theorem), of chemistry (direction of spontaneous reactions), and of cosmology (the arrow of time in an expanding universe).

Every second-law articulation specifies (1) the system and its thermodynamic state; (2) a formulation — Kelvin-Planck <!-- FACT-D11-022 -->[^kelvin-1851] (no cyclic device converts heat entirely to work), Clausius <!-- FACT-D11-021 -->[^clausius-1850] (no cyclic device transfers heat from cold to hot without external work), or entropy (ΔS_iso ≥ 0 for any process in an isolated system), all equivalent; (3) the scope — applicable strictly only to macroscopic systems near equilibrium and to coarse-grained descriptions; and (4) the consequences — Carnot efficiency bound η_max = 1 − T_cold/T_hot, direction of spontaneous processes, free-energy functions for thermodynamic potentials (Helmholtz and Gibbs free energy) and their role in determining equilibrium. The construct was developed by Carnot (1824) <!-- FACT-D11-020 -->[^carnot-1824], Clausius (1850s) <!-- FACT-D11-021 -->[^clausius-1850], and Kelvin (1852) <!-- FACT-D11-022 -->[^kelvin-1851], with statistical foundation from Boltzmann <!-- FACT-D11-024 -->[^boltzmann-1872] and Gibbs in the late 19th century. The H-theorem and Boltzmann's probabilistic interpretation <!-- FACT-D11-025 -->[^boltzmann-1877] established that the second law emerges from molecular chaos and is fundamentally statistical in nature.

structural_signature: *Total entropy non-decrease principle* — For an isolated macroscopic system undergoing any process, the entropy change satisfies ΔS ≥ 0, with equality only for reversible processes. *Closed-system framing / isolated boundary condition* — For a system exchanging heat Q with a reservoir at temperature T, the entropy change of the combined system-plus-reservoir satisfies ΔS_total ≥ 0. *Quantitative entropy bookkeeping* — In the Clausius inequality form, ∮ δQ/T ≤ 0 for any cyclic process. *Disorder-direction commitment / arrow of time* — Equivalent formulations emphasize different aspects but are provably equivalent: Kelvin-Planck prohibits perfect heat-to-work conversion; Clausius prohibits spontaneous cold-to-hot heat flow; the entropy formulation captures both. *Macroscopic certainty from microscopic statistics* — Carathéodory's axiomatic approach <!-- FACT-D11-023 -->[^caratheo-1909] to thermodynamics provides a rigorous geometrical foundation for these equivalences, replacing them with the principle of inaccessibility of certain states.

what_it_is_not:
**Common misclassification**: Treating the second law as stating that entropy always increases in every subsystem. The law applies to *isolated* systems; entropy can and does decrease locally when a subsystem exchanges entropy with its environment. Refrigerators decrease the entropy of their cold compartment; living organisms maintain low entropy by exporting entropy to the environment. The total (system + environment) entropy increases.

**Not a violation of time-reversal symmetry at the microscopic level**: the underlying classical and quantum mechanical laws are time-reversal symmetric (with CPT caveat). The macroscopic arrow of time emerges statistically from the overwhelming number of high-entropy microstates versus low-entropy ones, not from asymmetric microscopic dynamics. This reflects Loschmidt's paradox <!-- FACT-D11-026 -->[^loschmidt-1876], the tension between microscopic reversibility and macroscopic irreversibility.

**Not an absolute prohibition**: the second law is statistical, and for small systems or short times, fluctuations that temporarily decrease entropy are not only possible but calculable. Modern fluctuation theorems <!-- FACT-D11-034 -->[^jarzynski-1997] rigorously quantify such phenomena. Nanoscale experiments regularly observe such fluctuations.

**Not equivalent to "disorder increases"**: the disorder gloss is pedagogically useful but often misleading (see *entropy_thermodynamic_sense*). The precise statement is about microstate counting under macroscopic constraints.

**Not directly applicable to non-equilibrium systems**: the traditional second law formulations (Clausius, Kelvin) assume thermodynamic equilibrium or quasi-static processes. Non-equilibrium domains require extended formalism (Prigogine 1947 entropy production<!-- FACT-D11-029 -->[^prigogine-1947]; Jarzynski 1997 fluctuation theorem<!-- FACT-D11-034 -->[^jarzynski-1997]) where the second law manifests as an inequality consequence of equality.

**Not a statement about information without qualification**: while Landauer's principle <!-- FACT-D11-031 -->[^landauer-1961] establishes a link between information erasure and thermodynamic entropy production (k_B T ln 2 of heat per erased bit), the second law proper is a statement about thermodynamic entropy, not Shannon information. The link is substantive but narrow. This was resolved in the Maxwell demon paradox <!-- FACT-D11-027 -->[^maxwell-1867] through the Szilard-Bennett analysis <!-- FACT-D11-028, FACT-D11-032 -->[^szilard-1929][^bennett-1982].

Cross-references: see *entropy_thermodynamic_sense* (the state function whose behavior the second law specifies); see *irreversibility* (the macroscopic consequence); see *equilibrium* (the state of maximum entropy under constraints); see *conservation_laws* (the second law is not a conservation law but a direction-of-process law); see *thermodynamic_equilibrium* (G1 sibling); see *phase_space* (DP-10 G2, microscopic foundation of entropy).

broad_use: The second law appears in thermodynamics (heat engines, refrigerators, heat pumps with Carnot-bounded efficiency); in chemistry (direction of spontaneous reactions, chemical equilibrium, Gibbs free energy minimization); in statistical mechanics (as the statistical consequence of microstate-counting); in cosmology (the arrow of time in an expanding universe; initial-state questions about low-entropy Big Bang); in biology (metabolic thermodynamics, energy flow through ecosystems; life as an entropy-exporting process); in economics (as a metaphor for systems-level irreversibility, with care); in information theory (Landauer's principle linking information erasure to entropy production); in black-hole physics (generalized second law including black-hole entropy); and in engineering (all heat-exchanging devices operate within second-law constraints).

clarity: The second law is clarifying because it provides a quantitative, universal constraint on what physical processes can and cannot happen spontaneously, unifying heat flow, diffusion, chemical reaction, and phase transition under a single principle. It also establishes time-asymmetry as a macroscopic emergent feature from time-symmetric microscopic dynamics — a deep insight about the relationship between microscopic and macroscopic descriptions. Eddington's concept of the "arrow of time" <!-- FACT-D11-030 -->[^eddington-1928] made this asymmetry central to understanding the direction of physical processes and the structure of time itself in the universe.

manages_complexity: The construct manages the complexity of macroscopic many-particle systems by providing a single universal principle — entropy maximization under constraints — from which vast ranges of phenomena can be derived: heat-flow direction, chemical spontaneity, phase-equilibrium conditions, efficiency bounds on engines, reaction extents, and more. Instead of treating each case separately, the second law and its free-energy implementations provide a unified framework.

abstract_reasoning: Second-law reasoning proceeds by identifying the system and its environment, computing entropy changes (of the system, environment, and total), applying the second law to determine direction of spontaneous processes, and using free-energy functions (Helmholtz for fixed-T-V, Gibbs for fixed-T-P) to quantify thermodynamic spontaneity. It licenses the apparatus of Carnot analysis, free-energy thermodynamics, statistical mechanics ensembles, and non-equilibrium generalizations (Prigogine, Jarzynski-Crooks). It underlies engineering design of all heat-exchanging devices.

knowledge_transfer:
| Role | Clausius form | Kelvin-Planck form | Entropy form | Statistical form |
|---|---|---|---|---|
| Statement | No cyclic cold→hot heat flow | No cyclic 100% heat→work | ΔS_iso ≥ 0 | Macroscopic states evolve toward higher-Ω regions |
| Primary use | Refrigeration analysis | Heat-engine analysis | Direction of spontaneous processes | Bridging micro and macro |
| Bound | COP_max for refrigerators | η_max = 1 − T_cold/T_hot | Entropy maximization | Boltzmann equation, H-theorem |
| Violation regime | None at macroscopic scale | None at macroscopic scale | Small-system fluctuations allowed | Reverse direction of overwhelming improbability |
| Cross-link | Equivalent to others | Equivalent to others | Equivalent to others | Grounds the thermodynamic formulations |

A thermodynamicist's second-law reasoning transfers to chemistry (Gibbs-energy analysis of chemical spontaneity), to cosmology (arrow of time in expanding universes), to information theory (Landauer's bound on heat per bit), to biology (entropy export by living systems), and to engineering (efficiency bounds on all heat engines). The structural core is the direction of spontaneous processes set by the growth of accessible microstates; what varies is the substrate and the specific formulation.

prose: *(Pass B — deferred.)*

notes: Held at High confidence. Foundational law of physics with multiple equivalent formulations. Entry notes the applicability conditions (isolation, macroscopic scale, equilibrium or quasi-static processes) and flags the mis-applications (subsystem-only analysis, nanoscale absoluteness claims, non-equilibrium naive extension). Cross-links to entropy_thermodynamic_sense (the state function), irreversibility (the macroscopic consequence), thermodynamic_equilibrium (G1 sibling), and phase_space (microscopic foundation). Tensions T1–T6 capture the deepest open questions: the micro-macro bridge, reversibility idealization, information costs, time-asymmetry paradoxes, gravitational scope, and cosmological origins.