# Project 03 — low-coverage rerun: does the coverage discipline earn its keep off the ceiling? RESULTS (2026-05-25)

**Question.** Project 02 (+ B′) found the coverage-enforcing lift inert on quality — but only at the
high-coverage ceiling. Is it inert *in general*, or does it earn its keep once baseline coverage is genuinely
LOW? This builds the low-coverage regime two ways — (a) deliberately hard, under-specified, multi-trap,
distant scenarios; (b) a weaker solver (Haiku) crossed with the frontier — and measures coverage with an
objective, ceiling-breaking instrument: a **catalog-derived coverage key** per scenario (the load-bearing
components, with the 2–4 most-droppable flagged [HARD]/"trap"), scored COVERED/MISSING by independent audit.

**Status.** Rep-1 of the matrix run: B′ (free planner, coverage NOT foregrounded) vs C′ (scoped/adaptive
coverage — single operative pattern, aggressive `correctly_declined`, fixing project-02 C's over-build) ×
{frontier=opus, weak=haiku} × 5 scenarios = 20 cells. Coverage keys were derived from the catalog by 5
independent passes (NOT hand-authored). Holistic 6-dim grading deferred (the coverage-keyed metric answers
the question; quality tied throughout the program). n=1/cell this rep — directional.

## The premise check (B′ vs the keys) — the regime split exactly on §9.3

| solver | overall coverage | trap ([HARD]) coverage | traps dropped |
|---|---|---|---|
| **frontier (opus)** | 61/67 = 91% | **20/20 = 100%** | none |
| **weak (haiku)** | 52/67 = 78% | **15/20 = 75%** | L1 error-cost posture; L4 truthfulness-scoring; L5 (hidden-state estimation, uncertainty, behavior-monitoring) |

Even on deliberately hard/oblique/multi-trap scenarios, the **frontier free planner covers every trap with
no scaffolding** — the low-coverage regime cannot be reached at the frontier (it "already has the grammar
internalized", §9.3). The **weak solver does show real gaps** (75% trap coverage), so the headroom the
coverage check needs exists only in the weak arm. That made the frontier arm a predicted null and the weak
arm the live test.

## The decisive result: the coverage discipline did NOT raise coverage, even where there was headroom

| arm | overall coverage | trap coverage |
|---|---|---|
| B′ frontier | 61/67 (91%) | 20/20 (100%) |
| **C′ frontier** | 63/67 (94%) | **20/20 (100%)** |
| B′ weak | 52/67 (78%) | 15/20 (75%) |
| **C′ weak** | 53/67 (79%) | **14/20 (70%)** |

- **Frontier: null, as predicted.** C′ ≈ B′ (both ~100% traps); no headroom.
- **Weak: flat.** Forcing the scoped coverage check moved overall coverage 78%→79% and trap coverage
  **75%→70%** — i.e. **no improvement** (within n=1 noise, slightly worse). C′-weak *recovered* L1's dropped
  trap but *lost* traps on L2 and L3 — it shuffled which traps were missed rather than closing the gap.

So the coverage-enforcing lift does **not** earn its keep as a runtime coverage/quality mechanism in *either*
regime. The workflow-02/B′ "inert at the ceiling" finding extends to the low-coverage regime: still inert.

## Why — the mechanism (the real finding)

Every C′ cell, including the weak ones, dutifully produced a coverage map with **0 missing** — but **against
the single operative pattern the solver itself selected**, not against the load-bearing requirements of the
problem. The weak solver's trap misses are where its self-chosen pattern simply *does not contain* the trap:
- **L5:** weak C′ anchored on `tragedy_of_the_commons` and reported full coverage of it — but the dropped
  traps (hidden-state estimation, estimate-uncertainty, un-meterable behavior monitoring) belong to a
  *State-Estimation* pattern it never selected. The frontier cell brought State-Estimation in and covered
  them. A coverage check over commons-governance **structurally cannot surface a state-estimation component.**
- **L2:** weak C′ chose `Dependency Exposure`, covered it, and missed the point-of-no-return trap (an
  *Irreversible-Commitment-Management* component it didn't select).
- **L3:** weak C′ chose `Tacit Knowledge Elicitation`, covered it, missed the deadline-anchoring trap.

**The binding constraint is pattern SELECTION, not coverage-enforcement.** A coverage check can only verify
the components of a pattern that is already on the table; it cannot recover a component that lives in a
pattern the solver never retrieved or chose. The frontier model out-covers the weak model because it
*selects broader / the right* operative patterns, not because it checks coverage more diligently (B′-frontier,
which ran no coverage check at all, already hit 100% traps).

**And the scoping is a genuine dilemma, not a bug.** C′ was deliberately scoped to ONE pattern to avoid
project-02 C's over-build (instantiate-everything across many patterns → sprawl that graders penalized).
But scoping to one pattern is exactly what makes C′ miss cross-pattern traps. So the coverage check is caught
between two failure modes: **scope-to-one → miss cross-pattern components; instantiate-all → over-build.**
Neither setting makes coverage-enforcement a net win. The way out is upstream: select the right *set* of
operative patterns (2–3), then cover those — which is a retrieval/composition problem, not an enforcement one.

## Verdict for the engine

Across projects 02 + B′ + 03, the explicit coverage-enforcing lift has now failed to earn its keep at the
ceiling (inert), in the high-coverage frontier regime (inert), and in the low-coverage weak regime (flat).
**Retire it as an always-on or even hard-case-routed runtime discipline.** What actually discriminates a
complete design from an incomplete one is whether the *right operative pattern(s) were selected and composed*
— i.e. retrieval/transport breadth (project 01) and multi-pattern composition — not whether a chosen
pattern's components were ticked off. This is the direct, experimentally-grounded case for §9.3: **the
catalog's leverage is in teaching a model to surface and select the apt structures (training-data
generation), not in runtime coverage-enforcement.** A weak model's gap is a selection gap, and the catalog is
exactly the supervision signal that would close it in training.

## Caveats
- **n=1/cell this rep**; the frontier-null (100%=100%) is robust, the weak-arm flat result rests on 5 cells/arm
  — directional. A rep-2 would firm the weak-arm numbers (cheap; harness ready).
- Coverage keys are catalog-derived (anti-confound) but the COVERED/MISSING audit is a single same-model
  judge per cell (function-not-vocabulary instruction); a second auditor would tighten it.
- C′ was scoped-to-one-pattern by design; an "instantiate-all" variant would trade the selection-miss failure
  for the over-build failure (project-02 C) — neither wins, but the matrix doesn't include it.
- Holistic 6-dim quality grading not run (coverage-keyed answered the question); available if a quality
  cross-check is wanted, though quality has tied across the whole program.
- Same model class for solver/auditor/deriver (different grader model deferred per Kurt). Directional (Hubbard).

## Next (if pursued)
1. **Rep-2** of the 20 cells to firm the weak-arm flat result (harness ready: `exp_project03.py`).
2. **Test the selection hypothesis directly:** give the weak solver the *frontier's selected pattern set* and
   re-run — if its coverage jumps, selection (not enforcement) is confirmed as the binding constraint.
3. **Pivot to the §9.3 training use:** generate (hard-scenario → apt-pattern-set) supervision from the catalog
   and test whether a fine-tuned/weak retriever's *selection* improves — the leverage this program keeps
   pointing at.

## Reproduce
`python scripts/exp_project03.py build|status|pending` (40-cell matrix; this rep ran the 20 rep-1 cells).
Solver strength set via the Agent `model` param (frontier=opus, weak=haiku). Coverage keys:
`coverage_keys/*.md` (catalog-derived) + `*_audit.md` (B′) + `*_audit_Cprime.md` (C′). Scenarios:
`scenarios_lowcov_DRAFT.json`. Outputs: `outputs/d*.md`. Manifest: `manifest_PRIVATE.json`.
