# Project 02 — Verb-engine A/B/C control-structure ablation: RESULTS (2026-05-25)

**Definition-of-done deliverable.** Built a verb library + a structured-state planner, and ran the clean
A/B/C control-structure ablation (vary only how verbs are sequenced; hold catalog, verb set, and model class
fixed). 5 scenarios × 3 conditions × 2 reps = 30 target-blind designs, each graded by 2 independent blinded
graders on the six-dimension rubric (/60); the 10 condition-C designs additionally passed a separate
independent coverage-verifier. **Run against the upgraded project-01 semantic/facet/neighborhood retrieval**
(the earlier lexical-search run was discarded as invalid — Kurt's catch).

Conditions: **A** = fixed nine-step pipeline (baseline); **B** = free planner (choose/repeat/skip/stop, no
enforced checks); **C** = flexibility-within-discipline (free planner, but the externalization
post-conditions — referential integrity, lift witness-check, and the lift **coverage check**, plus the gate —
must hold before any recommendation; the coverage check enforced by a separate verifier pass).

## Headline

**Once retrieval is good and the solver is a capable frontier model, control structure barely moves
aggregate design quality, and the §9.2 fear that a free planner would erode the discipline did not
materialize.** Pooled means are a three-way tie within noise — **A 50.35, B 50.70, C 49.80 /60** — while
inter-grader agreement is tight (mean |Δ| ≤ 1.2/60), so the near-equality is real, not grader noise. The
pre-registered ordering **C ≥ A > B is not supported**; the observed ordering is closer to **B ≳ A ≈ C**, with
**B the most robust and cheapest** controller and **C the highest-variance** one.

## The numbers

| Scenario | maturity / obliqueness / transport | A | B | C | C−A | B−A |
|---|---|---:|---:|---:|---:|---:|
| S1 fishery | strong / low / near | **52.75** | 51.25 | 46.25 | **−6.50** | −1.50 |
| S2 service-reliability | strong / low / near | 49.00 | 50.00 | 49.75 | +0.75 | +1.00 |
| S3 autonomous-agent deletion | mixed / high / distant | 47.25 | 49.75 | **51.25** | **+4.00** | +2.50 |
| S4 skill-exchange | mixed / high / moderate | 50.25 | 50.25 | 48.75 | −1.50 | 0.00 |
| S5 oss-burnout | strong / high / near | 52.50 | 52.25 | 53.00 | +0.50 | −0.25 |
| **Pooled (n=20/cond)** | | **50.35** | **50.70** | **49.80** | −0.55 | +0.35 |
| pooled SD | | 2.52 | **1.73** | 2.98 | | |

Catalog-aligned dims {coherence, failure-modes, equity} vs neutral dims {implementability, mechanism,
threshold} were ~equal across conditions (pooled CA: A 25.35 / B 25.30 / C 24.80; NT: A 25.0 / B 25.40 /
C 25.0), so the rubric-overlap confound does not drive the result. Inter-grader mean |Δ|: S1 0.83, S2 0.83,
S3 1.17, S4 0.50, S5 0.83 (/60). Top-pick condition by grader: S1 A/A, S2 C/C, S3 C/C, S4 A/B, S5 C/A.

## What actually happened (four findings)

**1. The free planner did NOT erode the discipline — it adopted it. (Refutes the §9.2 worry, for this regime.)**
All **10/10 condition-B cells voluntarily ran the coverage check** even though nothing required it, across
both controls and oblique scenarios. The load-bearing caution was that flexibility would skip the helpful
step; instead, with the verbs in its toolkit, a capable planner chose the coverage check on its own. B was
the **most robust** condition (lowest variance, SD 1.73; never the worst condition in any single scenario)
and the **cheapest** (≈79k tokens/cell, ~5–9 catalog ops vs A's fixed nine steps at ~107k/13–24 ops). So the
discipline's value is real, but a free planner already captures it — the *enforced* part (C over B) bought
nothing on the mean (C−B = −0.9) and added variance and cost.

**2. Mandatory enforcement (C) is double-edged, and its downside is OVER-BUILDING.** C produced both the best
single scenario (S3, +4.0 over A) and the worst (S1, −6.5). The deficits trace to a concrete mechanism, not
noise: C dutifully instantiated *many* components across *multiple* retrieved patterns — the verifier logs
show C cells carrying 24–38 instantiated components (S1 folded commons_governance + tipping_point_prevention
+ distributional patterns; S4 folded incentive_compatible_rule_design + balancing_loop + commons) — and the
`correctly_declined` back-off was under-used. The result is design *sprawl* that graders penalized on
coherence and implementability. This is the **over-lift failure (P3 `symmetry`, P4 high-coverage regime)
reappearing at the design level**: when the baseline already covers, forcing the coverage check to instantiate
every component of every pattern degrades the design. The coverage map's escape hatch (`correctly_declined`,
back-off) is exactly the guard against this, and C did not lean on it hard enough.

**3. The one clear C win is the regime the coverage account predicts.** S3 — oblique + distant + a *latent
framed* operative pattern (independent review at the irreversible moment + durable record + consequence-scaled
authority) — is precisely where leaving components latent costs the most (P2 distance, P4 burial). There A
scored *lowest* (its fixed order surfaced the frame via facet search but under-instantiated it), C *highest*,
and both graders' top pick was a C cell. So the coverage discipline has **real but narrow** value: it pays on
the hardest oblique/distant transport with a genuine coverage gap, and is washed out or negative elsewhere.

**4. The predicted obliqueness→C-advantage interaction did NOT hold cleanly.** The clean S1(−6.5)/S3(+4.0)
pilot looked like a strong interaction, but it did not replicate: S4 is oblique yet C *lost* (−1.5, via
over-build), S5 is oblique yet flat (+0.5), S2 is a control yet C did *not* hurt (+0.75). The real predictor
of C's sign is **genuine coverage gap vs over-build**, not obliqueness per se. Honest reading: only S3 shows a
clear C advantage; only S1 a clear C deficit; the other three are within the n=2 noise band.

**Retrieval substitutability (P1) confirmed live.** With semantic + facet search on, even the fixed pipeline
(A) surfaced latent framed primes (S3-A's facet query surfaced `separation_of_powers`, `due_process`,
`reversibility_horizon`). The salience the lift would supply is partly supplied by good retrieval, which
shrinks A's disadvantage and is part of why the conditions tie.

## What this implies for the engine

- **Flexibility-within-discipline is the right shape, but the discipline should be the planner's *adaptive*
  judgment to run AND SCOPE the coverage check — not a mandate to fully instantiate every component of every
  retrieved pattern.** Concretely: keep the coverage-enforcing lift as a planner-invoked verb (B's behavior),
  scope it to the *single* operative pattern, and make `correctly_declined` / back-off first-class so the
  over-lift guard actually fires. A always-on, instantiate-everything mandate (C as run) buys nothing on
  average and risks sprawl.
- **The external coverage-verifier is a safety net, not an always-on gate.** All 10 C cells passed it on the
  first try with no revision — the solver's self-discipline already satisfied it — so its *binding* value was
  never exercised here, and it roughly doubled C's cost. Recommend reserving it for flagged
  high-stakes / low-coverage cases (oblique + distant, e.g. an S3-type scenario) rather than every run.
- **B (free planner) is the default to beat.** Best robustness, lowest cost, never worst. The verb engine's
  value over the fixed pipeline is *failure-tolerance and efficiency* (it skips steps the problem doesn't need
  and routes to facet retrieval when oblique), not a quality jump.

## Caveats (carry these)

- **n = 2 reps/cell; 2 graders.** Pooled differences < 1/60 are within noise; only |C−A| on S1 (−6.5) and S3
  (+4.0) exceed the rep/grader noise band. Directional, not proof (Hubbard).
- **Deep confound:** designer = solver = verifier = grader = same model class. Blinding, pre-registration, and
  the within-design comparison mitigate but do not remove it.
- **B's self-discipline may be partly a prompt artifact:** the shared verb library foregrounds the
  coverage-enforcing lift and the STATE schema lists `coverage_map`, which makes the check salient to B. The
  cleanest follow-up is a **B′** whose prompt does not foreground coverage — does the planner still run it?
- **Ceiling:** scores cluster 45–55/60; the rubric/scenarios don't fully separate strong frontier designs
  (same ceiling as Protocols 1–4). A harder, more under-specified scenario set would resolve the conditions
  better — and would widen the coverage gap C is built to close.

## Recommended next tests (one variable at a time)
1. **B′ (coverage not foregrounded)** — isolates whether B's 10/10 self-discipline is genuine policy or
   prompt-primed. The single most informative follow-up.
2. **C′ (adaptive/scoped coverage)** — coverage check scoped to one operative pattern with aggressive
   `correctly_declined`; test whether it keeps the S3 win while removing the S1/S4 over-build deficit.
3. **Harder scenarios** (deliberately under-specified / multi-gap) to break the ceiling and create real
   coverage gaps — the regime where mandatory discipline should finally separate from free.
4. **Verifier-as-safety-net** fired only on flagged oblique+distant cases; measure whether it ever forces a
   value-adding revision (it did not here).

---

## ADDENDUM — B′ follow-up: was B's self-discipline genuine, and does the coverage check matter? (2026-05-25)

The headline rested on B (free planner) *voluntarily* running the coverage check 10/10. The obvious confound:
B's shared verb library foregrounds the coverage-enforcing lift and lists `coverage_map` in STATE. **B′**
re-runs the free planner with that scaffolding removed — `lift` described as plain abstraction, no
`coverage_map` in STATE, the word "coverage" absent from all 10 prompts (verified) — same 5 scenarios × 2
reps, same retrieval/model. A separate **blind auditor** then read 4 mixed transcripts per scenario (2 B as a
positive control + 2 B′) and classified each for an explicit, enumerated, component-by-component coverage
check (FULL / PARTIAL / NONE). B vs B′ recommendations were graded head-to-head, blinded.

**Result 1 — B's self-discipline was largely a prompt artifact.** Auditor (which correctly flagged **10/10 B
cells as FULL** — the positive control held): **B′ ran 0/10 FULL coverage checks (7 NONE, 3 PARTIAL).** With
the coverage verb not foregrounded, the free planner falls back to plain lift + a failure-mode gate and does
**not** pose the operative pattern's components and check each. So the §9.2 erosion concern is **real after
all**: a genuinely unscaffolded free planner erodes the coverage discipline. B kept it only because its
toolkit advertised it. (My earlier "flexibility preserved the discipline" read was wrong — it was the
*prompt*, not the flexibility, that preserved it.)

**Result 2 — but skipping the coverage check did not cost quality.** Head-to-head (single blinded grader/
scenario, n=2): **B′ 50.9 ≈ B 49.6 /60** (B−B′ = −1.3, i.e. B′ marginally *higher*). Per scenario B−B′:
S1 +0.5, S2 −4.5, S3 +0.5, S4 −2.5, S5 −0.5. **Crucially, on S3 — the one oblique/distant scenario where the
coverage discipline produced C's only clear win in the main run — B (ran the check) and B′ (skipped it) tied
(49.5 vs 49.0).** Dropping the explicit coverage check changed nothing the graders could see.

**What this means (and the honest deflation).** In this regime — project-01's upgraded semantic/facet
retrieval + a frontier model + these scenarios — **the explicit coverage-enforcing lift is largely inert on
output quality**: it is skipped when not prompted (B′), forcing it adds variance/over-build (C), and prompting
it (B) yields designs no better than not prompting it (B′). A capable model evidently covers the operative
components *implicitly*; the explicit component-by-component ritual is neither spontaneously performed nor,
when performed, quality-additive here. This is the engine-level echo of P1's substitutability: **good
retrieval raises baseline coverage so high that the coverage-check has no headroom** (P4: "lift is neutral
when coverage is already complete"). Project-01's retrieval upgrade appears to have substituted for
project-02's coverage discipline.

**Where the discipline could still earn its keep (not refuted, just not exercised here):** genuinely
*low-coverage* regimes — much harder / under-specified problems, weaker solver models, or thinner retrieval —
which P2/P4 identify as where the coverage gap (and thus the lift's value) is large. This scenario set is
ceiling-bound (45–58/60) and, with strong retrieval, high-baseline-coverage throughout, so it cannot test the
low-coverage regime where the check was predicted to matter. **That is now the sharpest open question.**

**Caveats:** single grader for the head-to-head; n=2/cell; same model class throughout; the auditor measured
the *explicit ritual*, so B′ may cover components implicitly (which is precisely why quality holds). Directional.

**Revised engine takeaway.** Don't ship an always-on coverage mandate (C: variance/over-build) or assume a
free planner will self-discipline (B′: it won't). The explicit coverage verb is worth keeping in the toolkit
as a *prompted disposition for the hard, low-coverage cases* (where P2/P4 say it pays), routed in by a
difficulty/coverage trigger — not run on every problem, where it is inert or harmful. Next: re-test the
coverage check on a deliberately **under-specified / low-coverage** scenario set (and/or a weaker solver) that
breaks the ceiling — the only regime left where it is predicted to matter.

Artifacts: `bprime/` (prompts, outputs, manifest), `bprime/audit/` (blinded audit-sets + `audit_mapping_PRIVATE.json`
+ per-scenario classifications), `bprime/gradesets/` (head-to-head sets + grades + `label_mapping_PRIVATE.json`);
harness `scripts/exp_project02_bprime.py` build/status/gradesets/auditsets.

## Reproduce
`python scripts/exp_project02.py build` → run solvers (subagents write outputs/) → for C cells run the
verifier (verifier_prompt.md) → `python scripts/exp_project02.py gradesets` (idempotent; per-scenario seeded)
→ 2 graders/grade-set (grader_prompt.md) → `python scripts/exp_project02_score.py` (unblind + per-condition
means + catalog/neutral split + inter-grader agreement). Artifacts: DESIGN_PROPOSAL.md, PREREGISTRATION.md,
verbs.md, scenarios.json, prompts/ + outputs/ (30 cells + 10 *.verify.md), gradesets/ (5 sets + 10 grader
files + label_mapping_PRIVATE.json), manifest_PRIVATE.json.
