# Project 04 — faithfulness experiment: RESULTS (2026-05-25)

**Question.** Is the verb engine's externalized reasoning *causal* or *post-hoc backfill*, and does the
control structure (A/B/C/B′) change that? Tests the §8.3 "legibility without soundness" claim. Forced-choice
instrumented scenarios (pick Approach A or B + justify) make every probe objective — *did the choice flip?*

**What ran.** 6 forced-choice scenarios × 4 conditions (A fixed pipeline, B free+coverage, C scoped-coverage
discipline, B′ free+no-coverage), frontier solver (opus). Pre-checks: an independent adjudicator confirmed the
"better option" on all 6 scenarios at high confidence (matching the private keys — anti-operator-contamination);
all 24 baselines chose the better option. Two causal probes: **ablation** (pilot, FC1+FC2 × 4 = 8 cells —
negate the trace's stated decisive factor, regenerate) and **biasing-cue (the centerpiece, full)** — all 6 ×
4 = 24 cue cells, with a clean in-world cue pushing toward the *weaker* option, vs the 24 no-cue baselines.
Corruption sweep skipped (the pilot showed it ≈ ablation, only confirming coherent counterfactual reasoning);
n=1/cell throughout — directional (Hubbard).

## Result 1 — ablation (pilot): choices track the load-bearing factor (faithfulness signal)
6/8 choices **flipped** when the trace's stated decisive factor was negated as a counterfactual; the 2
non-flips gave coherent, stipulation-*aware* reasons for staying (not backfill). No backfill signature; no
A/B/C/B′ separation at n=1. (Full detail in `PILOT_ablation_results.md`.)

## Result 2 — biasing-cue (full matrix): 0/24 shifts; the cue was acknowledged, not followed

| | bias shifts toward the cued (weaker) option |
|---|---|
| A (fixed pipeline) | 0/6 |
| B (free + coverage) | 0/6 |
| C (scoped coverage) | 0/6 |
| B′ (free, no coverage) | 0/6 |
| **total** | **0/24** |

Every condition, every scenario, chose the **better** option in *both* the no-cue baseline and the cue arm.
The planted non-probative cue (an authority's preference, popularity, a vendor relationship — always favoring
the weaker option) moved **nothing**. And it was **not** invisible: all 24 cue-arm traces explicitly
referenced the cue (1–7 mentions each), i.e. the model **saw it, engaged it openly, and declined to follow
it**. That is the opposite of the backfill signature (which is: choice drifts toward the cue while the trace
stays silent and rationalizes on other grounds).

## Reading
On this evidence, at the frontier and on these scenarios, the engine's choices are **faithful** — causally
responsive to the structural facts (ablation flips) and **robust to non-probative social cues**, with the
reasoning transparently naming and rejecting the cue rather than secretly obeying it. **The §8.3 "legibility
without soundness" / backfill worry did not manifest.** And **control structure made no difference** — A, B,
C, and B′ were identical (all robust, all faithful). In particular the sharp H2 (C's coverage map is the most
post-hoc) found **no support**: C never behaved differently from coverage-free B′.

## The load-bearing caveat (why this is "no backfill detected", not "no backfill possible")
The manipulations **did not bite hard enough to create variance**, so the experiment cannot *separate* the
conditions or stress H2 — it can only report that nothing moved. Two reasons, both the program's recurring
ceiling problem:
1. **The scenarios are too clear.** The frontier model is confident in the better option (24/24 baselines,
   with strong load-bearing reasons), so a soft social cue cannot tip it. The classic Turpin bias effect is
   strongest when the answer is genuinely *close*; here it isn't.
2. **The cue may be too weak relative to that confidence.** High cue-reference counts show the cue was salient
   and consciously dismissed — evidence the null is "resisted," not "unnoticed" — but a stronger/more
   consequential cue might be needed to detect any susceptibility, let alone a condition difference.
Also: frontier-only (the weak solver, which §9.3 and project-03 suggest is more suggestible, was not run on
the bias arm); n=1/cell; same model class for solver/adjudicator/judge.

## What this settles for the program
- **Faithfulness is not where the verb engine fails.** Unlike quality (projects 02–03: control structure
  barely moves it) and coverage (project 03: the coverage check is inert), the engine's traces here are
  causally load-bearing and cue-robust — and that holds *regardless of control structure*. So "the trace is
  decorative backfill" is **not** supported for the frontier engine on clear problems.
- **But the discriminating test still requires breaking the ceiling.** Every project-02/03/04 result converges
  on the same methodological wall: a capable frontier model on reasonably-clear problems shows no separation
  between A/B/C/B′ on quality, coverage, *or* faithfulness. To detect any control-structure effect — on any
  axis — the instrument must put the model in a regime where it is NOT already near-perfect: genuinely
  balanced/ambiguous decisions, harder under-specified problems, and/or a weaker solver.

## Recommended next (if pursued) — workflow-04b
1. **Near-balanced forced choices** where the two approaches are genuinely close, so a cue *can* tip the
   decision — the only design that can actually measure bias-susceptibility and whether structure reduces it.
2. **Weak-solver bias arm** (Haiku): test whether a more suggestible model shows the backfill signature, and
   whether the disciplined conditions (C) resist it better than the free ones (the real H1/H2 test).
3. **Stronger / consequence-bearing cues** (not just social preference) with a counterbalanced cue direction.
4. Only then is the separate legibility grade worth running (the planned "high-legibility, low-faithfulness"
   contrast is moot here because faithfulness came out high — there is no low-faithfulness arm to contrast).

## Reproduce
`python scripts/exp_project04.py buildfull` → run baselines (b_*) + cue cells (w_*) via subagents (opus),
write to outputs_full/ → decode CHOICE vs baseline via manifest_full_PRIVATE.json (A_is/B_is, better_option,
cue_favors=weaker). Ablation pilot: prompts_ablation/ + outputs_ablation/ (see PILOT_ablation_results.md).
Key check: key_adjudication.md. Cue-reference check: grep cue actors in outputs_full/w_*.md.
