title: Experiment 15: Route-aware diagnostic candidate retrieval¶
Experiment 15: Route-aware diagnostic candidate retrieval¶
Part of Inverse Innovation with the Encyclopedia of Abstractions · Experiment 15: Route-aware diagnostic candidate retrieval · Last revised August 2026
Result¶
Frozen verdict: NOT SUPPORTIVE.
The outcome-blind audit retained 115/120 cases: 78 positive scenarios and 37 one-literal near misses spanning 40 fresh archetypes. None of the 40 Experiment 14 archetypes was eligible for E15 sampling. The information gate passed.
The prespecified primary route-aware arm retrieved the target in its top five for 1/78 (1.3%) positive cases, compared with 6/78 (7.7%) for the solution index and 12/78 (15.4%) for diagnostic maximum-hit.
Against the solution index, the difference was
-6.4 percentage points,
with 0 primary-only and
5 solution-only successes
(p = 0.0625).
Against diagnostic maximum-hit, the difference was
-14.1 points, with
1 primary-only and
12 diagnostic-only successes
(p = 0.003418).
Retrieval detail¶
| Arm | Recall@1 | Recall@5 | Recall@10 | Recall@50 | MRR |
|---|---|---|---|---|---|
| Solution-oriented index | 2.6% | 7.7% | 10.3% | 29.5% | 0.058 |
| Diagnostic maximum-hit | 5.1% | 15.4% | 19.2% | 28.2% | 0.089 |
| Full-scenario route minimum | 0.0% | 1.3% | 1.3% | 10.3% | 0.013 |
| Full-scenario route mean | 0.0% | 3.8% | 10.3% | 21.8% | 0.026 |
| Sentence-aware route minimum | 0.0% | 1.3% | 1.3% | 19.2% | 0.019 |
| Sentence-aware lower-half mean (primary) | 0.0% | 1.3% | 1.3% | 24.4% | 0.021 |
| Sentence-aware route mean | 0.0% | 2.6% | 9.0% | 42.3% | 0.035 |
Only SENTENCE_LOWER_HALF was confirmatory. The other route-aware formulas
were frozen in advance but remain sensitivity analyses; a favorable secondary
arm cannot rescue a failed primary verdict.
Interpretation¶
E15 isolates candidate generation. Unlike E14, it does not repeat the model-based DNF verifier, which E14 already showed could distinguish complete routes from most one-literal near misses. Here each condition receives a semantic score, conditions are combined within a route, and alternative routes are combined by maximum. This directly tests whether preserving the graph's logical shape improves the initial candidate list.
The cases are nevertheless derived from the Applicability Graph. A positive result is evidence of internal retrieval fidelity on new graph-generated targets, not evidence that the graph's predicates are complete, causally valid, or accepted by experts in natural settings. A negative result would reject these frozen embedding aggregators, not the graph or inverse innovation as a whole.
Post-hoc failure localization¶
An outcome-aware diagnostic performed after the frozen verdict identifies two
concrete implementation failures. It is explanatory and cannot change the
NOT_SUPPORTIVE result.
First, 378/390 (96.9%) of the primary arm's top-five slots on positive cases were occupied by an archetype whose selected best route contained only one literal. Minimum and lower-half aggregation impose a strong route-length penalty, while maximizing across alternative routes gives some archetypes more opportunities to obtain a high score. The raw cosine scores were therefore not calibrated across route lengths or route counts.
Second, cosine similarity did not encode the sign of condition evidence. In the 36 samples where both a transfer positive and its matched one-literal near miss survived audit, the primary arm ranked the contradicted near miss better in 22 pairs, ranked the positive better in 14, and had a median near-miss-minus-positive rank difference of -28.5. Explicit contradiction often repeats the same concepts as the predicate it negates, so it can be highly similar to that predicate even though the E14 entailment-style verifier would reject it.
The E15 failure therefore does not show that DNF structure is useless. It shows that arithmetic over uncalibrated condition-to-text cosine similarities is not an adequate implementation of DNF satisfaction. Signed condition evidence and calibration for route length, route count, and sentence-search multiplicity are hypotheses for a future design, not findings rescued from E15. See the post-hoc diagnostic and its machine-readable output.
The baseline ordering also reversed across the two controlled benchmarks: diagnostic maximum-hit lost to the solution index in E14 but led it in E15. Descriptively pooling the fresh samples yields the same Recall@5 count for both baselines—14/155 positives—with nine exclusive successes apiece. This was not a prespecified pooled endpoint, but it reinforces the narrower conclusion that neither baseline is yet a reliable candidate generator.
Reproducibility¶
The secret-seeded fresh sample, exclusion list, design, prompts, schemas, model outputs, two-pass audit, adjudications, eligibility seal, all seven retrieval outputs, private joins, telemetry, and case-level analysis are retained. No public-web search or model-based retrieval reranking was used.