title: Experiment 14: Applicability Graph retrieval and verification¶
Experiment 14: Applicability Graph retrieval and verification¶
Part of Inverse Innovation with the Encyclopedia of Abstractions · Experiment 14: Applicability Graph retrieval and verification · Last revised August 2026
Result¶
Frozen verdict: VERIFICATION ONLY.
The outcome-blind audit retained 114/120 cases: 77 positive scenarios and 37 one-literal near misses spanning 40 archetypes. The frozen information gate passed.
On positive cases, the solution-oriented index retrieved the target in its top
five for 8/77
(10.4%) cases. The diagnostic maximum-hit
search did so for 2/77
(2.6%), a difference of
-7.8 percentage points. There were
0 diagnostic-only and 6 solution-only top-five
successes (exact paired p = 0.03125). The frozen diagnostic-retrieval
gate did not pass.
The blinded DNF verifier accepted the target for 77/77 positives (100.0%) and correctly rejected it for 32/37 near misses (86.5%). Balanced accuracy was 93.2%. The frozen DNF-verification gate passed.
This separation is the central finding. The Applicability Graph was highly effective when asked whether a supplied archetype fit a case, but the current diagnostic search did not reliably place that archetype into the candidate set. The graph is therefore a strong verification layer in this internal benchmark, not yet a working end-to-end problem-to-archetype retrieval system.
Retrieval detail¶
| Arm | Recall@1 | Recall@5 | Recall@10 | Recall@50 | MRR |
|---|---|---|---|---|---|
| Solution-oriented index | 3.9% | 10.4% | 14.3% | 28.6% | 0.075 |
| Diagnostic maximum-hit | 0.0% | 2.6% | 5.2% | 16.9% | 0.013 |
| Diagnostic top-12 + DNF reranking | 5.2% | 5.2% | 5.2% | n/a | 0.052 |
The DNF reranker can only reorder the original diagnostic top 12. Its result is therefore secondary and should not be compared as though it searched the full catalog independently.
The target was present in the diagnostic top 12 for only 4/77 positives (5.2%). The shadow-target rule allowed target-route fidelity to be measured when retrieval missed, but did not insert that target into the reported reranking pool. This is why excellent verification cannot compensate for the retrieval bottleneck.
Outcome-blind case audit¶
Two independent blinded audit passes agreed on case validity for 114/120 (95.0%) cases. Eleven cases had at least one material-field disagreement and received fresh adjudication. Six cases were excluded before retrieval, leaving 38 direct positives, 39 transfer positives, and 37 near misses across all 40 sampled archetypes.
One exhausted audit batch contained complete substantive outputs whose two
positive-case validity flags conflicted with the frozen positive rule only
because an alternative route was also satisfied. The deterministic recovery in
AMENDMENT_001.md changed only those two flags and preserves
every original response.
Exploratory failure localization¶
This was not a frozen stratum-level endpoint, but the five near-miss false
acceptances were concentrated: four occurred among the eight eligible
MULTI_PARTIAL_OPEN near misses and one among seven ALTERNATIVE_ROUTES near
misses. None occurred in the other 22 eligible near misses. This suggests that
open or incompletely grounded route conditions deserve focused curation before
the graph is treated as a high-specificity rule base. The pattern is diagnostic
only and needs replication.
What this establishes¶
This experiment tests two different failure points. The retrieval comparison asks whether diagnostic-language documents are a better entry point than solution descriptions for problems generated from the graph. The DNF test asks whether explicit conjunctions prevent a highly similar but structurally incomplete case from being accepted.
Because the cases were constructed from the graph and all semantic judgments came from the same broad model family used elsewhere in the project, even a positive result establishes internal operational fidelity, not independent truth. It does not show that the predicates are complete or causally correct in real deployments. The appropriate next validation would use naturally occurring cases collected independently of the graph, ideally with domain-expert review.
The immediate engineering implication is narrower: preserve the DNF verifier, but redesign candidate generation. Plausible next approaches include retrieving and aggregating at the literal level, using route-aware multi-vector scoring instead of the strongest single diagnostic hit, and testing a learned or model-assisted query-to-route bridge. Those approaches should be compared on a fresh frozen case set so this benchmark does not become a tuning target.
Resource use¶
The canonical run completed 215 model calls with no failed final calls: case construction, duplicate blinded case auditing plus adjudication, and 114 blinded DNF verifications. It recorded 3,584,592 input tokens (1,880,832 cached), 912,450 output tokens, and 236,247 reasoning-output tokens. Summed call time was 5.14 hours; three-way concurrency reduced the first-to-last telemetry span to 1.79 hours. No external web search was used.
Reproducibility¶
The design, sample, prompts, schemas, case-audit decisions, retrieval outputs, opaque verification inputs, model responses, telemetry, private join keys, and analysis are retained in this directory. The case corpus and eligibility were sealed before retrieval; retrieval and verifier inputs were sealed before DNF measurement. The deterministic validator verified all 68 frozen design sources, 40 sampled archetypes, 120 constructed cases, 114 eligible retrieval cases, and 114 verifier outputs without error. No external web search was used.