{"schema_version":1,"experiment_id":"eoa_inverse_innovation_exp09_archetype_breadth150_20260804","cell_id":"prediction_error_learning_calibration__computer_science","arm":"BREADTH_PROBE_ONE_SHOT","candidate_id":"prediction_error_learning_calibration__computer_science__P1","proposal_index":1,"version":0,"title":"Residual-Guided CI Test Selection for Flaky-Test Budget Traps","problem":"An adaptive continuous-integration test selector learns which tests to run from raw pass and failure outcomes. Repeated failures from already-known flaky tests keep increasing their selection priority, while an unexpected failure in a test predicted to be safe is treated like any other failure. Under a fixed execution budget, the selector can spend capacity rediscovering expected noise instead of learning which change–test relationships reveal previously mispredicted regressions.","actors":["Software maintainers","CI platform owner","Adaptive test-selection service","Developers submitting code changes","Release engineer responsible for regression controls"],"observable_state":"For each code change, the selector produces test priorities but does not retain a pre-run failure probability for each change–test pair. Selection weights rise after raw failures, known flaky tests consume a recurring share of the execution budget, and full-suite or shadow runs reveal failures among low-priority tests that the selector had treated as low risk.","consequence":"CI capacity is allocated toward expected flaky failures while informative unexpected failures receive insufficiently differentiated updates, potentially delaying regression detection or forcing longer test runs.","affected_objective":"Detect behavior-changing regressions within a bounded CI execution budget while maintaining calibrated change–test risk estimates.","intervention":"Place a prediction-error calibration layer between test outcomes and updates to the selection policy. Before an eligible full-suite or shadow run, record each test's predicted failure probability for the current change. Compare the binary outcome with that prediction, preserve the signed residual, filter known flakiness and run-level infrastructure faults, trace the residual to changed files and test dependencies, and update selection weights in proportion to residual reliability. An unexpected failure receives more learning weight than an already-expected flaky failure; an unexpected pass can reduce an overstated risk estimate. No production gating changes occur until offline and shadow evidence passes predefined safeguards.","structural_mapping":[{"archetype_element":"Prior Prediction Record","domain_realization":"A timestamped, immutable failure-probability estimate for every evaluated change–test pair, emitted before test results are available."},{"archetype_element":"Value Reference Frame","domain_realization":"Regression-detection information under a fixed test-execution budget, with infrastructure failures and known non-product flakiness represented separately from product regressions."},{"archetype_element":"Received Outcome Record","domain_realization":"The test's pass, product-failure, flaky-failure, or infrastructure-failure result, accompanied by runtime and execution context."},{"archetype_element":"Signed Error Signal","domain_realization":"For a product-failure indicator y and prior probability p, the residual y-p; positive residuals represent more failure than expected and negative residuals represent more passing than expected."},{"archetype_element":"Credit Assignment Window","domain_realization":"A trace linking each reliable residual to the submitted change, touched files, dependency edges, test identity, runner image, and recent test edits."},{"archetype_element":"Noise and Volatility Filter","domain_realization":"Suppression or reduced gain for rerun-confirmed flakiness, shared runner outages, nondeterministic fixtures, sparse test histories, and abrupt environment changes."},{"archetype_element":"Learning Gain Rule","domain_realization":"A bounded update rate determined by prediction confidence, rerun evidence, history length, and context stability rather than by failure magnitude alone."},{"archetype_element":"Shortcut Learning Guard","domain_realization":"Holdouts that break incidental correlations such as directory name, change author, runner type, or historically flaky test identity while preserving meaningful code–test dependencies."},{"archetype_element":"Update Target","domain_realization":"The risk score used to rank tests for a change, not the merge decision, test assertion, or definition of correctness."}],"mechanism_mapping":[{"mechanism_slug":"prediction_outcome_delta_log","role":"Stores the pre-result probability, observed outcome, signed residual, context, credit decision, and applied update for each evaluated change–test pair.","counterfactual_removal":"Without the log, expectations can be reconstructed after the outcome, making it impossible to distinguish genuine surprise from hindsight-adjusted risk."},{"mechanism_slug":"credit_assignment_trace","role":"Determines whether a residual should update a code–test relationship, a flakiness estimate, or an infrastructure-state estimate.","counterfactual_removal":"Without the trace, a genuine surprise can increase the wrong test or feature weight merely because it was temporally adjacent to the change."},{"mechanism_slug":"shortcut_probe_holdout_set","role":"Evaluates whether residual-weighted updates learn stable code–test relationships rather than convenient metadata proxies.","counterfactual_removal":"Without the holdout, improved replay performance could result from author, directory, runner, or test-identity shortcuts that fail under changed conditions."},{"mechanism_slug":"calibration_curve_review","role":"Compares predicted failure probabilities with observed outcome frequencies before and after residual-guided updates.","counterfactual_removal":"Without calibration review, the selector could improve ranking on one replay while becoming systematically overconfident or underconfident."}],"causal_chain":["The selector records a failure probability before observing each eligible test result.","The test produces a context-labeled outcome that remains distinct from the prediction.","The system computes the signed prediction residual instead of treating every failure as an equal teaching event.","Flakiness, infrastructure faults, sparse histories, and context changes reduce or suppress unreliable residuals.","A credit trace assigns each remaining residual to the relevant code–test relation rather than the nearest visible cue.","A bounded learning-gain rule changes the test-priority model more for reliable unexpected outcomes and less for expected outcomes.","Shortcut holdouts and calibration review test whether the update improved stable risk estimation rather than proxy exploitation.","Only evidence-passing model changes become candidates for a separately authorized production test-selection pilot."],"baseline":"The current selector updates priorities from raw pass/failure counts, recent failure status, or empirical failure frequency without requiring an immutable pre-outcome probability or using the signed prediction residual as the teaching signal.","nearest_rivals":["Flaky-test detection or quarantine, which classifies unreliable tests but does not govern every selection update by the gap between prior risk and received outcome.","Static test-impact analysis, which maps changed code to tests but does not learn from signed expectation violations.","Failure-frequency ranking, which prioritizes tests with many failures regardless of whether those failures were already predicted.","Periodic model retraining on pass/failure labels, which can improve a selector without preserving pre-outcome predictions, residual sign, or event-level credit decisions."],"remaining_contrastive_claim":"The candidate's distinguishing commitment is that a test outcome may change selection weights only through a preserved pre-outcome prediction, signed residual, reliability filter, and explicit credit assignment. Merely using failure labels, flakiness scores, or learned risk rankings does not satisfy that commitment.","authority_safety":{"decision_authority":"The CI platform owner may authorize offline replay and a non-gating shadow evaluation; the repository's designated release-control owner retains authority over required tests, merge gates, and any production rollout.","authorized_first_step":"Export a bounded set of existing full-suite records, generate immutable out-of-sample predictions in replay, and compare residual-guided updates with the existing update rule without changing live test execution or merge decisions.","excluded_actions":["Removing required tests","Quarantining or disabling tests","Changing merge or release gates","Using developer identity as an update feature","Automatically rewriting test assertions","Deploying the residual-guided selector to production","Treating failed tests as proof of developer fault"],"halt_rollback":"Stop the evaluation and discard the candidate model if records cannot separate product failures from infrastructure or flaky failures, if prediction timestamps cannot establish that estimates preceded outcomes, if shortcut probes expose identity or environment proxies, or if the replay violates repository data controls. Because the first step is offline, rollback consists of deleting the derived model artifacts while retaining only authorized audit summaries."},"negative_tests":{"strongest_counterevidence":"On temporally separated full-suite runs, the raw-outcome baseline matches or exceeds the residual-guided rule on calibration and budget-constrained regression detection, while the proposed rule adds instability or repeatedly assigns residuals to the wrong code–test relationships.","problem_falsifier":"The problem is falsified if immutable logs show that the selector already updates from pre-outcome probabilities and signed residuals, known flaky tests do not consume disproportionate budget, and low-priority unexpected product failures are not systematically underweighted.","intervention_falsifier":"The intervention is falsified if, after reliable outcome filtering and credit tracing, residual magnitude or sign provides no reproducible guidance for updating change–test risk on a temporally held-out replay.","risks":["Developers or model owners could sandbag predictions to manufacture positive surprise.","Sparse binary outcomes can create high-variance residuals and unstable updates.","Flaky-test labels can conceal genuine intermittent product regressions.","Credit assignment can blame the submitted change when the cause is shared infrastructure or latent repository state.","The selector can learn shortcuts from author, directory, runner, or test-identity metadata.","Optimizing calibrated failure probability may still neglect severity, coverage, or long-horizon release risk.","Logging change–test traces may expose sensitive repository or personnel metadata.","Added prediction and audit steps may increase CI complexity and operational cost."]},"next_evidence_step":"Using no more than 200 recent full-suite runs from one repository, conduct an offline chronological replay. Fit or recover predictions using only information available before each run; compare the existing update rule with the residual-guided rule under the same simulated execution budget; inspect calibration, product-regression discovery, flaky-test capacity share, and shortcut-probe results. Predefine outcome labels and halt criteria, make no live CI changes, and treat the result only as evidence for or against a later shadow pilot.","prior_art_status":"UNSEARCHED","diversity_from_prior_proposals":"One-shot candidate created without inspecting prior proposals. Its domain realization is specifically an adaptive CI test selector whose learning signal is recalibrated at the change–test-pair level.","revision_record":{"parent_version":null,"progress_targets_addressed":["Initial one-shot formulation from the supplied archetype and computer-science domain card."],"conceptual_changes":["Instantiated prediction-error learning as residual-guided adaptation of CI test-selection risk rather than raw failure reinforcement."],"operational_changes":["Bounded the first step to offline chronological replay with no production gating changes."],"evidence_changes":["No external evidence or prior-art search was used."],"claim_changes":["Makes no claim of novelty, prevalence, demand, or effect size."]}}