{"schema_version":1,"experiment_id":"eoa_inverse_innovation_exp09_archetype_breadth150_20260804","research_id":"eoa_inverse_innovation_exp09_light_prior_art_20260804","cell_id":"prediction_error_learning_calibration__computer_science","search_lanes":{"direct_problem_and_intervention":{"queries":["adaptive regression test selection flaky tests predicted failure probability residual learning","CI test selection expected versus unexpected failures residual learning","\"signed residual\" \"test selection\" software","\"prediction error\" \"test case prioritization\""],"source_ids":["SRC1","SRC2"],"no_result_note":"No source found describing the proposal's complete preserved-prediction, signed-residual, reliability-filter, and explicit-credit update chain."},"synonyms_and_historical_terms":{"queries":["regression test prioritization fault detection history based flaky tests","test case prioritization Bayesian surprise calibration","online learning regression test prioritization test outcome logistic regression","reinforcement learning test case prioritization flaky tests failure"],"source_ids":["SRC1","SRC2"],"no_result_note":null},"products_practices_and_standards":{"queries":["Launchable predictive test selection flaky tests","Launchable smart subset drop flaky tests failure probability","Facebook predictive test selection probabilistic model regression tests","Google test selection flaky tests CI"],"source_ids":["SRC1","SRC3"],"no_result_note":null},"component_combination":{"queries":["test selection predicted failure probability calibration Brier residual","regression test prioritization change test dependency flaky test filtering machine learning","continuous integration test prioritization uncertainty prediction error online learning","flaky failure classifier true regression rerun"],"source_ids":["SRC1","SRC2","SRC3","SRC4"],"no_result_note":null}},"sources":[{"source_id":"SRC1","title":"Predictive Test Selection","publisher":"arXiv; authors affiliated with Facebook","url":"https://arxiv.org/abs/1810.05286","source_type":"PRIMARY_RESEARCH","claims_supported":["A production CI selector can estimate the likelihood that a particular test will fail for a particular code change and rank or select tests from that estimate.","Full execution of all potentially impacted tests is impractical at large scale, creating a cost-versus-failure-recall allocation problem.","Training on flaky outcomes can produce poor predictions and cause a selector to learn flaky failures rather than change-caused faults.","The reported method combines change-test features, historical outcomes, probabilistic prediction, sampled evaluation, and a de-flaking procedure, but does not disclose the proposal's event-level signed-residual update protocol."]},{"source_id":"SRC2","title":"Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration","publisher":"arXiv; ISSTA/ACM research paper","url":"https://arxiv.org/abs/1811.04122","source_type":"PRIMARY_RESEARCH","claims_supported":["Retecs adaptively prioritizes and selects CI tests under a time limit using execution duration, recency, and previous verdicts.","Its test-case failure reward directly reinforces an executed failing test while a passing test receives no specific reward, closely matching the raw-outcome baseline.","The paper recognizes that unexecuted-test failures are unavailable and that noisy industrial failures can mislead learning and produce performance fluctuations.","Retecs does not require a preserved pre-outcome probability, signed probability residual, flakiness-specific gain filter, or change-test credit trace."]},{"source_id":"SRC3","title":"Smart subset optimization targets","publisher":"Launchable","url":"https://help.launchableinc.com/features/predictive-test-selection/requesting-and-running-a-subset-of-tests/choosing-a-subset-optimization-target/smart-subset-optimization-targets/","source_type":"FIRST_PARTY_PRODUCT","claims_supported":["A commercial predictive-test-selection product offers bounded subsets combining likely-to-fail and not-recently-run tests.","The product can drop tests above a flakiness threshold and can exclude tests whose historical failure probability is too high, explicitly addressing noise and known unstable tests.","The same product can instead prioritize recently failed tests, demonstrating that raw recent failure reinforcement and flake suppression coexist as established configurable practices.","These documented rules filter or compose selections but do not document residual-weighted learning from immutable pre-outcome predictions."]},{"source_id":"SRC4","title":"230,439 Test Failures Later: An Empirical Evaluation of Flaky Failure Classifiers","publisher":"arXiv; empirical software-engineering researchers","url":"https://arxiv.org/abs/2401.15788","source_type":"PRIMARY_RESEARCH","claims_supported":["A flaky test can also detect a genuine defect, so failures from known flaky tests cannot safely be ignored wholesale.","Rerunning failures is common practice but can be ineffective and expensive, while failure deduplication can confuse genuine and flaky failures.","The study's 22-project dataset shows that flaky-versus-true failure classification effectiveness varies substantially by project.","These findings support reliability filtering and halt criteria, while warning that a flakiness label can conceal a real regression."]}],"problem_evidence":{"status":"PARTLY_SUPPORTED","finding":"The underlying allocation problem is visible: CI test budgets are constrained, adaptive and probabilistic selectors use historical failures, raw failure rewards can reinforce failing tests, and prior research explicitly warns that learning from flaky outcomes can optimize for flakes rather than change-caused faults. Product documentation also exposes both recently-failed prioritization and rules for suppressing flaky or high-failure-rate tests. The stronger prevalence claim—that deployed selectors routinely lack pre-run probabilities and systematically spend disproportionate budget rediscovering expected flakes while underweighting surprising regressions—was not directly demonstrated by the retained sources.","source_ids":["SRC1","SRC2","SRC3","SRC4"]},"closest_prior_art":[{"name":"Facebook Predictive Test Selection","source_ids":["SRC1"],"overlap":"Predicts failure likelihood for each code-change/test pair, uses change and test features, trains from historical outcomes, operates under CI resource constraints, samples outcomes for evaluation, and explicitly removes or compensates for flaky results.","remaining_difference":"The paper does not disclose an immutable event-level pre-outcome probability log, require every update to pass through signed residual y-p, assign each residual among code-test, flake, and infrastructure models, or condition rollout on residual-specific calibration and shortcut holdouts."},{"name":"Launchable smart predictive subsets","source_ids":["SRC3"],"overlap":"Combines probabilistic test selection with fixed-budget subsetting and configurable suppression of flaky or historically high-failure tests so known unstable tests do not dominate regression-oriented subsets.","remaining_difference":"The documented mechanism is selection-rule composition and threshold filtering, not an auditable learning rule in which each outcome updates risk only through its preserved prediction error, reliability weighting, and explicit change-test credit decision."},{"name":"Retecs reinforced test-case selection","source_ids":["SRC2"],"overlap":"Provides online adaptive CI test prioritization under a time budget, learns from prior cycles, and can reward each failed test individually.","remaining_difference":"Retecs is the closest baseline rather than the proposed correction: its failure reward is based on the verdict, passing can receive no test-specific reward, and the method lacks preserved failure probabilities, signed residuals, flake/infrastructure filtering, and change-test credit tracing."}],"prior_art_disposition":"ADJACENT_PRIOR_ART","contrastive_claim_remaining":"Holding repository, chronological full-suite records, model inputs, and simulated execution budget fixed, an update rule that admits each outcome only through a timestamped pre-outcome probability, signed residual, reliability classification, and explicit code-test credit decision will produce measurably better held-out calibration and budget-constrained product-regression discovery, or a lower flaky-test capacity share without worse discovery, than raw-verdict or recent-failure reinforcement. The retained prior art contains several components but does not disclose this complete event-audited update commitment.","contrastive_claim_falsifier":"The claim is falsified if a temporally held-out replay shows no reproducible improvement on the predefined calibration, product-regression discovery, or flaky-capacity measures; if improvements disappear when incidental metadata shortcuts are blocked; if residuals are repeatedly credited to the wrong change-test relationship; or if an opened prior-art implementation is shown to already require the same preserved prediction, signed residual, reliability filter, and explicit credit chain for every selection-weight update.","gates":{"adequate_source_search":{"status":"PASS","rationale":"The bounded search covered the proposal directly, older history-based and reinforcement-learning terminology, first-party predictive-selection and flake-filtering practices, and component combinations. Four opened sources from three publisher contexts were retained, including three primary research sources and one first-party source.","source_ids":["SRC1","SRC2","SRC3","SRC4"]},"supported_problem":{"status":"PASS","rationale":"The exact prevalence and magnitude are unquantified, but primary research and first-party documentation support the constituent problem: constrained CI selection, verdict-driven reinforcement, noisy flaky failures, and the risk of learning or selecting flakes instead of change-caused faults.","source_ids":["SRC1","SRC2","SRC3","SRC4"]},"distinct_testable_claim":{"status":"PASS","rationale":"The remaining claim specifies an observable update-chain constraint and comparative held-out outcomes. None of the retained sources documents the complete preserved-prediction, signed-residual, reliability-filter, and explicit-credit combination.","source_ids":["SRC1","SRC2","SRC3"]},"bounded_next_test":{"status":"PASS","rationale":"An offline chronological replay of no more than 200 authorized full-suite runs is bounded by repository, sample size, time order, fixed simulated budget, predefined metrics, baseline, shortcut probes, and halt criteria. It changes neither live execution nor merge gates and directly tests the contrastive claim.","source_ids":["SRC1","SRC2","SRC4"]},"no_obvious_safety_or_authority_stop":{"status":"PASS","rationale":"The proposed first step is offline and non-gating, preserves release-control authority, excludes developer identity and production rollout, and includes stops for inadequate timestamps, inseparable failure causes, shortcut leakage, and data-control violations. Flaky labels require caution because they can conceal genuine regressions, but that is a safeguard requirement rather than a categorical stop.","source_ids":["SRC1","SRC4"]}},"screen_survival":true,"world_novelty_boundary":"This bounded public-web screen establishes only that the complete contrastive mechanism was not found in the retained search results. It cannot establish world novelty, patentability, freedom to operate, market size, expert acceptance, implementation feasibility at scale, or realized value."}