{"schema_version":1,"research_id":"eoa_inverse_innovation_exp03_external48_20260801","source_assessment_id":"eoa_inverse_innovation_exp03_opportunity320_20260801","cell_id":"computability_boundary_mapping__data_science","selection_stratum":"MIDDLE_BAND_AUDIT_Q4","search_queries":["site:dl.acm.org machine learning data leakage detection pipeline paper","mlinspect pipeline inspection data leakage paper overlap train test","site:developers.google.com machine learning data leakage official guidance","Rice theorem static analysis undecidable semantic program properties official","\"Data leakage in notebooks: Static detection and better processes\" PDF","\"LeakageDetector\" open source data leakage analysis tool machine learning pipelines 2025","NIST AI RMF validation testing data leakage official","\"An abstract interpretation-based data leakage static analysis\"","machine learning pipeline DSL decidable data leakage checker formal verification","data leakage detection static analysis UNKNOWN unsupported pipeline","site:fda.gov machine learning independent test datasets representative intended population good machine learning practice","site:bls.gov/ooh computer and information research scientists median pay 2024","site:bls.gov/ooh software developers quality assurance analysts testers median pay 2024"],"sources":[{"source_id":"S1","title":"Data Leakage in Notebooks: Static Detection and Better Processes","publisher":"37th IEEE/ACM International Conference on Automated Software Engineering","url":"https://www.cs.cmu.edu/~ckaestne/pdf/ase22.pdf","source_class":"PRIMARY_RESEARCH","publication_date":"2022-10","accessed_at":"2026-08-02","claims_supported":["A static analyzer for common preprocessing, overlap, and repeated-test leakage reported at least one warning in 29.6% of 107,603 analyzed public GitHub notebooks.","Manual evaluation of 100 sampled notebooks produced 97.6% precision and 67.8% recall, showing useful but incomplete detection.","The implementation explicitly failed on syntax errors, unsupported constructs, external storage, unsupported libraries, and some timeouts rather than establishing universal coverage.","The authors describe the analysis as non-comprehensive and limited to statically observable notebook behavior and modeled libraries."]},{"source_id":"S2","title":"Static Analysis by Abstract Interpretation Against Data Leakage in Machine Learning","publisher":"Science of Computer Programming","url":"https://doi.org/10.1016/j.scico.2025.103338","source_class":"PRIMARY_RESEARCH","publication_date":"2025-12","accessed_at":"2026-08-02","claims_supported":["The work derives a sound, computable abstract analysis from a sound-and-complete but noncomputable collecting semantics to prove absence of specified train-test contamination.","Its formal policy tracks whether training and testing data originate from disjoint, untainted sources; target and group leakage are outside its evaluated scope.","The NBLyzer implementation was evaluated on 2,111 Kaggle notebooks and reported 25 true leakage instances with two false positives, described as 93% precision.","The paper formalizes a small data-frame language while the implementation also handles modeled branches, loops, and procedure calls."]},{"source_id":"S3","title":"Intensional Kleene and Rice Theorems for Abstract Program Semantics","publisher":"Information and Computation","url":"https://arxiv.org/abs/2105.14579","source_class":"PRIMARY_RESEARCH","publication_date":"2021-05-30","accessed_at":"2026-08-02","claims_supported":["Classical Rice results concern nontrivial extensional properties of partial computable functions.","The paper proves broader undecidability results only for abstract program semantics satisfying stated structural conditions, including strong s-m-n and a universal fair program.","Computability conclusions are therefore relative to a formal property and computation model, not automatic consequences of calling leakage semantic."]},{"source_id":"S4","title":"AI Risk Management Framework Core","publisher":"National Institute of Standards and Technology","url":"https://airc.nist.gov/airmf-resources/airmf/5-sec-core/","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2023-01-26","accessed_at":"2026-08-02","claims_supported":["NIST calls for documented, objective, repeatable or scalable testing, evaluation, verification, and validation processes.","NIST says limitations on generalizability beyond development conditions should be documented and systems should fail safely beyond their knowledge limits.","NIST recommends independent assessment and traceable risk-management decisions, supporting governance involvement beyond the platform developers."]},{"source_id":"S5","title":"Good Machine Learning Practice for Medical Device Development: Guiding Principles","publisher":"U.S. Food and Drug Administration","url":"https://www.fda.gov/media/154669/download","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2021-10-27","accessed_at":"2026-08-02","claims_supported":["FDA and partner regulators identify independence of training and test datasets as a good machine-learning-practice principle.","The principles also call for multidisciplinary expertise, clinically relevant testing, clear information for users, and management of retraining risk.","This supplies concrete regulatory stakeholder pull in medical-device settings but does not establish demand across all data-science platforms."]},{"source_id":"S6","title":"Datasets: Dividing the Original Dataset","publisher":"Google for Developers","url":"https://developers.google.com/machine-learning/crash-course/overfitting/dividing-datasets","source_class":"OFFICIAL_GUIDANCE","publication_date":"2025","accessed_at":"2026-08-02","claims_supported":["Google advises separating training, validation, and test data and warns that repeated use can wear out evaluation sets.","Google identifies duplicate train-test examples as invalidating a fair evaluation.","This independently supports the operational importance of preventing evaluation contamination."]},{"source_id":"S7","title":"Computer and Information Research Scientists","publisher":"U.S. Bureau of Labor Statistics","url":"https://www.bls.gov/ooh/computer-and-information-technology/computer-and-information-research-scientists.htm","source_class":"OFFICIAL_ORGANIZATION_DATA","publication_date":"2025-08-28","accessed_at":"2026-08-02","claims_supported":["The May 2024 median annual wage for U.S. computer and information research scientists was $140,910.","Formalization and computability review plausibly require specialized research labor, and wages exclude employer overhead and other project costs."]},{"source_id":"S8","title":"Software Developers, Quality Assurance Analysts, and Testers","publisher":"U.S. Bureau of Labor Statistics","url":"https://www.bls.gov/ooh/computer-and-information-technology/software-developers.htm","source_class":"OFFICIAL_ORGANIZATION_DATA","publication_date":"2025-08-28","accessed_at":"2026-08-02","claims_supported":["The May 2024 median annual wages were $133,080 for software developers and $102,610 for software quality-assurance analysts and testers.","These wage data provide labor anchors, but not loaded 2026 project prices or costs for governance, compliance, infrastructure, or integration."]}],"problem_evidence":{"support":"STRONG","rationale":"Independent research and official guidance establish that train-test, preprocessing, and repeated-evaluation leakage can invalidate model evaluation. S1 found leakage warnings in 29.6% of a large public-notebook corpus and documented incomplete detection, while S5 and S6 require or recommend independent evaluation data. However, no reviewed source established that a real platform currently promises an exact terminating Boolean verdict for arbitrary executable pipelines or maps timeout to NO_LEAK; that candidate-specific prevalence remains unverified.","source_ids":["S1","S5","S6"]},"stakeholder_evidence":{"support":"MODERATE","rationale":"NIST assigns testing, documentation, independent assessment, limitation disclosure, and safe-failure responsibilities across AI actors, while FDA explicitly requires independent training and test datasets in medical-device GMLP. These sources make platform engineering, validation, and risk-governance roles credible authorizers, but they do not show a named organization committed to this proposed checker or willingness to accept frequent UNKNOWN results.","source_ids":["S4","S5"]},"prior_art":{"proximity":"SUBSTANTIAL_COLLISION","closest_analogues":[{"name":"Static Analysis by Abstract Interpretation Against Data Leakage in Machine Learning","similarity":"This is a formal, sound, computable leakage analysis that can prove absence of a specified class of train-test contamination and has an implemented notebook analyzer. It already realizes much of the proposed restricted exact-or-conservative checking mechanism.","remaining_difference":"It does not present the candidate's model-matched impossibility reduction for all arbitrary executable pipelines, does not claim coverage of target or group leakage, and does not document a mechanically enforced platform fragment with standardized UNKNOWN/OUT_OF_SCOPE governance and a prohibition on timeout-to-negative conversion.","source_ids":["S2","S3"]},{"name":"Data Leakage in Notebooks: Static Detection and Better Processes","similarity":"It statically detects three common leakage patterns, was evaluated at large scale, exposes unsupported cases and timeouts, and explicitly treats warnings as review inputs rather than comprehensive verdicts.","remaining_difference":"Its analysis is heuristic and incomplete rather than an exact checker for an enforceable decidable fragment; it reports warnings but does not establish the proposed guarantee-label contract or an unrestricted impossibility theorem.","source_ids":["S1"]}],"distinctive_claim_remaining":"The remaining testable distinction is not automated leakage detection or even sound absence checking. It is the integrated contract: a reviewed reduction for one precisely defined unrestricted leakage predicate and computation model, mechanically enforced membership in a proved terminating-and-correct fragment, and machine-preserved UNKNOWN/OUT_OF_SCOPE labels for every unsupported, timed-out, or weaker analysis path. This bounded search found no source implementing and governing that entire composition.","confidence":"HIGH"},"implementation_evidence":{"support":"MODERATE","rationale":"S1 demonstrates scalable static leakage detection over more than 100,000 notebooks but with 67.8% recall and explicit unsupported cases. S2 demonstrates a formally derived sound abstraction and a 2,111-notebook implementation, substantially de-risking the checker component. Evidence is still missing for the candidate's exact policy, mechanically enforced fragment membership, explicit guarantee-label propagation, opaque services, and model-matched impossibility reduction.","source_ids":["S1","S2","S3"]},"scores":{"meaningful_impact":{"score":4,"rationale":"Leakage can make offline results overoptimistic, and both research and official guidance treat independent evaluation data as important. Impact is capped because the specific alleged universal analyzer and timeout-to-NO_LEAK behavior were not externally observed.","source_ids":["S1","S5","S6"]},"stakeholder_pull":{"score":4,"rationale":"NIST and FDA create strong validation, documentation, independent-review, and safe-failure incentives for risk owners and regulated ML teams. No specific adopter commitment or demand for a formal DSL was found.","source_ids":["S4","S5"]},"incremental_advantage":{"score":3,"rationale":"Explicit scope enforcement and UNKNOWN routing would improve honesty over an analyzer that silently returns NO_LEAK on unresolved inputs. The incremental technical advantage is narrower than initially assessed because S2 already supplies a sound computable absence analysis for a bounded leakage policy.","source_ids":["S1","S2","S4"]},"distinctiveness_plausibility":{"score":2,"rationale":"The closest prior art already formalizes leakage semantics, derives a sound computable abstraction, proves soundness, and implements it. Only the reviewed universal-boundary argument, enforced fragment contract, and governed label propagation remain plausibly distinctive.","source_ids":["S2","S3"]},"technical_implementability":{"score":4,"rationale":"Large-scale static analysis and a formally sound NBLyzer implementation show that the principal bounded-checker components are implementable. The exact DSL proof, platform integration, opaque calls, and impossibility reduction remain unresolved.","source_ids":["S1","S2"]},"adoption_authority_feasibility":{"score":3,"rationale":"NIST supports independent review, documentation, and risk-management decisions, making joint engineering and governance approval credible. No actual platform authority, production claim, integration surface, or commitment was independently verified.","source_ids":["S4","S5"]},"evidence_readiness":{"score":4,"rationale":"Published policies, analyzers, corpora, accuracy measures, and documented failure modes provide usable baselines for a frozen-corpus comparison. The candidate still lacks a platform inventory, formal reduction, fragment grammar, and interpretation testing for guarantee labels.","source_ids":["S1","S2","S3"]},"safety_net_benefit":{"score":5,"rationale":"UNKNOWN/OUT_OF_SCOPE routing, limitation disclosure, independent review, and fail-safe behavior directly address the risk of converting missing evidence into negative assurance and align with NIST's treatment of knowledge limits.","source_ids":["S4"]},"scalability":{"score":3,"rationale":"S1 analyzed 107,603 notebooks and S2 reports interactive-scale implementation evidence, supporting technical scaling for modeled constructs. Maintaining library models, handling external state, preserving labels downstream, and supporting routine pipelines remain material constraints.","source_ids":["S1","S2"]}},"score_confidence":"MODERATE","costs":{"first_evidence":{"band_2026_usd":"50K_TO_250K","scope":"An 8–16 week non-production study by a formal-methods researcher, platform engineer, and part-time ML validation reviewer: inventory one frozen pipeline corpus, specify one leakage policy and IR, independently review one reduction, adapt one prior-art checker, implement scope and UNKNOWN tests, and evaluate verdicts and label interpretation. Includes labor, corpus preparation, software/compute, coordination, security review, and evaluation.","confidence":"MODERATE","assumptions":["Existing code and a representative frozen corpus are accessible without new data acquisition.","The study reuses published analyzer concepts rather than creating a static-analysis framework from scratch.","BLS wages are base-labor anchors; employer burden, specialist premiums, management, and infrastructure raise resource-equivalent cost.","No production decisions or regulated submissions depend on the study."],"source_ids":["S1","S2","S7","S8"]},"initial_deployment_startup":{"band_2026_usd":"250K_TO_1M","scope":"Harden one restricted checker and IR, enforce scope membership, integrate versioned LEAK/NO_LEAK/UNKNOWN/OUT_OF_SCOPE records into one platform, build regression and adversarial suites, perform independent validation, and establish security, privacy, compliance, documentation, training, rollback, and audit controls for a limited non-authorizing release.","confidence":"LOW","assumptions":["One platform and a limited family of pipeline libraries are in scope.","Existing identity, audit, CI, storage, and review systems can be reused.","At least two engineering/research FTE-equivalents plus fractional governance, security, legal/compliance, and training effort are required.","Opaque external services are routed out of scope rather than modeled exactly."],"source_ids":["S2","S4","S7","S8"]},"operational_launch":{"band_2026_usd":"1M_TO_5M","scope":"Production launch across materially used supported workflows, including migration and compatibility tooling, policy and proof review, multi-team integration, monitoring for label collapse, reviewer queues for unresolved cases, incident response, independent assurance, compliance evidence, documentation, training, compute, storage, and outcome evaluation.","confidence":"LOW","assumptions":["Launch spans several workflow libraries and organizational teams but not every arbitrary executable environment.","Existing manual review remains mandatory for UNKNOWN and OUT_OF_SCOPE cases.","The organization requires production reliability, security, observability, on-call support, and model-risk sign-off.","A broader regulated or multi-platform rollout could exceed this band."],"source_ids":["S4","S5","S7","S8"]},"annual_recurring":{"band_2026_usd":"250K_TO_1M","scope":"Maintain the IR, library models, checker and proofs, policy versions, regression corpus, integrations, infrastructure, monitoring, audits, compliance evidence, training, and escalation capacity for unsupported pipelines.","confidence":"LOW","assumptions":["A small cross-functional team is retained with one to three FTE-equivalents plus infrastructure and independent-review capacity.","Pipeline-language, library, policy, and external-service changes trigger continuing validation work.","Most compute and storage use existing organizational infrastructure.","High UNKNOWN review volume or frequent language changes could move recurring cost above this band."],"source_ids":["S1","S2","S4","S7","S8"]}},"verified_pipeline_gates":{"externally_supported_problem":{"status":"YES","reason":"Leakage prevalence, incomplete automated detection, and evaluation harm are independently supported. The exact prevalence of universal Boolean overclaims and timeout-to-NO_LEAK conversion remains unknown but does not negate the underlying assurance problem.","source_ids":["S1","S5","S6"]},"externally_credible_adopter_or_authorizer":{"status":"YES","reason":"Platform engineering, independent validation, and model-risk governance are credible adopter and authorization roles under NIST's lifecycle governance and independent-assessment framework; FDA supplies a concrete regulated setting requiring train-test independence.","source_ids":["S4","S5"]},"distinct_testable_incremental_claim":{"status":"YES","reason":"The remaining claim is measurable against close prior art: mechanical scope enforcement plus preserved UNKNOWN labels should eliminate unsupported negative verdicts without reducing in-scope correctness relative to S2's sound-analysis baseline.","source_ids":["S1","S2","S4"]},"bounded_next_evidence_step":{"status":"YES","reason":"A frozen-corpus, non-production comparison of the candidate contract against the published S1 and S2 analysis patterns can measure false negative verdicts, coverage, UNKNOWN rate, runtime, and label interpretation without changing downstream decisions.","source_ids":["S1","S2"]},"no_unresolved_safety_or_authority_stop":{"status":"YES","reason":"The evidence step can remain read-only and non-authorizing, retain existing human review for all results, and halt on a false in-scope NO_LEAK or label collapse. Formal uncertainty blocks production claims but not this study.","source_ids":["S4"]},"credible_cost_scope_and_range":{"status":"YES","reason":"The bands enumerate specialized research, engineering, QA, data, infrastructure, integration, governance, compliance, training, and evaluation work. BLS wages anchor labor order of magnitude, while low confidence reflects absent architecture, corpus, staffing, and regulatory details.","source_ids":["S7","S8"]}},"next_evidence_step":"Run a 100-pipeline frozen-corpus differentiation study against the closest prior art: pre-register one leakage predicate and IR; have an independent reviewer check a halting reduction and its preservation argument; implement mechanical fragment membership and four-valued routing; then compare against a reproduction or faithful implementation of S2 and the current timeout analyzer on false NO_LEAK count, false alarms, UNKNOWN/OUT_OF_SCOPE rate, routine-pipeline coverage, and runtime. Falsify the intervention if any mechanically in-scope pipeline receives a false NO_LEAK, if the proposed reduction does not preserve the registered leakage predicate, if scope membership is bypassable, or if blinded reviewers or downstream test code interpret UNKNOWN as NO_LEAK.","blocking_evidence":["No external source or platform artifact confirms that the target platform accepts unrestricted executable pipelines, promises an exact terminating Boolean answer, or converts timeout and unsupported inputs to NO_LEAK.","A model-matched reduction for the selected leakage predicate has not been exhibited or independently reviewed; generic Rice-style results do not substitute for it.","The leakage policy must address whether target, group, temporal, repeated-test, external-state, and opaque-service channels are included; S2 explicitly leaves some of these outside scope.","No grammar, totality proof, correctness proof, or mechanical membership test exists yet for the proposed restricted fragment.","The candidate has not shown a measurable advantage over S2's already-sound computable abstraction.","No frozen-corpus evidence establishes routine-pipeline coverage, false-negative assurance, UNKNOWN volume, runtime, or reviewer workload.","No human-factors or interface evidence shows that callers and downstream systems preserve UNKNOWN and OUT_OF_SCOPE meanings.","No named adopter has committed engineering, governance, or validation resources."],"research_disposition":"PRIOR_ART_DIFFERENTIATION_STUDY","world_novelty_boundary":"This bounded search found substantial prior-art collision: formal, sound, computable leakage-absence analysis and large-scale static leakage detection already exist. It did not find the full combination of a leakage-specific reviewed impossibility reduction, mechanically enforced decidable platform fragment, and governed four-valued guarantee routing. That residual is only a testable differentiation hypothesis; it is not a world-novelty or patentability claim."}