{"closest_prior_art":[{"name":"Multiple Testing for IR and Recommendation System Experiments","overlap":"Directly identifies inflated false discoveries when more than two IR systems are evaluated and studies procedures controlling false-discovery rate.","remaining_difference":"It does not establish a library-discovery workflow that preregisters the complete configuration-and-metric family, records every attempt, assigns non-deployment statuses, and permits only one untouched confirmation before governance review.","source_ids":["SRC1"]},{"name":"The Information Retrieval Experiment Platform (TIREx)","overlap":"Provides standardized, reproducible and blinded IR experiments; archives experiments and results, records artifacts, uses immutable software, and can keep test data private in a sandbox.","remaining_difference":"It does not itself apply FDR screening to a frozen configuration-by-analysis family or impose library-specific candidate status, specialist veto and production-authorization gates.","source_ids":["SRC2"]},{"name":"CARLI Primo VE configuration-testing guidance plus WSU discovery-relevance framework","overlap":"CARLI documents locally adjustable ranking, boosting and deduplication settings, baseline searches, change documentation and stakeholder testing. WSU supplies a reproducible discovery-layer evaluation using multiple search strategies and four metrics and proposes A/B testing configurations.","remaining_difference":"These practices do not calibrate selection to the number of configurations and metric slices examined, require a complete append-only run inventory, or sequester a one-use confirmation set before deployment eligibility.","source_ids":["SRC3","SRC4"]}],"contrastive_claim_falsifier":"The contrast would be falsified by prior art showing an operational library-discovery process that, before tuning, freezes the complete configuration-and-analysis family, automatically preserves every run and status, applies multiplicity-aware candidate screening, tests at most one candidate once on inaccessible queries, and blocks production eligibility pending specialist and governance review. Experimentally, it would also fail if unregistered runs affect selection, confirmation data influence tuning, or adjusted-discovery results enter approval materials as confirmed.","contrastive_claim_remaining":"The remaining testable distinction is an end-to-end library-discovery governance protocol that joins a preregistered configuration-by-evaluation family, automatic complete run provenance, FDR-based exploratory status, one untouched confirmation, specialist veto and committee-controlled production eligibility. The retained prior art covers these elements separately but does not demonstrate that combined control loop for library ranking changes.","experiment_id":"eoa_inverse_innovation_exp13_second_slot_policy60_20260806","gates":{"adequate_source_search":{"rationale":"Four lanes were searched using direct wording, older IR/test-collection terminology, library products and practices, and combinations involving registries, multiplicity and blinded holdouts. Exactly four sources from four publishing organizations were opened, including three primary studies and official consortium guidance. No phrase miss was treated as novelty evidence.","source_ids":["SRC1","SRC2","SRC3","SRC4"],"status":"PASS"},"bounded_next_test":{"rationale":"A single-component, offline replay with a frozen family, automatic run capture, one selected candidate and one sequestered confirmation is finite and measurable. TIREx demonstrates that comprehensive artifact capture and blinded IR evaluation are technically feasible, while the library sources provide relevant query, metric and configuration-testing precedents.","source_ids":["SRC2","SRC3","SRC4"],"status":"PASS"},"distinct_testable_claim":{"rationale":"The proposed joint control loop is distinguishable from FDR-only analysis, reproducible/blinded IR infrastructure, and ordinary library configuration testing. Its claimed distinction can be tested through ledger completeness, candidate-status transitions, confirmation-set access records and whether any ineligible result reaches governance review.","source_ids":["SRC1","SRC2","SRC3","SRC4"],"status":"PASS"},"no_obvious_safety_or_authority_stop":{"rationale":"The authorized first step is read-only and nonproduction, and the proposal excludes patron identities, rare unsuppressed queries and live ranking changes. Blinded sandbox evaluation and test views have precedents. Privacy suppression and governance approval remain required controls rather than obvious reasons the bounded pilot cannot proceed.","source_ids":["SRC2","SRC3"],"status":"PASS"},"supported_problem":{"rationale":"IR research directly establishes that comparing many systems can inflate false discoveries. TIREx describes bias and leakage risks around reusable test data, while library sources show configurable rankings, repeated example-search testing, multiple strategies and multiple relevance metrics. Evidence of actual selective deployment reporting in libraries was not found, so support is partial rather than complete.","source_ids":["SRC1","SRC2","SRC3","SRC4"],"status":"PASS"}},"prior_art_disposition":"ADJACENT_PRIOR_ART","problem_evidence":{"finding":"The underlying opportunity for evidentiary inflation is visible: library discovery rankings expose several adjustable settings and are evaluated across reusable searches, strategies and metrics, while IR research shows that multiple system comparisons inflate false discoveries and that test-data exposure can bias evaluation. The retained sources do not directly document a library deploying a winner after concealing unsuccessful configurations or unfavorable slices.","source_ids":["SRC1","SRC2","SRC3","SRC4"],"status":"PARTLY_SUPPORTED"},"research_id":"eoa_inverse_innovation_exp13_light_screen_20260806","schema_version":1,"screen_id":"E13P036","screen_survival":true,"search_lanes":{"component_combination":{"no_result_note":"No opened result combined an immutable all-run ledger, FDR screening, a one-use holdout and library production governance in one protocol.","queries":["search ranking experiment append-only ledger holdout confirmation multiple testing","information retrieval evaluation preregistration ranking experiment registry","library discovery ranking evaluation holdout false discovery rate"],"source_ids":["SRC1","SRC2"]},"direct_problem_and_intervention":{"no_result_note":"No direct end-to-end match for the proposed library-specific protocol was found; adjacent results covered multiplicity, blinded evaluation or discovery-layer assessment separately.","queries":["library discovery layer relevance ranking evaluation tuning configurations metrics study","library discovery ranking evaluation holdout false discovery rate","information retrieval evaluation multiple comparisons false discovery rate ranking systems"],"source_ids":["SRC1","SRC2","SRC3","SRC4"]},"products_practices_and_standards":{"no_result_note":null,"queries":["site:carli.illinois.edu Primo VE \"Best Practices for Testing Configuration Changes\" ranking","library discovery relevancy evaluation ranking configurations A/B testing metrics","library discovery system relevance ranking configuration testing best practices"],"source_ids":["SRC3","SRC4"]},"synonyms_and_historical_terms":{"no_result_note":"Searches using test-collection reuse, blinded evaluation, reproducibility and preregistration terminology found adjacent IR controls but no library-specific complete protocol.","queries":["search ranking evaluation holdout query set overfitting repeated tuning test collection","\"Improvements that don't add up\" information retrieval test collection overfitting","IR evaluation pre-registration multiple comparisons test collection reuse"],"source_ids":["SRC1","SRC2"]}},"sources":[{"claims_supported":["Many IR experiments compare more than two systems, creating inflated false discoveries through the multiple-comparison problem.","FDR-controlling multiple-comparison procedures have been studied directly for IR and recommender evaluation."],"publisher":"Springer; record hosted by the National Science Foundation Public Access Repository","source_id":"SRC1","source_type":"PRIMARY_RESEARCH","title":"Multiple Testing for IR and Recommendation System Experiments","url":"https://par.nsf.gov/biblio/10497108"},{"claims_supported":["TIREx supports standardized, reproducible, scalable and blinded retrieval experiments.","The platform archives experiments and results, maintains comprehensive artifact records, uses immutable software and can hide test data in a sandbox.","The paper reports that preference reproducibility declines across dissimilar retrieval tasks, supporting fresh-task confirmation concerns."],"publisher":"International Joint Conferences on Artificial Intelligence Organization","source_id":"SRC2","source_type":"PRIMARY_RESEARCH","title":"The Information Retrieval Experiment Platform (Extended Abstract)","url":"https://www.ijcai.org/proceedings/2024/0931.pdf"},{"claims_supported":["Primo VE permits changes to field, date, resource-type and institution boosting, blended profiles, deduplication and related discovery behavior.","CARLI recommends baseline example searches, testing one small change at a time, documenting effects, involving stakeholders and documenting production decisions.","The guidance does not specify multiplicity adjustment or a sequestered confirmation set."],"publisher":"Consortium of Academic and Research Libraries in Illinois","source_id":"SRC3","source_type":"OFFICIAL_GUIDANCE","title":"CARLI/I-Share Office Hours: Primo VE and CDI—Improve Discoverability and Promote Efficient and Effective Searching","url":"https://www.carli.illinois.edu/sites/files/i-share/documentation/OfficeHours20210624-Slides.pdf"},{"claims_supported":["A production library discovery environment was assessed with a quantitative, reproducible relevance framework.","The framework compared search strategies using four metrics and proposed future A/B testing of commonly deployed discovery configurations and limiters.","The study cautioned that its preliminary results did not support statistical conclusions about one strategy's dominance."],"publisher":"Information Technology and Libraries (Core/ALA)","source_id":"SRC4","source_type":"PRIMARY_RESEARCH","title":"A Framework for Measuring Relevancy in Discovery Environments","url":"https://ital.corejournals.org/index.php/ital/article/download/12835/10387/28087"}],"world_novelty_boundary":"This bounded four-source screen supports only an adjacent-prior-art disposition. It cannot establish world novelty, patentability, market size, expert acceptance or realized value, and unsearched patents, internal library procedures, vendor documentation or differently worded governance systems could contain closer matches."}