{"schema_version":1,"research_id":"eoa_inverse_innovation_exp05_external_evaluation_20260803","source_assessment_id":"layer_decay_and_expiration_management__cognitive_science:P1:v0","cell_id":"layer_decay_and_expiration_management__cognitive_science","search_queries":["cognitive science experiment materials versioning reproducibility task stimuli scoring rules official guidance","OSF registrations immutable frozen files version control official","research data lifecycle retention archive restore digital preservation standard official","electronic laboratory notebook protocol version lifecycle superseded archived research data","site:psycnet.apa.org psychological task materials reproducibility stimuli analysis code study primary research","cognitive task measurement reliability different versions stimuli scoring norms reproducibility study","psychology open materials prevalence study reproducibility materials code","protocol version mismatch psychology experiment task instructions stimuli replication","protocols.io official versioning protocol versions DOI archive research protocols","RO-Crate specification version provenance research object official standard","research data lifecycle policy retention disposition university lab data official","digital preservation archive restore testing official guidance research data","ReproSchema cognitive task protocol versioning official","cognitive assessment protocol metadata versioning platform research ReproNim","psychology experiment task repository version controlled stimuli scoring platform","cognitive atlas task versions protocol materials","\"Estimating the Prevalence of Transparency\" Hardwicke full text DOI","Hardwicke Thibault Kosie 2021 psychology transparency reproducibility materials protocols 250 articles DOI"],"sources":[{"source_id":"S1","title":"Estimating the Prevalence of Transparency and Reproducibility-Related Research Practices in Psychology (2014–2017)","publisher":"Perspectives on Psychological Science / SAGE Publications","url":"https://journals.sagepub.com/doi/10.1177/1745691620979806","source_class":"PRIMARY_RESEARCH","publication_date":"2021-03-08","accessed_at":"2026-08-03","claims_supported":["A random sample of 250 psychology articles found research-material sharing in 26 of 183 eligible articles, protocol sharing in 0 of 188, and analysis-script sharing in 1 of 188.","Materials, protocols, data, and scripts are needed to evaluate, interpret, verify, and replicate research claims.","Reportedly available materials sometimes could not be retrieved because of broken links, showing that preservation and restoration cannot be assumed."]},{"source_id":"S2","title":"Introduction—ReproSchema Documentation","publisher":"ReproNim","url":"https://repronim.org/reproschema/introduction/","source_class":"OFFICIAL_PRODUCT_DOCUMENTATION","publication_date":"undated documentation, current at access","accessed_at":"2026-08-03","claims_supported":["Questionnaires and assessments are commonly altered for individual studies.","Researchers may be unable to determine which questionnaire version was used or exactly how it was scored.","ReproSchema already provides schemas and tools that preserve relationships, sources, and changes over time."]},{"source_id":"S3","title":"Welcome to Registrations & Preregistrations!","publisher":"Center for Open Science","url":"https://help.osf.io/article/330-welcome-to-registrations","source_class":"OFFICIAL_PRODUCT_DOCUMENTATION","publication_date":"2026-07","accessed_at":"2026-08-03","claims_supported":["OSF registrations create immutable frozen versions of projects and associated files.","Projects can continue evolving while registrations certify exact states at important lifecycle points.","Withdrawal is irreversible but leaves basic metadata, justification, and a tombstone; administrators approve withdrawal."]},{"source_id":"S4","title":"Bring Structure to Your Research","publisher":"protocols.io","url":"https://www.protocols.io/","source_class":"COMMERCIAL_FIRST_PARTY","publication_date":"undated, current at access","accessed_at":"2026-08-03","claims_supported":["protocols.io provides centralized, versionable methods with history, collaboration, audit-trail, approval, and signature features.","A UCSF principal investigator explicitly identifies exact protocol-version identification as valuable for reproducibility.","A UCSF library adopter reports substantial institutional uptake and expects more rigorous and reproducible research methods."]},{"source_id":"S5","title":"RO-Crate Specification 1.1—Introduction","publisher":"RO-Crate Community / University of Technology Sydney / University of Manchester","url":"https://www.researchobject.org/ro-crate/specification/1.1/introduction.html","source_class":"STANDARD","publication_date":"2024 documentation edition","accessed_at":"2026-08-03","claims_supported":["RO-Crate specifies aggregation of research data and associated metadata at package and individual-resource levels.","Its JSON-LD metadata can represent authorship, software, workflows, provenance, context, and relationships needed for reuse and long-term management.","A protocol-package manifest and dependency representation can therefore use an existing interoperable specification."]},{"source_id":"S6","title":"Intramural Electronic Lab Notebook Policy","publisher":"NIH Office of Intramural Research","url":"https://oir.nih.gov/sourcebook/intramural-program-oversight/electronic-lab-notebooks/intramural-electronic-lab-notebook-policy","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2024-06","accessed_at":"2026-08-03","claims_supported":["NIH requires approved electronic laboratory notebooks for new intramural research and applies records requirements across the document lifecycle.","ELNs must document rationale, methods, data locations, analysis, interpretation, and the complete research record sufficiently for reproduction.","NIH principal investigators and designated data managers have explicit responsibility to manage and safeguard research records."]},{"source_id":"S7","title":"Research Data Framework (RDaF)","publisher":"National Institute of Standards and Technology","url":"https://www.nist.gov/programs-projects/research-data-framework-rdaf","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2024, RDaF 2.0 page","accessed_at":"2026-08-03","claims_supported":["NIST identifies loss or mismanagement of research data as carrying potentially severe economic and social consequences.","RDaF organizes research-data management as a lifecycle including preservation and discard.","The framework addresses stakeholders, costs, benefits, risks, governance, use, reuse, and organizational strategy."]},{"source_id":"S8","title":"Research Data Stewardship Policy","publisher":"University of Michigan","url":"https://spg.umich.edu/policy/303.06","source_class":"OFFICIAL_GUIDANCE","publication_date":"2023","accessed_at":"2026-08-03","claims_supported":["Research data include supporting code, metadata, laboratory notebooks, and other documentation needed to interpret, reconstruct, evaluate, or replicate findings.","The university owns covered research data, while principal investigators or delegates manage and steward it on the university's behalf.","Principal investigators must retain research data and coordinate alternative archival solutions when ordinary storage becomes cost-prohibitive."]}],"problem_evidence":{"support":"MODERATE","rationale":"The general problem is visible and consequential: psychology frequently lacks accessible protocols, materials, and scripts; ReproSchema independently identifies uncertainty about assessment versions and scoring; and official policies treat lifecycle management and reconstructability as material obligations. However, no source measured the proposal's narrower event—researchers selecting mutually incompatible superseded layers from a laboratory repository—or its prevalence in cognitive-task laboratories.","source_ids":["S1","S2","S6","S7"]},"stakeholder_evidence":{"support":"MODERATE","rationale":"Identifiable authorizers and adopters exist. NIH assigns lifecycle and reproducibility duties to principal investigators and data managers; the University of Michigan assigns stewardship to PIs or delegates; and UCSF users explicitly value exact protocol-version identification. The evidence demonstrates adjacent institutional pull but not a stated demand for the proposed full lifecycle registry, dependency gate, quarantine, and restore-drill bundle.","source_ids":["S4","S6","S8"]},"prior_art":{"proximity":"SUBSTANTIAL_COLLISION","closest_analogues":[{"name":"ReproSchema","similarity":"Directly addresses evolving assessments, missing version and scoring information, formal relationships, provenance, and change tracking in a cognitive/neurobehavioral research context.","remaining_difference":"The documentation does not establish a human-authorized lifecycle with semantic staleness review, preservation exceptions, dependency-gated disposition, quarantine, tombstones, and tested archival restoration.","source_ids":["S2"]},{"name":"OSF projects and immutable registrations","similarity":"Separates an evolving project from immutable, timestamped snapshots; preserves file structure; uses administrative approval; and leaves withdrawal metadata and a tombstone.","remaining_difference":"It freezes whole project states rather than managing active authority and disposition across individual protocol layers, and it does not demonstrate dependency tracing or archive-restore drills.","source_ids":["S3"]},{"name":"protocols.io","similarity":"Offers centralized versionable methods, history, audit trails, approvals, signatures, and exact protocol-version identification for reproducibility.","remaining_difference":"The public evidence does not show an assembled multi-artifact lifecycle registry spanning stimuli, scoring rules, norm tables, exclusions, environments, dependencies, quarantine, and restoration testing.","source_ids":["S4"]},{"name":"RO-Crate","similarity":"Provides a standard package manifest and linked metadata for files, software, workflows, provenance, context, and resource relationships.","remaining_difference":"RO-Crate describes and packages research objects but does not itself supply lifecycle authority, stale-layer adjudication, reversible disposition, or empirical selection safeguards.","source_ids":["S5"]},{"name":"Institutional ELN and research-data stewardship programs","similarity":"Assign lifecycle, retention, security, archival, reproducibility, and stewardship duties to laboratories, PIs, and institutional authorities.","remaining_difference":"The policies establish authority and obligations but do not prescribe the proposed artifact-level user interface, scoring triage, dependency gate, or comparative reconstruction exercise.","source_ids":["S6","S7","S8"]}],"distinctive_claim_remaining":"For a cognitive-task family with multiple revisions, adding artifact-level lifecycle states, visible study bindings, dependency checks, reversible disposition metadata, and tested archive restoration will improve blinded researchers' correct package selection and successful reconstruction relative to both the existing repository and a lightweight immutable current-version manifest, without producing material false-staleness classifications or missed dependencies.","confidence":"MODERATE"},"implementation_evidence":{"support":"MODERATE","rationale":"Existing systems demonstrate feasible components: versioned assessment schemas, immutable snapshots, protocol versioning and approval, interoperable package manifests, and institutionally assigned stewardship. A read-only shadow registry is technically low risk. Feasibility remains unverified for semantic staleness classification, complete inbound-dependency discovery, integration across heterogeneous lab storage, access-control inheritance, long-term reader preservation, and reliable restoration of actual cognitive-task packages.","source_ids":["S2","S3","S4","S5","S6","S8"]},"scores":{"meaningful_impact":{"score":3,"rationale":"Correct protocol binding and reconstructability matter scientifically, but the frequency and severity of wrong-layer selection are not measured.","source_ids":["S1","S2","S6"]},"stakeholder_pull":{"score":3,"rationale":"PIs and institutional stewards have explicit lifecycle duties, and exact version identification has expressed user value; demand for this full bundle is unproven.","source_ids":["S4","S6","S8"]},"incremental_advantage":{"score":2,"rationale":"Most functional ingredients already exist in ReproSchema, OSF, protocols.io, RO-Crate, and institutional stewardship practice. Advantage over a manifest-plus-version-control baseline requires testing.","source_ids":["S2","S3","S4","S5"]},"distinctiveness_plausibility":{"score":2,"rationale":"The distinctive element is an integrated governance workflow and comparative outcome claim, not a clearly novel technical primitive.","source_ids":["S2","S3","S4","S5","S7"]},"technical_implementability":{"score":4,"rationale":"A read-only registry can reuse mature versioning, schema, packaging, and provenance approaches; automated semantic and dependency completeness remain difficult.","source_ids":["S2","S3","S5"]},"adoption_authority_feasibility":{"score":4,"rationale":"Official policies locate authority with PIs, designated data managers, institutional owners, and records authorities, matching the proposal's approval structure.","source_ids":["S6","S8"]},"evidence_readiness":{"score":3,"rationale":"The proposal defines bounded comparators and outcomes, but no laboratory inventory, gold-standard dependencies, usability data, or baseline error rate is available.","source_ids":["S1","S2"]},"safety_net_benefit":{"score":4,"rationale":"Shadow mode, immutable snapshots, approval, tombstones, quarantine, and restoration tests are plausible safeguards, though their performance is untested.","source_ids":["S3","S6"]},"scalability":{"score":3,"rationale":"Standards and reusable schemas favor replication, but human classification, exception review, dependency validation, and restoration drills create recurring stewardship load.","source_ids":["S2","S5","S7","S8"]}},"score_confidence":"MODERATE","costs":{"first_evidence":{"band_2026_usd":"10K_TO_50K","scope":"One consenting laboratory; one completed task family with three to five revisions; participant data excluded; read-only inventory, owner-verified gold standard, three or more restore samples, and a 12–24-person blinded comparison.","confidence":"LOW","assumptions":["Approximately 6–12 person-weeks across a research-data specialist, research assistant, task owner, and part-time technical support.","Existing repository access and ordinary institutional sandbox infrastructure are available.","No production connector, automatic deletion, participant-data migration, or formal validation is included.","This is a resource-equivalent estimate, not a vendor quote."],"source_ids":["S5","S6","S8"]},"initial_deployment_startup":{"band_2026_usd":"50K_TO_250K","scope":"Productionize the registry for one laboratory: identity resolution, repository connector, role-based access, study bindings, audit logging, retention matrix, archival packaging, restore runbook, training, and institutional review.","confidence":"LOW","assumptions":["Approximately 0.5–1.5 FTE-years of combined engineering, data stewardship, governance, and laboratory-owner effort.","Existing OSF, Git, ELN, or institutional storage remains the system of record.","The registry manages metadata and protocol artifacts, not participant data.","Security and records review do not require a new enterprise platform procurement."],"source_ids":["S3","S5","S6","S8"]},"operational_launch":{"band_2026_usd":"250K_TO_1M","scope":"Institutional or multi-laboratory launch across roughly 3–10 laboratories, including connector hardening, migration support, access-control testing, governance templates, service ownership, monitoring, restore exercises, and user support.","confidence":"LOW","assumptions":["Two to six FTE-equivalent contributors during launch, including engineering, data stewardship, security, records, and training.","Heterogeneous repositories require configuration but not full replacement.","Restricted and participant data remain out of initial scope.","No cost reduction from storage tiering is credited."],"source_ids":["S6","S7","S8"]},"annual_recurring":{"band_2026_usd":"50K_TO_250K","scope":"Annual operation for a small institutional service: metadata quality review, exception revalidation, dependency sampling, access reviews, archive-restore drills, incident response, software maintenance, and user support.","confidence":"LOW","assumptions":["Approximately 0.5–1.5 recurring FTE plus ordinary storage and compute.","Artifact volumes are moderate and cold-storage costs are not dominant.","Human review remains mandatory for disposition.","Large-scale format migration or legal discovery is excluded."],"source_ids":["S5","S7","S8"]}},"verified_pipeline_gates":{"externally_supported_problem":{"status":"YES","reason":"Independent empirical and official sources support missing research materials, uncertain assessment versions and scoring, and consequential research-record lifecycle obligations, although exact laboratory mismatch prevalence is unknown.","source_ids":["S1","S2","S6","S7"]},"externally_credible_adopter_or_authorizer":{"status":"YES","reason":"NIH principal investigators and data managers, university PIs or delegates, research-data stewards, and institutional records authorities are identifiable authorizers; UCSF provides evidence of adoption and expressed value for exact protocol versions.","source_ids":["S4","S6","S8"]},"distinct_testable_incremental_claim":{"status":"YES","reason":"Correct package selection and restoration can be compared against both the current repository and a lightweight immutable manifest, with false staleness, missed dependencies, and failed restores as falsifiers.","source_ids":["S2","S3","S4","S5"]},"bounded_next_evidence_step":{"status":"YES","reason":"A read-only one-laboratory, one-task-family exercise can be completed without moving or deleting source artifacts and has explicit comparators, outcomes, and halt conditions.","source_ids":["S3","S5","S6"]},"no_unresolved_safety_or_authority_stop":{"status":"YES","reason":"The proposed first step excludes participant data and disposition, preserves source artifacts, and places decisions with the PI, steward, and applicable institutional authority. Approval and access-control verification remain prerequisites.","source_ids":["S6","S8"]},"credible_cost_scope_and_range":{"status":"UNCERTAIN","reason":"Broad resource-equivalent bands and scopes can be stated, but no direct labor study, integration estimate, vendor quote, artifact count, or institutional security assessment was available. Cost confidence is therefore low.","source_ids":["S6","S7","S8"]}},"next_evidence_step":"In one consenting laboratory, choose one completed cognitive-task family with three to five revisions and exclude participant data. Inventory approximately 30–150 protocol artifacts in a read-only shadow registry. Have the task owner and data steward independently establish a gold standard for identities, study bindings, lifecycle labels, known inbound references, and preservation constraints. Sample at least one archived package from each available age or format band and attempt end-to-end restoration in a sandbox. Then run a randomized crossover exercise with 12–24 blinded researchers using identical reconstruction prompts under three conditions: the existing repository interface, a lightweight immutable current-version manifest, and the lifecycle view. Measure correct complete-package selection, time to selection, unresolved-dependency count, restoration success, false-staleness rate, restricted-metadata exposure, and steward effort. Falsify the incremental claim if the lifecycle view does not outperform the lightweight manifest on package-selection correctness or reconstruction success, if it misses any gold-standard high-risk dependency, if restoration fails for a supposedly supported package, or if false labels or access exposure trigger a halt. Do not move, quarantine, or delete source artifacts.","blocking_evidence":["No direct prevalence estimate for unlabeled or mutually incompatible cognitive-task artifact versions inside laboratories.","No observed baseline rate of wrong-package selection or failed reconstruction for the proposed adopter population.","No comparative evidence that the full lifecycle view outperforms an immutable study manifest or existing version control.","No measured recall for inbound-dependency discovery or false-positive rate for semantic staleness labels.","No evidence that archived cognitive-task packages remain executable across actual age and format bands.","No local records, ethics, intellectual-property, licensing, or access-control review for the selected artifact family.","No measured stewardship burden, integration estimate, or vendor and labor cost evidence."],"research_disposition":"PARTNERED_RESEARCH_PROGRAM","world_novelty_boundary":"This evaluation measures neither world novelty nor patentability, freedom to operate, market size, or realized impact. The search establishes substantial collision with known versioning, assessment-schema, research-object packaging, registration, ELN, retention, and stewardship practices. It does not establish whether the exact end-to-end combination has previously been implemented or claimed.","arm":"COMPLETE_PROPOSAL_PORTFOLIO","candidate_version":0,"controller_recommendation":{"action":"STOP_EMPIRICAL_RESEARCH_NEEDED","repairable":false,"material_progress_observed":false,"progress_targets":["Measure the prevalence and severity of stale or incompatible protocol-layer selection in a bounded cognitive-task family.","Run the three-condition blinded comparison against the existing interface and a lightweight immutable manifest.","Validate dependency-tracing recall against an owner-and-steward gold standard.","Demonstrate sandbox restoration across all represented age and format bands.","Obtain institutional records, ethics, security, licensing, and authority sign-off for the selected artifact scope.","Replace resource-equivalent cost assumptions with measured staff time, artifact counts, connector effort, and storage requirements."],"reason":"Bounded web research supports the general problem, credible authorities, implementation ingredients, and extensive adjacent prior art, but it cannot determine whether the integrated lifecycle workflow improves package selection or reconstruction over a simple manifest. That remaining claim requires proprietary repository inspection and a live laboratory comparison, so the evaluation must stop for empirical research."},"proposal_index":1}