{"schema_version":1,"research_id":"eoa_inverse_innovation_exp06_external_evaluation_20260803","source_assessment_id":"representation_independent_interface_contract__futurism_foresight:P2:v0","cell_id":"representation_independent_interface_contract__futurism_foresight","search_queries":["Delphi study data management anonymity controlled feedback rounds public health guidance","Delphi reporting guideline CREDES anonymity controlled feedback rounds primary research","Delphi method software platform limitations spreadsheet data management study","government foresight Delphi method toolkit public health agency","Delphi survey software platform rounds anonymity controlled feedback official documentation DelphiManager","DelphiManager software official round feedback anonymity platform","Delphi study data management software lifecycle participant withdrawal late response","Delphi platform systematic review software features web Delphi","site:bls.gov Occupational Outlook Handbook software developers median pay 2024","site:eur-lex.europa.eu GDPR Article 5 pseudonymisation data minimisation Article 25 privacy by design official","site:comet-initiative.org DelphiManager data security privacy GDPR anonymous Study ID","site:cdisc.org ODM standard audit trail clinical research official","ACCORD Accurate Consensus Reporting Document 2024 PLOS Medicine DOI","CREDES guidance conducting reporting Delphi studies 2017 palliative medicine DOI","Delphi studies poor reporting anonymity feedback response rates systematic review primary paper","REDCap longitudinal survey management audit trails official documentation"],"sources":[{"source_id":"s1","title":"The Futures Toolkit HTML","publisher":"UK Government Office for Science, GOV.UK","url":"https://www.gov.uk/government/publications/futures-toolkit-for-policy-makers-and-analysts/the-futures-toolkit-html","source_class":"OFFICIAL_GUIDANCE","publication_date":"2024-08-20","accessed_at":"2026-08-03","claims_supported":["Delphi is an established government foresight method using multiple questionnaire rounds and anonymised expert participation.","The UK Food Standards Agency is an identifiable governmental user: it invited 300 participants, used an online platform for rounds, and coded material in Excel.","The FSA used the resulting assessment to focus its research programme, showing that Delphi outputs can affect agency priorities."]},{"source_id":"s2","title":"Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review","publisher":"Palliative Medicine / PubMed","url":"https://pubmed.ncbi.nlm.nih.gov/28190381/","source_class":"AUTHORITATIVE_SECONDARY","publication_date":"2017-02-13","accessed_at":"2026-08-03","claims_supported":["A methodological systematic review found substantial variation in the conduct quality and reporting transparency of Delphi studies.","Delphi-derived guidance can affect decisions relevant to patient care.","The review supports the need to make consequential methodological choices explicit, but does not document software-migration state divergence."]},{"source_id":"s3","title":"ACCORD (ACcurate COnsensus Reporting Document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi","publisher":"PLOS Medicine","url":"https://journals.plos.org/plosmedicine/article/file?id=10.1371%2Fjournal.pmed.1004326&type=printable","source_class":"PRIMARY_RESEARCH","publication_date":"2024-01-23","accessed_at":"2026-08-03","claims_supported":["Consensus methods are often poorly reported despite informing health decisions.","The ACCORD guideline was developed through a systematic review, three Delphi rounds, and finalization meetings.","Transparent reporting is needed so readers can judge the robustness and credibility of consensus recommendations."]},{"source_id":"s4","title":"eDelphi 2026","publisher":"eDelphi / Metodix","url":"https://www.edelphi.org/index.page","source_class":"COMMERCIAL_FIRST_PARTY","publication_date":"2026-08-03","accessed_at":"2026-08-03","claims_supported":["A mature Delphi-specific product already supports one-to-four-round and roundless variants, anonymised feedback, response revision, stopping criteria, and process management.","The product reports 25 years of development with futures-research institutions and worldwide use.","Existing products already implement much of the proposed workflow, but this source does not describe a vendor-independent behavioral conformance contract."]},{"source_id":"s5","title":"DelphiManager Frequently Asked Questions","publisher":"COMET Initiative, University of Liverpool","url":"https://www.comet-initiative.org/delphimanager/docs/DelphiManager%20-%20Frequently%20Asked%20Questions.pdf","source_class":"OFFICIAL_PRODUCT_DOCUMENTATION","publication_date":"2022","accessed_at":"2026-08-03","claims_supported":["DelphiManager permits previewing rounds before launch and supports previous-round personal scores and bespoke feedback.","It assigns participants unique Study IDs and exports outcome ratings against those IDs to maintain anonymity.","It reports encryption in transit and at rest and institutionally hosted storage, demonstrating practical implementation of several proposed privacy and lifecycle features."]},{"source_id":"s6","title":"ODM v2.0","publisher":"Clinical Data Interchange Standards Consortium (CDISC)","url":"https://www.cdisc.org/standards/data-exchange/odm-xml/odm-v2-0","source_class":"STANDARD","publication_date":"2023-08-23","accessed_at":"2026-08-03","claims_supported":["ODM is a vendor-neutral, platform-independent exchange and archival format for study data, metadata, administrative data, reference data, and audit information.","ODM can be serialized in XML, JSON, or other formats and provides a base structure for extensions.","ODM is close adjacent prior art for portable study records, but its published scope does not supply Delphi-specific transition laws, aggregate-population semantics, or a substitutability oracle."]},{"source_id":"s7","title":"Regulation (EU) 2016/679 (General Data Protection Regulation)","publisher":"European Union, EUR-Lex","url":"https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1640080938208&uri=CELEX%3A32016R0679","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2016-04-27","accessed_at":"2026-08-03","claims_supported":["Personal-data processing must satisfy purpose limitation, data minimisation, accuracy, storage limitation, security, and accountability.","Public-authority processing requires an applicable legal basis; research and statistical processing remain subject to safeguards.","Pseudonymisation and encryption can be safeguards, but pseudonymous handles do not by themselves remove controller obligations or establish lawful identity recovery."]},{"source_id":"s8","title":"Software Developers, Quality Assurance Analysts, and Testers","publisher":"U.S. Bureau of Labor Statistics","url":"https://www.bls.gov/ooh/computer-and-information-technology/software-developers.htm","source_class":"OFFICIAL_ORGANIZATION_DATA","publication_date":"2025-08-28","accessed_at":"2026-08-03","claims_supported":["The May 2024 median annual wage was $133,080 for software developers and $102,610 for software quality-assurance analysts and testers.","Software development includes requirements analysis, modeling, maintenance, and testing; QA work includes test plans, automated tests, risk identification, and defect reporting.","These wages provide a labor anchor for broad 2026 resource-equivalent bands, subject to inflation, benefits, overhead, and jurisdictional variation."]}],"problem_evidence":{"support":"MODERATE","rationale":"Government and biomedical sources establish that Delphi work is consequential, frequently spans an online platform and Excel, and exhibits substantial methodological and reporting variation. This makes preservation of anonymity, feedback, population, and round semantics credible and important. No relied-upon source directly documents a platform migration or spreadsheet-to-script transfer silently changing the authoritative Delphi state, so the proposal's specific failure mode remains an inference rather than an observed prevalence estimate.","source_ids":["s1","s2","s3"]},"stakeholder_evidence":{"support":"WEAK","rationale":"The UK Food Standards Agency is an identifiable governmental user and beneficiary of a Delphi exercise conducted across an online platform and Excel. Delphi-specific products and standards bodies are also plausible technical partners. However, no source records the FSA, a national public-health agency, a data-protection officer, or a funder explicitly requesting a representation-independent Delphi behavioral contract or committing resources to it.","source_ids":["s1","s4","s5"]},"prior_art":{"proximity":"ADJACENT_PRIOR_ART","closest_analogues":[{"name":"eDelphi","similarity":"Implements Delphi rounds, anonymous feedback, revision, stopping criteria, and multiple Delphi variants in a mature dedicated platform.","remaining_difference":"It is a particular managed implementation; the source does not expose a portable abstract state, cross-vendor transition laws, or a common black-box acceptance suite.","source_ids":["s4"]},{"name":"DelphiManager","similarity":"Implements round preview, participant Study IDs, anonymised exports, feedback features, encryption, and institutional study administration.","remaining_difference":"Its documentation describes one product's workflow and privacy controls rather than a representation-independent contract governing spreadsheets, survey vendors, and analysis services.","source_ids":["s5"]},{"name":"CDISC ODM 2.0","similarity":"Provides a vendor-neutral, platform-independent model for exchanging study data, metadata, administrative data, and audit information.","remaining_difference":"It standardizes representable study information and interchange, not the proposed Delphi-specific eligibility, closure, replacement, withdrawal, feedback-provenance, identity-access, and error behavior.","source_ids":["s6"]},{"name":"CREDES and ACCORD guidance","similarity":"Makes Delphi and consensus-method choices explicit to improve transparency, credibility, and appraisal.","remaining_difference":"Reporting guidance does not make implementations behaviorally substitutable or provide an executable operation-sequence oracle.","source_ids":["s2","s3"]}],"distinctive_claim_remaining":"For one sponsor-approved synthetic Delphi protocol, two independently built implementations governed by a shared representation-independent state contract will produce identical contract-level eligibility, active-response, withdrawal, closure, aggregation-membership, feedback-provenance, typed-error, and identity-access observations for every predeclared sequence, while seeded implementations that violate any one of those rules will be rejected. The claim is falsified if an approved workflow cannot be expressed, a seeded semantic mutant passes, conforming implementations disagree, or an allowed output reveals a predeclared identity-linkage channel.","confidence":"MODERATE"},"implementation_evidence":{"support":"MODERATE","rationale":"Existing Delphi products demonstrate that multiround workflows, anonymised identifiers, feedback, revision, stopping rules, exports, and encryption are technically implementable; ODM demonstrates platform-neutral study representation. The candidate's offline synthetic reference model and adapter test therefore appear feasible with ordinary software and QA skills. Evidence is absent for the harder parts: a complete Delphi semantic contract, property-based generators that discriminate realistic defects, leakage testing, adapter effort for actual vendors, and preservation of facilitator-approved methodological variants.","source_ids":["s4","s5","s6","s7","s8"]},"scores":{"meaningful_impact":{"score":4,"rationale":"Delphi findings can focus agency research and biomedical consensus can influence care-related decisions; preserving population, feedback, and anonymity semantics could protect interpretability. Realized impact and incident prevalence are unmeasured.","source_ids":["s1","s2","s3"]},"stakeholder_pull":{"score":2,"rationale":"A governmental Delphi user and mature product ecosystem are identifiable, but no prospective adopter or authorizer explicitly asks for this contract or offers sponsorship.","source_ids":["s1","s4","s5"]},"incremental_advantage":{"score":3,"rationale":"The proposed behavioral oracle addresses semantics that reporting guidance, single-platform products, and a data-exchange standard do not jointly govern. Its advantage over a carefully configured authoritative platform plus protocol remains untested.","source_ids":["s2","s3","s4","s5","s6"]},"distinctiveness_plausibility":{"score":3,"rationale":"No close Delphi-specific cross-implementation behavioral contract was found in the eight-source review, while several components are established separately. This supports a plausible compositional distinction, not world novelty.","source_ids":["s2","s3","s4","s5","s6"]},"technical_implementability":{"score":4,"rationale":"Mature products already implement most required operations and privacy controls, and platform-neutral research-data modeling exists. Contract completeness, semantic generators, and leakage detection remain engineering uncertainties rather than apparent impossibilities.","source_ids":["s4","s5","s6","s8"]},"adoption_authority_feasibility":{"score":3,"rationale":"A study sponsor can approve methodological semantics and an agency controller or data-protection function can govern identity handling, but the candidate names no jurisdiction or committed authority. Software designers cannot legitimately settle contested withdrawal and aggregation rules alone.","source_ids":["s1","s7"]},"evidence_readiness":{"score":4,"rationale":"A synthetic, offline two-implementation trial can be bounded, instrumented, compared, and falsified without enrolling experts. No executable contract, adapter, or seeded-mutant result currently exists.","source_ids":["s4","s5","s6","s8"]},"safety_net_benefit":{"score":4,"rationale":"Synthetic records, non-live questions, isolated identity mappings, and rejection of nonconforming adapters permit rollback without modifying an active study. The safety claim falls if supposedly anonymous outputs remain linkable.","source_ids":["s5","s7"]},"scalability":{"score":3,"rationale":"A shared suite could be reused across implementations, but each vendor and methodological variant would require an adapter and sponsor-approved semantic profile; eDelphi's support for multiple variants shows why one universal state model may be difficult.","source_ids":["s4","s5","s6"]}},"score_confidence":"MODERATE","costs":{"first_evidence":{"band_2026_usd":"10K_TO_50K","scope":"Four to eight weeks to predeclare one synthetic protocol, specify the minimal state model, build an in-memory model and spreadsheet adapter, implement scripted/generated tests and seeded mutants, and produce a leakage register.","confidence":"MODERATE","assumptions":["Approximately 160-400 blended hours from a software/QA engineer, Delphi method owner, and privacy reviewer.","2024 BLS wages are uplifted for 2026 inflation and 20%-50% benefits/overhead.","No real participants, proprietary vendor integration, production hosting, or formal certification is included."],"source_ids":["s8"]},"initial_deployment_startup":{"band_2026_usd":"50K_TO_250K","scope":"Create a sponsor-approved versioned contract, reference implementation, conformance harness, documentation, privacy threat model, and two production-quality adapters for one agency protocol family.","confidence":"LOW","assumptions":["Roughly 0.5-1.5 loaded staff-years across engineering, QA, facilitation/methodology, privacy, and governance.","Existing platforms expose sufficient export or API access; procurement and licensing are excluded.","The agency supplies methodological decisions and test fixtures."],"source_ids":["s4","s5","s6","s7","s8"]},"operational_launch":{"band_2026_usd":"50K_TO_250K","scope":"Validate one non-active historical or shadow study, train administrators, complete agency security/privacy review, establish identity custody and rollback procedures, and authorize the first production adapter set.","confidence":"LOW","assumptions":["No migration of an active authoritative study occurs before shadow validation.","Jurisdiction-specific legal review and institutional security testing are required.","Major vendor customization, participant recruitment, and enterprise procurement could move the cost above this band."],"source_ids":["s5","s7","s8"]},"annual_recurring":{"band_2026_usd":"10K_TO_50K","scope":"Maintain the contract and tests, rerun conformance after platform changes, review semantic version changes, refresh leakage checks, and support one or two protocol families.","confidence":"LOW","assumptions":["Approximately 0.1-0.3 loaded technical FTE plus periodic method-owner and privacy review.","No dedicated full-time service team, high-availability infrastructure, or broad multi-agency certification program is included.","Vendor license fees are excluded because no comparable required configuration or price was verified."],"source_ids":["s4","s5","s8"]}},"verified_pipeline_gates":{"externally_supported_problem":{"status":"YES","reason":"Official and research sources show consequential Delphi use across online-platform and spreadsheet representations and substantial variation or poor transparency in Delphi conduct. The exact migration-divergence prevalence remains unmeasured but the underlying preservation problem is externally credible.","source_ids":["s1","s2","s3"]},"externally_credible_adopter_or_authorizer":{"status":"UNCERTAIN","reason":"The FSA is a credible governmental Delphi user and a study sponsor/controller would have relevant authority, but no identified organization expresses demand for this specific contract or agrees to authorize a trial.","source_ids":["s1","s7"]},"distinct_testable_incremental_claim":{"status":"YES","reason":"The proposal makes a contrastive claim beyond dedicated platforms, reporting guidance, and ODM: independently implemented Delphi state engines should agree under a shared semantic suite and seeded violations should fail.","source_ids":["s2","s3","s4","s5","s6"]},"bounded_next_evidence_step":{"status":"YES","reason":"A time-boxed synthetic comparison of an in-memory model and spreadsheet adapter, with predeclared sequences, seeded semantic mutants, and leakage channels, supplies explicit comparators and falsifiers without live participants.","source_ids":["s4","s5","s6","s8"]},"no_unresolved_safety_or_authority_stop":{"status":"YES","reason":"The proposed first step can use only synthetic identities and non-live questions, so it need not process participant data or alter an authoritative study. Any later real-data use remains contingent on sponsor, controller, privacy, and jurisdictional approval.","source_ids":["s7"]},"credible_cost_scope_and_range":{"status":"YES","reason":"The four bands state included work and exclusions and are anchored to official developer and QA wages, although adapter access, vendor fees, and legal-review costs remain low-confidence variables.","source_ids":["s8"]}},"next_evidence_step":"Within six weeks, have a Delphi method owner predeclare one synthetic two-round protocol covering enrollment and eligibility change, duplicate submission, replacement, missing answers, late submission, withdrawal before and after closure, failed closure, two aggregation rules, controlled feedback, and attempted identity resolution. Independently implement an in-memory reference and spreadsheet-backed adapter. Run identical scripted and generated histories against both; compare only eligibility, active-response membership, withdrawal state, closure state, aggregate membership/value, feedback provenance, typed errors, allowed side effects, and identity-access outcomes. Add at least eight single-rule mutants and require the suite to reject all eight. Audit handles, ordering, timestamps, filenames, errors, and exports against a predeclared linkage watchlist. Falsify the intervention if any approved workflow is inexpressible, either implementation passes while disagreeing on a contract observation, any mutant survives, or an ordinary output resolves or predictably links an identity. The comparator is the same protocol reconstructed through the current spreadsheet/export procedure without the shared oracle; no real experts, active study, or authoritative record may be used.","blocking_evidence":["No direct incident study or prevalence estimate shows that Delphi platform migration or spreadsheet/script transfer changes lifecycle semantics in practice.","No named agency sponsor, data-protection authority, or funder has expressed demand for the proposed contract or committed to the synthetic trial.","No executable state contract, independent adapter pair, seeded-mutant result, or leakage-audit result exists.","The applicable jurisdiction, legal basis, consent language, retention rules, withdrawal policy, and authorized identity-recovery procedure are unspecified.","Cost bands are labor-equivalent estimates; vendor access, procurement, licensing, security accreditation, and integration costs were not verified."],"research_disposition":"PARTNERED_RESEARCH_PROGRAM","world_novelty_boundary":"This evaluation found adjacent reporting guidance, dedicated Delphi platforms, and a vendor-neutral study-data standard, but did not measure world novelty, patentability, freedom to operate, market size, or realized impact. Absence of a Delphi-specific behavioral conformance contract from these eight sources is not evidence that none exists elsewhere.","arm":"COMPLETE_PROPOSAL_PORTFOLIO","candidate_version":0,"controller_recommendation":{"action":"STOP_EMPIRICAL_RESEARCH_NEEDED","repairable":false,"material_progress_observed":true,"progress_targets":["Secure a named agency study sponsor and privacy/data-protection authority who explicitly endorse the problem and authorize the synthetic protocol.","Complete the six-week independent-model versus spreadsheet-adapter experiment with all predeclared semantic comparators and falsifiers.","Demonstrate that the suite rejects every seeded rule violation and that conforming implementations agree on every contract observation.","Resolve jurisdiction-specific authority, consent, withdrawal, retention, identity-recovery, and audit requirements before any real-data trial.","Replace labor-only cost assumptions with adapter-access confirmation and written implementation, security-review, and maintenance estimates."],"reason":"Web evidence supports consequential Delphi use, methodological variability, feasible implementation components, and adjacent products and standards. It does not establish adopter pull or the proposal's central empirical claim that heterogeneous implementations actually diverge and that the proposed contract detects and prevents those divergences. Resolving that claim requires building and live-running the bounded synthetic conformance experiment rather than further web research; under the controller rule, this requires an empirical-research stop with repairable set to false."},"proposal_index":2}