{"schema_version":1,"research_id":"eoa_inverse_innovation_exp06_external_evaluation_20260803","source_assessment_id":"predictive_residual_processing__futurism_foresight:P2:v0","cell_id":"predictive_residual_processing__futurism_foresight","search_queries":["Delphi method respondent burden repeated rounds response rate attrition study","Delphi method anchoring feedback previous response study","Real-Time Delphi participants revise answers online prior responses software","Delphi survey prefilled previous round answers change tracking","site:rand.org Delphi methodological guidance panels PDF","Delphi method attrition burden systematic review primary study response rounds","Delphi feedback anchoring effect experiment expert judgments","IDEA protocol Investigate Discuss Estimate Aggregate expert judgment protocol full paper","\"The effects of feeding back experts’ own initial ratings\" full text","S0169207018300025 own initial ratings randomized trial results Delphi","Delphi panel respondent burden questionnaire length attrition empirical study","Delphi fatigue number rounds response burden empirical evidence","site:pubmed.ncbi.nlm.nih.gov \"effects of feeding back experts\"","site:research.tue.nl \"effects of feeding back experts\"","site:repository.ubn.ru.nl \"feeding back experts\" Delphi","doi 10.1016/j.ijforecast.2018.01.001","site:ico.org.uk pseudonymisation guidance personal data research participants","site:edpb.europa.eu pseudonymisation guidelines official 2025","site:nist.gov privacy framework pseudonymization survey research data","\"own initial ratings\" Delphi \"randomized trial\" authors","\"providing experts’ initial ratings\" Delphi experiment reduced percentage","\"The effects of feeding back\" Delphi Beiderbeck","\"own initial ratings\" \"29%\" Delphi"],"sources":[{"source_id":"S1","title":"RAND Methodological Guidance for Conducting and Critically Appraising Delphi Panels","publisher":"RAND Corporation","url":"https://www.rand.org/content/dam/rand/pubs/tools/TLA3000/TLA3082-1/RAND_TLA3082-1.pdf","source_class":"OFFICIAL_GUIDANCE","publication_date":"2023","accessed_at":"2026-08-03","claims_supported":["Delphi conventionally repeats questions across anonymous rounds and returns individualized and group feedback.","Panelist fatigue, attrition, task complexity, question volume, and forced-consensus risk are recognized methodological concerns.","Even small panels generate substantial data, and individualized reports can require weeks of preparation.","RAND recommends pilots, explicit stopping criteria, missing-data handling, stability analysis, anonymity, diverse representation, and human-subject protections."]},{"source_id":"S2","title":"Proposal for a critical appraisal tool for studies using the Delphi method (DCAT)","publisher":"The BMJ","url":"https://www.bmj.com/content/391/bmj-2025-084509","source_class":"PRIMARY_RESEARCH","publication_date":"2025","accessed_at":"2026-08-03","claims_supported":["Panelist nonresponse and attrition are common in Delphi panels and can impair result quality.","Minimizing participation burden is identified as one strategy for retaining panelists.","Attrition and the handling of missing responses should be reported explicitly rather than silently carried forward."]},{"source_id":"S3","title":"The effects of feeding back experts’ own initial ratings in Delphi studies: A randomized trial","publisher":"International Journal of Forecasting / Elsevier","url":"https://www.sciencedirect.com/science/article/pii/S0169207018300025","source_class":"PRIMARY_RESEARCH","publication_date":"2018-04","accessed_at":"2026-08-03","claims_supported":["In a real-world Delphi, 139 first-round experts were randomized to feedback with or without their own initial ratings; reported dropout was 29% in both conditions.","Experts shown their own initial ratings changed opinions on fewer questionnaire items than experts not shown those ratings.","No between-condition difference was found in the increase in overall agreement.","Displaying a participant-specific baseline can therefore alter reconsideration behavior, directly supporting the proposal’s anchoring risk and need for a from-scratch comparator."]},{"source_id":"S4","title":"Investigate Discuss Estimate Aggregate for structured expert judgement","publisher":"Monash University / International Journal of Forecasting","url":"https://research.monash.edu/en/publications/investigate-discuss-estimate-aggregate-for-structured-expert-judg/","source_class":"PRIMARY_RESEARCH","publication_date":"2017-01","accessed_at":"2026-08-03","claims_supported":["The IDEA protocol already uses an initial estimate, discussion of evidence and reasoning, and a second private anonymous estimate.","The study found value in discussion for resolving linguistic uncertainty and sharing evidence, although some comparisons lacked statistical power.","IDEA is a close methodological comparator because it protects an independently reconsidered second estimate and preserves reasoning rather than merely optimizing entry effort."]},{"source_id":"S5","title":"Real-Time Delphi","publisher":"The Millennium Project","url":"https://www.millennium-project.org/real-time-delphi/","source_class":"OFFICIAL_ORGANIZATION_DATA","publication_date":"undated","accessed_at":"2026-08-03","claims_supported":["The Millennium Project identifies months-long multi-round questionnaires as an efficiency problem and has adopted roundless Real-Time Delphi systems.","Participants can revisit their own answers, view evolving group feedback, revise answers, and justify revisions.","This is direct first-party evidence of an identifiable foresight adopter and demand for more efficient iterative elicitation."]},{"source_id":"S6","title":"2nd round – Providing Controlled Feedback to the active participants in a Welphi process","publisher":"Welphi","url":"https://www.welphi.com/2nd-round-providing-controlled-feedback-to-the-active-participants-in-a-welphi-process/","source_class":"OFFICIAL_PRODUCT_DOCUMENTATION","publication_date":"2019","accessed_at":"2026-08-03","claims_supported":["Welphi already presents participants with prior-round answers preselected beside anonymous group feedback and comments.","Participants may keep prior answers by advancing through the questionnaire or unlock and change them.","The product claims time- and cost-efficient automation while retaining anonymity, iteration, feedback, and aggregation.","This substantially collides with the proposal’s basic baseline-plus-change interface, though not with its audit, reconstruction, drift, protected-bypass, and fallback package."]},{"source_id":"S7","title":"Survey Rounds","publisher":"Calibrum Surveylet","url":"https://calibrum.net/surveylethelp/survey_rounds.htm","source_class":"OFFICIAL_PRODUCT_DOCUMENTATION","publication_date":"undated","accessed_at":"2026-08-03","claims_supported":["Surveylet can copy all prior-round answers so panelists need only change responses they believe need revision.","Copied answers are counted as real responses even if panelists never enter the new round, demonstrating the exact missingness-versus-confirmation hazard the proposal addresses.","The product supports round comparisons and snapshots but warns that survey edits affect prior rounds and that versions cannot be restored, leaving a meaningful version-governance gap."]},{"source_id":"S8","title":"Pseudonymisation","publisher":"UK Information Commissioner’s Office","url":"https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/data-sharing/anonymisation/pseudonymisation/","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"undated; current on access","accessed_at":"2026-08-03","claims_supported":["Pseudonymized longitudinal participant data remains personal data when it can be linked back using additional information.","The linkage information must be held separately and securely, and re-identification risk must be assessed and mitigated.","Pseudonymisation can support research, data minimization, security, and privacy by design but does not itself establish a lawful basis or eliminate consent, transparency, access-control, and risk-assessment duties.","Distinctive residuals and rationales can remain identifying even after direct identifiers are removed."]}],"problem_evidence":{"support":"MODERATE","rationale":"Authoritative guidance and recent methodological work visibly establish fatigue, attrition, participation burden, missing-data ambiguity, forced-consensus risk, and large facilitator data volumes in repeated Delphi rounds. The randomized trial also shows that displaying prior answers changes revision behavior. However, no located source measures the candidate’s narrower prevalence claim: the share of probability, timing, confidence, dependency, and rationale fields that remain stable in strategic-foresight Delphis, or the facilitator time specifically spent rediscovering changes.","source_ids":["S1","S2","S3","S5"]},"stakeholder_evidence":{"support":"MODERATE","rationale":"The Millennium Project is an identifiable foresight adopter that explicitly describes conventional multi-round duration as a problem and uses Real-Time Delphi. Welphi and Surveylet product documentation demonstrate demand for prefilled, revisable, lower-effort rounds, while RAND explicitly values burden reduction and automated individualized reports. None expresses demand for this proposal’s full governed package or commits to funding a trial.","source_ids":["S1","S5","S6","S7"]},"prior_art":{"proximity":"SUBSTANTIAL_COLLISION","closest_analogues":[{"name":"Surveylet Copy Last Round (with Answers)","similarity":"Copies every prior response into the next Delphi round so panelists only change answers when needed—the proposal’s core residual-entry behavior.","remaining_difference":"It does not require active confirmation, treat non-entry as missing, freeze an independently versioned prediction, verify semantic reconstruction, conceal from-scratch audit items, protect high-stakes fields, or trigger governed full-response fallback. Its documentation explicitly says copied values count as responses without panelist entry.","source_ids":["S7"]},{"name":"Welphi controlled-feedback second round","similarity":"Shows each participant’s prior answers preselected, provides anonymous group statistics and comments, and lets the participant keep or edit answers.","remaining_difference":"It uses the prior answer rather than a declared participant-specific next-response predictor and lacks the proposal’s independent scratch audit, checksum/version gate, scheduled full reset, drift monitoring, consequence bypass, and explicit nonresponse state.","source_ids":["S6"]},{"name":"Real-Time Delphi","similarity":"Eliminates conventional rounds and lets participants revisit, revise, and justify their own answers while seeing evolving group feedback, directly targeting duration and coordination cost.","remaining_difference":"It is continuous revision rather than reconstructive residual reporting and does not establish concealed baseline-free audit blocks, prediction-version synchronization, or residual-versus-full crossover controls.","source_ids":["S5","S1"]},{"name":"IDEA protocol","similarity":"Uses a first estimate, structured discussion, and a second private anonymous estimate, protecting independent reconsideration and explicit reasoning.","remaining_difference":"It recollects the second estimate rather than encoding only changes against a predicted participant baseline, so it does not claim entry-time savings or residual reconstruction.","source_ids":["S4"]}],"distinctive_claim_remaining":"Relative to both full re-entry and ordinary copy-forward Delphi, a frozen, versioned participant-proposition predictor combined with mandatory active confirmation, complete-response reconstruction, concealed from-scratch audits, protected full-response classes, explicit missingness, and triggered full fallback will reduce total expert-plus-facilitator-plus-maintenance time by at least 20% versus full re-entry without reducing independently reconsidered material revisions, rationale quality, disagreement or minority-position retention, and without increasing semantic reconstruction error or anchoring beyond preregistered noninferiority margins.","confidence":"HIGH"},"implementation_evidence":{"support":"MODERATE","rationale":"Existing Delphi products already implement authentication, anonymity, prior-answer display, copy-forward, editing, comments, rounds, snapshots, and analytics; the incremental prototype is therefore technically straightforward. RAND guidance supports pilots, individualized feedback, stability and missingness analysis, while ICO guidance makes a privacy-compliant pseudonymous architecture feasible with separate linkage keys, access controls, lawful basis, transparency, and risk assessment. Evidence is absent for automated semantic reconstruction of rationales, valid prediction rules beyond carrying forward the last answer, concealed audit logistics, threshold calibration, anonymity under distinctive residuals, and operator workload at scale.","source_ids":["S1","S6","S7","S8"]},"scores":{"meaningful_impact":{"score":3,"rationale":"Reducing repeated elicitation and comparison effort could improve retention and leave more attention for material reasoning, but the affected workflow is bounded and much of the efficiency is already available through copy-forward or real-time Delphi.","source_ids":["S1","S2","S5","S6","S7"]},"stakeholder_pull":{"score":3,"rationale":"Foresight and Delphi organizations visibly seek faster, lower-burden elicitation and already adopt relevant tools, but no adopter has requested or funded the governed residual package.","source_ids":["S1","S5","S6","S7"]},"incremental_advantage":{"score":2,"rationale":"Ordinary copy-forward already removes re-entry of stable answers. The remaining potential advantage is safer missingness, audit, dissent protection, reconstruction, and fallback—not the basic time-saving interface—and must be measured against the added governance cost.","source_ids":["S6","S7"]},"distinctiveness_plausibility":{"score":2,"rationale":"The integrated governance bundle is contrastive, but its central interaction substantially collides with established product features and its safeguards are familiar research and data-governance practices assembled around that interaction.","source_ids":["S1","S6","S7","S8"]},"technical_implementability":{"score":4,"rationale":"Round copying, editing, anonymous feedback, participant records, and analytics are existing product capabilities; version checks, active confirmation, audit assignment, and reconstruction checks are conventional software additions. Semantic rationale comparison remains harder.","source_ids":["S6","S7","S8"]},"adoption_authority_feasibility":{"score":3,"rationale":"A Delphi method owner can authorize a non-decision pilot and experts can retain control over their answers, but institutional ethics/privacy review, participant consent, and independent governance are needed and no named sponsor has committed.","source_ids":["S1","S5","S8"]},"evidence_readiness":{"score":3,"rationale":"A bounded randomized crossover is well specified and existing platforms can host it, but the decisive outcomes require recruiting experts and collecting new behavioral and workload data.","source_ids":["S1","S3","S6","S7"]},"safety_net_benefit":{"score":4,"rationale":"Explicit missingness, concealed scratch audits, protected full responses, participant-requested fallback, versioning, and complete-response verification directly address hazards exposed by copy-forward and own-rating feedback. Benefit remains prospective until tested.","source_ids":["S3","S7","S8"]},"scalability":{"score":3,"rationale":"Software automation is scalable, but participant-specific baselines, rationale review, audit sampling, privacy controls, and frequent fallback may create costs proportional to panel size and proposition count.","source_ids":["S1","S6","S7","S8"]}},"score_confidence":"MODERATE","costs":{"first_evidence":{"band_2026_usd":"50K_TO_250K","scope":"One preregistered, two-round, non-decision-bearing crossover with approximately 48–72 paid experts, 24 mock propositions, three round-two conditions, concealed audit blocks, blinded rationale scoring, privacy/ethics review, analysis, and a lightweight portal extension.","confidence":"LOW","assumptions":["Uses an existing survey/Delphi platform rather than building a production system.","Includes expert honoraria, facilitator and analyst labor, limited engineering, independent scoring, and ethics/privacy work.","Resource-equivalent estimate, not a vendor quote; no direct 2026 price source was located."],"source_ids":["S1","S3","S6","S7"]},"initial_deployment_startup":{"band_2026_usd":"250K_TO_1M","scope":"Production-grade module for one organization: baseline/version service, active confirmation and reconstruction UI, missingness state, audit randomization, protected-field policy, fallback workflow, pseudonymous identity store, access controls, monitoring, testing, and governance documentation.","confidence":"LOW","assumptions":["Approximately 3–8 full-time-equivalent months across product, engineering, research methods, privacy/security, and quality assurance.","Integrates with an existing Delphi platform and identity/security environment.","Excludes acquisition of a commercial platform, enterprise-wide data migration, and regulated clinical validation."],"source_ids":["S1","S6","S7","S8"]},"operational_launch":{"band_2026_usd":"50K_TO_250K","scope":"Configure and run the first production, low-consequence Delphi after successful testing, including proposition classification, threshold approval, participant onboarding, training, privacy review, independent audit, facilitation, and post-round evaluation.","confidence":"LOW","assumptions":["One panel of roughly 30–100 experts and up to 50 propositions.","No live safety-critical or legally determinative decision use.","Expert compensation and facilitation intensity vary materially by domain."],"source_ids":["S1","S5","S8"]},"annual_recurring":{"band_2026_usd":"250K_TO_1M","scope":"Operate 4–8 panels per year with platform support, facilitator and audit labor, security/privacy administration, predictor and threshold review, expert honoraria, full-response resets, incident handling, and annual independent evaluation.","confidence":"LOW","assumptions":["One organization with a continuing foresight program rather than a single study.","Safeguards are retained even if they reduce the apparent efficiency gain.","Does not include consequential downstream strategy implementation or organization-wide foresight staffing."],"source_ids":["S1","S5","S6","S8"]}},"verified_pipeline_gates":{"externally_supported_problem":{"status":"YES","reason":"Independent guidance and research establish repeated-round fatigue, attrition, burden, missing-data concerns, forced-consensus risk, and high analysis volume, although stable-field prevalence in foresight remains unmeasured.","source_ids":["S1","S2","S3"]},"externally_credible_adopter_or_authorizer":{"status":"YES","reason":"The Millennium Project is an identifiable foresight adopter that moved to Real-Time Delphi to address months-long multi-round processes; RAND and commercial Delphi method owners are credible pilot authorizers. Demand is for efficiency generally, not yet for this exact package.","source_ids":["S1","S5","S6","S7"]},"distinct_testable_incremental_claim":{"status":"YES","reason":"The package can be compared against both full re-entry and established copy-forward, with preregistered measures of total effort, independent revision, reconstruction, rationale quality, anchoring, dissent retention, missingness, fallback, and privacy incidents.","source_ids":["S3","S6","S7"]},"bounded_next_evidence_step":{"status":"YES","reason":"A two-round mock-proposition crossover with fixed sample, conditions, outcomes, margins, halt rules, and no live decision use is bounded and executable.","source_ids":["S1","S3","S6","S7"]},"no_unresolved_safety_or_authority_stop":{"status":"YES","reason":"No categorical prohibition was found for a consensual, non-decision research pilot. Proceeding still requires the method owner’s authorization, applicable ethics review, lawful personal-data basis, transparency, separate identity keys, access controls, and immediate full-response rollback.","source_ids":["S1","S8"]},"credible_cost_scope_and_range":{"status":"UNCERTAIN","reason":"Scopes and labor assumptions can be bounded, but no direct quotes, platform integration estimate, expert-honorarium schedule, or observed audit/fallback workload was found. Bands are resource-equivalent planning estimates only.","source_ids":["S1","S6","S7","S8"]}},"next_evidence_step":"Pre-register one two-round, non-decision-bearing crossover with 48–72 experts and 24 fixed mock long-horizon propositions. After a common full first round, randomize matched proposition blocks within participant to: (A) complete full re-entry, (B) ordinary prior-answer copy-forward, or (C) the governed residual protocol. Freeze all round-two baselines before feedback; require active confirmation in C; collect a concealed baseline-free answer on a random 20% audit subset before revealing prior values; and blind evaluators of rationale quality and minority-position retention. Primary success requires at least 20% lower total expert, facilitator, maintenance, audit, and fallback minutes than A, with a confidence interval excluding zero, and a measurable advantage over B after safeguard costs. Noninferiority margins should include no more than a 5-percentage-point reduction in independently reconsidered material revisions or dissent retention, no more than 0.25 standard deviations lower blinded rationale quality, at least 99.5% semantic reconstruction agreement and 100% exact structured-field reconstruction after confirmation, and fewer than 2% audit-discovered material changes absent from residual submissions. Falsify the claim if C is not faster after all costs, is not incrementally safer or more accurate than B, increases anchoring beyond the margin, ambiguates any nonresponse, loses a protected/minority rationale, causes an anonymity or version failure, or requires fallback for more than 15% of in-scope responses. Halt immediately on any wrong-baseline reconstruction, protected-field compression, re-identification event, or use of outputs in a live decision.","blocking_evidence":["No field-level estimate of how much foresight-Delphi content remains stable between rounds.","No measured expert or facilitator time saved beyond what ordinary copy-forward already provides.","No evidence that baseline prediction beyond simply carrying forward the last answer improves net effort or fidelity.","No crossover evidence on anchoring, independent reconsideration, rationale quality, disagreement, or minority-position retention.","No observed semantic reconstruction error, concealed-audit miss rate, fallback rate, or maintenance workload.","No adopter commitment, procurement signal, or named funder for the governed package.","No jurisdiction-specific ethics, consent, retention, cross-border transfer, or records-management determination.","No vendor quote or bottom-up engineering estimate supporting the 2026 cost bands."],"research_disposition":"PARTNERED_RESEARCH_PROGRAM","world_novelty_boundary":"This assessment establishes only that conventional Delphi, Real-Time Delphi, Welphi prior-answer confirmation/editing, Surveylet copy-forward, and IDEA substantially overlap the proposal. It does not establish world novelty, patentability, freedom to operate, market size, realized impact, the absence of unpublished or proprietary implementations, or the absence of relevant non-English practices. Patent databases, source code, procurement records, and proprietary platform behavior were not examined.","arm":"COMPLETE_PROPOSAL_PORTFOLIO","candidate_version":0,"controller_recommendation":{"action":"STOP_EMPIRICAL_RESEARCH_NEEDED","repairable":false,"material_progress_observed":true,"progress_targets":["Complete the preregistered three-condition crossover and publish condition-level expert, facilitator, maintenance, audit, and fallback minutes.","Demonstrate at least 20% net effort reduction versus full re-entry and a positive incremental advantage versus ordinary copy-forward after all safeguard costs.","Meet preregistered noninferiority margins for independent revision, anchoring, rationale quality, dissent and minority retention, reconstruction, missingness, anonymity, and protected-signal preservation.","Obtain a documented pilot authorization, ethics/privacy determination, lawful-basis and consent plan, data-retention schedule, and independent audit owner.","Replace resource-equivalent cost estimates with an observed pilot cost ledger and a bottom-up production estimate.","Stop the concept if copy-forward performs equivalently, if safeguards erase the efficiency gain, or if any anchoring, semantic-loss, missingness, anonymity, or protected-signal falsifier is triggered."],"reason":"Web research materially narrowed the proposal to a testable governance advantage over established copy-forward practice and identified a credible adopter class, but it cannot answer the decisive incremental-effect claim. Existing products already implement the core prior-answer-plus-change interaction, while a randomized Delphi trial shows that displaying one’s prior rating can suppress changes. Only participant fieldwork and live workflow measurement can determine whether the proposed audits, reconstruction, and fallback preserve independent reasoning while producing net savings."},"proposal_index":2}