{"schema_version":1,"research_id":"eoa_inverse_innovation_exp06_external_evaluation_20260803","source_assessment_id":"catalytic_pathway_enablement__futurism_foresight:P2:v0","cell_id":"catalytic_pathway_enablement__futurism_foresight","search_queries":["site:fema.gov emergency management planning expert judgment uncertainty scenario planning guidance","site:gao.gov strategic foresight scenario planning local government emergency management challenges","Delphi method emergency preparedness disaster risk management expert elicitation study","standing expert panel Delphi emergency management repeated elicitation","RAND ExpertLens official modified Delphi platform expert elicitation","site:rand.org Delphi expert elicitation anonymity controlled feedback RAND","site:epa.gov expert elicitation guidance uncertainty experts protocol","site:gov.uk futures toolkit Delphi expert elicitation government office science","multi county emergency management consortium strategic foresight cascading infrastructure risks regional planning official","site:marc.org emergency preparedness regional hazard mitigation infrastructure interdependencies","site:mwcog.org emergency preparedness council regional risk scenario planning expert","regional emergency management consortium cascading risk scenario planning uncertainty","site:rand.org ExpertLens system eliciting opinions large pool non-collocated experts diverse knowledge","site:rand.org ExpertLens online modified Delphi platform how it works","Delphi panel attrition response rate systematic review methodological quality primary research","expert elicitation confidentiality anonymity small panel identification risk guidance"],"sources":[{"source_id":"S1","title":"Final Report Summary—CASCEFF: Modelling of Dependencies and Cascading Effects for Emergency Management in Crisis Situations","publisher":"European Commission CORDIS","url":"https://cordis.europa.eu/project/id/607665/reporting/it","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2018-07-25","accessed_at":"2026-08-03","claims_supported":["Cascading incidents create high indeterminacy because knowledge and authority are fragmented across systems and organizations.","Collecting relevant information from many actors and disciplines is difficult and requires a facilitating structure.","Preparedness benefits from relationships, common protocols, terminology alignment, structured scenarios, and cross-disciplinary cooperation established before incidents.","A related structured methodology was tested through tabletop exercises against a baseline, showing that bounded comparative testing is feasible."]},{"source_id":"S2","title":"Emergency Preparedness Council","publisher":"Metropolitan Washington Council of Governments","url":"https://www.mwcog.org/committees/ncr-emergency-preparedness-council/","source_class":"OFFICIAL_ORGANIZATION_DATA","publication_date":"n.d.","accessed_at":"2026-08-03","claims_supported":["A real regional emergency-preparedness council coordinates governments, emergency managers, transportation, public health, private-sector, nonprofit, and federal participants.","The council oversees a regional coordination plan, working-group relationships, training, and tests.","The council can add institutions and individuals to working groups, making it a credible prospective authorizer or host for a non-operational probe."]},{"source_id":"S3","title":"The Futures Toolkit","publisher":"UK Government Office for Science","url":"https://www.gov.uk/government/publications/futures-toolkit-for-policy-makers-and-analysts/the-futures-toolkit-html","source_class":"OFFICIAL_GUIDANCE","publication_date":"2024-08-29","accessed_at":"2026-08-03","claims_supported":["Delphi is an established foresight tool for eliciting agreement and disagreement through repeated questionnaires.","Guidance calls for a varied panel, careful response-rate monitoring, and cross-disciplinary participation.","A typical exercise uses one or two organizers, approximately 12–16 experts, and may take three to six months.","The UK Food Standards Agency used an online Delphi process with participants from government, academia, industry, and the third sector."]},{"source_id":"S4","title":"RAND Methodological Guidance for Conducting and Critically Appraising Delphi Panels","publisher":"RAND Corporation","url":"https://www.rand.org/pubs/tools/TLA3082-1.html","source_class":"OFFICIAL_GUIDANCE","publication_date":"2023-12-29","accessed_at":"2026-08-03","claims_supported":["Delphi is an iterative, anonymous, structured expert-elicitation process for decisions under uncertainty and incomplete information.","Repeated questions and controlled feedback are established practice rather than a new mechanism.","Delphi is widely used for forecasting, probability estimation, policy prioritization, and stakeholder engagement.","RAND provides a prospective critical-appraisal tool for designing and assessing such panels."]},{"source_id":"S5","title":"How CFA Uses the Expert Elicitation Process to Inform Our Work","publisher":"U.S. Centers for Disease Control and Prevention","url":"https://www.cdc.gov/cfa-qualitative-assessments/php/about/expert-elicitation-methods.html","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2025-12-16","accessed_at":"2026-08-03","claims_supported":["A standing government forecasting center already convenes cross-disciplinary experts for questions with insufficient data.","Its outputs include scenario estimates, uncertainty, disagreement, and rationales used as inputs to analysis and modeling.","Protocol design, recruitment, briefing material, structured feedback, and aggregation are existing institutional practices.","This is a close institutional analogue to the proposed relay, although it does not establish the proposal's claimed multi-cycle regeneration advantage."]},{"source_id":"S6","title":"Peer Review of Expert Elicitation","publisher":"U.S. Environmental Protection Agency / RTI International","url":"https://www3.epa.gov/ttnecas1/regdata/Benefits/memo_7.30.04.pdf","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2004-07-30","accessed_at":"2026-08-03","claims_supported":["Defensible elicitation requires expert-selection criteria, breadth of views, documentation, clear framing, briefing, bias controls, and careful encoding.","Reviewers cautioned that no universally agreed method exists for combining expert judgments and sometimes preferred retaining individual distributions.","Pre- and post-elicitation interaction can improve conditioning and reconsideration, but consensus can be artificial or unnecessary.","The source identifies motivational bias, anchoring, representativeness, and aggregation as implementation risks."]},{"source_id":"S7","title":"Using the Delphi Technique to Determine Which Outcomes to Measure in Clinical Trials: Recommendations for the Future Based on a Systematic Review of Existing Studies","publisher":"PLOS Medicine","url":"https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1000393","source_class":"PRIMARY_RESEARCH","publication_date":"2011-01-25","accessed_at":"2026-08-03","claims_supported":["Anonymous sequential questionnaires can reduce domination relative to round-table discussion.","Panel composition, question design, information supplied, feedback, interaction, and consensus definitions materially affect validity.","Published Delphi studies often inadequately report anonymity, feedback, distributions, and attrition.","Minority-view attrition can create false consensus, and aggregate cutoffs can conceal strong disagreement."]},{"source_id":"S8","title":"How to Choose? Using the Delphi Method to Develop Consensus Triggers and Indicators for Disaster Response","publisher":"Cambridge University Press / Disaster Medicine and Public Health Preparedness","url":"https://www.cambridge.org/core/journals/disaster-medicine-and-public-health-preparedness/article/abs/how-to-choose-using-the-delphi-method-to-develop-consensus-triggers-and-indicators-for-disaster-response/D008562679BDEFCC90FE30A82E7BF430","source_class":"PRIMARY_RESEARCH","publication_date":"2017-02-03","accessed_at":"2026-08-03","claims_supported":["A regional Washington healthcare-response network used classic and modified Delphi processes with clinicians, public-health officials, and healthcare coalitions.","The process produced standardized regional and state-level emergency-response triggers and indicators.","The study demonstrates direct emergency-management prior art for cross-institutional expert elicitation supporting regional decisions.","Not all candidate items reached consensus, reinforcing the need to preserve distributions and disagreement."]}],"problem_evidence":{"support":"STRONG","rationale":"CORDIS directly documents that cascading incidents involve fragmented knowledge, high indeterminacy, difficult multi-actor coordination, terminology barriers, and a need for pre-established relationships and protocols. Regional disaster-response research demonstrates that cross-institutional Delphi elicitation addresses actual emergency-planning questions. The evidence supports the general problem, but it does not establish the candidate's local prevalence claims, coordinator-hour burden, or frequency of eligible long-range questions in any named consortium.","source_ids":["S1","S8"]},"stakeholder_evidence":{"support":"MODERATE","rationale":"The Metropolitan Washington Council of Governments' Emergency Preparedness Council is an identifiable regional body with authority over coordination-plan development, working-group relationships, training, tests, and participant expansion. CORDIS expresses a need for structures that facilitate transdisciplinary coordination. Neither source records a request, budget commitment, or stated preference for this specific standing relay, so adopter credibility is stronger than demonstrated pull.","source_ids":["S1","S2"]},"prior_art":{"proximity":"ESTABLISHED_PRACTICE","closest_analogues":[{"name":"Institutional Delphi and online expert-elicitation services","similarity":"RAND documents a mature, structured, anonymous, iterative method with controlled feedback, panel selection, implementation guidance, and prospective appraisal—the proposal's core elicitation cycle.","remaining_difference":"The proposal adds an explicit regional emergency-management service wrapper with maintained relationships, burden monitoring, conflict refresh, recovery, and a claimed second-cycle reuse advantage.","source_ids":["S3","S4"]},{"name":"CDC Center for Forecasting and Outbreak Analytics expert elicitation","similarity":"A standing public forecasting organization convenes cross-disciplinary experts to answer data-poor scenario questions and reports estimates, uncertainty, disagreement, and rationales for modeling.","remaining_difference":"The public description does not establish a maintained regional roster, formal regeneration assay, participant-rest rules, or comparative evidence that later cycles reduce setup burden.","source_ids":["S5"]},{"name":"Northwest Healthcare Response Network disaster-response Delphi","similarity":"A regional emergency-response network used classic and modified Delphi panels across clinical, public-health, and coalition stakeholders to develop standardized decision inputs.","remaining_difference":"It concerns emergency-response triggers rather than recurring long-range cascading-risk distributions and does not test a standing relay against repeated ad hoc panels.","source_ids":["S8"]},{"name":"CascEff Incident Evolution Methodology and Tool","similarity":"CascEff addresses cascading-risk indeterminacy through a reusable structured method, cross-disciplinary information collection, advance relationships, common protocols, and scenario work.","remaining_difference":"It is a scenario/dependency methodology and software tool, not an anonymous expert-judgment relay preserving individual distributions and dissent.","source_ids":["S1"]}],"distinctive_claim_remaining":"For a stable class of eligible regional cascading-risk questions, reusing a governed convener's relationships, participation terms, translation capability, and elicitation infrastructure will reduce second-and-later-cycle setup elapsed time and coordinator touch time relative to matched ad hoc consultation, without reducing perspective coverage, dissent retention, traceability, confidentiality, participant welfare, or output usability. Failure occurs if savings disappear after fully counting relay maintenance and participant labor, or if any quality, privacy, neutrality, or recovery threshold is breached.","confidence":"HIGH"},"implementation_evidence":{"support":"MODERATE","rationale":"The elicitation workflow is technically conventional: bounded questions, diverse panel selection, sequential surveys, controlled feedback, distributions, rationales, and audit documentation all have established guidance and operational examples. A regional council has plausible authority to conduct training or tests and enlist participants. Principal unresolved issues are organizational rather than technical: jurisdiction-specific public-records and procurement rules, research/ethics classification, compensation authority, secure identity separation, small-specialty reidentification, sponsor influence, attrition, and the untested capacity to refresh relationships without proportional recurring labor. These issues can be contained in an archived-question shadow probe but require local review before invitations.","source_ids":["S2","S3","S4","S5","S6","S7"]},"scores":{"meaningful_impact":{"score":3,"rationale":"Cascading-risk indeterminacy and fragmented knowledge matter for regional preparedness, but the candidate provides a non-decisive planning input and neither local delay prevalence nor downstream decision improvement is quantified.","source_ids":["S1","S8"]},"stakeholder_pull":{"score":3,"rationale":"A clearly identifiable regional council has relevant coordination, training, testing, and membership authority, but no source shows that it requested or funded this relay.","source_ids":["S1","S2"]},"incremental_advantage":{"score":2,"rationale":"The proposed advantage is a plausible reduction in repeated setup burden, but no source compares successive standing-relay cycles with matched ad hoc panels after counting maintenance and expert burden.","source_ids":["S3","S4","S5","S8"]},"distinctiveness_plausibility":{"score":2,"rationale":"Delphi, structured elicitation, online facilitation, regional emergency panels, and standing government elicitation are established. The remaining distinction is a narrow operating-and-regeneration claim rather than a new method.","source_ids":["S3","S4","S5","S8"]},"technical_implementability":{"score":4,"rationale":"The workflow can be implemented with standard survey, secure communications, facilitation, and records systems; authoritative guidance supplies the core protocol. Human-capacity and governance controls remain substantial.","source_ids":["S3","S4","S5","S6"]},"adoption_authority_feasibility":{"score":3,"rationale":"The NCREPC has credible regional planning and test authority and can add institutions or individuals, but actual budget, procurement, records, privacy, and compensation authority were not verified.","source_ids":["S2"]},"evidence_readiness":{"score":3,"rationale":"A safe archived-question comparison can be bounded and preregistered, and precedent exists for tabletop baseline testing. Local baseline data, partner commitment, and valid matching are not yet available.","source_ids":["S1","S4","S6","S7"]},"safety_net_benefit":{"score":2,"rationale":"Better cascading-risk planning may benefit populations dependent on public infrastructure, and the design can include community perspectives, but no source establishes a differential benefit for underserved or high-vulnerability groups.","source_ids":["S1","S5","S7"]},"scalability":{"score":3,"rationale":"Digital multi-round elicitation is geographically scalable, but panel recruitment, organizer time, expert burden, attrition, confidentiality, and representation impose human limits that automation does not remove.","source_ids":["S3","S4","S7"]}},"score_confidence":"MODERATE","costs":{"first_evidence":{"band_2026_usd":"10K_TO_50K","scope":"Six-to-eight-week local process audit and probe design: verify question volume and delay causes, interview planners and records/procurement staff, inventory prior consultations, define the eligible question class, and preregister matching, metrics, and halt criteria.","confidence":"LOW","assumptions":["Approximately 0.15–0.35 fully loaded FTE-year across planning, facilitation, evaluation, privacy/records, and administration.","Uses existing records and systems; excludes participant recruitment and live elicitation.","Resource-equivalent estimate, not a vendor quote."],"source_ids":["S2","S3","S4"]},"initial_deployment_startup":{"band_2026_usd":"50K_TO_250K","scope":"Build and validate the minimum relay: governance charter, question interface, panel and conflict database, participation and compensation terms, secure survey workflow, anonymization tests, reviewer rubric, training, and shadow-probe preparation.","confidence":"LOW","assumptions":["Uses commercial or existing government survey and identity-management tools rather than custom software.","Includes legal, privacy, records, accessibility, and procurement review.","Includes setup for a small 12-person compensated volunteer panel but not a full operating year."],"source_ids":["S2","S3","S4","S6"]},"operational_launch":{"band_2026_usd":"250K_TO_1M","scope":"One-year bounded launch with a relay lead, analyst/facilitator, part-time administration and security/records support, expert compensation, independent quality review, and several non-operational or advisory question cycles.","confidence":"LOW","assumptions":["Approximately 1.5–3.0 fully loaded staff equivalents plus specialist support.","Panels generally contain 12–16 experts and several rounds lasting up to three to six months.","No custom high-assurance platform development or operational emergency-response reliance."],"source_ids":["S3","S4","S5"]},"annual_recurring":{"band_2026_usd":"250K_TO_1M","scope":"Maintain a small regional relay: staffing, expert compensation, recruitment and relationship refresh, accessibility and translation, secure tooling, conflict and records updates, protocol testing, independent audits, incident remediation, and contributor rotation.","confidence":"LOW","assumptions":["Several overlapping question cycles per year within explicit participant-burden caps.","Expert time remains a paid cofactor rather than being treated as free reusable capacity.","Costs rise materially if the service requires 24/7 availability, sensitive operational data, custom hosting, or broad community representation."],"source_ids":["S3","S4","S7"]}},"verified_pipeline_gates":{"externally_supported_problem":{"status":"YES","reason":"External project evidence directly identifies cascading-risk indeterminacy, fragmented cross-disciplinary knowledge, terminology barriers, and the need for facilitating structures and pre-established relationships.","source_ids":["S1","S8"]},"externally_credible_adopter_or_authorizer":{"status":"YES","reason":"The NCREPC is a real regional emergency-preparedness body empowered to oversee coordination planning, working-group relationships, training, tests, and participant expansion.","source_ids":["S2"]},"distinct_testable_incremental_claim":{"status":"YES","reason":"The proposal can test second-cycle setup and coordinator-effort reductions against matched ad hoc consultation while holding panel breadth, compensation, safeguards, and output criteria constant.","source_ids":["S3","S4","S7"]},"bounded_next_evidence_step":{"status":"YES","reason":"Four archived, non-live questions can support a small matched shadow comparison with blinded output review, burden caps, explicit privacy controls, and precommitted pass and halt criteria.","source_ids":["S1","S4","S6","S7"]},"no_unresolved_safety_or_authority_stop":{"status":"UNCERTAIN","reason":"A non-operational probe is containable, but governing jurisdiction, public-records treatment, compensation and procurement authority, research/ethics classification, withdrawal/deletion limits, and small-specialty reidentification risk have not been verified.","source_ids":["S2","S4","S6","S7"]},"credible_cost_scope_and_range":{"status":"UNCERTAIN","reason":"The bands are scoped resource-equivalent estimates anchored to published organizer, panel-size, and duration requirements, but no local salary, compensation, secure-tooling, legal-review, or procurement data were found.","source_ids":["S3","S4"]}},"next_evidence_step":"With a willing regional council, preregister and run one shadow study on four archived, redacted, non-live cascading-risk questions. Match questions before assignment; process two through successive cycles of the same test relay and two through a documented ad hoc-consultation comparator, then cross over or repeat if contamination can be prevented. Cap the panel at 12 compensated volunteers and prohibit operational use. Measure setup elapsed time, coordinator touch time, recruitment contacts, participant hours, attrition, perspective coverage, conflict hits, dissent retention, distribution traceability, identity-risk events, reviewer-rated usability, rework, willingness to participate again, maintenance labor, and time to verified readiness. Blinded reviewers apply a frozen rubric. Falsify incremental advantage if the relay's second cycle fails to reduce setup time and coordinator effort by at least 25% after counting maintenance, or if it performs worse than baseline on any prespecified coverage, dissent, traceability, confidentiality, burden, neutrality, or recovery threshold. Halt immediately for disclosure, retaliation concern, sponsor interference, unconsented reuse, irreconstructible aggregation, or systematic suppression of minority judgments.","blocking_evidence":["No local process audit shows that expert search, trust, translation, and elicitation setup—rather than unclear sponsor decisions, unresolved values, missing authority, or infrequent questions—is the rate-limiting burden.","No named regional council has expressed demand for, committed staff to, or authorized procurement and compensation for this specific relay.","No comparative field evidence shows a net second-cycle reuse advantage after counting relationship maintenance, facilitator labor, expert compensation, attrition, and recovery.","Jurisdiction-specific public-records, confidentiality, procurement, research/ethics, consent, withdrawal, deletion, and compensation requirements remain unverified.","No live or shadow evidence establishes that anonymity, dissent, disciplinary/community coverage, neutrality, and participant welfare remain intact across repeated cycles."],"research_disposition":"PARTNERED_RESEARCH_PROGRAM","world_novelty_boundary":"The search establishes extensive prior practice for Delphi, online structured expert elicitation, government forecasting-center elicitation, regional disaster-response Delphi panels, and reusable cascading-risk methodologies. It does not measure world novelty, patentability, freedom to operate, market size, or realized impact. The only surviving novelty boundary is the empirical operating claim that a governed regional relay's maintained relationships and explicit regeneration cycle create a net, quality-preserving advantage on second and later matched questions.","arm":"COMPLETE_PROPOSAL_PORTFOLIO","candidate_version":0,"controller_recommendation":{"action":"STOP_EMPIRICAL_RESEARCH_NEEDED","repairable":false,"material_progress_observed":true,"progress_targets":["Obtain written interest and probe authority from a named regional emergency-preparedness council, including staff owner, records/privacy owner, procurement path, and participant-compensation authority.","Complete a retrospective workload audit quantifying eligible-question frequency, setup elapsed time, coordinator touch time, recruitment failure, attrition, coverage gaps, and downstream rework under the ad hoc baseline.","Resolve jurisdiction-specific public-records, confidentiality, ethics/research, consent, withdrawal/deletion, procurement, accessibility, and compensation requirements before recruitment.","Preregister the matched archived-question probe, including question matching, panel-coverage rules, maintenance-cost accounting, blinded review, a minimum 25% second-cycle burden-reduction threshold, noninferiority thresholds for quality and welfare, and automatic safety halts.","Produce comparative second-cycle evidence showing net reusable savings without narrower participation, suppressed dissent, false consensus, identity leakage, sponsor capture, disproportionate participant burden, or failure to return to a verified ready state."],"reason":"Bounded web research verifies the problem, a plausible regional authorizer, technical feasibility, and extensive established prior art, but it cannot determine whether maintained relationships produce a net multi-cycle advantage in the intended workflow. That claim, adopter commitment, local baseline prevalence, and cross-cycle safety require proprietary workflow data and a partnered shadow test. Under the evaluation rule, the appropriate terminal recommendation is therefore an empirical-research stop; all STOP recommendations are non-repairable."},"proposal_index":2}