{"schema_version":1,"research_id":"eoa_inverse_innovation_exp06_external_evaluation_20260803","source_assessment_id":"catalytic_pathway_enablement__futurism_foresight:P3:v0","cell_id":"catalytic_pathway_enablement__futurism_foresight","search_queries":["site:fema.gov HSEEP exercise design development conduct evaluation improvement planning 2020 PDF","site:cisa.gov tabletop exercise packages exercise planner facilitator guide customizable scenarios","site:uscg.mil port exercise program Area Maritime Security exercise AMSTEP","site:ecfr.gov 33 CFR port facility security exercises 105.220","port emergency exercise simulation platform reusable modular scenario exercise port authority","site:apec.org maritime security exercises port facilities manual exercise planning injects evaluation","port authority tabletop exercise emergency preparedness scenario exercise after action port official","research tabletop exercise reusable simulation emergency management scenario exercise evaluation decision making","official exercise management software injects simulation logs after action report tabletop platform","site:everbridge.com crisis exercise simulation inject exercise management platform","site:c3pathways.com tabletop exercise platform injects after action reporting","site:exonaut.com exercise management system inject log evaluation","\"Enhancing the design, conduct and evaluation\" emergency preparedness exercises rapid review DOI","site:pubmed.ncbi.nlm.nih.gov \"Enhancing the design, conduct and evaluation\"","eCFR 33 CFR 103.515 Area Maritime Security Plan exercise each calendar year 18 months"],"sources":[{"source_id":"S1","title":"Navigation and Vessel Inspection Circular No. 09-02, Change 6: Guidelines for Area Maritime Security Committees and Area Maritime Security Plans Required for U.S. Ports","publisher":"United States Coast Guard","url":"https://www.dco.uscg.mil/Portals/9/DCO%20Documents/5p/5ps/NVIC/2002/NVIC%2009-02%20CH%206.pdf","source_class":"OFFICIAL_GUIDANCE","publication_date":"2023-07-20","accessed_at":"2026-08-03","claims_supported":["Captains of the Port and Area Maritime Security Committees are responsible for exercises testing Area Maritime Security Plans.","Exercises are used to validate port plans, procedures, command and control, and recovery arrangements.","HSEEP is the Coast Guard's standardized exercise methodology.","AMSTEP requires stakeholder participation, exercise evaluation, after-action work, sensitive-information controls, and corrective action."]},{"source_id":"S2","title":"Building partnerships through required area exercises to keep the port safe","publisher":"United States Coast Guard, MyCG","url":"https://www.mycg.uscg.mil/News/Article/3240027/building-partnerships-through-required-area-exercises-to-keep-the-port-safe/","source_class":"OFFICIAL_ORGANIZATION_DATA","publication_date":"2022-12-19","accessed_at":"2026-08-03","claims_supported":["A named Coast Guard sector and port partners conducted three complex exercises over seven months.","The sector described large-scale exercise execution as requiring creativity, hard work, and substantial preparation.","The exercises involved multiple agencies, vessels, role stations, simulated events, and unified-command discussion.","Named sector leaders expressed need for recurring exercises to strengthen coordination and response."]},{"source_id":"S3","title":"Homeland Security Exercise and Evaluation Program Policy and Guidance","publisher":"Federal Emergency Management Agency","url":"https://preptoolkit.fema.gov/web/hseep-resources/policy-and-guidance","source_class":"OFFICIAL_GUIDANCE","publication_date":"2020","accessed_at":"2026-08-03","claims_supported":["HSEEP already supplies a common lifecycle for exercise program management, design, development, conduct, evaluation, and improvement planning.","FEMA provides reusable templates and exercise starter kits for organizations conducting discussion-based exercises."]},{"source_id":"S4","title":"Technical Guidance for the Specification and Development of Homeland Security Simulation Applications, NIST Technical Note 1742","publisher":"National Institute of Standards and Technology","url":"https://www.govinfo.gov/content/pkg/GOVPUB-C13-0031323c3ab5afe0752aae5db8544f45/pdf/GOVPUB-C13-0031323c3ab5afe0752aae5db8544f45.pdf","source_class":"STANDARD","publication_date":"2012-05","accessed_at":"2026-08-03","claims_supported":["Standard exercise artifacts already include situation manuals, controller/evaluator handbooks, master scenario event lists, evaluation guides, and after-action plans.","Automated tabletop software was already specified to manage inject libraries, clocks, pause/rollback controls, scenarios, messages, decisions, and exercise data.","Large event lists and exercise-data capture can overwhelm manual control, supporting the claimed workflow burden.","The principal technical components of the proposed rehearsal bed have long-established specifications."]},{"source_id":"S5","title":"INJECT Exercise Platform Documentation","publisher":"Masaryk University","url":"https://docs.inject.muni.cz/","source_class":"OFFICIAL_PRODUCT_DOCUMENTATION","publication_date":"2026-05-27","accessed_at":"2026-08-03","claims_supported":["A maintained open-source platform already supports interactive tabletop exercises.","The platform is explicitly intended to simplify exercise preparation, execution, and evaluation and reduce organizer workload.","Current documentation covers objectives, scenarios, injects, execution, reflection, logs, security, APIs, and deployable architecture.","Reusable, configurable exercise-platform technology is technically available rather than hypothetical."]},{"source_id":"S6","title":"Enhancing the design, conduct and evaluation of public health emergency preparedness exercises: a rapid review","publisher":"BMC Public Health / Springer Nature","url":"https://link.springer.com/article/10.1186/s12889-025-23270-6","source_class":"AUTHORITATIVE_SECONDARY","publication_date":"2025-07-03","accessed_at":"2026-08-03","claims_supported":["Exercise development and implementation are resource-intensive.","Evidence-informed design depends on scenario development, participant selection, trained facilitators, fit-for-purpose evaluation, and pathways from findings to improvement.","The evidence base remains methodologically limited and needs more rigorous evaluations and indicators.","Evidence from public-health exercises is adjacent rather than direct evidence for a port rehearsal bed."]},{"source_id":"S7","title":"Occupational Employment and Wages, May 2023: Project Management Specialists","publisher":"U.S. Bureau of Labor Statistics","url":"https://www.bls.gov/oes/2023/may/oes131082.htm","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2024","accessed_at":"2026-08-03","claims_supported":["The national mean wage for project-management specialists was $50.44 per hour and $104,920 annually in May 2023.","Technical consulting and computer-systems industries reported mean annual wages above $112,000, providing an official labor anchor for resource-equivalent estimates.","The estimates exclude benefits, overhead, participant opportunity cost, equipment, security review, and 2023-to-2026 escalation."]},{"source_id":"S8","title":"33 CFR Part 103, Subpart E: Area Maritime Security Plans, including §103.515 Exercises","publisher":"U.S. Government Publishing Office","url":"https://www.govinfo.gov/link/cfr/33/103?link-type=pdf&sectionnum=505&year=mostrecent","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2025-07-01","accessed_at":"2026-08-03","claims_supported":["The Captain of the Port coordinates with the Area Maritime Security Committee to conduct or participate in an exercise at least annually, with no more than 18 months between exercises.","A tabletop exercise may validate the Area Maritime Security Plan without equipment or personnel deployment.","Area plans cover command, response, recovery, communications, jurisdiction, security resources, sensitive information, and periodic exercise and updating.","The Captain of the Port develops the plan with the committee; District and Area Commanders retain plan approval authority, and exercise documentation must be retained."]}],"problem_evidence":{"support":"MODERATE","rationale":"The core burden is visible: official port practice involves complex multi-agency exercises that require substantial preparation, while broader research calls exercise development resource-intensive. The operational importance is strong because exercises validate plans and command arrangements and are required regularly. However, no direct source quantified setup hours, scenario-to-rehearsal delays, deferrals, or an accumulation of approved-but-unrehearsed foresight scenarios at a particular port. The candidate's exact prevalence and bottleneck attribution therefore remain unverified.","source_ids":["S1","S2","S6","S8"]},"stakeholder_evidence":{"support":"STRONG","rationale":"An identifiable authorizer-adopter chain exists: the Captain of the Port/Federal Maritime Security Coordinator and Area Maritime Security Committee coordinate required exercises, while port stakeholders participate. Coast Guard Sector Long Island Sound provides a concrete example with named leaders expressing the need for recurring, coordinated exercises and acknowledging the preparation burden. A local port authority could fund or host the system, but federal exercise credit, sensitive-information treatment, and plan implications remain with the applicable authorities.","source_ids":["S1","S2","S8"]},"prior_art":{"proximity":"ESTABLISHED_PRACTICE","closest_analogues":[{"name":"HSEEP exercise lifecycle and reusable templates","similarity":"Already standardizes exercise design, conduct, evaluation, after-action reporting, and improvement planning—the proposed bed's principal workflow.","remaining_difference":"HSEEP is doctrine and reusable documentation, not a verified port-specific modular installation converting long-range foresight scenarios into comparable decision traces.","source_ids":["S1","S3"]},{"name":"NIST automated tabletop-exercise architecture","similarity":"Specifies inject libraries, scenario editing, timing, pause/rollback controls, decision and message databases, and exercise-data capture, closely matching the proposed inject controller and logging layer.","remaining_difference":"It is general technical guidance rather than a locally governed port configuration with participant-protection and reset assays.","source_ids":["S4"]},{"name":"INJECT Exercise Platform","similarity":"A maintained open-source platform supports configurable objectives, scenarios, injects, execution, evaluation, logs, security, and repeatable deployment while claiming reduced organizer workload.","remaining_difference":"The externally visible documentation does not establish port-domain modules, maritime authority boundaries, synthetic port-data interfaces, or matched effectiveness against bespoke port exercises.","source_ids":["S5"]},{"name":"USCG AMSTEP port exercise program","similarity":"An established recurring port-security exercise program already tests plans, command, coordination, recovery, and stakeholder behavior under HSEEP-aligned governance.","remaining_difference":"AMSTEP does not itself establish that a fixed modular bed reduces recurring preparation or validly extends from security-plan testing to plural long-range foresight scenarios.","source_ids":["S1","S2","S8"]}],"distinctive_claim_remaining":"The defensible remaining claim is local and incremental, not categorical novelty: for a predeclared class of compatible, non-live port scenarios, a port-configured modular bed assembled from established exercise components will reduce staff setup hours to a valid rehearsal dossier by at least 30% versus both a bespoke HSEEP tabletop and an off-the-shelf configurable platform, while keeping decision-log completeness within five percentage points, producing no additional safety or disclosure events, avoiding higher participant burden, and reproducing the benefit after a documented reset. Hidden bespoke work, easier-case selection, reduced validity, protection erosion, rig-attributable findings, or failure on the second turnover falsifies the claim.","confidence":"HIGH"},"implementation_evidence":{"support":"MODERATE","rationale":"Technical feasibility is high because doctrine, automated tabletop specifications, deployable open-source software, inject control, logging, evaluation, and after-action workflows already exist. Workflow and authority feasibility are also plausible under existing tabletop authority. What remains unverified is the integration of port-specific modules, isolated synthetic data, records and sensitive-security-information controls, accessibility and labor protections, evaluator reliability, scenario validity, reset integrity, and downstream findings capacity at an actual port. No source demonstrates the proposed integrated system's comparative port performance.","source_ids":["S1","S3","S4","S5","S6","S8"]},"scores":{"meaningful_impact":{"score":4,"rationale":"Better repeated testing of port decisions could improve plan validation and coordination in a safety- and commerce-critical setting, but realized readiness impact is unmeasured and exercises cannot prove real-world readiness.","source_ids":["S1","S2","S8"]},"stakeholder_pull":{"score":4,"rationale":"Recurring exercises have regulatory pull and named Coast Guard/port stakeholders, although no identified port has requested or funded this specific bed.","source_ids":["S1","S2","S8"]},"incremental_advantage":{"score":2,"rationale":"Reduced recurring setup is plausible, but no comparative evidence shows advantage over HSEEP templates, automated tabletop software, or the current open-source INJECT platform.","source_ids":["S3","S4","S5","S6"]},"distinctiveness_plausibility":{"score":2,"rationale":"Nearly all functional elements are established. Distinctiveness is confined to port-specific integration, governance, and a measurable multi-cycle performance claim.","source_ids":["S1","S3","S4","S5"]},"technical_implementability":{"score":4,"rationale":"Existing specifications and software make a synthetic, isolated prototype straightforward; secure port integration and reliable attribution remain substantive engineering tasks.","source_ids":["S4","S5"]},"adoption_authority_feasibility":{"score":4,"rationale":"The COTP/AMSC exercise pathway and tabletop authority are explicit, but local port funding, labor, information-security, records, accessibility, and findings-use approvals still must be secured.","source_ids":["S1","S8"]},"evidence_readiness":{"score":2,"rationale":"A credible comparator study can be designed, but there are no port-specific baselines, setup logs, matched trials, reliability data, safety data, or turnover results.","source_ids":["S2","S6"]},"safety_net_benefit":{"score":4,"rationale":"Synthetic, isolated tabletop rehearsal offers a bounded way to expose coordination failures without live-system deployment, provided findings are not misrepresented as readiness and sensitive data remain protected.","source_ids":["S1","S4","S8"]},"scalability":{"score":3,"rationale":"Reusable software and templates favor replication, but scenario customization, participant time, facilitator skill, security classification, and findings-remediation capacity limit low-marginal-cost scaling.","source_ids":["S3","S4","S5","S6"]}},"score_confidence":"MODERATE","costs":{"first_evidence":{"band_2026_usd":"50K_TO_250K","scope":"One partnered, pre-registered shadow study with three frozen synthetic scenario slices, three comparable teams, three counterbalanced conditions, independent observation, secure data handling, reset testing, analysis, and a decision memorandum.","confidence":"MODERATE","assumptions":["Approximately 500-1,500 combined specialist, technical, evaluator, participant, and governance hours.","2023 BLS wages are escalated to 2026 and converted to loaded resource-equivalent rates including benefits, overhead, and participant opportunity cost.","Existing meeting space and an open-source or already licensed exercise platform are available.","No live-system interface, classified-data accreditation, or capital construction is included."],"source_ids":["S4","S5","S6","S7"]},"initial_deployment_startup":{"band_2026_usd":"250K_TO_1M","scope":"Port-specific configuration, synthetic-data environment, modular scenario library, role and authority model, logging and evaluation layer, access controls, security and records review, accessibility work, training, testing, and operational documentation.","confidence":"LOW","assumptions":["A hardened local deployment requires roughly two to five staff-equivalent years across exercise design, software/configuration, cybersecurity, data governance, evaluation, and program management.","Existing facilities and endpoint hardware are reused.","The band excludes high-fidelity digital twins, live operational integration, classified-system accreditation, and construction.","No vendor quotations or port-specific salary schedule were found."],"source_ids":["S1","S4","S5","S7","S8"]},"operational_launch":{"band_2026_usd":"50K_TO_250K","scope":"Readiness review, facilitator and observer qualification, participant onboarding, synthetic-data validation, first two governed operational exercises, reset verification, after-action review, and launch contingency.","confidence":"MODERATE","assumptions":["The platform and port modules are already accepted from startup.","Exercises remain tabletop or bounded functional rehearsals with synthetic data.","Participant opportunity cost and protective-officer time are included as resource equivalents.","Major remediation of plans or facilities discovered by exercises is excluded."],"source_ids":["S1","S3","S6","S7","S8"]},"annual_recurring":{"band_2026_usd":"250K_TO_1M","scope":"A standing exercise-design and stewardship function, platform maintenance, security updates, scenario and module refresh, four to twelve bounded exercises, participant and evaluator time, records handling, reset testing, and findings tracking.","confidence":"LOW","assumptions":["Approximately two to five dedicated staff equivalents plus distributed participant, legal, labor, accessibility, information-security, and operational-review time.","Exercise volume and scenario complexity dominate the range.","The estimate includes opportunity cost but excludes remediation projects and full-scale live drills.","Open-source software reduces licensing cost but not exercise design, validation, governance, or facilitation labor."],"source_ids":["S1","S3","S5","S6","S7"]}},"verified_pipeline_gates":{"externally_supported_problem":{"status":"YES","reason":"Official port evidence and broader review literature support the material preparation burden and importance of recurring exercises, although the proposed port's exact backlog and bottleneck prevalence remain unmeasured.","source_ids":["S1","S2","S6","S8"]},"externally_credible_adopter_or_authorizer":{"status":"YES","reason":"The COTP/FMSC and AMSC are explicit exercise authorizers and coordinators, with port stakeholders as credible adopters and participants.","source_ids":["S1","S2","S8"]},"distinct_testable_incremental_claim":{"status":"YES","reason":"Setup-hour reduction, noninferiority on logging and burden, safety events, rig-attributable errors, and successful second-cycle reset can be pre-registered against established comparators.","source_ids":["S3","S4","S5","S6"]},"bounded_next_evidence_step":{"status":"YES","reason":"A small, synthetic, counterbalanced shadow study with fixed scenarios, time caps, comparators, stop rules, and explicit falsifiers is bounded and reversible.","source_ids":["S3","S4","S6","S8"]},"no_unresolved_safety_or_authority_stop":{"status":"UNCERTAIN","reason":"Existing authority permits tabletop exercises and the candidate states strong exclusions, but local approval for labor participation, accessibility, sensitive-security-information handling, records retention, synthetic data, participant welfare, and findings use has not been demonstrated.","source_ids":["S1","S8"]},"credible_cost_scope_and_range":{"status":"YES","reason":"The four broad bands are credible as resource-equivalent planning ranges anchored to official labor data and documented exercise complexity, but confidence is limited by absent vendor quotations and port-specific staffing, security, and facility information.","source_ids":["S1","S4","S5","S7"]}},"next_evidence_step":"Partner with one willing port authority and its COTP/AMSC, first auditing 12-24 months of exercise records to verify that bespoke setup is a material bottleneck. If it is, pre-register a three-scenario, three-team Latin-square shadow study. Freeze three approved non-live scenario slices and have every team complete: (A) a bespoke HSEEP tabletop, (B) the current INJECT or comparable off-the-shelf configurable platform without custom port modules, and (C) the full port rehearsal bed, with order counterbalanced. Use identical synthetic data, 90-minute play limits, preparation and debrief caps, protected constraints, evaluator rubrics, and downstream review. The primary endpoint is total staff setup hours to an independently accepted dossier. Pass only if C reduces setup by at least 30% against both comparators, repeats the benefit after reset, keeps decision-log completeness within five percentage points, attains evaluator agreement of at least kappa 0.70, adds no safety/privacy/live-system event, does not increase participant burden, and resets within 25% of first-run setup time. Falsify for hidden bespoke preparation, easier scenarios, missing or manufactured decision traces, evaluator disagreement, rig-attributable findings, protection erosion, findings-owner overload, or failed second-cycle readiness.","blocking_evidence":["No port-specific measurement shows that exercise architecture, rather than participant release, leadership commitment, scheduling, obsolete plans, or absent findings owners, is the rate-limiting barrier.","No matched evidence estimates setup-hour reduction against either bespoke HSEEP practice or a current configurable exercise platform.","No evidence validates the proposed decision traces as evidence of contingency behavior rather than artifacts of synthetic data, compressed time, facilitator behavior, or module design.","No second-cycle result demonstrates successful reset, absence of residual case contamination, or stable performance after reuse.","No participating port, COTP/AMSC, labor representative, safety officer, information-security officer, accessibility officer, or records authority has approved the probe.","No port-specific resource inventory, vendor quotation, loaded labor schedule, information-classification requirement, or remediation-capacity estimate is available.","No evidence demonstrates that exercise findings can be investigated and closed without creating a downstream backlog."],"research_disposition":"KNOWN_PRACTICE_DIFFUSION","world_novelty_boundary":"The search establishes recurring port exercises, standardized HSEEP workflows, automated tabletop architectures, and a maintained open-source exercise platform as existing practice. It does not establish whether any organization has implemented the exact port-foresight configuration, governance bundle, or proposed performance thresholds. World novelty, patentability, freedom to operate, market size, and realized impact remain unmeasured.","arm":"COMPLETE_PROPOSAL_PORTFOLIO","candidate_version":0,"controller_recommendation":{"action":"STOP_EMPIRICAL_RESEARCH_NEEDED","repairable":false,"material_progress_observed":true,"progress_targets":["Obtain a written partnered-research commitment from one port authority and the applicable COTP/AMSC, including labor, safety, accessibility, information-security, records, and participant-protection approvals.","Audit 12-24 months of exercise preparation records and show that bespoke architecture consumes at least 20% of total preparation effort or materially causes delay or deferral.","Pre-register the three-condition counterbalanced study, outcome definitions, attribution rules, noninferiority margins, safety stops, reset criteria, and analysis before configuration begins.","Measure all custom configuration, participant preparation, observer labor, synthetic-data work, rework, and downstream findings effort so work shifting cannot masquerade as catalytic gain.","Demonstrate at least 30% lower setup hours than both bespoke HSEEP and an off-the-shelf platform while meeting the stated quality, safety, participant-burden, evaluator-reliability, and second-turnover thresholds.","Produce a port-specific total-cost-of-ownership range and capacity model using local loaded labor rates, deployment/security requirements, annual scenario volume, and findings-remediation capacity."],"reason":"Bounded web research has resolved that the need, authorizer pathway, technical building blocks, and most functional prior art are real. It cannot resolve the only remaining incremental claim—whether a port-specific modular configuration reduces total recurring work without degrading validity, safety, protections, or reset performance. That evidence requires local records, authority approvals, participant fieldwork, and comparative live rehearsal testing."},"proposal_index":3}