{"schema_version":1,"research_id":"eoa_inverse_innovation_exp04_external_evaluation_20260802","source_assessment_id":"negative_space_design__behavioral_economics:SENTINEL:v0","cell_id":"negative_space_design__behavioral_economics","search_queries":["site:who.int data quality missing values true zero reports PDF","site:cdc.gov NHSN data quality dashboard missing survey incomplete survey PDF","site:grafana.com/docs missing data No Data MissingSeries alerting","site:hl7.org/fhir Observation dataAbsentReason","site:osr.statisticsauthority.gov.uk guidance dashboards data quality limitations user needs","site:cdc.gov/nssp data quality tools stopped sending data alerts","visualizing missing values decision making study Song Fu Saket Stasko 2022","Toward Systematic Considerations of Missingness in Visual Analytics Sun missingness","site:w3.org/WAI/WCAG21 Understanding use of color status messages official","site:nist.gov automation bias decision support human factors guidance","regulatory dashboard missing data decision false reassurance study missing zero","cause-specific empty state design action research missing data dashboard"],"sources":[{"source_id":"S1","title":"Data Quality Assurance, Module 2: Desk Review of Data Quality","publisher":"World Health Organization","url":"https://cdn.who.int/media/docs/default-source/data-quality-pages/2021_-dqa_module-2_desk-review-of-data-quality.pdf","source_class":"OFFICIAL_GUIDANCE","publication_date":"2020-12","accessed_at":"2026-08-02","claims_supported":["Missing data should be clearly differentiated from true zero values in facility and district reports.","Assigning zero to missing entries makes no-event periods indistinguishable from reporting failures.","Poor-quality data can produce incorrect program decisions and poor targeting of resources.","Routine data-quality monitoring and follow-up are established managerial responsibilities."]},{"source_id":"S2","title":"Quick Reference Guide to the NHSN PSC Annual Survey Data Quality Dashboard","publisher":"U.S. Centers for Disease Control and Prevention","url":"https://www.cdc.gov/nhsn/pdfs/pscmanual/PSC-Survey-Data-Quality-Dashboard-QRG.pdf","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2025","accessed_at":"2026-08-02","claims_supported":["A deployed surveillance dashboard distinguishes missing or incomplete surveys, non-operational facilities, uncalculable comparisons, potential quality issues, and resolved states.","The dashboard preserves current and prior reporting context.","Displayed conditions lead to actions including review, edit, submit, confirm, and resolve."]},{"source_id":"S3","title":"Data Quality Tools","publisher":"U.S. Centers for Disease Control and Prevention, National Syndromic Surveillance Program","url":"https://www.cdc.gov/nssp/php/onboarding-toolkits/data-quality.html","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2026-01-13","accessed_at":"2026-08-02","claims_supported":["NSSP operates tools for monitoring data flow, timeliness, completeness, validity, processing problems, and anomalies.","Users can create anomaly rules and alerts, and site administrators are urged to review processing status frequently.","NSSP sends alerts when active facilities stop sending data and offers service-desk and inspector support for corrective action.","CDC surveillance data-quality staff are an identifiable adopter or authorizer class with an expressed operational need to detect and act on missing reporting."]},{"source_id":"S4","title":"Handle Missing Data in Grafana Alerting","publisher":"Grafana Labs","url":"https://grafana.com/docs/grafana/latest/alerting/guides/missing-data/","source_class":"OFFICIAL_PRODUCT_DOCUMENTATION","publication_date":"2026","accessed_at":"2026-08-02","claims_supported":["Grafana distinguishes query failure, No Data, and MissingSeries rather than treating all absence as a normal zero.","It exposes state-reason annotations and permits different handling and notification behavior.","The documentation identifies silent monitoring failure when a source disappears and alerts are not explicitly configured for missingness.","Missingness handling must account for benign causes and alert noise, showing a risk of excessive escalation."]},{"source_id":"S5","title":"Observation—FHIR v4.0.1: Detailed Descriptions","publisher":"Health Level Seven International","url":"https://hl7.org/fhir/R4/observation-definitions.html","source_class":"STANDARD","publication_date":"2019-10-30","accessed_at":"2026-08-02","claims_supported":["FHIR defines dataAbsentReason to record why an expected observation value is missing.","The standard requires dataAbsentReason to be present only when the actual value is absent.","FHIR notes that use-case agreements are necessary to interpret null or exceptional values, limiting the sufficiency of generic cause codes alone."]},{"source_id":"S6","title":"Understanding the Effects of Visualizing Missing Values on Visual Data Exploration","publisher":"Hayeong Song, Yu Fu, Bahador Saket, and John Stasko; arXiv","url":"https://arxiv.org/abs/2109.08723","source_class":"PRIMARY_RESEARCH","publication_date":"2021-09-17","accessed_at":"2026-08-02","claims_supported":["A controlled experiment compared omission of records with explicit missing-value visualization.","Missing-value presentation changed participants' decision-making workflow.","The study did not test regulator-like closure, data-request, or investigation decisions or the proposed matched-action comparison."]},{"source_id":"S7","title":"Toward Systematic Considerations of Missingness in Visual Analytics","publisher":"Maoyuan Sun et al.; arXiv","url":"https://arxiv.org/abs/2108.04931","source_class":"PRIMARY_RESEARCH","publication_date":"2021-08-10","accessed_at":"2026-08-02","claims_supported":["Missingness can arise from system failure, network interruption, intentional hiding, and bias.","Missingness affects human sensemaking and decisions and may be observed, inferred, or ignored.","The work supplies a conceptual taxonomy rather than causal evidence for cause-matched oversight actions."]},{"source_id":"S8","title":"Web Content Accessibility Guidelines (WCAG) 2.1","publisher":"World Wide Web Consortium","url":"https://www.w3.org/TR/WCAG21/","source_class":"STANDARD","publication_date":"2025-05-06","accessed_at":"2026-08-02","claims_supported":["Web interfaces must not rely on color alone to convey information or actions.","Interface controls require accessible names, roles, and values, and status messages must be programmatically determinable for assistive technologies.","The proposed state label and action can be implemented with established accessibility requirements."]}],"problem_evidence":{"support":"STRONG","rationale":"WHO explicitly identifies the consequential missing-versus-zero failure and links poor data quality to incorrect decisions and resource allocation. CDC and Grafana document deployed monitoring workflows where missing reports, stopped feeds, backlogs, and other absence states require review or alerts. Research shows missingness presentation can affect decision workflow. No direct prevalence estimate was found for false no-incident closure in regulator-like dashboards, so the precise outcome frequency remains unmeasured.","source_ids":["S1","S2","S3","S4","S6","S7"]},"stakeholder_evidence":{"support":"STRONG","rationale":"CDC NHSN and NSSP are identifiable operational adopters: they maintain data-quality dashboards, distinguish reporting conditions, recommend frequent review, alert on stopped reporting, and connect anomalies to corrective workflows. Their documentation establishes expressed need and relevant product/data-governance authority, but not commitment to this exact intervention or experiment.","source_ids":["S2","S3"]},"prior_art":{"proximity":"SUBSTANTIAL_COLLISION","closest_analogues":[{"name":"CDC NHSN PSC Annual Survey Data Quality Dashboard","similarity":"A deployed surveillance dashboard distinguishes several missingness and quality states, retains current and prior context, and connects states to review, submission, confirmation, and resolution actions.","remaining_difference":"The source reports neither regulator-facing false-closure outcomes nor a controlled comparison between one cause-matched action and an equally salient generic missing-data warning.","source_ids":["S2"]},{"name":"CDC NSSP data-flow and facility-silence monitoring","similarity":"The practice monitors completeness, timeliness, validity, backlogs, processing failures, and facilities that stop reporting, then alerts staff and supports investigation or correction.","remaining_difference":"Public documentation does not establish the proposed entity-period panel or validate its behavioral advantage over a generic warning.","source_ids":["S3"]},{"name":"Grafana No Data and MissingSeries handling","similarity":"It represents absence as explicit operational states, exposes state reasons, preserves state history, and supports differentiated responses.","remaining_difference":"It concerns technical infrastructure, not regulated entities or strategic under-reporting, and does not test closure or oversight follow-up outcomes.","source_ids":["S4"]},{"name":"WHO missing-versus-zero rule and FHIR dataAbsentReason","similarity":"The guidance and standard already establish the core representational moves: separate true zero from missing data and encode a cause for an absent expected value.","remaining_difference":"They do not establish that presenting exactly one matched oversight action improves decisions beyond an equally salient generic missingness warning.","source_ids":["S1","S5"]},{"name":"Missingness-aware visualization research","similarity":"The research identifies heterogeneous causes of missingness and provides experimental evidence that its visual presentation can alter decision workflow.","remaining_difference":"It does not evaluate regulator-like closure, requests, escalation, inappropriate investigation, or cause-matched actions.","source_ids":["S6","S7"]}],"distinctive_claim_remaining":"In regulator-like entity-period decisions, pairing each reliably diagnosed absence cause with one matched oversight action reduces false no-incident closure and increases predefined appropriate follow-up more than an equally salient generic missing-data warning with the same context and available actions.","confidence":"HIGH"},"implementation_evidence":{"support":"MODERATE","rationale":"Existing CDC and Grafana systems demonstrate technical feasibility for state classification, state-specific messaging, history, alerts, and workflow links; FHIR demonstrates machine-readable absence reasons; WCAG supplies accessibility constraints. Implementation nevertheless depends on each adopter having sufficiently accurate and timely metadata to distinguish confirmed zero, unreceived report, delay, permission block, and system failure. Legal authority remains with the host regulator, and a non-production simulation avoids changing real cases. Material safety risks are wrong cause classification, automation bias from a single recommendation, disproportionate scrutiny of weak reporters, permission leakage, and excessive escalation; these require human authority, visible uncertainty and provenance, alternative-action recoverability, subgroup analysis, and rollback.","source_ids":["S2","S3","S4","S5","S8"]},"scores":{"meaningful_impact":{"score":4,"rationale":"False reassurance in oversight can misallocate attention, and official guidance links poor or ambiguous data to incorrect decisions; actual false-closure prevalence is not measured.","source_ids":["S1","S3"]},"stakeholder_pull":{"score":4,"rationale":"CDC surveillance programs operate closely related dashboards, alerts, and corrective workflows, evidencing strong need but not demand for this exact comparison.","source_ids":["S2","S3"]},"incremental_advantage":{"score":3,"rationale":"A cause-matched action could improve follow-up beyond a generic warning, but closely related deployed systems already pair conditions with actions and no comparative effect size exists.","source_ids":["S2","S3","S4"]},"distinctiveness_plausibility":{"score":2,"rationale":"Cause-coded absence, missing-versus-zero distinctions, explicit states, and differentiated actions are established; only the narrow controlled behavioral comparison remains distinctive.","source_ids":["S1","S2","S4","S5","S6","S7"]},"technical_implementability":{"score":4,"rationale":"State fields, conditional panels, audit context, and action routing are conventional dashboard capabilities, conditional on trustworthy source metadata and integration.","source_ids":["S2","S3","S4","S5"]},"adoption_authority_feasibility":{"score":3,"rationale":"Dashboard and data-governance owners can authorize a prototype and simulation, but live regulator deployment requires agency-specific workflow, privacy, accessibility, records, and decision-authority review.","source_ids":["S2","S3","S8"]},"evidence_readiness":{"score":3,"rationale":"A controlled synthetic-case experiment is well bounded and relevant comparators exist, but outcome prevalence, recruitment access, cause ground truth, and effect size are unavailable.","source_ids":["S2","S6"]},"safety_net_benefit":{"score":4,"rationale":"The panel could prevent unsupported closure and silent reporting failures while preserving human authority, though excessive or inequitable escalation is a counter-risk.","source_ids":["S1","S3","S4"]},"scalability":{"score":4,"rationale":"The rule-based pattern is reusable across dashboard platforms once state metadata and workflow mappings exist; heterogeneous taxonomies and legal processes limit plug-and-play transfer.","source_ids":["S3","S4","S5","S8"]}},"score_confidence":"MODERATE","costs":{"first_evidence":{"band_2026_usd":"50K_TO_250K","scope":"Design and preregister one synthetic-case randomized simulation; develop two accessible prototypes; obtain independent cause/action adjudication; recruit regulator-like participants; analyze primary, safety, and subgroup outcomes.","confidence":"LOW","assumptions":["Approximately 80–200 participants with specialized recruitment premiums","No live agency integration or real case data","Includes research ethics, accessibility review, participant compensation, design, engineering, and analysis","Resource-equivalent estimate, not a vendor quote"],"source_ids":["S6","S8"]},"initial_deployment_startup":{"band_2026_usd":"250K_TO_1M","scope":"One-agency preproduction implementation covering state taxonomy, metadata validation, interface development, audit logging, accessibility, security/privacy review, and workflow configuration.","confidence":"LOW","assumptions":["Existing dashboard, identity system, and data feeds are reused","Five cause states and a small number of workflows","No automated enforcement or case disposition","Includes independent classification audit and rollback tooling"],"source_ids":["S2","S3","S4","S8"]},"operational_launch":{"band_2026_usd":"1M_TO_5M","scope":"Authorized production launch across one oversight program, including integrations, migration, validation, training, change management, records and privacy review, monitoring, staged rollout, and evaluation.","confidence":"LOW","assumptions":["Multiple feeds and user roles but a single agency or program","Human officials retain all consequential authority","Launch includes accessibility conformance and incident-response procedures","Excludes nationwide multi-agency replacement of legacy systems"],"source_ids":["S2","S3","S8"]},"annual_recurring":{"band_2026_usd":"250K_TO_1M","scope":"Annual product ownership, rule and taxonomy maintenance, data-quality monitoring, accessibility regression testing, security support, audit review, user support, and outcome/safety surveillance for one program.","confidence":"LOW","assumptions":["Existing hosting and core dashboard operations continue","Small cross-functional maintenance allocation rather than a dedicated large platform team","Material source-system changes or multi-agency expansion are excluded","Resource-equivalent estimate, not documented expenditure"],"source_ids":["S3","S4","S8"]}},"verified_pipeline_gates":{"externally_supported_problem":{"status":"YES","reason":"Official guidance and operational documentation establish that missing data can be confused with zero, that sources stop reporting, and that these states require monitoring and corrective action.","source_ids":["S1","S3","S4"]},"externally_credible_adopter_or_authorizer":{"status":"YES","reason":"CDC NHSN and NSSP dashboard, surveillance, and data-governance staff are identifiable organizations already authorized to operate closely related data-quality workflows.","source_ids":["S2","S3"]},"distinct_testable_incremental_claim":{"status":"YES","reason":"The residual claim compares cause-matched actions against an equally salient generic warning while holding context, available actions, and nonfocal spacing constant; false closure and predefined appropriate follow-up are measurable.","source_ids":["S2","S6"]},"bounded_next_evidence_step":{"status":"YES","reason":"A preregistered, non-production randomized synthetic-case simulation has a defined population, five cause states, active comparator, primary outcome, safety outcomes, and explicit falsifiers.","source_ids":["S6","S7"]},"no_unresolved_safety_or_authority_stop":{"status":"YES","reason":"The first step changes no live entity status, uses synthetic records, preserves human authority, and can halt on misclassification, increased inappropriate escalation, inaccessible context, permission leakage, or accessibility failure. Organizational ethics review remains required but is not an intrinsic stop.","source_ids":["S3","S8"]},"credible_cost_scope_and_range":{"status":"UNCERTAIN","reason":"The four ranges are scoped resource-equivalent estimates with explicit assumptions, but no adopter architecture, procurement quote, staffing plan, sample quote, or integration inventory was publicly available.","source_ids":["S2","S3","S4"]}},"next_evidence_step":"With an authorized oversight or surveillance partner, preregister one non-production randomized simulation using synthetic entity-period records spanning confirmed no events, report not received, delayed feed, permission block, and system failure. Randomize regulator-like participants between (A) an equally salient generic missing-data panel preserving entity, period, prior values, timestamp, provenance, history, and the same underlying action set and (B) the identical display plus a validated cause label and exactly one matched recommended action. Independently adjudicate the correct response and cause before launch. Primary outcome: false no-incident closure. Secondary outcomes: predefined correct follow-up, decision time, cause comprehension, confidence calibration, context retrieval, and inappropriate escalation, analyzed by cause, expertise, and reporter-resource scenario. Falsify the claim if the matched-action condition has no practically material advantage on false closure and correct follow-up, or if any advantage is offset by materially higher inappropriate escalation, delay, inequitable scrutiny, context loss, or cause confusion. Halt if cause-label accuracy, permissions, privacy, or WCAG checks fail.","blocking_evidence":["No representative estimate of false no-incident closure under current or generic-warning interfaces was found.","No controlled study of cause-matched oversight actions versus an equally salient generic missing-data warning was found.","The accuracy and timeliness with which real systems can classify the five proposed causes are unknown.","Recruitment access, sample-size inputs, effect-size threshold, and ethics arrangements require an adopter or research partner.","Agency-specific legal authority, records, privacy, permission-disclosure, and workflow requirements remain unverified.","Cost ranges lack architecture inventories, staffing plans, procurement quotes, and specialized participant-recruitment quotes.","Live-workload transfer, strategic reporter adaptation, and distributional effects on lower-capacity reporting entities require later field evidence."],"research_disposition":"PARTNERED_RESEARCH_PROGRAM","world_novelty_boundary":"This bounded English-language web review establishes substantial collision with public guidance, standards, products, practices, and research but does not measure world novelty, patentability, freedom to operate, market size, or realized impact. Internal regulator systems, procurement records, unpublished evaluations, patents, non-English materials, and later publications may contain an equal or closer match.","arm":"SENTINEL","candidate_version":0,"controller_recommendation":{"action":"STOP_EMPIRICAL_RESEARCH_NEEDED","repairable":false,"material_progress_observed":true,"progress_targets":["Secure an authorized oversight or surveillance partner and confirm product, data-governance, privacy, records, accessibility, and research-ethics authority.","Measure baseline false no-incident closure and cause comprehension using the current and equally salient generic-warning comparators.","Validate the five-state cause classifier against independently adjudicated synthetic cases before exposing recommendations.","Preregister the minimum practically important reduction in false closure and the non-inferiority or harm thresholds for inappropriate escalation, delay, context retrieval, and subgroup disparity.","Obtain recruitment and engineering quotes and replace resource-equivalent cost assumptions with an adopter-specific budget.","Run the bounded randomized simulation and report results by cause, expertise, and reporter-resource scenario."],"reason":"Bounded web research has answered the public problem, adopter, prior-art, standards, and feasibility questions. The remaining incremental claim is causal and can only be resolved through participant testing with partner-specific workflow and authority inputs; additional general web search cannot establish it. Because this is a stop recommendation, repairable is correctly false."}}