{"schema_version":1,"research_id":"eoa_inverse_innovation_exp06_external_evaluation_20260803","source_assessment_id":"predictive_residual_processing__sociology_anthropology:P1:v0","cell_id":"predictive_residual_processing__sociology_anthropology","search_queries":["site:acf.hhs.gov qualitative research public benefits service delivery interviews observations timely analysis","ethnography qualitative analysis time consuming full field notes rapid assessment framework method primary research","Framework Method qualitative data matrix prior art Gale Heath Cameron Rashid Redwood 2013","critical incident technique qualitative research unusual events prior art","site:acf.hhs.gov opre qualitative methods public benefits human services program evaluation qualitative data analysis burden","site:acf.hhs.gov opre ethnography benefit access intake qualitative research","site:aspe.hhs.gov qualitative research public benefits access administrative burden","site:gao.gov public benefits administrative burden access application processes report","OMB M-22-10 qualitative research customer experience public benefits official","rapid qualitative analysis ethnography time savings primary study analyst hours","rapid ethnographic assessment procedures public health time sensitive primary research","COREQ checklist field notes context reflexivity qualitative research official","eCFR 45 CFR 46 IRB changes approved research prior approval official","HHS OHRP coded private information deidentified secondary research IRB official","NIH qualitative data sharing confidentiality ethnographic data official guidance","anthropological ethics code do no harm informed consent official AAA","\"Rapid versus traditional qualitative analysis\" Veterans Affairs DOI","\"Can rapid approaches to qualitative analysis deliver timely, valid findings\" DOI","\"Rapid ethnographic assessment\" Bentley Pelto PubMed","site:americananthro.org \"Code of Ethics\" do no harm informed consent"],"sources":[{"source_id":"S1","title":"Administrative Burden: OMB Should Update Instructions to Help Agency Assessment Efforts","publisher":"U.S. Government Accountability Office","url":"https://www.gao.gov/products/gao-25-107239","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2025-04-21","accessed_at":"2026-08-03","claims_supported":["Benefit-access burden visibly matters: GAO reports that eligible Americans forgo more than $140 billion in benefits partly because of administrative burdens.","USDA, VA, and SSA have customer-experience offices that collect feedback and identify burden-reduction solutions, providing identifiable potential adopters and funders of qualitative service research.","Agencies may be missing burdens and opportunities to prioritize improvements."]},{"source_id":"S2","title":"Ethnographic study: qualitative studies","publisher":"UK Office for Health Improvement and Disparities","url":"https://www.gov.uk/guidance/ethnographic-study-qualitative-studies","source_class":"OFFICIAL_GUIDANCE","publication_date":"2020-05-04","accessed_at":"2026-08-03","claims_supported":["Ethnography can uncover local practices, marginalized perspectives, and unintended consequences that other methods miss.","Ethnographic fieldwork and repeated interpretive analysis are time-consuming and skill-dependent.","Field notes contain factual material plus researcher interpretation; reflexivity, team sense-checking, participant validation, and constant comparison protect validity.","Rapid ethnographic approaches already exist."]},{"source_id":"S3","title":"Rapid versus traditional qualitative analysis using the Consolidated Framework for Implementation Research (CFIR)","publisher":"Implementation Science / Springer Nature","url":"https://link.springer.com/article/10.1186/s13012-021-01111-5","source_class":"PRIMARY_RESEARCH","publication_date":"2021-07-02","accessed_at":"2026-08-03","claims_supported":["Traditional qualitative analysis is resource-intensive and can conflict with constrained timelines, funding, and staffing.","A structured rapid approach using notes, audio verification, and a facility-by-framework matrix required 409.5 analyst hours versus 683 hours and eliminated $7,250 in transcription costs in the studied VHA evaluations.","Rapid qualitative analysis using templates and matrices is established adjacent practice, not a novel category.","The study emphasizes expertise, required detail, data mode, and tradeoffs, and notes that participant privacy limited dataset availability."]},{"source_id":"S4","title":"Can rapid approaches to qualitative analysis deliver timely, valid findings to clinical leaders? A mixed methods study comparing rapid and thematic analysis","publisher":"University of Warwick Research Archive; article published in BMJ Open","url":"https://wrap.warwick.ac.uk/id/eprint/175956/","source_class":"PRIMARY_RESEARCH","publication_date":"2018-10-08","accessed_at":"2026-08-03","claims_supported":["Rapid analysis data management took 43 hours versus 116.5 hours for thematic analysis, but total time savings were modest: 100 versus 126.5 hours.","Rapid-analysis findings overlapped with 79% of thematic-analysis findings, while recommendation overlap was only 55%, showing that speed does not establish interpretive equivalence.","The authors call for further comparisons across additional datasets and comparable teams, supporting a paired empirical benchmark."]},{"source_id":"S5","title":"Using the framework method for the analysis of qualitative data in multi-disciplinary health research","publisher":"University of Manchester Research Explorer; article published in BMC Medical Research Methodology","url":"https://research.manchester.ac.uk/en/publications/using-the-framework-method-for-the-analysis-of-qualitative-data-i/","source_class":"PRIMARY_RESEARCH","publication_date":"2013-09-18","accessed_at":"2026-08-03","claims_supported":["The Framework Method is an established systematic and flexible way to manage and analyze qualitative data in multidisciplinary teams.","Existing framework-based analysis is a close analogue to site-scoped encounter templates and cross-case matrices.","Effective use still requires leadership by an experienced qualitative researcher."]},{"source_id":"S6","title":"Rapid ethnographic assessment of the COVID-19 pandemic April 2020 'surge' and its impact on service delivery in an Acute Care Medical Emergency Department and Trauma Center","publisher":"U.S. National Library of Medicine / PubMed; article published in BMJ Open","url":"https://pubmed.ncbi.nlm.nih.gov/33082198/","source_class":"PRIMARY_RESEARCH","publication_date":"2020-10-20","accessed_at":"2026-08-03","claims_supported":["A structured protocol for rapid ethnographic observation and analysis of service-delivery encounters is technically and methodologically feasible.","Rapid clinical ethnography can identify procedural, organizational, provider, and patient changes during a changing service context.","The study was limited to one setting, illustrating the scope and generalizability limits relevant to site-specific templates."]},{"source_id":"S7","title":"Anthropological Ethics","publisher":"American Anthropological Association","url":"https://americananthro.org/about/anthropological-ethics/","source_class":"STANDARD","publication_date":"n.d.","accessed_at":"2026-08-03","claims_supported":["Professional anthropology ethics require avoiding harm, openness, informed consent and permissions, weighing obligations to affected parties, protecting records, and respectful relationships.","Efficiency cannot override participant welfare, consent, confidentiality, or record stewardship."]},{"source_id":"S8","title":"Institutional Review Board (IRB) Written Procedures: Guidance for Institutions and IRBs","publisher":"U.S. Department of Health and Human Services, Office for Human Research Protections","url":"https://www.hhs.gov/ohrp/regulations-and-policy/requests-for-comments/guidance-for-institutions-and-irbs/index.html","source_class":"OFFICIAL_GUIDANCE","publication_date":"2025-06-06 (content last reviewed)","accessed_at":"2026-08-03","claims_supported":["IRBs assess risk minimization, informed consent, privacy, confidentiality, safeguards for vulnerability to coercion, and study changes.","Changes to approved research generally may not be initiated without prior IRB review and approval except to eliminate immediate hazards.","The institution determines whether a study is subject to IRB review or exempt; a PI cannot self-authorize a material analytic change.","IRBs can approve, require modifications, disapprove, suspend, or terminate research."]}],"problem_evidence":{"support":"MODERATE","rationale":"The general problem is visible: official guidance describes ethnography as time-consuming and interpretively demanding, and two comparative studies quantify substantial qualitative-analysis labor. Benefit-access encounters also matter because agencies are actively trying to detect and reduce administrative burdens. However, no opened source establishes the candidate's specific prevalence claim that multi-site benefit-intake ethnographies routinely have review queues in which consequential deviations are discovered late after initially reviewing repetitive narratives.","source_ids":["S1","S2","S3","S4"]},"stakeholder_evidence":{"support":"MODERATE","rationale":"Identifiable adopter classes exist. GAO reports that USDA, VA, and SSA have customer-experience offices collecting feedback to identify burden-reduction solutions, while VHA has funded and used structured rapid qualitative analysis. An IRB is the identifiable authorizer for a covered protocol change. No source shows an organization requesting, purchasing, funding, or committing to this particular prediction-plus-residual design, so demonstrated stakeholder pull remains weak despite a credible adoption and authorization path.","source_ids":["S1","S3","S8"]},"prior_art":{"proximity":"ADJACENT_PRIOR_ART","closest_analogues":[{"name":"Rapid CFIR-based qualitative analysis","similarity":"Uses detailed contemporaneous notes, a predefined framework, facility-by-construct matrices, secondary verification, and an explicit goal of reducing analyst time.","remaining_difference":"It summarizes all material into framework matrices; it does not create a pre-observation expected encounter, transmit signed reconstructive edits against a synchronized model version, or govern suppression through protected bypasses and random raw-note audits.","source_ids":["S3"]},{"name":"Framework Method","similarity":"Provides systematic matrix-based organization and cross-case comparison using an analytic framework in multidisciplinary qualitative teams.","remaining_difference":"Framework coding is not a prediction-residual codec: categories need not be prospective expectations, matrices need not reconstruct baseline-plus-edit narratives, and model synchronization, fallback, and suppression-error budgets are absent.","source_ids":["S5"]},{"name":"Rapid clinical ethnography with a structured protocol","similarity":"Uses structured, team-based rapid observation and analysis to identify procedural and interactional changes in service delivery.","remaining_difference":"It accelerates ethnography but does not route only model-relative deviations or evaluate what a maintained predictor suppressed.","source_ids":["S6"]},{"name":"Rapid versus thematic analysis","similarity":"Directly compares a streamlined qualitative workflow against fuller thematic analysis using time and output-agreement measures.","remaining_difference":"It is a comparator study, not a versioned expected-encounter system; its incomplete finding and recommendation overlap underscores rather than solves contextual-fidelity risk.","source_ids":["S4"]}],"distinctive_claim_remaining":"Compared with both full-note chronological review and conventional rapid template/matrix analysis, a locked, site-scoped expected-encounter model plus reconstructive signed edits, full-context protected bypasses, and independent raw-note audits will reduce total analyst time-to-escalation by at least 20% while keeping overall material-deviation recall within 5 percentage points of independent full-note review and producing zero missed rights-, consent-, coercion-, discrimination-, or safeguarding-class events in the tested sample. This is contrastive and falsifiable but wholly unverified.","confidence":"HIGH"},"implementation_evidence":{"support":"MODERATE","rationale":"Structured templates, matrices, rapid encounter ethnography, secondary verification, and timed comparison against full analysis are all demonstrated. Versioning, secure raw-note retention, locked templates, audit sampling, and fallback are technically straightforward on existing qualitative-research infrastructure. The exact prospective encounter predictor, signed edit schema, consequence gate, reconstruction interface, subgroup audit, and protected bypass have not been field-tested. Workflow feasibility depends on expert coding consistency; data feasibility depends on authorized access to sensitive notes; authority depends on prior IRB or institutional determination; and safety depends on demonstrating that apparently routine context and minority experiences are not suppressed.","source_ids":["S2","S3","S4","S5","S6","S7","S8"]},"scores":{"meaningful_impact":{"score":3,"rationale":"On a 1-5 scale, earlier recognition of access barriers could matter, but the proposed pilot changes only research review, not benefit decisions, and realized impact is unmeasured.","source_ids":["S1","S2"]},"stakeholder_pull":{"score":2,"rationale":"Agencies and VHA visibly need timely qualitative feedback and burden-reduction insight, but no external stakeholder has expressed demand for the residual architecture itself.","source_ids":["S1","S3"]},"incremental_advantage":{"score":3,"rationale":"The governance package could improve auditability and protected-context retention over ordinary rapid matrices, but it adds maintenance and audit costs and has no demonstrated advantage yet.","source_ids":["S3","S4","S5"]},"distinctiveness_plausibility":{"score":3,"rationale":"The exact prospective baseline-plus-edit reconstruction with version checks, bypasses, and raw audits was not found in the eight-source review, although most constituent practices are established and tightly adjacent.","source_ids":["S3","S5","S6"]},"technical_implementability":{"score":4,"rationale":"A retrospective shadow prototype can be built with ordinary secure qualitative-data, form, matrix, versioning, and review tools; the difficult issue is interpretive reliability rather than software feasibility.","source_ids":["S3","S5","S6"]},"adoption_authority_feasibility":{"score":3,"rationale":"A PI can sponsor the analytic pilot and an IRB or institutional office can authorize it, but prior review may be required and no participating study or agency has committed data or staff.","source_ids":["S1","S8"]},"evidence_readiness":{"score":3,"rationale":"The proposed outcomes and comparator methods are clear and existing authorized notes could support a retrospective test, but access to suitable proprietary notes and independent analysts is not established.","source_ids":["S3","S4","S8"]},"safety_net_benefit":{"score":4,"rationale":"Raw-note preservation, unrestricted full-note access, protected full-context routing, audit samples, and rollback directly address known ethical and interpretive hazards, provided the bypass is empirically validated.","source_ids":["S2","S7","S8"]},"scalability":{"score":3,"rationale":"Templates and matrices can scale across analysts, but every site and service context requires local calibration, expert oversight, ethics review, drift monitoring, and audits; these costs may offset attention savings.","source_ids":["S3","S5","S6"]}},"score_confidence":"MODERATE","costs":{"first_evidence":{"band_2026_usd":"50K_TO_250K","scope":"Preregister, obtain institutional/IRB determination, prepare up to 60 authorized notes, build and lock templates, run three-arm shadow review, adjudicate outcomes, analyze subgroup and rights-class misses, and report results over roughly 8-14 weeks.","confidence":"MODERATE","assumptions":["Existing secure storage and qualitative software are available.","The study supplies up to 60 already-authorized notes across no more than three sites.","Two independent analysts, one adjudicating senior qualitative researcher, a part-time data steward, statistician, and IRB liaison contribute approximately 500-1,200 fully loaded hours.","No new participant recruitment, transcription, field collection, or operational benefit-system integration occurs.","The band is resource-equivalent, not a vendor quote; comparative studies support analyst-hour materiality but not this pilot's exact cost."],"source_ids":["S3","S4","S8"]},"initial_deployment_startup":{"band_2026_usd":"250K_TO_1M","scope":"Convert a successful prototype into a secure three-site research workflow: configurable templates, access controls, model-version logging, full-note retrieval, bypass rules, audit sampling, training, validation, governance documentation, and protocol amendments.","confidence":"LOW","assumptions":["Deployment remains within research analysis and does not influence eligibility or service delivery.","Existing institutional identity, storage, and qualitative-research platforms can be extended rather than replaced.","Costs include privacy/security review, community-advisory work, training, validation, and contingency but exclude new enterprise procurement.","Local site adaptation and expert review dominate software cost."],"source_ids":["S2","S3","S7","S8"]},"operational_launch":{"band_2026_usd":"250K_TO_1M","scope":"Launch across the bounded multi-site study with parallel full-note record-of-record review, staffed escalation, audit and fallback operations, monitoring, and an independent evaluation period.","confidence":"LOW","assumptions":["Launch lasts 6-12 months and retains full-note review until noninferiority and safety criteria are met.","At least two qualitative analysts, a senior methodologist, data steward, PI effort, community-advisory compensation, and technical support are funded.","No use is made in participant credibility, eligibility, enforcement, or other benefit decisions.","A wider agency or cross-program launch would exceed this band."],"source_ids":["S1","S3","S7","S8"]},"annual_recurring":{"band_2026_usd":"250K_TO_1M","scope":"Annual operation for up to three sites: analyst triage, raw-note audits, template review, model-version administration, privacy/security support, community and ethics governance, drift reviews, retraining, and independent quality checks.","confidence":"LOW","assumptions":["One to three analyst FTE equivalents plus fractional methodologist, data-steward, PI, governance, and technical-support effort are required.","All raw notes continue to be stored under existing retention rules.","Quarterly full-baseline reviews and continuous protected-class audits are retained.","Savings cannot be credited until total maintenance and audit labor is measured in the empirical pilot."],"source_ids":["S3","S4","S7","S8"]}},"verified_pipeline_gates":{"externally_supported_problem":{"status":"YES","reason":"External official and primary sources show that ethnography and conventional qualitative analysis are resource-intensive and that benefit agencies need better insight into access burdens. Exact queue prevalence in benefit-intake ethnography remains unmeasured but does not negate the supported general problem.","source_ids":["S1","S2","S3","S4"]},"externally_credible_adopter_or_authorizer":{"status":"YES","reason":"USDA, VA, and SSA customer-experience offices are identifiable potential adopters; VHA has already funded rapid qualitative-analysis work; and an IRB or institutional human-research office is the identifiable authorizer. No adopter commitment to this design is established.","source_ids":["S1","S3","S8"]},"distinct_testable_incremental_claim":{"status":"YES","reason":"The residual design can be tested against full-note review and conventional rapid matrix analysis using preregistered time, recall, interpretation, subgroup, rights-class, fallback, and privacy outcomes.","source_ids":["S3","S4","S5"]},"bounded_next_evidence_step":{"status":"YES","reason":"A retrospective shadow study of at most 60 already-authorized notes across at most three sites is bounded and does not alter participant treatment or benefit decisions.","source_ids":["S3","S4","S8"]},"no_unresolved_safety_or_authority_stop":{"status":"UNCERTAIN","reason":"The design contains strong safeguards, but the relevant institution has not determined whether the new analysis is within the approved protocol, exempt, or requires amendment. The protected bypass and subgroup/context fidelity also require empirical validation before any residual-only workflow.","source_ids":["S2","S7","S8"]},"credible_cost_scope_and_range":{"status":"YES","reason":"The scopes are bounded and labor-dominated, with 2026 resource-equivalent ranges anchored qualitatively by published analyst-hour comparisons. Confidence is only moderate for the first test and low for deployment because no site-specific labor rates, data volumes, integration requirements, or vendor quotes are available.","source_ids":["S3","S4"]}},"next_evidence_step":"After obtaining an institutional or IRB determination, preregister a retrospective, counterbalanced shadow study using no more than 60 deidentified, already-authorized encounter notes from no more than three sites. Derive the site templates only from a temporally earlier training subset and lock the model and bypass rules before testing. Compare three conditions on the held-out notes: (A) independent full-note review as the record-of-record reference, (B) established rapid template/framework-matrix analysis, and (C) the proposed prediction-plus-signed-residual packet with unrestricted full-note retrieval. Counterbalance trained analysts across B and C and use a blinded senior panel to adjudicate material deviations. Measure total minutes including template creation, maintenance, audits, fallback, interpretation, and write-up; time to first escalation; overall and subgroup-specific material-deviation recall; rights/consent/coercion/discrimination/safeguarding recall; interpretation agreement; false escalation; full-note request rate; uncodable-content rate; version failures; reidentification/privacy incidents; and analyst acceptability. Falsify the incremental claim if C fails to save at least 20% total time versus the faster of A or B, loses more than 5 percentage points of overall material-deviation recall versus A, misses any protected-class event in the tested sample, produces a greater than 10% consequential interpretation-disagreement rate, creates a site/subgroup-specific miss cluster, causes any material privacy incident, or consumes its savings through maintenance, audit, and fallback work. Do not replace full-note review or change benefit operations during this test.","blocking_evidence":["No external prevalence estimate for repetitive full-note review, queue latency, or late discovery of consequential changes in public-benefit encounter ethnographies.","No participating study, agency, data steward, community advisory body, or qualitative-research team has committed notes, staff, funds, or adoption authority.","No paired evidence compares prediction-plus-residual ethnographic review with both full-note review and conventional rapid framework analysis.","No empirical evidence establishes protected-class recall, subgroup parity, contextual reconstruction fidelity, audit sampling adequacy, or fallback burden.","No institution-specific determination establishes whether the proposed secondary analysis is already authorized, exempt, or requires prior IRB amendment.","No site-specific cost data, note volumes, analyst rates, integration estimates, security assessment, or privacy/reidentification assessment are available."],"research_disposition":"PARTNERED_RESEARCH_PROGRAM","world_novelty_boundary":"This bounded eight-source review found established rapid qualitative analysis, framework matrices, and structured rapid ethnography, but did not find the exact governed prediction-plus-reconstructive-edit workflow. That absence is not evidence of world novelty. World novelty, patentability, freedom to operate, market size, comprehensive literature coverage, and realized impact remain unmeasured.","arm":"COMPLETE_PROPOSAL_PORTFOLIO","candidate_version":0,"controller_recommendation":{"action":"STOP_EMPIRICAL_RESEARCH_NEEDED","repairable":false,"material_progress_observed":true,"progress_targets":["Secure a written IRB or institutional determination covering reuse of the notes, residual packet creation, analyst access, audit sampling, retention, and community oversight.","Obtain a committed study partner with up to 60 authorized notes across no more than three sites, qualified independent analysts, a data steward, and advisory-panel participation.","Preregister the three-condition comparator protocol, temporal template lock, adjudication rules, 20% total-time advantage threshold, 5-percentage-point noninferiority margin, and zero protected-class-miss rule.","Measure total workflow cost, contextual reconstruction disagreement, subgroup and rights-class recall, false escalation, fallback and audit burden, version failures, privacy incidents, and analyst acceptability.","Stop or redesign if maintenance and audit work erase savings, any protected-class event is missed, contextual disagreement exceeds the bound, or errors cluster by site or subgroup."],"reason":"Web research establishes a meaningful general problem, credible adopter and authorizer classes, adjacent prior art, a distinct testable contrast, and a bounded protocol. It cannot establish whether the design saves net attention or preserves contextual, subgroup, and rights-bearing meaning. Those decisive facts require authorized proprietary notes, independent human interpretation, and live comparative testing. Under the controller rule, this is an empirical-research stop and therefore repairable is false."},"proposal_index":1}