{"schema_version":1,"research_id":"eoa_inverse_innovation_exp04_external_evaluation_20260802","source_assessment_id":"computability_boundary_mapping__ethnography_qualitative_methods:RETRIEVAL_FIRST:v0","cell_id":"computability_boundary_mapping__ethnography_qualitative_methods","search_queries":["site:plos.org \"A simple method to assess and report thematic saturation\"","\"Code Saturation Versus Meaning Saturation\" Hennink Kaiser Marconi","\"Saturation in qualitative research\" Saunders Sim Kingstone Baker Waterfield Bartlam Burroughs Jinks","site:fda.gov qualitative saturation grid interviews no new concepts","Nelson 2017 conceptual depth criteria reaching saturation qualitative research PDF","Q-FORS qualitative framework operationalizing respondent sampling data adequacy 2025","site:fda.gov \"Methods to Identify What Is Important to Patients\" saturation grid","site:cdc.gov field epidemiology manual collecting analyzing qualitative data saturation purposive","10.1177/1468794116679873 publication date","site:fda.gov/media/119241 E-RS COPD staff review date","10.1016/j.socscimed.2025.118781 publication date","NVivo saturation dashboard qualitative research saturation feature","MAXQDA saturation analysis feature code saturation","qualitative research software saturation grid stopping decision","site:maxqda.com/help/visual-tools/code-trends \"2026\" \"Code Trends\"","\"Using conceptual depth criteria\" \"First Published\""],"sources":[{"source_id":"S1","title":"A simple method to assess and report thematic saturation in qualitative research","publisher":"PLOS ONE","url":"https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0232076","source_class":"PRIMARY_RESEARCH","publication_date":"2020-05-05","accessed_at":"2026-08-02","claims_supported":["Prospective thematic-saturation assessment already uses a base size, run length, and new-information threshold.","Retrospective calculations on a fixed dataset necessarily reach 100 percent and do not establish future completeness.","The method was tested with bootstrapping on three previously coded interview datasets.","The method supplies bounded, study-objective-relative stopping evidence rather than a finite-frame exhaustion certificate."]},{"source_id":"S2","title":"Code Saturation Versus Meaning Saturation: How Many Interviews Are Enough?","publisher":"SAGE / Qualitative Health Research","url":"https://escholarship.org/content/qt2q0136v7/qt2q0136v7_noSplash_55041ffc42fa99e82f2f4f1e5265288d.pdf?t=rv9rwe","source_class":"PRIMARY_RESEARCH","publication_date":"2016-09-25","accessed_at":"2026-08-02","claims_supported":["In the examined 25-interview dataset, code saturation occurred at nine interviews while meaning saturation required 16 to 24 interviews.","Saturation depends on study purpose, population, code characteristics, and codebook complexity and stability.","The paper reports continuing uncertainty and weak operationalization around saturation.","Chronological code development and multiple-analyst checks demonstrate that relevant audit data can be retained."]},{"source_id":"S3","title":"Saturation in qualitative research: exploring its conceptualization and operationalization","publisher":"Springer Nature / Quality & Quantity","url":"https://eresearch.qmu.ac.uk/server/api/core/bitstreams/e19fcf6b-5ba2-457d-9cdb-3023e70fdc97/content","source_class":"AUTHORITATIVE_SECONDARY","publication_date":"2017-09-14","accessed_at":"2026-08-02","claims_supported":["Saturation is widely accepted but conceptualized and used inconsistently.","The review distinguishes four approaches to saturation with different inductive or deductive logics and different locations in data collection and analysis.","A saturation statement often supports the consequential judgment that further collection or analysis is unnecessary, despite uncertainty about its meaning.","The review does not establish literal completeness over all future admissible cases."]},{"source_id":"S4","title":"Using conceptual depth criteria: addressing the challenge of reaching saturation in qualitative research","publisher":"SAGE / Qualitative Research; Queen's University Belfast repository","url":"https://pureadmin.qub.ac.uk/ws/portalfiles/portal/121940009/ConceptPURE_Document.pdf","source_class":"PRIMARY_RESEARCH","publication_date":"2016-12-14","accessed_at":"2026-08-02","claims_supported":["Saturation is described as nebulous, insufficiently systematized, and nevertheless commonly required by funders and reviewers.","Conceptual-depth criteria provide structured evidence for a researcher’s study-specific theoretical-sufficiency judgment.","The criteria assess progress toward theory building rather than mechanically certifying that no future case can matter.","The framework demonstrates an established alternative to literal completeness claims."]},{"source_id":"S5","title":"Evaluating Respiratory Symptoms (E-RS) in COPD: Clinical Outcome Assessment Staff Review","publisher":"U.S. Food and Drug Administration","url":"https://www.fda.gov/media/119241/download","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2016-03-10","accessed_at":"2026-08-02","claims_supported":["FDA staff reviewed consequential saturation evidence defined by consecutive focus-group or interview groups producing no new concepts.","The reviewed evidence used a chronological saturation grid.","Additional interviews were undertaken to improve racial and ethnic diversity, showing that observed concept recurrence does not eliminate sampling-coverage concerns.","This is an implemented regulatory practice using bounded no-new-concept evidence, not finite-frame exhaustion."]},{"source_id":"S6","title":"Patient-Focused Drug Development: Methods to Identify What Is Important to Patients","publisher":"U.S. Food and Drug Administration","url":"https://www.fda.gov/media/131230/download","source_class":"OFFICIAL_GUIDANCE","publication_date":"2022-02-01","accessed_at":"2026-08-02","claims_supported":["FDA identifies patients, researchers, medical-product developers, and regulatory staff as stakeholders in transparent qualitative evidence collection.","The guidance defines saturation through iterative rounds in which no new relevant information emerges, while stating that there are no set evaluation criteria or methodology.","It describes sequential comparison of interview sets, additional interviews when necessary, saturation grids, demographic review, quality checks, audit trails, and chronological analysis.","The guidance leaves stopping dependent on research objectives and professional judgment."]},{"source_id":"S7","title":"Collecting and Analyzing Qualitative Data","publisher":"U.S. Centers for Disease Control and Prevention","url":"https://www.cdc.gov/field-epi-manual/php/chapters/qualitative-data.html","source_class":"OFFICIAL_GUIDANCE","publication_date":"2024-08-08","accessed_at":"2026-08-02","claims_supported":["Qualitative studies commonly use small purposive samples that do not support statistical generalization.","CDC describes saturation operationally as continuing interviews until additional interviews yield no additional insights, while recommending flexible sample-size ranges.","Clear eligibility criteria, ethics approvals, careful data organization, de-identification where required, and version control are workflow requirements.","Iterative coding and investigator judgment remain integral to qualitative analysis."]},{"source_id":"S8","title":"QDA Software Comparison: MAXQDA vs. NVivo vs. ATLAS.ti","publisher":"MAXQDA / VERBI Software","url":"https://www.maxqda.com/blogpost/qda-software-comparison","source_class":"COMMERCIAL_FIRST_PARTY","publication_date":"2026-06-10","accessed_at":"2026-08-02","claims_supported":["A deployed qualitative-analysis product already supports hierarchical codes, code and document variables, memos, code trends, summaries, project reports, and project interchange.","Code Trends can display code frequency across ordered documents, providing part of the data substrate needed for chronological novelty monitoring.","The product emphasizes traceability and researcher control over AI-assisted analysis.","The opened first-party description does not document the candidate's finite-frame-only exhaustion certificate or mandatory UNKNOWN_AFTER_BOUND state."]}],"problem_evidence":{"support":"STRONG","rationale":"Primary studies, methodological reviews, FDA practice, and official guidance consistently show that saturation is consequential, inconsistently operationalized, dependent on study purpose and coding depth, and commonly inferred from bounded runs with no new concepts. Code saturation can precede meaning saturation substantially, and fixed-dataset calculations cannot establish future completeness. The evidence strongly supports a real risk of overinterpreting bounded no-novelty observations, although it does not measure the prevalence of actual premature stopping or downstream harm.","source_ids":["S1","S2","S3","S4","S5","S6","S7"]},"stakeholder_evidence":{"support":"MODERATE","rationale":"FDA is an identifiable authorizer of patient-experience evidence and explicitly asks researchers and medical-product developers to use transparent, chronological saturation evidence, quality checks, audit trails, and demographic review. Funders and reviewers are also reported to demand saturation justifications. This establishes stakeholder pull for auditable stopping evidence, but no source expresses demand for the candidate's exact three-state semantics.","source_ids":["S4","S5","S6"]},"prior_art":{"proximity":"SUBSTANTIAL_COLLISION","closest_analogues":[{"name":"Guest, Namey, and Chen prospective thematic-saturation assessment","similarity":"Already replaces intuitive stopping with declared base size, run length, a new-information threshold, chronological evidence, and an auditable bounded calculation.","remaining_difference":"It continues to report saturation and does not reserve exact exhaustion for an enforceably finite eligible frame or mandate UNKNOWN_AFTER_BOUND for open-ended frames.","source_ids":["S1"]},{"name":"Code-saturation versus meaning-saturation assessment","similarity":"Already distinguishes shallow code discovery from richer conceptual understanding and records chronological code development, directly challenging a single binary saturation label.","remaining_difference":"It refines the meaning and evidentiary threshold of saturation but does not implement the proposed three-state output contract or finite-frame certificate.","source_ids":["S2"]},{"name":"Conceptual-depth and theoretical-sufficiency criteria","similarity":"Already rejects simple completeness implications and provides structured evidence for a study-relative adequacy judgment.","remaining_difference":"It remains an interpretive framework rather than a machine-enforced protocol that structurally preserves UNKNOWN and separately certifies finite enumeration.","source_ids":["S4"]},{"name":"FDA saturation grids and sequential interview-set comparisons","similarity":"Already provide chronological, consequential, auditable stopping evidence based on bounded runs with no new concepts and permit additional sampling when evidence or subgroup coverage is inadequate.","remaining_difference":"They neither require a finite, enforceable sampling frame before an exhaustion certificate nor prohibit labeling an open-ended bounded result as saturation.","source_ids":["S5","S6"]},{"name":"MAXQDA coding, memo, variable, and code-trend workflow","similarity":"A deployed product already stores and visualizes much of the chronological coding evidence that the proposed protocol would consume.","remaining_difference":"The reviewed first-party description supplies the data and workflow substrate but not the candidate's three decision states, frame-classification gate, or exhaustion-certificate rules.","source_ids":["S8"]}],"distinctive_claim_remaining":"On matched sampling histories containing admissible negative or theory-altering cases delayed until after an apparent no-new-code plateau, the mandatory three-state protocol will produce fewer premature terminal saturation calls than an ordinary bounded saturation workflow while issuing zero invalid FINITE_FRAME_EXHAUSTED certificates. The claim is falsified if premature calls do not decrease, any nonexhausted frame is certified, or reviewers routinely relabel or bypass UNKNOWN_AFTER_BOUND.","confidence":"HIGH"},"implementation_evidence":{"support":"MODERATE","rationale":"The state-transition logic and record format are technically straightforward, and existing practice already produces chronological grids, codebooks, memos, variables, and audit trails. Exact exhaustion can be mechanically checked only after a human-governed eligibility rule establishes a genuinely finite, enumerable frame and every member's disposition is recorded. Novelty, theoretical importance, and substantive adequacy remain interpretive judgments and must not be delegated to the state machine. Archived human-subject data require applicable approval, access control, minimization, and de-identification; a synthetic-only pilot avoids most legal risk. Shadow mode preserves study-lead authority and makes rollback simple. The unresolved implementation questions are reviewer agreement on frame classification, interoperability with real project records, and whether users respect rather than relabel UNKNOWN.","source_ids":["S1","S2","S5","S6","S7","S8"]},"scores":{"meaningful_impact":{"score":4,"rationale":"Preventing premature closure could improve representational adequacy and the trustworthiness of consequential qualitative findings, but the frequency and realized magnitude of harm have not been measured.","source_ids":["S2","S3","S5","S6"]},"stakeholder_pull":{"score":4,"rationale":"FDA, researchers, funders, and reviewers visibly need transparent saturation rationales and audit trails, although pull for this specific protocol is not demonstrated.","source_ids":["S4","S5","S6"]},"incremental_advantage":{"score":3,"rationale":"Mandatory UNKNOWN semantics and finite-frame-only exhaustion add a clear safeguard, but most data collection, novelty monitoring, documentation, and sufficiency reasoning already exist.","source_ids":["S1","S2","S4","S5","S6"]},"distinctiveness_plausibility":{"score":3,"rationale":"The enforced state semantics appear distinguishable within the bounded review, but adjacent methods and software are dense and world novelty remains unmeasured.","source_ids":["S1","S3","S4","S6","S8"]},"technical_implementability":{"score":4,"rationale":"A deterministic rules layer over structured frame, bound, chronology, and disposition records is feasible; semantic judgments and frame validity must remain human-governed.","source_ids":["S5","S6","S7","S8"]},"adoption_authority_feasibility":{"score":3,"rationale":"Study leads can authorize a shadow-mode workflow and FDA-facing researchers already maintain related evidence, but software maintainers, research governance, and methodologists must coordinate before live enforcement.","source_ids":["S5","S6","S8"]},"evidence_readiness":{"score":3,"rationale":"Comparators, outcomes, and falsifiers are well defined, but matched traces, qualified reviewers, and prospective protocol adherence data are not yet available.","source_ids":["S1","S2","S5","S6"]},"safety_net_benefit":{"score":4,"rationale":"Preserving UNKNOWN and prohibiting autonomous recruitment decisions is a meaningful non-deceptive fallback, provided finite-frame mistakes and indefinite-recruitment interpretations are actively monitored.","source_ids":["S3","S4","S6"]},"scalability":{"score":4,"rationale":"The state machine and record schema could be embedded in common qualitative-analysis workflows, though eligibility and novelty criteria remain study-specific and limit full automation.","source_ids":["S6","S8"]}},"score_confidence":"MODERATE","costs":{"first_evidence":{"band_2026_usd":"10K_TO_50K","scope":"Preregister, construct or curate 20 paired archived or synthetic trajectories, compensate two qualified blinded reviewers, run both protocols, and adjudicate disagreements.","confidence":"MODERATE","assumptions":["Existing de-identified traces or synthetic trajectories are available without new participant recruitment.","Two reviewers and one methodologist can complete the bounded exercise within roughly four to eight person-weeks.","No production-software integration is included."],"source_ids":[]},"initial_deployment_startup":{"band_2026_usd":"50K_TO_250K","scope":"Build a shadow-mode prototype, frame and chronology schema, rule engine, audit export, access controls, interoperability adapters, and researcher-facing warnings.","confidence":"LOW","assumptions":["One qualitative-analysis platform or export format is targeted initially.","The system does not perform autonomous coding or recruitment.","Institutional security and privacy review are limited to a nonproduction pilot."],"source_ids":["S6","S8"]},"operational_launch":{"band_2026_usd":"250K_TO_1M","scope":"Production integration, security and privacy validation, governance documentation, user research, training, support, monitoring, and multi-study rollout.","confidence":"LOW","assumptions":["Launch spans several research teams but not a regulated clinical decision system.","Human study leads retain stopping authority.","Existing identity, storage, and audit infrastructure can be reused."],"source_ids":["S6","S7","S8"]},"annual_recurring":{"band_2026_usd":"50K_TO_250K","scope":"Maintenance, support, rule and schema updates, access review, audit retention, user training, and periodic monitoring for bypass, relabeling, or invalid certificates.","confidence":"LOW","assumptions":["Deployment remains organizational rather than global.","No new participant recruitment is charged to software operations.","One to two engineering or research-operations staff equivalents support the service."],"source_ids":[]}},"verified_pipeline_gates":{"externally_supported_problem":{"status":"YES","reason":"Multiple direct primary, methodological, regulatory, and guidance sources show consequential but inconsistent saturation judgments based on bounded no-new-concept evidence.","source_ids":["S1","S2","S3","S4","S5","S6","S7"]},"externally_credible_adopter_or_authorizer":{"status":"YES","reason":"FDA is an identifiable authorizer of patient-experience evidence, and its guidance directly addresses researchers and medical-product developers who collect and submit qualitative evidence.","source_ids":["S5","S6"]},"distinct_testable_incremental_claim":{"status":"YES","reason":"The paired delayed-negative-case comparison can directly test premature terminal calls, invalid exhaustion certificates, UNKNOWN frequency, and relabeling behavior against established bounded saturation workflows.","source_ids":["S1","S5","S6"]},"bounded_next_evidence_step":{"status":"YES","reason":"A 20-trajectory, shadow-mode, paired review has a fixed sample, explicit comparators, outcomes, advance conditions, falsifiers, and no effect on live recruitment.","source_ids":["S1","S2","S5","S6"]},"no_unresolved_safety_or_authority_stop":{"status":"YES","reason":"The authorized first step is observational shadow evaluation. It leaves recruitment and interpretive authority with the study lead, can use synthetic or properly governed de-identified records, and can be rolled back without changing source data or live decisions.","source_ids":["S6","S7"]},"credible_cost_scope_and_range":{"status":"UNCERTAIN","reason":"The four bands are transparently scoped resource-equivalent estimates, but no direct labor, integration, licensing, privacy-review, or partner-specific cost benchmarks were obtained.","source_ids":[]}},"next_evidence_step":"Preregister and run one paired shadow evaluation on exactly 20 archived or synthetic sampling trajectories: 10 with an explicitly finite candidate ledger and 10 with an open-ended frame. Where the trace permits, place an admissible theory-altering or negative case after an apparent no-new-code plateau. Have blinded qualified reviewers apply (A) the Guest/FDA-style bounded saturation workflow and (B) the three-state protocol to identical round histories. Measure premature terminal calls before the delayed case, invalid FINITE_FRAME_EXHAUSTED certificates, UNKNOWN_AFTER_BOUND frequency, agreement on finite/open frame classification, additional rounds requested, and reviewer attempts to relabel or bypass UNKNOWN. Qualitatively adjudicate every disagreement and stop after 20 paired traces. Advance only if the candidate issues zero invalid exhaustion certificates and reduces premature terminal calls without routine UNKNOWN relabeling. Falsify the intervention if either primary condition fails; do not change live recruitment or published findings.","blocking_evidence":["No paired evaluation has measured whether the protocol reduces premature saturation calls relative to established bounded workflows.","No evidence establishes reliable agreement among qualified researchers on finite versus open-ended frame classification.","No behavioral evidence shows that researchers will preserve UNKNOWN_AFTER_BOUND rather than relabel, bypass, or treat it as a command for indefinite recruitment.","No software maintainer, research organization, regulator, or funder has committed to adopt or fund the protocol.","Real archived trajectories may require data-use agreements, ethics review, de-identification, and access controls that have not been arranged.","The bounded product search did not establish whether another deployed tool already implements both mandatory UNKNOWN and finite-frame-only exhaustion.","Labor, integration, governance, and recurring-cost estimates lack external benchmarks."],"research_disposition":"PARTNERED_RESEARCH_PROGRAM","world_novelty_boundary":"Established prior art includes prospective bounded saturation calculations, code-versus-meaning saturation, conceptual-depth and theoretical-sufficiency frameworks, chronological saturation grids, regulatory use of consecutive no-new-concept runs, and qualitative-analysis products with codes, trends, variables, memos, and audit-oriented reporting. The only residual distinction supported by this bounded review is enforcement of a three-state output contract that reserves exact exhaustion for an enforceably finite, fully accounted frame and otherwise preserves bounded non-discovery as UNKNOWN. Absence of a direct match in these searches is not evidence of world novelty. Patentability, freedom to operate, exhaustive product or standards coverage, market size, and realized impact remain unmeasured.","arm":"RETRIEVAL_FIRST","candidate_version":0,"controller_recommendation":{"action":"STOP_EMPIRICAL_RESEARCH_NEEDED","repairable":true,"material_progress_observed":true,"progress_targets":["Secure access to 20 suitable de-identified archived or synthetic trajectories and document whether each supports a defensible finite/open frame classification.","Preregister the paired comparator, delayed-negative-case construction, primary metrics, zero-invalid-certificate requirement, and premature-call improvement criterion.","Recruit blinded qualified reviewers and measure inter-reviewer agreement on frame classification and protocol outputs.","Demonstrate zero invalid FINITE_FRAME_EXHAUSTED certificates and fewer premature terminal calls than the bounded saturation baseline.","Measure and qualitatively investigate every UNKNOWN relabeling, bypass, or indefinite-recruitment interpretation.","Obtain required data-governance approvals for archived records or complete the pilot with synthetic records only.","Replace broad deployment-cost assumptions with partner-specific labor, integration, security, training, and maintenance estimates."],"reason":"Web research materially verifies the problem, adopter/authorizer context, dense prior art, and a narrow testable residual claim. The decisive remaining evidence concerns comparative behavior on sampling trajectories, reviewer agreement, and workflow response to UNKNOWN; those questions require proprietary or constructed traces and live human evaluation and cannot be resolved by further bounded web search."}}