{"schema_version":1,"research_id":"eoa_inverse_innovation_exp06_external_evaluation_20260803","source_assessment_id":"representation_independent_interface_contract__sociology_anthropology:P4:v0","cell_id":"representation_independent_interface_contract__sociology_anthropology","search_queries":["site:aapor.org sampling frame coverage duplication household surveys report","site:unece.org GSIM sampling frame sample statistical official","site:unstats.un.org household sample surveys sampling frame duplicates coverage","probability sampling frame duplicates inclusion probability research paper","official software sampling frame management reproducible random seed survey sample draw audit","site:statcan.gc.ca sampling frame management duplicates identifiers survey official","site:abs.gov.au sampling frame business register duplicate units official","survey sampling software reproducible samples random seed inclusion probabilities package documentation","UNECE GSIM Sampling Frame Sample official specification","site:unece.org \"Sampling Frame\" \"GSIM\"","DDI lifecycle sampling frame specification sample official","survey sampling frame data model standard sampling unit inclusion probability","official guidance pseudonymised identifiers remain personal data research participants handles","site:ico.org.uk pseudonymisation personal data research identifiers guidance","site:hhs.gov research identifiable private information IRB coded data guidance","site:aapor.org sampling frame coverage error duplicate listings probability sample official report HTML","site:pewresearch.org sampling frame coverage error address based sampling report","site:cdc.gov sampling frame duplicates survey official methodology"],"sources":[{"source_id":"S1","title":"Statistical Quality Standard A3: Developing and Implementing a Sample Design","publisher":"U.S. Census Bureau","url":"https://www.census.gov/about/policies/quality/standards/standarda3.html","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2021","accessed_at":"2026-08-03","claims_supported":["Sampling frames must address accuracy, completeness, eligibility, coverage, updating, and unduplication.","Sampling systems and code must be verified and tested against specifications.","Selection probabilities, design information, and documentation needed for replication must be retained.","Protected or administratively restricted frame information must not be released without authorization.","The Census Bureau is an identifiable institutional authorizer whose personnel and funded contractors must follow these requirements."]},{"source_id":"S2","title":"Statistics Canada Quality Guidelines: Coverage and frames","publisher":"Statistics Canada","url":"https://www150.statcan.gc.ca/n1/pub/12-539-x/steps-etapes/4147788-eng.htm","source_class":"OFFICIAL_GUIDANCE","publication_date":"undated archived guidance","accessed_at":"2026-08-03","claims_supported":["Frame omissions, erroneous inclusions, duplicates, and misclassifications are recognized coverage errors.","Frame defects can bias or reduce the reliability of estimates and increase collection costs.","Official guidance calls for duplicate elimination, updating, quality monitoring, alternative-source matching, and documentation.","Frame identifiers and characteristics are used across selection, collection, linkage, estimation, and analysis workflows."]},{"source_id":"S3","title":"Methodology: 2022–23 Survey of Asian Americans","publisher":"Pew Research Center","url":"https://www.pewresearch.org/2023/07/19/aa-global-views-methodology/","source_class":"PRIMARY_RESEARCH","publication_date":"2023-07-19","accessed_at":"2026-08-03","claims_supported":["A real probability survey used overlapping address and supplemental list frames.","Addresses appearing in multiple frames had multiple selection opportunities, requiring base-weight adjustment for their higher selection probability.","The workflow explicitly distinguished completed, incomplete, ineligible, and unknown eligibility states.","Frame limitations such as vacant, nonexistent, or multi-household drop-point addresses affected operational treatment."]},{"source_id":"S4","title":"Survey Development — DDI Lifecycle 3.3 Technical Guide","publisher":"DDI Alliance","url":"https://ddi-lifecycle-technical-guide.readthedocs.io/en/latest/Specific%20Structures/Survey%20Development.html","source_class":"STANDARD","publication_date":"2020","accessed_at":"2026-08-03","claims_supported":["DDI already models SamplingPlan, SampleFrame, SampleStep, SamplingStage, Sample, and weighting information.","Sampling-stage metadata can describe sampling units, frame requirements, probability of selection, stratification, commands, and processes.","This is close prior art for representation-independent documentation, but the opened guide does not specify opaque handles or cross-backend behavioral conformance."]},{"source_id":"S5","title":"sampling: Survey Sampling, version 2.11","publisher":"Comprehensive R Archive Network","url":"https://cran.r-project.org/web/packages/sampling/index.html","source_class":"OFFICIAL_PRODUCT_DOCUMENTATION","publication_date":"2025-07-10","accessed_at":"2026-08-03","claims_supported":["Open-source software already implements multiple sample-drawing schemes, unequal-probability designs, calibration weights, estimators, and variance estimators.","Existing software substantially reduces the algorithm-development burden for a prototype.","The package documentation does not present the proposed opaque frame abstraction or a backend-neutral frame conformance contract."]},{"source_id":"S6","title":"PROC SURVEYSELECT Statement, SAS/STAT 13.1 User's Guide","publisher":"SAS Institute","url":"https://support.sas.com/documentation/cdl/en/statug/66859/HTML/default/statug_surveyselect_syntax01.htm","source_class":"OFFICIAL_PRODUCT_DOCUMENTATION","publication_date":"2014","accessed_at":"2026-08-03","claims_supported":["Commercial software already supports stratified and unequal-probability sampling, explicit sample sizes or rates, replicates, and persisted design information.","A stored seed, unchanged input data, unchanged selection parameters, and compatible random-number generator can reproduce a sample.","The documentation shows that input ordering and strata ordering can be operationally significant for some procedures, while generator-version changes can affect reproducibility.","Seeded reproducibility alone is therefore narrower than representation-independent equivalence."]},{"source_id":"S7","title":"Pseudonymisation","publisher":"UK Information Commissioner's Office","url":"https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/data-sharing/anonymisation/pseudonymisation/","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"undated current guidance; under review in 2026","accessed_at":"2026-08-03","claims_supported":["Replacing identifiers with reference numbers and storing identifying information separately is pseudonymisation.","Pseudonymised personal data remains within data-protection law rather than becoming anonymous.","Technical and organizational controls and separation of reidentification information remain necessary.","Opaque recruitment handles reduce risk but do not by themselves eliminate legal or security obligations."]},{"source_id":"S8","title":"U.S. Survey Methodology","publisher":"Pew Research Center","url":"https://www.pewresearch.org/u-s-survey-methodology/","source_class":"OFFICIAL_ORGANIZATION_DATA","publication_date":"undated current methodology","accessed_at":"2026-08-03","claims_supported":["A major survey organization explicitly treats differences between the frame population and target population as coverage error.","Probability-based survey practice depends on identifiable frames, while special populations can be difficult to frame and contact consistently.","This supports the importance of frame governance but does not establish the prevalence of representation-coupled software migrations."]}],"problem_evidence":{"support":"MODERATE","rationale":"The underlying problem clearly exists and matters: official guidance identifies duplication, omission, erroneous inclusion, misclassification, outdated attributes, and confidentiality as consequential frame risks, while Pew documents a real overlapping-frame workflow in which multiple appearances changed selection opportunity and required weight correction. Census requires verified sampling systems and reproducible documentation. However, no opened source directly measures how often a storage-only migration, row reordering, blank convention, or identifier rewrite changes a multi-wave household draw. The broad sampling-frame integrity problem is externally verified; the proposal's specific representation-coupling prevalence remains an evidence gap.","source_ids":["S1","S2","S3","S8"]},"stakeholder_evidence":{"support":"MODERATE","rationale":"The U.S. Census Bureau is an identifiable authorizer and adopter class: its binding quality standard requires sound frames, unduplication, protected-information controls, specifications, code validation, integrated testing, and replication documentation. Statistics Canada and Pew demonstrate comparable operational needs. This is credible institutional pull for the functions, but no named study team, field office, funder, or governance body has expressed demand for this particular opaque cross-backend contract.","source_ids":["S1","S2","S3"]},"prior_art":{"proximity":"ADJACENT_PRIOR_ART","closest_analogues":[{"name":"U.S. Census Bureau Statistical Quality Standard A3","similarity":"Requires sampling-frame specifications, unduplication, protection of restricted information, code validation, integrated testing, monitoring, and replication documentation.","remaining_difference":"It states organizational quality obligations but does not define an opaque abstract-unit API, representation laws, or one conformance oracle for substitutable backends.","source_ids":["S1"]},{"name":"DDI Lifecycle 3.3 sampling information model","similarity":"Provides standardized objects for sampling plans, frames, stages, units, selection probabilities, processes, samples, and weighting.","remaining_difference":"It is primarily a metadata and documentation standard; the opened specification does not make storage permutation invariance, deterministic cross-backend draws, opaque handles, or behavioral tests an acceptance rule.","source_ids":["S4"]},{"name":"SAS PROC SURVEYSELECT and the R sampling package","similarity":"Implement established probability-sampling algorithms, strata, unequal probabilities, weights, replicates, and seeded reproducibility.","remaining_difference":"They operate on concrete input data and parameters rather than certifying that independently represented frames denote identical abstract units and remain behaviorally substitutable.","source_ids":["S5","S6"]},{"name":"Statistics Canada frame-quality practice","similarity":"Calls for duplicate removal, frame updating, quality monitoring, source matching, and documentation of frame coverage and selection history.","remaining_difference":"It is a quality-management practice rather than a machine-enforced representation-independent interface and conformance suite.","source_ids":["S2"]}],"distinctive_claim_remaining":"Given a governance-approved mapping from source records to unique abstract units, two independently implemented frame backends that pass a shared black-box, property-based, metamorphic, and leakage oracle will produce identical contract-visible frozen snapshots, opaque selections, inclusion probabilities, errors, and non-contact side effects for the same design version and seed, despite row permutation, lossless reserialization, and internal-key replacement. This is falsified by any held-out representation-only transformation that changes those observables, any accepted backend pair that diverges, or any valid frame state that cannot be mapped without concealing unresolved substantive judgment.","confidence":"MODERATE"},"implementation_evidence":{"support":"MODERATE","rationale":"The constituent techniques are technically credible: mature R and SAS tools implement relevant sampling algorithms and seeded execution; DDI supplies standardized sampling concepts; official quality systems already require specifications, testing, documentation, and confidentiality controls. A fictional 200-unit dual-backend prototype is therefore implementable with ordinary data-engineering and statistical skills. The unverified portions are the abstraction function for contested household or address identity, completeness of the behavioral oracle, stable cross-language pseudorandom semantics, leakage resistance of handles and metadata, integration with recruitment workflows, and jurisdiction-specific review of pseudonymised participant data.","source_ids":["S1","S4","S5","S6","S7"]},"scores":{"meaningful_impact":{"score":4,"rationale":"Frame duplication, exclusion, misclassification, or changed selection probabilities can affect inference, recruitment, burden, and privacy. The consequence is meaningful, although the frequency and magnitude of representation-only failures are unmeasured.","source_ids":["S1","S2","S3"]},"stakeholder_pull":{"score":3,"rationale":"Official statistical organizations express strong need for frame integrity, testing, documentation, and confidentiality, but no prospective adopter has requested the complete proposed intervention.","source_ids":["S1","S2"]},"incremental_advantage":{"score":3,"rationale":"A backend-neutral behavioral oracle could detect migration errors earlier and more repeatably than file freezing, row-count checks, or manual reconciliation. Its advantage over rigorous existing specifications, validation, DDI metadata, and reproducible scripts has not been measured.","source_ids":["S1","S2","S4","S6"]},"distinctiveness_plausibility":{"score":3,"rationale":"The integrated opaque-unit and cross-backend conformance claim is not directly matched in the opened sources, but most components and neighboring practices are established. World novelty is not assessed.","source_ids":["S1","S4","S5","S6"]},"technical_implementability":{"score":4,"rationale":"A small synthetic implementation is feasible using mature sampling libraries, explicit seeds, standardized sampling metadata, two simple storage adapters, and ordinary test tooling. Production-grade identity resolution, deterministic portability, and leakage control remain harder.","source_ids":["S4","S5","S6","S7"]},"adoption_authority_feasibility":{"score":3,"rationale":"Sampling statisticians and study governance bodies have recognizable authority over eligibility, design, and protected data, while Census demonstrates that institutional sampling standards can be mandatory. Cross-office agreement on identity rules and version governance may still be difficult.","source_ids":["S1","S7"]},"evidence_readiness":{"score":3,"rationale":"The proposal has explicit operations, comparators, transformations, falsifiers, and a synthetic trial. It lacks an adopter commitment, an implemented oracle, observed baseline divergences, and outcome data.","source_ids":["S1","S4","S5","S6"]},"safety_net_benefit":{"score":4,"rationale":"Synthetic testing, immutable snapshots, no-contact side-effect rules, explicit errors, protected identifiers, and rollback to the certified prior draw path form a useful safety net. Opaque handles remain pseudonymous rather than automatically anonymous.","source_ids":["S1","S7"]},"scalability":{"score":3,"rationale":"A reusable contract and conformance suite could scale across backends and waves, and mature algorithm libraries lower marginal technical effort. Each study still requires substantive identity, eligibility, design, security, and governance decisions.","source_ids":["S1","S4","S5","S6","S7"]}},"score_confidence":"MODERATE","costs":{"first_evidence":{"band_2026_usd":"10K_TO_50K","scope":"Two-week fictional 200-unit dual-backend exercise, contract workshop, test harness, deliberately broken adapters, leakage review, and results memo.","confidence":"MODERATE","assumptions":["Four roles participate part-time rather than full-time for two weeks.","Existing R or comparable open-source sampling software is reused.","No live participant records, recruitment, procurement, or production integration occurs.","The range is a resource-equivalent labor estimate, not a sourced vendor quote."],"source_ids":["S1","S4","S5"]},"initial_deployment_startup":{"band_2026_usd":"50K_TO_250K","scope":"One-study production design: governance-approved unit model, secure handle service, two backend adapters, contract and version registry, conformance suite, security review, documentation, and staff training.","confidence":"LOW","assumptions":["One bounded study with two existing backends and a cooperative data team.","No wholesale replacement of source-frame systems.","Existing identity-resolution policy can be formalized rather than invented from scratch.","Privacy and legal review is available internally.","No direct price evidence was found in the eight sources."],"source_ids":["S1","S4","S7"]},"operational_launch":{"band_2026_usd":"50K_TO_250K","scope":"Controlled launch for one wave, including historical-frame reconciliation, parallel draw comparison, incident and rollback procedures, user acceptance, monitoring, and governance approval.","confidence":"LOW","assumptions":["Launch covers one study and one wave.","The incumbent draw path remains available for rollback.","No participant contact is initiated until statistical and governance approval.","The frame is moderate in size and does not require a new enterprise identity platform."],"source_ids":["S1","S2","S7"]},"annual_recurring":{"band_2026_usd":"50K_TO_250K","scope":"Contract stewardship, adapter maintenance, regression and leakage testing, access review, version review, audits, incident response, and support for subsequent waves.","confidence":"LOW","assumptions":["One to several related studies share the platform.","Approximately 0.5 to 1.5 resource-equivalent technical/governance staff-years plus infrastructure are required.","Substantive frame rebuilding and field enumeration are excluded.","No direct recurring-cost benchmark was identified."],"source_ids":["S1","S2","S7"]}},"verified_pipeline_gates":{"externally_supported_problem":{"status":"YES","reason":"Multiple official and first-party sources establish that duplicate, omitted, misclassified, overlapping, or stale frame units can alter selection opportunities, weights, reliability, and cost. The narrower prevalence of storage-representation failures remains unmeasured but does not negate the verified underlying problem.","source_ids":["S1","S2","S3","S8"]},"externally_credible_adopter_or_authorizer":{"status":"YES","reason":"The U.S. Census Bureau is an identifiable institutional authorizer with mandatory requirements covering frame construction, unduplication, protected information, testing, monitoring, and replication documentation. This verifies a credible adopter class, not commitment to this specific proposal.","source_ids":["S1"]},"distinct_testable_incremental_claim":{"status":"YES","reason":"The proposal predicts invariant snapshots, selections, probabilities, errors, and side effects across independently implemented backends and enumerated representation-only transformations; observable divergence falsifies it.","source_ids":["S4","S5","S6"]},"bounded_next_evidence_step":{"status":"YES","reason":"A two-week, 200-fictional-unit, two-backend exercise has defined actors, transformations, comparators, injected faults, outputs, and stop conditions and requires no live recruitment.","source_ids":["S1","S4","S5","S6"]},"no_unresolved_safety_or_authority_stop":{"status":"YES","reason":"The first step uses fictional units, no direct identifiers, no participant selection or contact, and preserves substantive decisions for authorized statistical and governance personnel. Any later use of opaque handles with real people still requires privacy and authority review because pseudonymisation does not remove legal obligations.","source_ids":["S1","S7"]},"credible_cost_scope_and_range":{"status":"UNCERTAIN","reason":"The four ranges are scope-bounded resource-equivalent estimates and reuse of mature software is plausible, but no direct labor, procurement, integration, or maintenance price evidence was found. Production costs depend heavily on frame size, identity-resolution complexity, security controls, and governance time.","source_ids":["S1","S5","S6","S7"]}},"next_evidence_step":"Run the proposed two-week synthetic trial with 200 fictional units represented independently in a shuffled flat file and a normalized relational store. Freeze the contract before results are seen. Primary comparator: the current file-coupled scripted workflow applied to both encodings. Intervention comparator: both adapters accessed only through the opaque contract. For at least 1,000 prespecified seeds and fixed design versions, compare canonical abstract-unit sets, eligibility states, selected opaque handles, first-order inclusion probabilities, error codes, mutation logs, and contact side effects under row permutation, lossless reserialization, internal-key replacement, insertion of ineligible records, and consolidation of duplicate physical records. Include mutant adapters that depend on row order, permit duplicate chances, collapse unresolved into ineligible, emit partial samples on error, or leak source identifiers. Pass only if all legitimate representation transformations preserve every contract-visible result, every selected unit is eligible, no failure yields a partial sample or contact event, every mutant is detected, and reviewers can map all valid states without concealing a substantive identity judgment. Falsify or redesign if any legitimate transformation changes a draw or probability, a mutant survives, a leakage probe recovers source information, or an authorized reviewer finds an unmappable or improperly collapsed state. Record labor hours to replace the unsupported cost estimates.","blocking_evidence":["No externally observed baseline rate or case series quantifies representation-only sampling-frame divergences across multi-wave social research.","No named study, field office, funder, or governance body has committed to adopting or evaluating the proposed contract.","No dual-backend conformance suite has been implemented or run, so the incremental preservation claim remains untested.","No evidence shows that the proposed oracle detects held-out defects without over-specifying legitimate implementation differences.","No jurisdiction-specific privacy, research-ethics, records-retention, or security review has been completed for live opaque recruitment handles.","Production startup, launch, and recurring cost bands lack direct price or observed labor evidence.","World novelty, patentability, freedom to operate, market size, and realized impact remain unmeasured."],"research_disposition":"PILOT_OR_ADOPTION_INQUIRY","world_novelty_boundary":"This evaluation found established frame-quality standards, sampling metadata models, mature selection software, seeded reproducibility, and pseudonymisation guidance. It did not find, within the bounded search, a direct source combining unique abstract sampling units, opaque recruitment handles, immutable snapshots, representation-only metamorphic transformations, and one behavioral acceptance oracle for substitutable frame backends. That absence is not evidence of world novelty. Patentability, freedom to operate, market size, and realized impact were not assessed.","arm":"COMPLETE_PROPOSAL_PORTFOLIO","candidate_version":0,"controller_recommendation":{"action":"STOP_EMPIRICAL_RESEARCH_NEEDED","repairable":false,"material_progress_observed":true,"progress_targets":["Obtain a written evaluation commitment from one named multi-wave study and identify its authorized sampling, data-custody, and ethics or community-governance decision makers.","Implement the frozen synthetic contract, two independent adapters, comparators, leakage probes, and at least five deliberately defective adapters.","Demonstrate zero contract-visible divergence across prespecified representation transformations and seeds while detecting every injected defect.","Document every unmappable identity or eligibility state and show that technical staff do not resolve substantive cases without authorization.","Complete a privacy and security determination for opaque handles before any live-data pilot.","Measure role-specific labor hours and infrastructure requirements to replace low-confidence cost estimates.","If the synthetic trial passes, pre-register a non-contact shadow comparison on a historical or governance-approved frame with rollback and semantic acceptance thresholds."],"reason":"Web research verifies the general frame-integrity problem, credible authorizers, adjacent prior art, and feasibility of the components, but it cannot establish the proposal's decisive incremental claim. That evidence requires implementing and running independent backends, mutant tests, and leakage checks, followed eventually by an authorized shadow comparison. Under the controller rule, evidence requiring live technical testing warrants STOP_EMPIRICAL_RESEARCH_NEEDED, and every STOP is non-repairable within this evaluation cycle."},"proposal_index":4}