{"schema_version":1,"research_id":"eoa_inverse_innovation_exp05_external_evaluation_20260803","source_assessment_id":"invariant_mode_decomposition_design__information_theory:P3:v0","cell_id":"invariant_mode_decomposition_design__information_theory","search_queries":["site:osti.gov scientific data storage resilience correlated failures erasure coding archives","numerically robust erasure codes real-valued frames quantization small singular value distributed storage","scientific arrays lossy compression error bounds archival data DOE need","Ceph erasure coding failure domain rack official documentation correlated failure","site:science.osti.gov scientific data management policy preservation storage data official DOE","site:energy.gov scientific data archive storage reliability data preservation DOE Office Science","site:alcf.anl.gov storage campaign failures scientific data archive erasure coding","site:osti.gov distributed scientific data resilience storage failure erasure coding","frame codes erasures quantization distributed storage smallest singular value parity design","optimal frames multiple erasures quantization reconstruction error linear encoder","survivor matrix smallest singular value erasure code numerical conditioning scientific data","scenario aware erasure coding correlated failures placement optimization storage","NSTC Desirable Characteristics of Data Repositories integrity sustainability official PDF","site:whitehouse.gov desirable characteristics data repositories integrity preservation PDF","site:nist.gov desirable characteristics data repositories integrity authenticity availability","\"RAPIDS: Reconciling Availability, Accuracy, and Performance\"","\"Reconciling Availability, Accuracy, and Performance\" scientific data storage"],"sources":[{"source_id":"S1","title":"DOE Requirements and Guidance for Digital Research Data Management","publisher":"U.S. Department of Energy","url":"https://www.energy.gov/datamanagement/doe-requirements-and-guidance-digital-research-data-management","source_class":"OFFICIAL_GUIDANCE","publication_date":"2023-06","accessed_at":"2026-08-03","claims_supported":["DOE-funded research is subject to an approved Data Management and Sharing Plan addressing preservation, validation, repository selection, resources, protections, and limitations.","Facility approval is required when a plan commits resources beyond those conventionally available.","Preservation must be balanced against cost and administrative burden, and applicable privacy, security, intellectual-property, and other legal limitations remain relevant."]},{"source_id":"S2","title":"Department of Energy to Provide $10 Million for Research on Data Reduction for Science","publisher":"U.S. Department of Energy Office of Science","url":"https://www.energy.gov/science/articles/department-energy-provide-10-million-research-data-reduction-science","source_class":"OFFICIAL_ORGANIZATION_DATA","publication_date":"2021-04-15","accessed_at":"2026-08-03","claims_supported":["DOE stated that scientific facilities were producing data beyond available storage, analysis, streaming, and archival capacity.","DOE expressed a need for trusted methods that preserve scientifically important information and announced $10 million for related foundational research.","DOE Office of Science is an identifiable funder, although this announcement did not request weak-mode parity specifically."]},{"source_id":"S3","title":"Erasure code","publisher":"Ceph Project","url":"https://docs.ceph.com/en/umbrella/rados/operations/erasure-code/","source_class":"OFFICIAL_PRODUCT_DOCUMENTATION","publication_date":"n.d.","accessed_at":"2026-08-03","claims_supported":["Operational distributed storage already uses parity chunks, configurable k/m profiles, and host or rack failure domains.","Ceph describes storage, performance, recovery, and overlapping-failure tradeoffs and warns that an erasure-code profile cannot be modified in place; data must be moved to a new pool.","At least k surviving shards are required for recovery, showing that mainstream practice emphasizes rank/count guarantees rather than real-valued weak-direction fidelity."]},{"source_id":"S4","title":"Quantized Frame Expansions with Erasures","publisher":"Nokia Bell Labs","url":"https://www.nokia.com/bell-labs/publications-and-media/publications/quantized-frame-expansions-with-erasures/","source_class":"PRIMARY_RESEARCH","publication_date":"2001-05-01","accessed_at":"2026-08-03","claims_supported":["Prior research directly treats quantized linear frame coefficients, coefficient erasures, additive noise, stable reconstruction, and encoder-design optimization.","Tight-frame and other designs were analyzed to minimize distortion after erasures.","This substantially collides with the proposal's general idea of orienting real-valued redundant projections to reduce erasure-plus-quantization reconstruction error."]},{"source_id":"S5","title":"Numerically erasure-robust frames","publisher":"arXiv / authors Matthew Fickus and Dustin G. Mixon","url":"https://arxiv.org/abs/1202.4525","source_class":"PRIMARY_RESEARCH","publication_date":"2012-02-21","accessed_at":"2026-08-03","claims_supported":["Numerically erasure-robust frames explicitly design linear encoders for stable reconstruction under additive noise and adversarial erasures.","The work evaluates erased submatrices through condition-number behavior and demonstrates both constructions and fundamental limits.","It directly establishes that retained coefficient count alone does not ensure numerically stable reconstruction."]},{"source_id":"S6","title":"Frames, Graphs and Erasures","publisher":"arXiv / authors Bernhard G. Bodmann and Vern I. Paulsen","url":"https://arxiv.org/abs/math/0406134","source_class":"PRIMARY_RESEARCH","publication_date":"2004-06-08","accessed_at":"2026-08-03","claims_supported":["Prior work treats frames as vector codes and optimizes numerical reconstruction error when arbitrary sets of frame coefficients are lost.","It compares frame constructions under multiple erasures, further narrowing the proposal's distinctiveness as a coding-design method."]},{"source_id":"S7","title":"RAPIDS: Reconciling Availability, Accuracy, and Performance in Managing Geo-Distributed Scientific Data","publisher":"Oak Ridge National Laboratory; paper published by ACM SIGARCH","url":"https://www.ornl.gov/publication/rapids-reconciling-availability-accuracy-and-performance-managing-geo-distributed","source_class":"PRIMARY_RESEARCH","publication_date":"2023-08","accessed_at":"2026-08-03","claims_supported":["Outages and maintenance can make large scientific data unavailable and impede discovery.","RAPIDS already combines error-bounded scientific-data reduction with erasure coding and optimizes fault-tolerance configuration under accuracy, availability, storage, and network constraints.","The reported implementation and experiments support feasibility of bounded offline comparators but do not test survivor-operator weak-subspace parity replacement."]},{"source_id":"S8","title":"Desirable Characteristics of Data Repositories for Federally Funded Research","publisher":"National Science and Technology Council, Subcommittee on Open Science; Executive Office of the President","url":"https://rosap.ntl.bts.gov/view/dot/62310","source_class":"OFFICIAL_GUIDANCE","publication_date":"2022-05-01","accessed_at":"2026-08-03","claims_supported":["Federal repository guidance identifies preservation, quality, utility, privacy, security, and other protections as desirable repository characteristics.","The guidance supports governance and integrity requirements around any archive-code migration but does not prescribe the proposed parity design."]}],"problem_evidence":{"support":"MODERATE","rationale":"The general problem is visible and consequential: DOE documents preservation and capacity pressures, ORNL researchers report scientific-data unavailability from outages or maintenance, and frame research proves that surviving coefficient count can coexist with unstable reconstruction under erasure and noise. However, no opened source documents an actual scientific archive incident in which a reproducible weak singular direction defeated an otherwise-passing shard-count threshold; that candidate-specific prevalence remains unverified.","source_ids":["S1","S2","S4","S5","S7"]},"stakeholder_evidence":{"support":"MODERATE","rationale":"DOE is an identifiable funder and policy authorizer expressing needs for trusted preservation under resource constraints, while DOE guidance requires facility approval for extraordinary data-management resources. ORNL is an identifiable scientific-computing organization researching availability, accuracy, and overhead. None of the eight sources records an archive operator asking specifically for scenario-conditioned weak-mode parity, so pull for the precise intervention is inferred rather than demonstrated.","source_ids":["S1","S2","S7","S8"]},"prior_art":{"proximity":"SUBSTANTIAL_COLLISION","closest_analogues":[{"name":"Quantized frame expansions with erasures","similarity":"Directly combines real-valued redundant projections, quantization noise, coefficient loss, reconstruction distortion, and optimization of frame design.","remaining_difference":"The proposal adds an archive workflow based on enumerated physical failure domains, stable weak subspaces, parity placement, held-out scientific-fidelity tests, and reversible migration.","source_ids":["S4"]},{"name":"Numerically erasure-robust frames","similarity":"Designs linear encoders so erased survivor submatrices remain well-conditioned under additive noise; this is nearly the proposal's mathematical core.","remaining_difference":"The proposal targets selected consequential scenarios and replaces a bounded parity subset instead of seeking uniform robustness to all erasure subsets; it also adds placement and operational governance.","source_ids":["S5"]},{"name":"Frames, Graphs and Erasures","similarity":"Optimizes vector-coding frames against reconstruction error for multiple erased coefficients.","remaining_difference":"It does not establish the proposal's specific domain-aware migration, weak-subspace persistence test, or scientific quantity-of-interest acceptance workflow.","source_ids":["S6"]},{"name":"RAPIDS","similarity":"Addresses geo-distributed scientific-data availability and jointly optimizes accuracy, erasure protection, storage, and network overhead.","remaining_difference":"RAPIDS allocates fault tolerance across hierarchical compressed-data levels rather than redesigning real-valued parity rows around weak right-singular subspaces of survivor operators.","source_ids":["S7"]},{"name":"Ceph erasure coding and CRUSH failure-domain practice","similarity":"Implements configurable parity, recovery, and rack/host-aware placement with explicit overhead and migration constraints.","remaining_difference":"The documentation exposes k, m, rank/count, and placement controls but not real-valued weak-mode or structured-residual optimization.","source_ids":["S3"]}],"distinctive_claim_remaining":"For a real-valued projection archive and a predeclared catalog of correlated failure-domain outages, replacing only a fixed-budget subset of generic parity rows with versioned rows selected to strengthen reproducible, scientifically consequential weak survivor subspaces will improve held-out reconstruction fidelity over standard erasure coding, rack-aware replication, random parity, and direct worst-case conditioning optimization, without worsening approved-scenario rank, repair bandwidth, placement independence, or precision limits.","confidence":"HIGH"},"implementation_evidence":{"support":"MODERATE","rationale":"SVD, erased-submatrix conditioning, quantized-frame reconstruction, and error/overhead optimization are established research methods, and Ceph demonstrates operational parity and failure-domain controls. An offline matrix-and-decoder study is technically straightforward. Deployment is not yet externally validated because mainstream storage codes commonly operate over finite fields, the proposal assumes an actual real-valued projection encoder, archive dimensions and precision are unspecified, and changing an erasure profile can require creating a new pool and moving data. Authority is feasible only with data-owner, storage-operator, reliability-review, and possibly facility approval; production safety requires versioning, independent decoding, checksums, retention of the old representation, and rollback.","source_ids":["S1","S3","S4","S5","S7","S8"]},"scores":{"meaningful_impact":{"score":4,"rationale":"If the failure mode exists, preventing structured corruption or loss of irreplaceable scientific arrays has high value; preservation and availability matter to DOE science. Realized impact and prevalence are unmeasured.","source_ids":["S1","S2","S7"]},"stakeholder_pull":{"score":3,"rationale":"DOE and ORNL visibly value trusted scientific-data preservation and efficient availability, but no source expresses demand for weak-mode parity itself.","source_ids":["S1","S2","S7"]},"incremental_advantage":{"score":3,"rationale":"Scenario- and quantity-of-interest-aware parity replacement could beat uniform robustness at fixed overhead, but the advantage requires matched-budget empirical comparison.","source_ids":["S4","S5","S7"]},"distinctiveness_plausibility":{"score":2,"rationale":"The mathematical core substantially overlaps decades of quantized-frame and numerically erasure-robust-frame research; only the domain-aware operational combination remains plausibly distinctive.","source_ids":["S4","S5","S6","S7"]},"technical_implementability":{"score":4,"rationale":"The offline calculations and decoder comparisons use established linear-algebra and coding methods. Production integration, finite-precision behavior, and migration remain system-specific.","source_ids":["S3","S4","S5","S7"]},"adoption_authority_feasibility":{"score":3,"rationale":"DOE policy supplies a credible approval path and the candidate assigns relevant operational roles, but no specific repository has committed authority, data, or staff.","source_ids":["S1","S8"]},"evidence_readiness":{"score":3,"rationale":"The proposal specifies comparators, metrics, failure scenarios, held-out tests, and falsifiers, but requires a real encoder, placement map, non-production arrays, and facility participation.","source_ids":["S3","S7"]},"safety_net_benefit":{"score":4,"rationale":"Versioned parity, dual decoding, retention of old shards, rank checks, checksums, and rollback could add a useful safety layer if implemented; official guidance supports integrity and protection controls.","source_ids":["S3","S8"]},"scalability":{"score":3,"rationale":"Per-scenario SVD and offline optimization can scale with sparse or batched methods, but scenario explosion, parity migration volume, re-encoding, and continuous drift monitoring could become costly at archive scale.","source_ids":["S3","S5","S7"]}},"score_confidence":"MODERATE","costs":{"first_evidence":{"band_2026_usd":"10K_TO_50K","scope":"Offline study for one frozen non-production encoder, placement map, numerical format, bounded failure catalog, and modest held-out scientific-array set; includes matrix extraction, SVD/conditioning analysis, parity search, comparator decoders, and a short review report.","confidence":"MODERATE","assumptions":["Existing compute and non-production samples are available at no acquisition cost.","One preservation or storage engineer and one numerical-computing researcher contribute roughly 4-8 person-weeks total.","No production storage is altered and no new hardware is purchased."],"source_ids":["S1","S4","S5","S7"]},"initial_deployment_startup":{"band_2026_usd":"50K_TO_250K","scope":"Build a non-production, versioned dual-decoder prototype with metadata, integrity checks, failure injection, telemetry, rollback, and operator/reliability review.","confidence":"LOW","assumptions":["A compatible real-valued projection archive already exists.","Prototype scale is tens to hundreds of terabytes rather than a full facility archive.","Existing test-cluster capacity and staff security processes are reused."],"source_ids":["S1","S3","S8"]},"operational_launch":{"band_2026_usd":"250K_TO_1M","scope":"Production engineering, capacity headroom for old and new shards, staged parity migration, independent validation, operational documentation, monitoring integration, and rollback readiness for one archive service.","confidence":"LOW","assumptions":["Migration is limited to a bounded parity subset rather than complete data re-encoding.","Temporary dual storage, network transfer, and staff time dominate cost.","The archive is institution-scale; exabyte-scale migration could exceed this band."],"source_ids":["S1","S3","S8"]},"annual_recurring":{"band_2026_usd":"50K_TO_250K","scope":"Periodic scenario recomputation, weak-subspace and residual monitoring, regression tests after topology or encoder changes, incident review, metadata stewardship, and incremental parity refreshes.","confidence":"LOW","assumptions":["Monitoring uses existing observability and compute infrastructure.","One service is covered with fractional engineer and reliability-review effort.","Major topology changes or repeated full migrations are excluded."],"source_ids":["S1","S3","S8"]}},"verified_pipeline_gates":{"externally_supported_problem":{"status":"YES","reason":"Scientific-data availability and preservation pressures are externally documented, and primary frame research establishes the claimed count-versus-conditioning failure mechanism in linear coefficient systems.","source_ids":["S1","S2","S4","S5","S7"]},"externally_credible_adopter_or_authorizer":{"status":"YES","reason":"DOE is a credible funder and policy authorizer for scientific-data preservation; its guidance identifies facility approval requirements, and ORNL is an identifiable scientific-computing institution working on the adjacent problem. Specific interest in this intervention is not established.","source_ids":["S1","S2","S7"]},"distinct_testable_incremental_claim":{"status":"YES","reason":"Despite substantial prior-art overlap, the remaining matched-budget claim distinguishes scenario-specific weak-subspace parity replacement from uniform frame design, standard erasure coding, rack-aware replication, random parity, and direct worst-case conditioning optimization.","source_ids":["S3","S4","S5","S6","S7"]},"bounded_next_evidence_step":{"status":"YES","reason":"The proposed frozen-encoder, non-production experiment has bounded inputs, explicit comparators, held-out metrics, authorization limits, and outcome falsifiers.","source_ids":["S3","S4","S5","S7"]},"no_unresolved_safety_or_authority_stop":{"status":"YES","reason":"The first step is offline and non-production. Production changes remain contingent on operator, reliability, data-owner, and facility approval, with old shards retained and rollback required. Legal and security limits can be handled through approved non-production data and existing governance.","source_ids":["S1","S3","S8"]},"credible_cost_scope_and_range":{"status":"UNCERTAIN","reason":"Resource-equivalent bands can be bounded for the offline study, but archive size, matrix dimensions, migration volume, staffing rates, hardware headroom, and existing test infrastructure are unspecified, making deployment and recurring ranges low-confidence.","source_ids":["S1","S3","S7"]}},"next_evidence_step":"With a willing archive partner, freeze one non-production real-valued encoder, placement map, coefficient format, fidelity/QoI specification, storage and repair budgets, and no more than 20 approved rack/zone/controller failure scenarios. Use design partitions to select at most 2-4 replacement parity rows. On untouched arrays and failure realizations, compare: (A) unchanged generic parity, (B) standard MDS/erasure-code configuration, (C) rack-aware replication, (D) random parity rows, (E) direct minimax conditioning optimization without retained modal interpretation, and (F) proposed weak-subspace targeting. Predeclare exact-rank preservation, worst-case and QoI residuals, residual alignment, minimum singular value/condition number, coefficient-quantization sensitivity, repair bytes, decode time, and physical failure-domain independence. Falsify the intervention if weak subspaces do not reproduce across partitions, any approved scenario loses rank, any comparator is non-inferior on all fidelity measures at matched resources, proposed gains disappear at implementation precision, structured residuals persist, or repair/placement limits are breached. Successful results authorize only a reversible non-production dual-decoder trial.","blocking_evidence":["No opened source demonstrates that a production scientific archive using real-valued projection shards has experienced the proposed weak-direction failure while passing shard-count thresholds.","No specific repository operator has expressed demand for, supplied data to, or committed authority to this intervention.","The actual encoder matrix, numerical precision, topology map, approved correlated-failure catalog, scientific fidelity measure, and resource budgets are unavailable publicly.","Matched-budget performance against standard MDS erasure coding, rack-aware replication, random frames, numerically erasure-robust frames, RAPIDS-like fault-tolerance allocation, and direct minimax conditioning optimization is unknown.","Production-scale migration cost, temporary capacity, repair load, metadata compatibility, and weak-subspace drift are unmeasured.","Applicability to finite-field erasure codes is excluded unless a separate algebraic analysis establishes an equivalent mechanism."],"research_disposition":"PARTNERED_RESEARCH_PROGRAM","world_novelty_boundary":"The search establishes substantial collision with quantized frame expansions, optimal frames under erasures, numerically erasure-robust frames, domain-aware erasure-coded storage, and scientific-data fidelity/availability optimization. It did not establish whether the specific combination of correlated-failure survivor SVD, stable consequential weak-subspace targeting, bounded parity-row replacement, physical placement, and reversible archive migration has previously been implemented or claimed. World novelty, patentability, freedom to operate, market size, and realized impact remain unmeasured.","arm":"COMPLETE_PROPOSAL_PORTFOLIO","candidate_version":0,"controller_recommendation":{"action":"STOP_EMPIRICAL_RESEARCH_NEEDED","repairable":false,"material_progress_observed":false,"progress_targets":["Secure a named scientific-archive partner with a real-valued projection encoder and written authorization for a non-production study.","Obtain a frozen encoder, numerical format, placement topology, approved failure catalog, non-production arrays, QoI/fidelity thresholds, and matched storage/repair budgets.","Demonstrate that a weak survivor subspace is reproducible across data and failure partitions and explains held-out structured error beyond shard count, corruption, and decoder defects.","Run the predeclared matched-budget comparison against standard erasure coding, rack-aware replication, random parity, established erasure-robust frame constructions, RAPIDS-like allocation, and direct minimax conditioning optimization.","Show that any improvement survives coefficient quantization and implementation precision without rank loss, cross-mode degradation, placement dependence, or excess repair cost.","Produce a deployment-grade migration, integrity, dual-decoder, metadata, monitoring, authority, and rollback assessment with site-specific cost estimates."],"reason":"Bounded web research resolves the existence of the general problem and reveals substantial prior-art collision, but it cannot establish the candidate's remaining advantage. That claim requires proprietary archive matrices and topology, held-out scientific data, failure injection, decoder execution, and matched-resource live or offline field testing. Under the controller rule, this requires an empirical-research stop; all STOP recommendations are non-repairable in the current evaluation cycle."},"proposal_index":3}