{"schema_version":1,"research_id":"eoa_inverse_innovation_exp05_external_evaluation_20260803","source_assessment_id":"negative_space_design__astronomy_astrophysics:P2:v0","cell_id":"negative_space_design__astronomy_astrophysics","search_queries":["astronomy image viewer overlays annotations segmentation contours bias morphology classification faint structures","astronomer visual inspection image cutouts overlays catalog symbols segmentation maps interface","Rubin Science Platform image visualization overlays catalog regions Firefly requirements","astronomy viewer toggle overlays regions annotations DS9 Aladin Lite official documentation","\"How Do Observational Astronomers Learn to Inspect Imaging Data\" published","astronomers visual inspection faint low surface brightness structures artifacts survey candidate cutouts human classification","graphical overlay contour bias visual perception image segmentation outline human judgment study","astronomy visual inspection interface requirements overlays hide catalog sources morphology","site:lsst.io requirements image display overlay catalog visualization science platform Portal cutout","site:rubinobs.org \"visual inspection\" images candidate","site:github.com/Caltech-IPAC/firefly overlay hide region issue","Rubin Science Platform Portal image viewer overlay catalog cutout documentation Firefly","site:bls.gov/oes/2025/may software developers astronomers national occupational employment wage estimates","site:bls.gov/ooh physicists and astronomers pay 2025 software developers pay","BLS May 2025 National Occupational Employment Wage Estimates software developers astronomers","DeepShadows Separating Low Surface Brightness Galaxies from Artifacts using Deep Learning journal published","\"DeepShadows\" low surface brightness galaxies artifacts journal"],"sources":[{"source_id":"S1","title":"How Do Observational Astronomers Learn to Inspect Imaging Data?","publisher":"Astronomy Education Journal","url":"https://astroedjournal.org/index.php/ijae/article/download/115/75/1551","source_class":"PRIMARY_RESEARCH","publication_date":"2024-09-26","accessed_at":"2026-08-03","claims_supported":["Visual inspection and visual interpretation are consequential components of astronomical image-based discovery.","A survey of 70 professional observational astronomers found substantial reliance on formal or informal visual-inspection training, while 27% reported neither.","The effectiveness and methods of astronomical visual inspection remain under-investigated, limiting confidence in interface-design assumptions."]},{"source_id":"S2","title":"Census and classification of low-surface-brightness structures in nearby early-type galaxies from the MATLAS survey","publisher":"arXiv; subsequently Monthly Notices of the Royal Astronomical Society","url":"https://arxiv.org/abs/2007.13772","source_class":"PRIMARY_RESEARCH","publication_date":"2020-07-27","accessed_at":"2026-08-03","claims_supported":["Faint structures such as streams, shells, tails, disturbed isophotes, and cirrus were assessed through visual inspection of deep images.","Every galaxy was inspected at least twice, the median received six votes, and experienced participants agreed better than the full participant group.","The study used a dedicated web navigation tool and explicitly evaluated contaminating halos and cirrus, showing that reviewer workflow and contextual contamination matter."]},{"source_id":"S3","title":"DeepShadows: Separating Low Surface Brightness Galaxies from Artifacts using Deep Learning","publisher":"arXiv; subsequently Astronomy and Computing","url":"https://arxiv.org/abs/2011.12437","source_class":"PRIMARY_RESEARCH","publication_date":"2020-11-24","accessed_at":"2026-08-03","claims_supported":["Low-surface-brightness candidates are difficult to characterize and can be dominated by blended objects, Galactic cirrus, star-forming knots, and tidal ejecta.","Artifact rejection through visual inspection is time-consuming and supplied labels for the reported classifier.","The reported 92% classifier accuracy and acknowledged label-noise uncertainty show both the importance of reliable review and the availability of automated alternatives."]},{"source_id":"S4","title":"Image Display Working Group Report (DMTN-126)","publisher":"Vera C. Rubin Observatory","url":"https://dmtn-126.lsst.io/DMTN-126.pdf","source_class":"OFFICIAL_GUIDANCE","publication_date":"2020-02-25","accessed_at":"2026-08-03","claims_supported":["Rubin Science Pipelines, Camera, Commissioning, AP, and SST groups expressed needs for detailed individual-image inspection, marked regions, masks, overlays, and direct image comparison.","All groups polled requested locked-coordinate side-by-side or blinking image comparison.","The report says Firefly and DS9 already support mask overlays and describes Firefly as feature-rich and extensible at relatively modest cost.","The report identifies Rubin visualization stakeholders capable of authorizing an internal, read-only prototype, although it does not request an annotation-free moat."]},{"source_id":"S5","title":"Exploring SAOImageDS9-like Functionality in the Firefly-based RSP Portal","publisher":"Vera C. Rubin Observatory","url":"https://dp0-2.lsst.io/tutorials-examples/Portal-6.html","source_class":"OFFICIAL_PRODUCT_DOCUMENTATION","publication_date":"2024-10-22","accessed_at":"2026-08-03","claims_supported":["Rubin's Firefly-based Portal displays astronomical images with compass, grid, ruler, region, marker, and instrument-footprint overlays.","Individual layers can be removed or temporarily disabled with existing controls.","The documented implementation establishes a concrete product surface and maintainer workflow in which the proposed prototype could be tested."]},{"source_id":"S6","title":"SAOImage DS9 Regions Reference","publisher":"Smithsonian Astrophysical Observatory","url":"https://ds9.si.edu/doc/ref/region.html","source_class":"OFFICIAL_PRODUCT_DOCUMENTATION","publication_date":"n.d.","accessed_at":"2026-08-03","claims_supported":["DS9 supports graphical circles, polygons, points, text, rulers, compasses, and other regions displayed over image pixels.","Region geometry, rendering, tags, coordinate systems, persistence, and user manipulation are already mature practice.","DS9 supplies recoverable region context but does not document a default candidate-footprint moat that spatially excludes selected annotation classes."]},{"source_id":"S7","title":"Aladin Lite v2 Documentation","publisher":"Centre de Données astronomiques de Strasbourg","url":"https://aladin.cds.unistra.fr/AladinLite/v2-doc/","source_class":"OFFICIAL_PRODUCT_DOCUMENTATION","publication_date":"2019-02","accessed_at":"2026-08-03","claims_supported":["Aladin Lite superimposes catalogs and footprints and exposes APIs for catalog and overlay layers.","Its control panel can show or hide graphical overlays and catalogs, demonstrating established reversible layer suppression.","The documented practice is whole-layer control, not footprint-derived local suppression with a protected admission rule."]},{"source_id":"S8","title":"National employment and wage data by occupation, May 2025","publisher":"U.S. Bureau of Labor Statistics","url":"https://www.bls.gov/news.release/ocwage.t01.htm","source_class":"GOVERNMENT_OR_REGULATOR","publication_date":"2026-05-15","accessed_at":"2026-08-03","claims_supported":["The May 2025 national mean wage was $148,100 annually for software developers and $132,920 for astronomers.","These wage benchmarks support resource-equivalent labor estimates but are not vendor quotes and exclude project-specific overhead."]}],"problem_evidence":{"support":"MODERATE","rationale":"Primary studies show that astronomers make consequential, variable judgments about difficult faint morphology and contamination, while official product documentation visibly confirms that regions, catalog symbols, masks, labels, and footprints are routinely drawn over astronomical images. The precise causal assertion—that intersecting overlays materially conceal or impose morphology often enough to affect decisions—was not directly tested by any located source. Thus the broader problem exists and matters, but the proposal-specific prevalence and harm remain empirical gaps.","source_ids":["S1","S2","S3","S4","S5","S6","S7"]},"stakeholder_evidence":{"support":"MODERATE","rationale":"Rubin's Image Display Working Group records expressed needs from identifiable Science Pipelines, Camera, Commissioning, AP, and SST stakeholders for detailed inspection, synchronized comparison, masks, overlays, and interactive manipulation. Rubin's deployed Firefly Portal is a concrete adoption surface. No source records stakeholder demand for a default footprint-centered annotation-free moat, so pull for the incremental intervention is unverified.","source_ids":["S4","S5"]},"prior_art":{"proximity":"ADJACENT_PRIOR_ART","closest_analogues":[{"name":"Rubin Firefly layer controls","similarity":"Firefly already places regions, grids, markers, and footprints over images and lets users remove or temporarily disable individual layers.","remaining_difference":"It documents user-initiated whole-layer control, not default spatial suppression inside a candidate-derived boundary, a persistent omitted-layer inventory, or a layer-admission rule.","source_ids":["S5"]},{"name":"SAOImage DS9 regions and display workflow","similarity":"DS9 provides mature overlay geometry, coordinate registration, tagging, persistence, and interactive display/manipulation of regions over FITS imagery.","remaining_difference":"The documentation does not establish a protected annotation-free candidate zone or a controlled first-look morphology state with automatic restoration logging.","source_ids":["S6"]},{"name":"Aladin Lite overlay stack","similarity":"Catalogs, footprints, MOCs, and graphical overlays are separate layers that can be shown or hidden through a control panel or API.","remaining_difference":"Suppression applies to layers, not selectively to marks intersecting a candidate footprint; it lacks the proposed warning exemptions and anti-layer-creep policy.","source_ids":["S7"]},{"name":"Rubin synchronized comparison and mask-overlay practice","similarity":"Rubin stakeholders requested coordinate-locked side-by-side or blinking comparisons and adjustable mask overlays, closely matching the proposal's strongest comparator and recoverability requirement.","remaining_difference":"The practice compares whole views and manages transparency; it does not make a bounded clean region the default first-look state.","source_ids":["S4"]}],"distinctive_claim_remaining":"For a fixed archived-cutout workflow, automatically suppressing only noncritical annotations that intersect a declared candidate-footprint margin, while displaying an explicit omission inventory and preserving one-action exact-registration restoration, will improve the correctness and traceability of pixel-grounded morphology judgments over both ordinary manual layer toggles and synchronized raw/annotated paired views, without increasing missed critical warnings, contamination errors, or review time beyond prespecified margins.","confidence":"MODERATE"},"implementation_evidence":{"support":"MODERATE","rationale":"Technical feasibility is credible because Firefly, DS9, and Aladin already maintain registered overlay geometry and reversible layer visibility, while Rubin explicitly supports masks, comparison, and source-sized cutouts. A client-side footprint-intersection filter, immutable image-state hash, overlay inventory, and restoration control are bounded extensions. Workflow and data feasibility require access to archived cutouts, footprint geometry, annotation provenance, and qualified reviewers. Authority can be limited to a read-only duplicate under the visualization lead, with the data-quality lead controlling non-suppressible warnings. No production classifications, source pixels, catalogs, or observing priorities need change. Safety depends on exact pixel identity and registration, visible missing-data and quality states, keyboard-operable restoration, and stopping on any hidden critical warning. These safeguards and the proposed local-suppression behavior have not undergone live accessibility or scientific-safety testing.","source_ids":["S4","S5","S6","S7"]},"scores":{"meaningful_impact":{"score":3,"rationale":"Faint-structure and artifact decisions matter scientifically, but the frequency and magnitude of overlay-induced errors are unknown and automation may reduce the affected review volume.","source_ids":["S1","S2","S3"]},"stakeholder_pull":{"score":2,"rationale":"Rubin stakeholders explicitly want capable image inspection, comparison, masks, and overlays, but no source asks for annotation-free candidate moats or reports overlay interference as a priority.","source_ids":["S4","S5"]},"incremental_advantage":{"score":2,"rationale":"The moat adds a consistent default, local boundary, omission inventory, and anti-creep rule, but existing toggles and synchronized paired views may capture most of the benefit.","source_ids":["S4","S5","S7"]},"distinctiveness_plausibility":{"score":3,"rationale":"No close match was found for footprint-local default suppression plus admission policy and warning exemptions; however, this is a bounded search and the concept composes established layer and comparison mechanisms.","source_ids":["S4","S5","S6","S7"]},"technical_implementability":{"score":4,"rationale":"Existing viewers already expose registered shapes, catalogs, masks, layer toggles, and APIs. The main engineering risks are exact-state preservation, heterogeneous overlay types, and responsive intersection calculations rather than novel science infrastructure.","source_ids":["S4","S5","S6","S7"]},"adoption_authority_feasibility":{"score":3,"rationale":"Rubin/IPAC visualization and data-quality leads are identifiable plausible authorizers for a nonoperational prototype, but no written authorization, owner commitment, or approved critical-warning list was found.","source_ids":["S4","S5"]},"evidence_readiness":{"score":2,"rationale":"Archived cutouts and mature viewers make a study design possible, but overlay-intersection prevalence, reference labels, reviewer availability, power, and direct causal evidence are absent.","source_ids":["S1","S2","S3","S5"]},"safety_net_benefit":{"score":3,"rationale":"Read-only operation, immutable pixels, explicit omission disclosure, warning exemptions, and instant restoration bound harm and improve auditability; the interface could still hide contamination context or anchor reviewers to the moat boundary.","source_ids":["S4","S5","S6"]},"scalability":{"score":4,"rationale":"Once implemented, a deterministic footprint-and-layer rule could render across many cutouts at low marginal cost, although margin policies must vary with source scale, band, PSF, and display size.","source_ids":["S4","S5","S7"]}},"score_confidence":"MODERATE","costs":{"first_evidence":{"band_2026_usd":"10K_TO_50K","scope":"Build a nonoperational prototype, prepare at most 60 frozen cutouts, recruit qualified reviewers, run a randomized crossover, audit warning visibility and accessibility, and analyze results.","confidence":"MODERATE","assumptions":["Approximately 120-240 software/design hours and 60-120 astronomer/reviewer hours.","Existing Firefly or equivalent viewer, archived data, and institutional study infrastructure are reused.","Resource-equivalent rates apply BLS national wages with roughly 1.5-2.0 times salary allowance for benefits, overhead, coordination, and computing.","No procurement, production deployment, or observing-time allocation is included."],"source_ids":["S4","S5","S8"]},"initial_deployment_startup":{"band_2026_usd":"10K_TO_50K","scope":"Convert the research prototype into a maintained feature branch with configurable footprint margins, layer classification, immutable-view checks, omission logging, restoration controls, tests, and documentation.","confidence":"LOW","assumptions":["One to two developer-months plus part-time visualization scientist and data-quality review.","The host viewer's overlay model exposes geometry and visibility state without architectural replacement.","Security, privacy, and data-access controls are inherited from the existing read-only viewer."],"source_ids":["S4","S5","S6","S7","S8"]},"operational_launch":{"band_2026_usd":"50K_TO_250K","scope":"Production integration, multi-browser and accessibility validation, workflow-specific warning governance, performance and regression testing, user training, release management, and monitored rollout.","confidence":"LOW","assumptions":["Two to eight person-months across software, UX/accessibility, scientific validation, data quality, and operations.","Deployment occurs in one existing survey-review platform rather than a new standalone service.","The range allows for remediation if overlay provenance or geometry is inconsistent."],"source_ids":["S4","S5","S8"]},"annual_recurring":{"band_2026_usd":"10K_TO_50K","scope":"Maintain layer-admission rules, warning exemptions, regression fixtures, browser compatibility, support, and periodic audit of newly introduced overlay classes.","confidence":"LOW","assumptions":["Roughly 0.1-0.3 combined FTE across engineering, visualization science, and data quality.","Existing platform hosting and observability are reused.","Major viewer rewrites or expansion across multiple institutions are excluded."],"source_ids":["S4","S5","S8"]}},"verified_pipeline_gates":{"externally_supported_problem":{"status":"UNCERTAIN","reason":"The importance and difficulty of faint-structure visual review and the coexistence of overlays are externally supported, but no direct study establishes the prevalence or decision effect of overlays intersecting candidate morphology.","source_ids":["S1","S2","S3","S4","S5","S6","S7"]},"externally_credible_adopter_or_authorizer":{"status":"YES","reason":"Rubin's visualization, Science Pipelines, Camera, Commissioning, AP, and SST organizations are identifiable stakeholders, and the Firefly Portal is a concrete authorized product surface. Their expressed need is adjacent rather than moat-specific.","source_ids":["S4","S5"]},"distinct_testable_incremental_claim":{"status":"YES","reason":"The footprint-local default suppression rule can be compared directly with fully annotated manual toggles and synchronized raw/annotated paired views using fixed images and prespecified outcomes.","source_ids":["S4","S5","S7"]},"bounded_next_evidence_step":{"status":"YES","reason":"A read-only randomized crossover on no more than 60 frozen archived cutouts, with three explicit conditions, logged endpoints, safety audits, and preregistered falsifiers is bounded.","source_ids":["S1","S2","S4","S5"]},"no_unresolved_safety_or_authority_stop":{"status":"YES","reason":"The next study can be isolated from production decisions and halted on pixel mismatch, registration drift, undisclosed omission, or hidden critical warning. Production adoption would still require data-quality approval and accessibility validation.","source_ids":["S4","S5","S6"]},"credible_cost_scope_and_range":{"status":"YES","reason":"The proposed phases are scoped in person-hours or person-months and anchored to current national software-developer and astronomer wage data, although organization-specific loaded rates and integration complexity remain uncertain.","source_ids":["S4","S5","S8"]}},"next_evidence_step":"With written approval from one survey visualization lead and one data-quality lead, select at most 60 archived cutouts stratified by overlay-intersection density, angular extent, PSF/band, and known mask conditions. Freeze and hash pixels, stretch, zoom, WCS, annotations, and reference data. Run a preregistered randomized crossover with qualified reviewers under: (A) fully annotated baseline with ordinary layer toggles, (B) synchronized annotation-free and fully annotated paired views, and (C) the footprint-local moat with explicit omitted-layer inventory and one-action restoration. Record pre-restoration morphology descriptions and boundary traces, artifact/contamination explanations, classifications, confidence, time, toggles/restorations, and all missed quality warnings. Establish reference adjudication independently from deeper or multiband data and mask provenance where available. Audit pixel identity, registration, omission disclosure, keyboard operation, non-color-only warning cues, and exact restoration. Treat the intervention as falsified for this workflow if condition C does not outperform the better comparator on preregistered pixel-grounded correctness or inter-reviewer reliability, or if it increases missed critical warnings/contamination errors by more than 5 percentage points, median review time by more than 20%, or any immutable-view audit failure occurs. Use results only to decide whether a powered prospective evaluation is warranted.","blocking_evidence":["No measured prevalence distribution for nonessential overlays intersecting the actual morphology-review footprint.","No direct experiment showing that such overlays alter astronomical morphology, artifact, or follow-up judgments.","No written stakeholder commitment to adopt, fund, or authorize the moat-specific intervention.","No approved survey-specific taxonomy separating suppressible annotations from non-suppressible quality and missing-data warnings.","No confirmed reference-truth strategy, reviewer pool, power analysis, or workflow-specific noninferiority margins.","No live accessibility, registration, performance, or critical-warning safety evaluation.","Archived-cutout access, annotation provenance, and any restrictions on nonpublic survey data remain unverified."],"research_disposition":"PARTNERED_RESEARCH_PROGRAM","world_novelty_boundary":"This eight-source bounded search found established layer toggles, overlay stacks, registered regions, masks, and synchronized image comparison, but no documented candidate-footprint-local default suppression system with an omission inventory, critical-warning exemptions, and a layer-admission rule. This is only a contrast against the searched sources: world novelty, patentability, freedom to operate, market size, and realized impact were not measured.","arm":"COMPLETE_PROPOSAL_PORTFOLIO","candidate_version":0,"controller_recommendation":{"action":"STOP_EMPIRICAL_RESEARCH_NEEDED","repairable":false,"material_progress_observed":true,"progress_targets":["Obtain written prototype authority and expressed interest from a named Rubin/IPAC visualization owner plus approval of the non-suppressible warning taxonomy from a data-quality owner.","Measure overlay-footprint intersection prevalence and ordinary toggle behavior in a representative archived workflow before estimating impact.","Implement and verify immutable pixel, stretch, zoom, WCS, registration, omission-log, and one-action restoration checks.","Preregister the three-condition crossover, reference adjudication method, primary endpoints, power analysis, and safety/time noninferiority margins.","Complete the at-most-60-cutout reviewer study and accessibility/safety audit; stop on any hidden critical warning or state mismatch.","Reject the incremental claim if the moat fails against the better of manual toggles and synchronized paired views, or breaches the prespecified warning, contamination, time, or accessibility limits."],"reason":"Web research established a real high-consequence review context, credible authorizers, feasible implementation, and adjacent prior art, but it could not determine the proposal's key causal quantity: whether footprint-intersecting annotations are prevalent and whether the protected moat improves judgments beyond existing toggles or paired views without safety losses. Those questions require archived-workflow measurement and live reviewer testing, so bounded web research is exhausted and the required next evidence is empirical."},"proposal_index":2}