Signal/Noise Review¶

Review — instantiates Model-Guided Signal Separation

A human adjudication step where reviewers judge whether an extracted signal is real and fit for its use — or an artifact dressed up as signal — before it is allowed to drive a decision.

Signal/Noise Review is the human judgment gate at the end of a separation: a person or panel looks at the extracted signal and asks whether it is real, whether it means what the pipeline claims, and whether it is good enough for the decision it will feed. Where a Held-Out Sample Test delivers a metric, this delivers a verdict — it weighs the extraction against its intended use, against known instrument artifacts, and against what domain sense says a real signal should look like. Its defining role is adjudication under the reality that no automated score catches everything: a result can pass every statistical check and still be an artifact a knowledgeable reviewer recognizes on sight. It is the mechanism that decides whether an extraction is trusted, sent back, or set aside for a cruder but safer fallback.

Example¶

An observatory's pipeline flags a candidate signal — a faint chirp rising out of the detector noise. Before anything is claimed, it goes to review. The panel checks the mundane explanations first: was there a known instrumental glitch, a nearby seismic event, a data-quality flag on that segment? It requires the candidate to appear, consistently, across independent detectors rather than one. It weighs the candidate against the look-elsewhere effect — that scanning enough data will inevitably throw up chance-shaped bumps.

Only a candidate that survives every mundane explanation and clears a pre-agreed evidence bar is treated as signal; anything short is logged as a candidate, not a detection. The statistics got it onto the table; the review decides whether it is real enough to act on.

How it works¶

What distinguishes it from a metric-based test is that it is judgment, exercised against several standards at once:

Fitness for use — does the extraction actually serve the decision it was made for, at the quality that decision needs?
Artifact check — has it been compared against the catalogue of known instrument and processing artifacts that mimic signal?
Domain plausibility — does a real signal of this kind look like this, to someone who knows the domain?
Verdict — accept, return for rework, or fall back to a cruder trusted measurement.

The output is a decision about the extraction, not a number describing it.

Tuning parameters¶

Evidence bar — how strong the case must be before "signal" is declared; set by the cost of a false claim versus a missed one.
Reviewer independence / number — a single expert versus an adversarial panel; more independence resists confirmation bias but costs time.
Artifact-catalogue depth — how exhaustively known nuisances are checked before acceptance.
Blinding — whether reviewers see the hoped-for result while judging.
Fallback trigger — how weak the case must be before defaulting to a cruder trusted measurement.

When it helps, and when it misleads¶

Its strength is that it catches artifacts no metric encodes, ties the accept/reject decision to the actual downstream use, and provides an accountable place where a questionable extraction is stopped or downgraded.

Human review imports human bias: reviewers who know the desired answer find reasons to accept it, and the more candidates are scanned, the more chance-shaped artifacts appear — the look-elsewhere effect.^[1] The classic misuse is to convene the review only to ratify a result already announced, or to move the evidence bar after seeing the data. The discipline is to fix the evidence bar and the artifact checklist before the result is known, blind reviewers to the hoped-for outcome where feasible, and keep the fallback option genuinely on the table.

How it implements the components¶

Signal/Noise Review fills the human-judgment and decision-linkage slice of the archetype:

extraction_use_contract — it judges the extraction against what it is actually for, holding the result to its intended use.
evidence_threshold — it applies, and enforces the pre-setting of, the bar the evidence must clear to be called signal.
decision_link — it is the gate connecting a validated extraction to the decision it is allowed to drive.
fallback_measurement — when the extraction fails review, it invokes the cruder but trusted alternative rather than passing bad signal downstream.

It is qualitative adjudication, not quantitative validation — the generalization metric is a Held-Out Sample Test — and it builds and separates nothing itself.

Instantiates: Model-Guided Signal Separation — the human gate that decides whether an extraction is trusted enough to act on.
Sibling mechanisms: Held-Out Sample Test · Signal Injection–Recovery Test · Residual Leakage and Whiteness Check · Feature Selection · Latent Variable Model

Notes¶

Signal/Noise Review and Held-Out Sample Test are the two validators in this set, and they cover different blind spots: the test certifies generalization with a number, while the review certifies realism and fitness for use with judgment. Each catches what the other misses — a result can generalize to held-out data and still be an artifact both share, or look real to a reviewer and still fail out of sample — so a careful pipeline runs both rather than treating either as sufficient.

References¶

[1] The look-elsewhere effect: when many independent locations (frequencies, time windows, trials) are searched, the probability of a chance fluctuation somewhere crossing a fixed threshold rises with the number of places looked. A local significance must be corrected for the size of the search before it is read as a global detection. ↩