Source Confusion Matrix Review¶

Retrospective error review — instantiates Use-Time Source Attribution Calibration

A retrospective review that tabulates which source classes get mistaken for which — reading the off-diagonal cells to find systematic, directional misattributions and feed the fixes back.

Individual attribution errors look like carelessness; in aggregate they have shape. Source Confusion Matrix Review is the periodic look-back that lays recorded source attributions against their true source class in a grid, so that the errors stop being scattered mistakes and become a pattern of specific, directional confusions — which class keeps getting mistaken for which. Its defining move is reading the off-diagonal cells: not the headline "how often were we right," but the particular cell where, say, simulated keeps being logged as measured. A single confusion cell usually has a single, fixable cause; a lone error rarely does. The review turns a pile of corrected items into a diagnosis of the attribution system, and routes that diagnosis back into the cues, formats, and prompts that produced it.

Example¶

A research lab logs values into a shared database — some measured directly on an instrument, some produced by a simulation, some copied from a vendor datasheet, some interpolated. Each quarter, a data-integrity review samples logged entries, reconstructs each one's true source from lab notebooks and instrument logs, and crosses recorded-source against true-source in a grid.

The diagonal (agreements) looks reassuringly high. But one off-diagonal cell lights up: simulated values are repeatedly logged as measured, and almost never the reverse — a directional confusion, not random noise. That single cell points straight at a cause: the simulation tool exports in the same numeric format as the instrument, so people paste its output into the "measured" column without thinking. The fix is concrete — change the export format and add a source dropdown — and it is fed back rather than just corrected row by row. The review also notes a second pattern: the older an entry, the more likely its source tag is wrong, because whoever logged it has since forgotten. That decay signal says the fix must be capture-time, not cleanup-time.

How it works¶

Reconstruct ground truth for a sample. Take a batch of already-recorded attributions and independently establish each item's true source class from records, custody trails, or re-tracing — the answer key the recorded labels will be scored against.
Cross recorded against true. Tabulate the two into a grid: rows the source recorded, columns the source that was actually correct. Agreements fall on the diagonal; every error lands in a specific off-diagonal cell.
Read the off-diagonal, not the total. Interpret the concentrated cells as directional confusions with nameable causes (a shared format, a misleading interface, a cultural bias toward one label) rather than as a single accuracy score.
Watch confusion against age. Track whether specific confusions grow as items get older, flagging where source cues are decaying between capture and use — which distinguishes a capture problem from a classification problem.
Feed the pattern back. Route each identified confusion to its lever: cue design, storage format, prompts, interface warnings, or thresholds — correcting the generator of the error, not only the sampled rows.

What distinguishes it is the unit of analysis: it works on the population of past attributions to find systematic bias, where its siblings each act on one item at a time.

Tuning parameters¶

Review cadence — frequent small reviews versus rare deep ones. Frequent catches a drifting confusion early; deep gives enough sample to see rare but dangerous cells clearly.
Ground-truth rigor — how hard the true source is re-established per item. Higher rigor makes the matrix trustworthy but is costly, capping how many items you can review.
Sample selection — random draw versus targeting high-stakes or previously-suspect items. Random measures the true error profile; targeted finds the dangerous confusions faster but skews the picture.
Cell-salience threshold — how concentrated a cell must be before it counts as a systematic confusion worth acting on, rather than noise. Set it low and you chase flukes; set it high and you miss a slow-building bias.
Decay window — the age span over which confusion-versus-age is tracked. A short window catches fast cue loss; a long one reveals slow reconstruction drift but needs long-lived records.

When it helps, and when it misleads¶

Its strength is turning scattered misattributions into a small number of named, directional confusions, each with a traceable cause and a specific fix — and, by watching confusion rise with age, telling apart we classify badly from our cues rot before use. Because it also enforces looking at symmetric errors — internal-mistaken-for-external and the reverse — it resists fixing only the culturally salient direction while its mirror runs unchecked.

Its failure modes cluster on how the matrix is read. Collapsing it to a single accuracy figure hides the one off-diagonal cell that matters — a 95%-correct system can still be systematically laundering simulations into measurements in the dangerous 5%.^[1] It is retrospective, so it improves the next items, never the ones already misused. And its classic misuse is running it backward — producing a reassuring headline number for an audit or a stakeholder rather than to hunt the confusions. The discipline that keeps it honest is to read the off-diagonal cells rather than the diagonal, to insist on both error directions, and to close the loop into the error's cause instead of stopping at a corrected sample.

How it implements the components¶

Source Confusion Matrix Review fills the detect-and-correct-the-pattern side of the archetype — the components that operate on the population of past attributions:

source_attribution_error_profile — the matrix is the profile: which source classes are confused for which, in which direction, and how often.
source_cue_decay_monitor — tracking confusion against item age surfaces where source cues are decaying between capture and use, separating cue rot from misclassification.
attribution_feedback_loop — it closes the loop, routing each identified confusion back into cue design, formats, and thresholds so the generator of the error is fixed, not just its outputs.

It does NOT hold the gold-labeled commingled_item_inventory it scores against (the Source Attribution Training Set does), apply the live use_time_attribution_rule (the Reality Monitoring Checklist does), or attach a source_confidence_annotation to a single item (the Source Attribution Confidence Rubric does).

Instantiates: Use-Time Source Attribution Calibration — this review is the feedback organ that turns confirmed misattributions into fixes to the archetype's cues and thresholds.
Consumes: Source Attribution Training Set — its gold labels serve as the ground truth the matrix scores recorded attributions against.
Sibling mechanisms: Source Attribution Training Set · Reality Monitoring Checklist · Source Attribution Confidence Rubric · Memory Source Probe · Borrowed Idea Attribution Scan · Chain-of-Custody or Lineage Check · Generated Content Disclosure Gate · Hallucination Intrusion Triage · Observation Recheck or Replication · Provenance Lookup Before Publication · Source-Label Preserving Summary Template

References¶

[1] A confusion matrix — a table crossing the class recorded (rows) against the true class (columns), so that agreements fall on the diagonal and each off-diagonal cell names a specific, directional error such as "simulated logged as measured." Reading those off-diagonal cells, rather than the headline agreement rate, is the whole point of the review. ↩