Skip to content

Context Confusion Matrix

Diagnostic — instantiates Context-Keyed Representation Switching

Tabulates how often each true context is served the wrong representation — a rows-are-truth, columns-are-selected grid that turns "switching feels flaky" into a map of exactly which contexts get mistaken for which.

"The system sometimes loads the wrong map" is not a fixable statement. Context Confusion Matrix turns it into one. It is a diagnostic that lays the selection pipeline's behavior out as a grid — rows are the true context, columns are the representation actually selected — over a labeled sample of real cases. The diagonal is where the right map was chosen; every off-diagonal cell is a specific confusion, named and counted. Its defining value is localization: instead of a single accuracy number, it shows precisely which contexts get mistaken for which, and in which direction, so a fix can target the one cue boundary at fault rather than "improve routing" in general. It does not select or correct anything — it measures and points.

Example

A bilingual call center runs an agent-assist tool that loads a script-and-knowledge map keyed by the caller's language and register — Spanish-formal, Spanish-casual, English-formal, English-casual. Agents report the tool "sometimes feels off." The team builds a confusion matrix over a few hundred labeled calls: rows are the true context, cells count which map the tool actually loaded. The diagonal is mostly healthy, but one off-diagonal cell stands out — Spanish-formal calls are frequently served the English-casual map, and not the reverse. That asymmetric confusion points straight at a specific cue: the tool is latching onto occasional English loanwords in formal Spanish speech (a code-switching artifact). Now the fix is targeted at that boundary instead of the whole router.

How it works

Its distinguishing property is that it grades the selection pipeline without being part of it: it takes ground-truth context on one axis and the pipeline's actual choice on the other, so the off-diagonal structure is the error, decomposed by pair. Asymmetry is first-class — A-mistaken-for-B is a different problem than B-for-A and usually has a different cause. Because it is a measurement, its honesty rides entirely on the quality of the "true context" labels, and its usefulness rides on the context bins being fine enough to isolate the real confusion.

Tuning parameters

  • Labeling source — how ground-truth context is obtained (human review, downstream corrections, held-out gold). The matrix is only as trustworthy as its truth column; a biased label source produces a confident, wrong diagnosis.
  • Context granularity — how finely contexts are binned. Finer bins localize the confusion but need more data per cell before the counts mean anything.
  • Cost weighting — whether cells are weighted by the harm of that particular confusion, not just its frequency, so a rare-but-dangerous mix-up is not buried under common harmless ones.
  • Sampling window — recent versus cumulative traffic. Watching the matrix shift over windows is how drift in the context distribution shows up.

When it helps, and when it misleads

It shines at converting vague "flaky switching" complaints into targeted engineering, and at monitoring drift — a confusion that grows over weeks is visible as a cell filling in.

Its central trap is aggregation hiding the catastrophic tail: under class imbalance, a 99% diagonal can still contain the one off-diagonal confusion that is unacceptable, and a headline accuracy number sails right past it. The classic misuse is reading an overall score off the matrix and declaring the router "good enough" while a specific harmful confusion persists. The discipline is to weight by cost and inspect the worst cell, not the average — the matrix's whole reason for existing is that not all errors are the same error.[1]

How it implements the components

  • context_disambiguation_rule — it measures how well cues are resolved to the correct context, exposing exactly where the disambiguation breaks down and in which direction.
  • representation_selection_rule — the columns record which representation was actually selected per true context, grading the end-to-end selection outcome the rule produces.

It does NOT build the cue model it grades — targeted cue probing is Minimal-Pair Context Probe's and the live gate is Gated Expert Router's — and it does not measure cross-map contamination, which is Cross-Map Interference Regression Suite's domain.

Notes

It grades the selector; it is not one. Its hardest and most load-bearing input is a trustworthy source of true context — if that is wrong or biased, the matrix will diagnose confidently and incorrectly, sending fixes at the wrong boundary.

References

[1] A confusion matrix is the standard evaluation grid of true-versus-predicted class, whose off-diagonal cells name each misclassification. Weighting those cells by cost, rather than reading a single accuracy off the diagonal, is the discipline that keeps a rare, harmful confusion from hiding behind a good average. ↩