Skip to content

Minimal-Pair Context Probe

Diagnostic probe — instantiates Context-Keyed Representation Switching

Feeds the selector pairs of contexts that differ in exactly one cue, to find the single cue it is deaf to and pinpoint where it picks the wrong map.

Minimal-Pair Context Probe borrows the linguist's minimal pair — two words that differ in one sound, like bat and pat — and turns it on the map selector. You construct two contexts that are identical in every respect except a single cue, where that one cue should flip which representation is activated, and you check that the selector actually flips. Its distinguishing move is holding everything else constant: because the pair differs in exactly one place, a failure names the precise cue the system under- or over-weights, which an aggregate error rate never can.

Example

A customer-support assistant keeps separate intent-maps for a billing context and a technical context on one shared model. The word "charge" is the trap: it means a fee under billing and a battery level under technical. The probe builds a minimal pair — two messages identical except one cue: "my card won't charge" versus "my phone won't charge." Only the object noun differs, and it should route the first to billing and the second to technical. The selector sends both to billing. Because the pair isolates a single cue, the diagnosis is exact: the selector reads the verb "charge" and ignores the possessive-object cue that actually disambiguates. That is a targeted fix — reweight one cue — not a vague "improve accuracy."

How it works

  • Build the pair. Hold the whole context fixed and vary one cue that should change the selected map.
  • Present both and record the selection each produces.
  • Score the flip. A pass is the single cue flipping the map correctly; a fail is the cue being ignored (no flip) or triggering a spurious flip.
  • Sweep cues to rank which the selector genuinely uses versus which it is blind to.

Tuning parameters

  • Cue isolated — which single feature you vary. Aim at cues you suspect are load-bearing or dangerously ignored.
  • Pair distance — how minimal the difference is. A truly minimal pair probes the sharpest boundary; a looser pair tests robustness to bigger cue shifts.
  • Coverage versus depth — many shallow pairs across cues, or a few deep sweeps of one cue's sensitivity curve.
  • Boundary framing — honest near-neighbours versus adversarially chosen cues that sit just across the decision boundary, probing exploitable ambiguity.

When it helps, and when it misleads

Its strength is surgical localization: it names which cue the selector mishandles, where an aggregate metric can only say confusion happens. It also maps the adversarial cue boundary[1] — the smallest perturbation that flips the active map — which is exactly what an adversary would hunt for. Its failure modes come from where the pairs originate: hand-built pairs test the designer's imagination rather than the real cue distribution, so you probe the confusions you already suspect and miss the ones you don't; and fixes can overfit to the probe set, passing the pairs while the field behaviour is unchanged. The classic misuse is curating pairs the system already handles to demonstrate that it "works." The discipline that guards against it is to draw pairs from real misroute logs, and to pair the probe with an aggregate detector that surfaces confusions you never thought to test.

How it implements the components

Minimal-Pair Context Probe fills the front-end discrimination subset — the machinery that decides which map, tested at its sharpest edge:

  • context_cue_model — the probe stresses exactly this: which cues the model reads and how it weights them; a failed pair exposes an under-weighted cue.
  • context_disambiguation_rule — minimal pairs sit on the disambiguation boundary and test whether the rule resolves the single differing cue the right way.
  • adversarial_cue_boundary — the minimal perturbation that flips the selected map is that boundary; the probe locates it so it can be defended.

It does not aggregate confusions across a whole traffic sample (that's the Context Confusion Matrix), define the routing that maps cues to maps (that's the Context-to-Map Routing Table), or check that a selected map is internally intact (that's Map Difference and Integrity Check).

Notes

This probes selection correctness — was the right map chosen — and says nothing about whether the chosen map is itself intact; that back-end question belongs to Map Difference and Integrity Check. Where the Context Confusion Matrix finds that two contexts get confused across a sample, the minimal pair explains why by isolating the single responsible cue; they are the aggregate and the surgical view of the same boundary.

References

[1] Minimal pair — from phonology, two forms differing in exactly one contrastive feature. Used here as designed: a paired stimulus that isolates one cue's discriminative effect on which representation is selected. ↩