Feature Ablation or Occlusion Test¶
Diagnostic intervention — instantiates Shortcut-Reliance Mitigation
Masks, removes, or permutes a suspected cue while holding everything else fixed, and reads the drop in performance as the model's reliance on that exact cue.
Once you suspect a specific crutch, you want to know two things: is the model actually using it, and how much. Feature Ablation or Occlusion Test answers both with a controlled intervention on the input itself. It takes one named cue — a background region, a metadata field, a header token, a color channel — masks or removes or permutes it while holding the rest of the input constant, and re-scores. The change in performance is a direct, quantified attribution: a large drop means the model was leaning on that cue; near-perfect accuracy from the cue alone means it was leaning on almost nothing else. Its defining move, and what separates it from a stress test, is surgical locality — it edits the contents of the input to isolate a single feature's causal contribution, rather than swapping the whole surrounding context. That locality is what turns "the model is brittle" into "the model gets 0.9 of its score from the timestamp."
Example¶
A camera-trap classifier that tells deer from wild boar scores an illustrative 96% on its validation set. A reviewer suspects habitat, not animal, is doing the work — boar were photographed mostly in dense undergrowth, deer mostly in open clearings. Two ablations settle it. First, occlude the animal and leave the background: accuracy stays around 89%, which should be impossible if the model were reading the animal. Second, occlude the background (inpaint it to neutral) and leave the animal sharp and centered: accuracy falls to about 61%. The reliance score is unambiguous — the classifier is mostly a habitat detector wearing an animal-classifier's label, and it will fail the moment a boar wanders into a clearing. The test does not fix anything and does not decide whether to ship; it hands a specific, sized finding to remediation and to the model card.
How it works¶
What distinguishes it is the paired, single-variable intervention. You draw a cue from the shortcut inventory, choose an intervention (mask, blur, remove, permute across examples, or substitute a counterfactual value), and apply it to that cue only, keeping every other feature fixed, so the score change is attributable to that one edit and nothing else. Two complementary readings matter: performance with the cue removed (how much the model loses without it) and performance from the cue alone (how much it can do with only the cue). Run across the inventory, this yields a ranked reliance score per suspected feature — the quantitative core the rest of the appraisal reasons over.
Tuning parameters¶
- Intervention type — masking, permutation, deletion, or counterfactual substitution. Permutation preserves the marginal distribution; crude masking can create inputs unlike anything in training and mislead.
- Replacement baseline — what you put where the cue was: zeros, dataset mean, blur, or realistic inpainting. An unrealistic fill makes the input itself the problem and inflates the apparent drop.
- Granularity — ablate one feature or a correlated group at once. Removing a single member of a correlated cluster lets the model route around it and hides the reliance.
- Paired vs. marginal — measure each cue in isolation, or leave-one-group-out against a full-input baseline. Isolation attributes; grouping catches redundancy.
- Aggregation — how per-example score changes roll up into the headline reliance number (mean, worst-case, or distribution).
When it helps, and when it misleads¶
Its strength is precision and cheapness: no retraining, and an answer of the form "this exact cue accounts for this much of the score," which is what remediation and documentation actually need. It is the most direct way to convert a suspicion into a measured reliance.
Its sharpest failure mode is correlated features. If the suspected cue has a redundant partner, removing it alone lets the model lean on the partner and the test under-reports reliance — the classic suppressor problem. The opposite error is an out-of-distribution artifact: a clumsy mask produces an input so unnatural that performance collapses for reasons unrelated to reliance, over-reporting it. And like any attribution, an occlusion map can be run backwards — the cue and mask chosen after the fact to produce the saliency picture someone wanted.[1] The discipline is realistic replacements, ablating correlated cues as a group, and corroborating a positive finding with an invariance probe rather than trusting a single map.
How it implements the components¶
Feature Ablation or Occlusion Test realizes the measurement side of the archetype — turning a named suspicion into a sized number:
feature_ablation_and_occlusion_plan— it specifies which cues to intervene on, by what method, against which baseline: the plan that makes the test repeatable and attributable.shortcut_reliance_score— the performance delta under intervention is the reliance score, per cue and aggregated, that the acceptance gate and the model card consume.
It does not construct the shifted operating environments or set the ship bar — that is Domain-Shift Stress Test's matrix and gate — and it is a behavioral probe, not the human judgment of whether the features are semantically legitimate, which belongs to Causal Feature Review Panel.
Related¶
- Instantiates: Shortcut-Reliance Mitigation — supplies the quantified, per-cue reliance measurement the appraisal reasons over.
- Consumes: Artifact Red-Team Review — its inventory of suspected shortcuts is the list of cues this test ablates.
- Sibling mechanisms: Domain-Shift Stress Test · Invariance Probe · Causal Feature Review Panel · Hard-Negative Data Augmentation · Shortcut-Risk Model Card Section
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Feature Ablation or Occlusion Test operates as a bounded trial, probe, simulation, or rehearsal that generates evidence from performance because it masks, removes, or permutes a suspected cue while holding everything else fixed, and reads the drop in performance as the model's reliance on that exact cue.
Independent corroboration: The frozen evidence defines Feature Ablation or Occlusion Test as 'Masks, removes, or permutes a suspected cue while holding everything else fixed, and reads the drop in performance as the model's reliance on that exact cue', so its operative form is Experiment, Test & Rehearsal.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Convergent development
Present-day reach: Specialized
Rationale: Computer science is primary because ablation and occlusion became operational tests in machine learning and computer vision: remove or mask an input and measure the model response. Data science and cognitive science contribute parallel feature-importance and lesion-style reasoning, but the named test remains a specialized computational technique.
Related originating lineages:
- Cognitive Science — Cognitive vision research materially shaped cue-isolation and recognition tests.
- Data Science & Analytics — Ablation, masking, permutation, and occlusion tests developed as machine-learning interpretability diagnostics.
Review resolution: Computer science is primary because ablation and occlusion became operational tests in machine learning and computer vision: remove or mask an input and measure the model response. Data science and cognitive science contribute parallel feature-importance and lesion-style reasoning, but the named test remains a specialized computational technique.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
References¶
[1] Ann-Kathrin Dombrowski et al. "Explanations can be manipulated and geometry is to blame". Advances in Neural Information Processing Systems 32, 2019. Shows that six post-hoc explanation methods—not occlusion maps—can be manipulated toward arbitrary target maps with hardly perceptible input perturbations while model output remains approximately constant. registry ↩