Explainability Review¶
Test or assessment — instantiates Black-Box / White-Box Selection
Asks not whether a system is correct but whether its reasons are legible — whether the explanation it offers is understandable, and faithful, enough for the people who must rely on or contest the decision.
An Explainability Review evaluates the comprehensibility of a system's reasoning for a specific audience and decision. It sets aside whether the output is right — that is behavior testing — and whether the machinery is sound — that is internal audit — and asks a third question entirely: can the person who must act on, justify, or contest this decision actually understand why it was made? Its defining commitment is audience-relative legibility. The same model can be perfectly explainable to a machine-learning engineer and utterly opaque to the applicant it just rejected, and the review judges against the audience that matters for the decision — not against the most expert reader in the room.
Example¶
A bank's model-risk team reviews a machine-learning credit-decisioning model before it is allowed to issue adverse-action notices. Under fair-lending rules, a rejected applicant must receive specific, accurate principal reasons for the denial.[n1] The review does not re-test the model's accuracy and does not audit its training pipeline. It takes a set of real declined applications and checks whether the reasons the system surfaces — "high revolving-credit utilization," "short credit history" — are (a) intelligible to an ordinary applicant, (b) specific enough to act on, and © faithful to what actually drove the score rather than a plausible-sounding story pasted on afterward.
They find that the model's feature-attribution output is faithful but phrased in modeling jargon no applicant could parse, and they send it back for a plain-language rewrite keyed to the applicant audience. The model's correctness is never touched; what the review certifies is whether its explanations are fit — or unfit — for the people who depend on them.
How it works¶
- Fix the audience and the decision first. Legibility is meaningless in the abstract; the review names exactly who must understand and for what purpose (to act, to appeal, to sign off).
- Assess intelligibility, specificity, and actionability. A reason must be understandable to that audience, specific enough to be useful, and expressed in their vocabulary.
- Test faithfulness, not just plausibility. An explanation that sounds convincing but does not reflect the system's actual basis is judged a failure — it is worse than silence, because it manufactures false confidence.
- Judge sufficiency for this decision. The bar is "enough to support the decision," not maximum transparency.
Tuning parameters¶
- Target audience — expert reviewer through to affected layperson; the further toward layperson, the higher the legibility bar.
- Faithfulness standard — how tightly the explanation must track the system's actual computation, versus being merely consistent with its outputs.
- Depth of reason — a single headline factor, a ranked list, or a full counterfactual ("what would have changed the outcome").
- Explanation modality — natural language, worked examples, or feature attributions; each trades intelligibility against precision.
- Contestability threshold — whether the audience is merely informed or is actually equipped to challenge the decision.
When it helps, and when it misleads¶
Its strength is that it addresses a need neither behavior tests nor internal audits cover: decisions that must be justified, contested, or trusted by people, not merely be correct. Wherever accountability, consent, or appeal is at stake, a correct-but-inscrutable system can still fail its purpose.
Its failure mode is that an explanation can be plausible and satisfying yet unfaithful — a confident post-hoc rationalization that does not reflect the system's real basis. This is more dangerous than opacity because it feels like understanding while delivering none; feature-attribution methods such as LIME and SHAP are useful precisely to the extent their faithfulness is checked rather than assumed.[1] The classic misuse is an explanation produced for marketing or reassurance rather than to change any decision. The guarding discipline is to hold explanations to a faithfulness standard, probe them against counterfactual inputs, and always state the audience the judgment is relative to.
How it implements the components¶
explanation_audience— its defining component: it fixes who must understand and grades the system's reasons against that audience's comprehension, not an expert's.evaluation_goal— it ties the legibility bar to the specific decision the explanation must support (act on, appeal, certify), so "explainable enough" has a concrete meaning.
It does not verify that the outputs are correct by exercising the system — that is behavior_test, the work of Black-Box Test — and it does not itself pry open the code and controls to confirm the reasons are true at the source, which is internal_mechanism_access and belongs to White-Box Audit; an explainability review judges legibility, its two twins judge behavior and internals.
Related¶
- Instantiates: Black-Box / White-Box Selection — it supplies the comprehensibility evidence that a chosen visibility mode may require for accountability.
- Sibling mechanisms: Black-Box Test · White-Box Audit · Inspection / Outcome Matrix · Process Audit · Certification Regime · Tiered Audit Protocol · Transparency Report
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Explainability Review operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it asks not whether a system is correct but whether its reasons are legible — whether the explanation it offers is understandable, and faithful, enough for the people who must rely on or contest the decision.
Independent corroboration: The frozen evidence defines Explainability Review as 'Asks not whether a system is correct but whether its reasons are legible — whether the explanation it offers is understandable, and faithful, enough for the people who must rely on or contest the decision', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Ethics of Technology & AI Governance
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Reviewing whether consequential system reasons are faithful, understandable, and contestable is canonical explainable-AI governance.
Related originating lineages:
- Computer Science & Software Engineering — Interpretable-machine-learning methods materially shape fidelity testing and system-level explanation generation.
- Human-Computer Interaction (HCI) — Human-centered explanation evaluation materially supplies audience comprehensibility and reliance criteria. User comprehension and usability evaluation materially shape whether explanations are understandable to affected people.
Review resolution: Both reviewers agree that tech_ethics_ai_governance is primary. I retain human_computer_interaction, computer_science only as formative origin lineages; cross_disciplinary_synthesis is appropriate because the final form materially combines the agreed primary with the retained formative lineages. Reach is multi_domain because the structure transfers across several fields but is not a near-universal human pattern, an applicability judgment kept separate from provenance. Encyclopedia synthesis is false because the artifact is already established enough that encyclopedia-specific synthesis is not required. No unresolved historical ambiguity remains after reconciling the secondary fields.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Under the U.S. Equal Credit Opportunity Act and its Regulation B, a creditor that denies an application must give the applicant specific principal reasons for the adverse action. The requirement is a real-world instance of audience-relative explainability: the reasons must be intelligible and actionable to the applicant, not merely present in the model. ↩
References¶
[1] Slack, Dylan, Hilgard, Sophie, Jia, Emily, Singh, Sameer, and Lakkaraju, Himabindu. "Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods". Proceedings of the 2020 AAAI/ACM Conference on AI, Ethics, and Society, 180–186 (2020). Demonstrates that LIME and SHAP can produce plausible feature attributions that do not faithfully reflect a model's actual behavior. registry ↩ Show verification details
Supported in partVerified against the work's full text
Slack et al. demonstrate that LIME and SHAP explanations can be made to look innocuous for a biased classifier, so their faithfulness cannot be assumed; the 'more dangerous than opacity' claim is not tested.
“post hoc explanations techniques that rely on input perturbations, such as LIME and SHAP, are not reliable.”