Explainability Review¶
Test or assessment — instantiates Black-Box / White-Box Selection
Asks not whether a system is correct but whether its reasons are legible — whether the explanation it offers is understandable, and faithful, enough for the people who must rely on or contest the decision.
An Explainability Review evaluates the comprehensibility of a system's reasoning for a specific audience and decision. It sets aside whether the output is right — that is behavior testing — and whether the machinery is sound — that is internal audit — and asks a third question entirely: can the person who must act on, justify, or contest this decision actually understand why it was made? Its defining commitment is audience-relative legibility. The same model can be perfectly explainable to a machine-learning engineer and utterly opaque to the applicant it just rejected, and the review judges against the audience that matters for the decision — not against the most expert reader in the room.
Example¶
A bank's model-risk team reviews a machine-learning credit-decisioning model before it is allowed to issue adverse-action notices. Under fair-lending rules, a rejected applicant must receive specific, accurate principal reasons for the denial.[n1] The review does not re-test the model's accuracy and does not audit its training pipeline. It takes a set of real declined applications and checks whether the reasons the system surfaces — "high revolving-credit utilization," "short credit history" — are (a) intelligible to an ordinary applicant, (b) specific enough to act on, and © faithful to what actually drove the score rather than a plausible-sounding story pasted on afterward.
They find that the model's feature-attribution output is faithful but phrased in modeling jargon no applicant could parse, and they send it back for a plain-language rewrite keyed to the applicant audience. The model's correctness is never touched; what the review certifies is whether its explanations are fit — or unfit — for the people who depend on them.
How it works¶
- Fix the audience and the decision first. Legibility is meaningless in the abstract; the review names exactly who must understand and for what purpose (to act, to appeal, to sign off).
- Assess intelligibility, specificity, and actionability. A reason must be understandable to that audience, specific enough to be useful, and expressed in their vocabulary.
- Test faithfulness, not just plausibility. An explanation that sounds convincing but does not reflect the system's actual basis is judged a failure — it is worse than silence, because it manufactures false confidence.
- Judge sufficiency for this decision. The bar is "enough to support the decision," not maximum transparency.
Tuning parameters¶
- Target audience — expert reviewer through to affected layperson; the further toward layperson, the higher the legibility bar.
- Faithfulness standard — how tightly the explanation must track the system's actual computation, versus being merely consistent with its outputs.
- Depth of reason — a single headline factor, a ranked list, or a full counterfactual ("what would have changed the outcome").
- Explanation modality — natural language, worked examples, or feature attributions; each trades intelligibility against precision.
- Contestability threshold — whether the audience is merely informed or is actually equipped to challenge the decision.
When it helps, and when it misleads¶
Its strength is that it addresses a need neither behavior tests nor internal audits cover: decisions that must be justified, contested, or trusted by people, not merely be correct. Wherever accountability, consent, or appeal is at stake, a correct-but-inscrutable system can still fail its purpose.
Its failure mode is that an explanation can be plausible and satisfying yet unfaithful — a confident post-hoc rationalization that does not reflect the system's real basis. This is more dangerous than opacity because it feels like understanding while delivering none; feature-attribution methods such as LIME and SHAP are useful precisely to the extent their faithfulness is checked rather than assumed.[n2] The classic misuse is an explanation produced for marketing or reassurance rather than to change any decision. The guarding discipline is to hold explanations to a faithfulness standard, probe them against counterfactual inputs, and always state the audience the judgment is relative to.
How it implements the components¶
explanation_audience— its defining component: it fixes who must understand and grades the system's reasons against that audience's comprehension, not an expert's.evaluation_goal— it ties the legibility bar to the specific decision the explanation must support (act on, appeal, certify), so "explainable enough" has a concrete meaning.
It does not verify that the outputs are correct by exercising the system — that is behavior_test, the work of Black-Box Test — and it does not itself pry open the code and controls to confirm the reasons are true at the source, which is internal_mechanism_access and belongs to White-Box Audit; an explainability review judges legibility, its two twins judge behavior and internals.
Related¶
- Instantiates: Black-Box / White-Box Selection — it supplies the comprehensibility evidence that a chosen visibility mode may require for accountability.
- Sibling mechanisms: Black-Box Test · White-Box Audit · Inspection / Outcome Matrix · Process Audit · Certification Regime · Tiered Audit Protocol · Transparency Report
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Explainability Review operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it asks not whether a system is correct but whether its reasons are legible — whether the explanation it offers is understandable, and faithful, enough for the people who must rely on or contest the decision.
Independent corroboration: The frozen evidence defines Explainability Review as 'Asks not whether a system is correct but whether its reasons are legible — whether the explanation it offers is understandable, and faithful, enough for the people who must rely on or contest the decision', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Ethics of Technology & AI Governance
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Reviewing whether consequential system reasons are faithful, understandable, and contestable is canonical explainable-AI governance.
Related originating lineages:
- Computer Science & Software Engineering — Interpretable-machine-learning methods materially shape fidelity testing and system-level explanation generation.
- Human-Computer Interaction — Human-centered explanation evaluation materially supplies audience comprehensibility and reliance criteria. User comprehension and usability evaluation materially shape whether explanations are understandable to affected people.
Review resolution: Both reviewers agree that tech_ethics_ai_governance is primary. I retain human_computer_interaction, computer_science only as formative origin lineages; cross_disciplinary_synthesis is appropriate because the final form materially combines the agreed primary with the retained formative lineages. Reach is multi_domain because the structure transfers across several fields but is not a near-universal human pattern, an applicability judgment kept separate from provenance. Encyclopedia synthesis is false because the artifact is already established enough that encyclopedia-specific synthesis is not required. No unresolved historical ambiguity remains after reconciling the secondary fields.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Under the U.S. Equal Credit Opportunity Act and its Regulation B, a creditor that denies an application must give the applicant specific principal reasons for the adverse action. The requirement is a real-world instance of audience-relative explainability: the reasons must be intelligible and actionable to the applicant, not merely present in the model. ↩
[n2] In interpretability work, an explanation's faithfulness — how accurately it reflects the system's actual reasoning — is distinct from its plausibility — how convincing it seems. Popular post-hoc methods (e.g., LIME, SHAP) produce feature attributions whose faithfulness must be verified for the model at hand rather than presumed, which is why an explainability review tests explanations against counterfactuals rather than accepting them at face value. ↩