Inspection / Outcome Matrix¶
Checklist — instantiates Black-Box / White-Box Selection
A grid that lists every question the evaluation must answer and assigns each to the evidence source that can answer it — behavior test, internal inspection, or both — so no question is orphaned and no evidence is collected without a question.
An Inspection / Outcome Matrix is the routing artifact that decides, question by question, which evidence mode applies. Down one axis it lists the questions the decision actually requires; across the other it marks whether each is answered by observable behavior, by internal inspection, or by both in agreement. Its defining purpose is coverage and non-duplication: it exists to guarantee that every decision-relevant question has an assigned evidence source and that scattered tests and audits are not run for their own sake. Crucially, it is not itself a test or an audit — it produces no evidence about the system. It organizes the demand for evidence and, just as importantly, exposes the cells where no adequate source has been assigned.
Example¶
A hospital is evaluating a diagnostic imaging model before deployment, and the evaluation lead builds a matrix rather than letting each reviewer run their own tests. The rows are the questions that matter: overall accuracy, false-negative rate on rare pathologies, subgroup performance across age and skin tone, training-data provenance, the human-override pathway, and failure-alerting behavior. The columns are behavior test / internal inspection / both.
Accuracy and false-negative rate map to behavior tests on a held-out set. Subgroup fairness maps to "both" — behavior stratified by group, plus internal review of training-data composition — with an explicit rule that if the two disagree, the internal finding governs. Data provenance maps to internal inspection only. One row, model drift after deployment, has no adequate evidence source available today, so it is not given a checkmark: it is logged as residual uncertainty and assigned a monitoring trigger instead. The outcome is a one-page map that tells everyone exactly what evidence is owed, by whom, and where the genuine unknowns sit.
How it works¶
- Enumerate the questions from the goal, one per row. The rows come from what the decision needs to know, not from what is convenient to test — this is what keeps the evaluation on target.
- Assign each question an evidence source. Behavior, internal, or both; the assignment is the hybrid evidence map.
- Require agreement in "both" cells. Where behavior and internal evidence must jointly support a conclusion, the matrix states what happens if they conflict, so a contradiction cannot be silently ignored.
- Turn empty cells into logged unknowns. Any question with no adequate source becomes an explicit residual-uncertainty item with a review trigger — never a silent gap.
Tuning parameters¶
- Row granularity — coarse questions are quick to fill but hide sub-questions; fine ones give real coverage at the cost of a sprawling grid.
- Source strictness — how strong the assigned evidence must be before a cell counts as "answered."
- Conflict rule — what a "both" cell does when behavior and internal evidence disagree (internal governs, escalate, or block the decision).
- Completeness bar — whether the decision may proceed with open cells, or every row must be resolved or explicitly waived first.
- Ownership — whether each cell names a responsible party, which turns the matrix from a map into an accountability tool.
When it helps, and when it misleads¶
Its strength is that it prevents the two opposite pathologies the parent archetype warns about: an evidence free-for-all where tests and audits pile up disconnected from any question, and a coverage gap where a decisive question quietly goes unexamined. It is essentially a requirements-traceability matrix applied to evaluation evidence — every question traced to the source that answers it.[n1]
Its failure mode is checkbox theater: a fully green matrix that looks complete because every cell has a mark, while the marks point at weak or irrelevant evidence. A matrix records that a source was assigned, not that the source was any good — so a filled grid can launder shallow testing into apparent rigor. The guarding discipline is to pair each assignment with an evidence-strength judgment and to treat empty and weak cells as the most informative parts of the grid, not the embarrassing ones.
How it implements the components¶
hybrid_evidence_map— it is that map: the grid that assigns each question to behavioral evidence, internal inspection, or both, and states how the two reconcile.evaluation_goal— its rows are the goal decomposed into answerable questions, which is what keeps the evidence-gathering aimed at the decision.residual_uncertainty_register— every unassignable or unresolved cell becomes a logged unknown with a review trigger, so gaps are recorded rather than hidden.
It assigns questions to evidence sources but performs neither: exercising inputs and outputs is behavior_test (Black-Box Test) and inspecting internals is internal_mechanism_access (White-Box Audit). The matrix maps; they measure.
Related¶
- Instantiates: Black-Box / White-Box Selection — it operationalizes the mode-selection decision into a per-question evidence plan.
- Consumes: Black-Box Test and White-Box Audit supply the actual evidence the matrix routes questions to.
- Sibling mechanisms: Black-Box Test · White-Box Audit · Explainability Review · Process Audit · Certification Regime · Tiered Audit Protocol · Transparency Report
Editorial Notes¶
Form Classification¶
Form family: Representation, Specification & Plan
Rationale: Inspection / Outcome Matrix operates as a non-executable information artifact that externalizes static or prospective structure because it a grid that lists every question the evaluation must answer and assigns each to the evidence source that can answer it — behavior test, internal inspection, or both — so no question is orphaned and no evidence is collected without a question
Independent corroboration: The frozen evidence defines Inspection / Outcome Matrix as 'A grid that lists every question the evaluation must answer and assigns each to the evidence source that can answer it — behavior test, internal inspection, or both — so no question is orphaned and no evidence is collected without a question', so its operative form is Representation, Specification & Plan.
Nearest alternative: Analysis, Modeling & Optimization — The matrix externalizes a prospective question-to-evidence map and stops short of computing or judging the answers.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Engineering & Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Mapping each evaluation question to behavioral tests, internal inspection, or both derives from systems-engineering traceability and verification planning.
Related originating lineages:
- Statistics & Experimental Design — Evidence-design logic materially shapes whether each question has a valid and sufficient observation source.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Independent reviewer agreement; high confidence.
Notes¶
[n1] A requirements traceability matrix is a standard systems-engineering artifact that maps each requirement to the design element or test that satisfies it, so that no requirement is left unverified and no test exists without a requirement behind it. The Inspection / Outcome Matrix applies the same coverage discipline to the choice of evaluation evidence. ↩