Test Case Matrix¶
Validation matrix — instantiates Predicate Criterion Formalization
Pins a grid of inputs to their expected verdicts so a predicate's implementation can be validated and re-checked for regressions.
A Test Case Matrix pairs a curated grid of inputs with the verdict each one should produce, then runs the predicate against them to confirm that it does — a validation and regression instrument. Its defining orientation is confirmation: the expected outcomes are fixed in advance as an oracle, and the matrix's job is to certify that the implemented predicate reproduces them and keeps reproducing them as it changes. It is organized as a coverage grid — input classes crossed against each other, chosen to exercise obvious, borderline, and adversarial cases — where each cell ties an input to both an expected verdict and the running implementation that must match it. It asks: does the predicate still give the right answer on the cases we've blessed?
Example¶
A team ships an is_strong_password predicate and builds a test case matrix to validate it. The rows are input classes: length (7 / 8 / 20 characters), character variety (lower-only, mixed, with symbols), and known-bad patterns (dictionary word, repeated characters, keyboard walk). Each combination is a cell paired with its expected verdict — "abcdefgh" (8 chars, lower-only, dictionary-ish) → expected FALSE; "Tr0ub4dour&3" → expected TRUE; a 7-character strong-looking string → expected FALSE because it is under the length floor.
Running the matrix, one cell fails: the predicate accepts a 20-character all-lowercase string the team expected to reject. The failure is a regression signal — the implementation diverged from the intended criterion at a boundary they had blessed. They fix the code and the matrix goes green, then it stays in continuous integration so a future refactor cannot silently re-break that boundary. Setup to outcome: an "it seems to work" predicate becomes a grid of pinned, re-runnable expectations that catch drift between the criterion and its code.
How it works¶
- Cross input classes into a grid. Choose the dimensions that matter (length, type, adversarial patterns) and cover their combinations, prioritizing boundaries and known-hard cases.
- Fix the expected verdict (oracle). Each cell records the correct answer independently of the implementation, so the matrix tests the code against intent, not against itself.
- Run and compare. Execute the predicate on every cell; a mismatch between actual and expected is a defect or a regression.
- Keep it green. The matrix persists as an automated suite, so future changes must preserve the blessed verdicts.
Tuning parameters¶
- Coverage strategy — exhaustive combinations versus pairwise or boundary sampling. Exhaustive is thorough but explodes; sampling is cheap but can miss interactions.
- Oracle source — hand-specified expected values versus a trusted reference implementation. Hand-specified is authoritative but laborious; a reference oracle scales but inherits the reference's errors.
- Boundary emphasis — how heavily edge and adversarial cases are weighted relative to typical cases.
- Refresh policy — how new failures discovered in the wild are folded in as permanent rows (regression capture).
When it helps, and when it misleads¶
Its strength is that it certifies a predicate's implementation matches its intended verdicts and stays matched — it catches drift between criterion and code, and turns each fixed bug into a permanent guard against its return.
Its failure mode is the pesticide paradox[n1] — a fixed matrix only ever catches the failures it already encodes, so a predicate can pass a green suite while being wrong across a whole class of cases nobody thought to add. A green matrix proves consistency with its own expectations, not correctness. The classic misuse is treating "all tests pass" as "the predicate is right" when the oracle itself was mis-specified. The discipline that guards against this is to keep adding new and adversarial cases — especially real failures found in production — and to periodically re-examine whether the expected verdicts still encode the intended criterion.
How it implements the components¶
boundary_case_library— the matrix IS a curated, organized library of cases (obvious, borderline, adversarial) with their expected verdicts, doubling as a coverage map and a reviewer-calibration set.implementation_trace— each cell binds an input and its expected verdict to the running predicate, so the matrix continuously verifies the implementation against intent and localizes where the two diverge.
It does NOT accumulate the surprising cases that FALSIFY the criterion or route them toward revising it — the refutation-driven drift_monitor and the appeal_or_override_path from a counterexample to a rule change are Counterexample Register; the matrix confirms known verdicts, it does not hunt for new refutations.
Related¶
- Instantiates: Predicate Criterion Formalization — the matrix is the validation-and-regression instance, certifying that the implemented predicate matches intent.
- Consumes: Counterexample Register — refutations logged there become new permanent rows in the regression grid.
- Sibling mechanisms: Boolean Guard Clause · SQL WHERE Clause or Query Filter · Truth Table · Decision Table · Eligibility Criteria Checklist · Policy Definition of Terms · Predicate Version Registry · Counterexample Register · Unknown-State Routing Rule
Editorial Notes¶
Form Classification¶
Form family: Representation, Specification & Plan
Rationale: Test Case Matrix is defined in the frozen evidence as: Pins a grid of inputs to their expected verdicts so a predicate's implementation can be validated and re-checked for regressions. Its operative deployed or enacted form is therefore Representation, Specification & Plan.
Nearest alternative: Experiment, Test & Rehearsal — Experiment, Test & Rehearsal can support this mechanism, but the evidence centers the concrete operation described above rather than the alternative family's defining operation.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Universal
Rationale: Test case matrix derives most directly from computer science's software, data-system, and algorithmic tradition; its defining operation is to pins a grid of inputs to their expected verdicts so a predicate's implementation can be validated and re-checked for regressions.
Related originating lineages:
- Engineering & Design — Engineering design, reliability, and systems-safety practice supplies a parallel or contributing lineage for the mechanism's defining operation: pins a grid of inputs to their expected verdicts so a predicate's implementation can be validated and re-checked for regressions.
- Security Studies & Intelligence Analysis — Security's adversarial analysis, integrity, and incident-response tradition provides a formative adjacent lineage for the same test case matrix operation.
Review resolution: Both blind reviewers independently select computer_science as the primary historical origin for the concrete operation—Pins a grid of inputs to their expected verdicts so a predicate's implementation can be validated and re-checked for regressions. The queued differences concern alternate origin disagreement, origin mode disagreement, domain reach disagreement, encyclopedia synthesis disagreement, not the primary lineage. I retain every alternate that either reviewer explains, without a numeric cap, and choose origin_mode=cross_disciplinary_synthesis because the reviewers' combined evidence identifies material construction from multiple disciplines. domain_reach=universal records later portability rather than multiplying historical origins; confidence=high is the conservative shared evidentiary level, and encyclopedia_synthesis=true preserves either reviewer's affirmative synthesis finding.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] The pesticide paradox (Boris Beizer): running the same tests repeatedly, like spraying the same pesticide, eventually stops finding bugs because the survivors are immune to that test set. A test case matrix that is never extended certifies only the cases it already contains — which is why fresh and adversarial cases must keep being added. ↩