Test Coverage Audit¶
Test / assessment — instantiates Completeness Audit
Checks whether tests cover intended functions, branches, conditions, requirements, risks, or user paths, and then identifies untested regions.
A Test Coverage Audit measures whether an automated test suite actually exercises the behavioral units it is supposed to protect — functions, branches, conditions, paths — by mapping which tests execute which units, adversarially probing whether nominally-covered code is truly verified, and emitting a coverage figure as standing evidence. Its defining idea, and what separates it from its twin the Data Completeness Check, is that its coverage universe is executed behavior: it asks "does a test drive this code path, and would it notice if the behavior broke?" — not, as the Data Completeness Check asks, whether the underlying records and fields are present. It maps behavior exercised, not data supplied.
Example¶
A team maintaining an open-source payments library runs its coverage tooling before a release. Line coverage reads a comfortable 88%, but branch coverage on the refund-with-partial-capture path is only 40%, and — more insidiously — the currency-rounding function shows 100% line coverage while a mutation test reveals that flipping a <= to < inside it breaks no test. The tests execute the code but assert nothing meaningful about it: covered on the map, unverified in fact. The audit maps tests to branches, runs mutation testing to catch that "covered-but-unverified" illusion, and publishes the resulting coverage and mutation scores as the release's evidence record. The uncovered refund branch is flagged red on the map; whether that red branch is an acceptable gap, an infeasible path, or a must-fix hole is a disposition left to downstream judgment.[n1]
How it works¶
The mechanism instruments the running suite to record which behavioral units each test executes — building the coverage map — then runs mutation or other adversarial probes to test whether the "covered" units are genuinely asserted upon rather than merely executed. It computes and publishes a coverage-and-mutation metric that stands as the durable evidence of what the suite exercises. What distinguishes it is that its completeness claim is a measured percentage backed by adversarial probing, over executable behavior. It surfaces uncovered units directly off the map, but it stops there: classifying and disposing of those red regions is not its job.
Tuning parameters¶
- Coverage criterion — line versus branch versus condition/MC-DC versus path coverage. Stronger criteria catch more but are exponentially harder to satisfy.
- Probe strength — no mutation, a small operator set, or aggressive mutation. More probing exposes assertion-free "coverage" but costs runtime.
- Metric threshold — the coverage percentage that gates a release; set too high, it invites gaming rather than testing.
- Suite scope — unit tests only versus integration and end-to-end. Wider mapping, higher upkeep.
- Evidence retention — a one-off number versus a coverage record tracked over time as regression evidence.
When it helps, and when it misleads¶
Its strength is turning "we have tests" into a measured, adversarially-checked map of what is actually exercised, where mutation probing exposes the high-coverage-yet-untested illusion that a raw percentage hides. Its failure mode is that coverage is a proxy: 100% line coverage with weak assertions verifies nothing, and a coverage number invites Goodhart's law — once the percentage becomes the target, teams write tests that lift it without testing anything.[n1] The classic misuse is chasing a coverage figure as the goal rather than reading it as a signal of the untested regions worth attention. The guarding discipline is to pair coverage with mutation scoring so that "covered" means "verified," and to treat the metric as a map to untested risk, never as proof of correctness.
How it implements the components¶
coverage_map— it maps each test to the behavioral units (functions, branches, paths) it executes, so uncovered units are read directly off the map.sampling_probe— mutation testing and adversarial inputs probe whether nominally-covered code is genuinely verified, sampling the failure space rather than exhaustively enumerating it.completeness_evidence_record— the published coverage and mutation metrics stand as the durable evidence of what the suite exercises, for review and release gating.
It audits executed behavior, not present data: it does not check whether the required records and fields exist (intended_coverage_space), classify each uncovered region into true-gap versus declared-exclusion (gap_analysis), or keep the declared-missing register (exclusion_register) — all three belong to its twin Data Completeness Check.
Related¶
- Instantiates: Completeness Audit — it supplies a measured, adversarially-checked coverage map over executable behavior.
- Sibling mechanisms: Coverage Checklist Walkthrough · Data Completeness Check · Policy Gap Analysis · Risk Register Review · Scenario Tabletop Review · Stakeholder Inclusion Review · Requirements Traceability Matrix
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Test Coverage Audit operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it checks whether tests cover intended functions, branches, conditions, requirements, risks, or user paths, and then identifies untested regions.
Independent corroboration: The frozen evidence defines Test Coverage Audit as 'Checks whether tests cover intended functions, branches, conditions, requirements, risks, or user paths, and then identifies untested regions', so its operative form is Assessment, Review & Assurance.
Nearest alternative: Experiment, Test & Rehearsal — Test Coverage Audit includes features of an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation, but its defining operation is a bounded evaluation of existing evidence or work that produces a finding or disposition.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: The defining operation is: Checks whether tests cover intended functions, branches, conditions, requirements, risks, or user paths, and then identifies untested regions. In the computer_science lineage, that operation is specifically evidenced by authoritative or primary work that defines functional, code, branch, risk, and requirements coverage and requires evidence of untested regions. This makes computer_science the best historical origin, while the retained alternates document contributing methods and later applications rather than being mistaken for coequal origins.
Related originating lineages:
- Engineering & Design — engineering_design supplies a historically relevant parallel or contributing practice for the defining operation—Checks whether tests cover intended functions, branches, conditions, requirements, risks, or user paths, and then identifies untested regions—but the evidence does not make it the best primary lineage.
- Security Studies & Intelligence Analysis — Security's adversarial analysis, integrity, and incident-response tradition provides a formative adjacent lineage for the same test coverage audit operation.
- Statistics & Experimental Design — Statistics, experimental design, and measurement theory supplies a parallel or contributing lineage for the mechanism's defining operation: checks whether tests cover intended functions, branches, conditions, requirements, risks, or user paths, and then identifies untested regions.
- Systems Thinking & Cybernetics — Systems thinking, feedback control, and cybernetics supplies a parallel or contributing lineage for the mechanism's defining operation: checks whether tests cover intended functions, branches, conditions, requirements, risks, or user paths, and then identifies untested regions.
Review resolution: The blind reviewers disagree on primary lineage (computer_science versus engineering_design), so I adjudicated the mechanism rather than inheriting either label. The defining operation is: Checks whether tests cover intended functions, branches, conditions, requirements, risks, or user paths, and then identifies untested regions. In the computer_science lineage, that operation is specifically evidenced by authoritative or primary work that defines functional, code, branch, risk, and requirements coverage and requires evidence of untested regions. This makes computer_science the best historical origin, while the retained alternates document contributing methods and later applications rather than being mistaken for coequal origins. The cited NASA Software Test Plan: Test Coverage directly supports the mechanism-specific operation and its disciplinary lineage. I retain all independently explained historical alternates without a numeric cap. origin_mode=single_lineage records how the mechanism arose; domain_reach=multi_domain separately records how broadly it can now be applied.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
Notes¶
A coverage percentage and a mutation score answer different questions: the first says which code was run by a test, the second whether that test would catch a change. Reporting coverage without mutation is the single most common way a Test Coverage Audit certifies a suite that verifies far less than its number implies.
[n1] Goodhart's law — "when a measure becomes a target, it ceases to be a good measure" (after economist Charles Goodhart). Test coverage is a textbook case: a coverage percentage is a useful signal until it becomes the objective, at which point teams optimize the number rather than the verification it was meant to indicate. Mutation testing is the usual corrective, because it scores whether tests actually detect injected faults. ↩a ↩b