Skip to content

Black-Box Contract Test Suite

Conformance test suite — instantiates Representation-Independent Interface Contract

One reusable battery of tests written only against the public contract — no test may peek at internals — so that any implementation which passes it is accepted as a valid substitute.

A Black-Box Contract Test Suite encodes the expected behavior of a contract as executable checks that touch only the public surface, and it is parameterized over "the implementation under test" so the same battery runs against every implementation. Its two jobs give it its identity: it is the oracle that decides pass or fail on behavior, and it is the acceptance rule — passing the suite is the definition of a conforming, substitutable implementation. The load-bearing constraint is the no-peeking rule: the moment a test reads an internal field it stops being a contract test and becomes a test of one implementation. Where the Metamorphic Behavior Test checks relations without knowing the right answer and the Reference-Implementation Differential Test compares against a trusted implementation, this suite pins concrete, curated expected outcomes that every implementation must reproduce.

Example

A team owns a blob-storage interface with two implementations: a local-filesystem store and a cloud (S3-style) store. They write one suite against the interface, not against either backend — put(key, bytes) then get(key) returns those exact bytes; get of an absent key raises NotFound; list(prefix) returns exactly the keys under that prefix; overwriting a key returns the latest bytes. No test opens a file path or touches a bucket. Both implementations must pass the identical battery.

When they later add an in-memory cache-backed implementation, acceptance is simply does it pass the suite? — the suite has become the operational definition of "a conforming blob store." A subtle divergence surfaces cleanly: the cloud store is eventually consistent, so an immediate get-after-put occasionally misses, and that failure forces a real contract decision — is read-after-write consistency part of the promise, or not? Either answer is fine, but the suite made the question explicit instead of letting it hide in one backend. This is the Liskov Substitution Principle made runnable: a replacement must be usable everywhere the type is expected, and the suite is where that is checked.[1]

How it works

  • Derive checks from the contract, not any implementation — every assertion traces to a promised behavior, never to "what the current code does."
  • Forbid white-box access — tests may only call public operations and observe public results; this is what makes them survive a re-representation.
  • Parameterize over the implementation — one battery, run against many implementations, so the suite is portable across every candidate.
  • Treat passing as acceptance — a new or changed implementation is admitted exactly when it turns the suite green; the suite is the gate.

Tuning parameters

  • Coverage versus speed — how exhaustively the contract's cases and error paths are exercised. More coverage catches more divergence but costs authoring and runtime.
  • Curated versus generated cases — how much is hand-picked scenarios versus handed off to property or metamorphic generation. Curated cases are legible; generated ones find the surprises.
  • Guarantee pinning — whether the suite asserts timing, ordering, and consistency, or leaves them unspecified. Pinning them constrains implementations (a legitimate choice); over-pinning turns an incidental behavior into a required one.
  • Gate strictness — must-pass-all versus tiered core-and-optional features, which lets partial implementations be accepted for a subset of the contract.
  • Fixture realism — how production-like the setup is, trading fidelity against test speed and determinism.

When it helps, and when it misleads

Its strength is a single, implementation-independent definition of "conforming" that makes substitution safe and every swap auditable — and it catches divergence between implementations that reading either one in isolation would miss. But a suite is only as good as the cases it encodes: gaps let a non-conforming implementation pass (a green suite is not a correct implementation), and over-specified assertions that accidentally lock in an incidental behavior — a particular error string, a particular ordering — freeze the very representation the abstraction was meant to free. The classic misuse is writing the suite to match the current implementation's behavior — golden/snapshot tests of whatever it does today — which cements bugs as contract and blocks valid alternatives. The discipline: derive tests from the contract, review them for accidental over-specification, and when a second implementation legitimately fails, first ask whether the test or the contract is wrong before "fixing" the implementation.

How it implements the components

This mechanism supplies the verification and acceptance side of the archetype:

  • conformance_oracle — the suite's assertions are the pass/fail oracle for behavior at the public surface.
  • substitutability_acceptance_rule — "passes the suite" is the rule by which a replacement implementation is accepted as a valid substitute.

It does not author the contract it checks (the Abstract Data Type Specification and Design-by-Contract Clause do), supply the relational laws for the no-oracle case (the Metamorphic Behavior Test and Property-Based Conformance Test), or compare against a trusted reference build (the Reference-Implementation Differential Test).

References

[1] The Liskov Substitution Principle holds that objects of a subtype must be usable anywhere the supertype is expected without breaking the program's correctness. A contract test suite is one operational form of that principle: it defines, in runnable terms, what "usable in place of" means for a given interface. ↩