Combinatorial Sampling Strategy¶
Test-design method — instantiates Composability Testing and Validation
Spends a finite test budget across an unbounded combination space, choosing which combinations to run by risk, factorial coverage, and domain theory rather than testing all of them.
Once a system has more than a handful of components, the set of possible combinations explodes past what anyone can test. Combinatorial Sampling Strategy is the mechanism that turns that impossible space into a defensible, finite list. Its defining move is principled selection under a budget: given that you can afford to run N combinations and not the full 2^k, it decides which N maximize the information gained per test — using coverage guarantees (e.g. every pair of settings appears together at least once), risk weighting (test the high-coupling, high-consequence combinations first), factorial structure, and domain theory about where interactions are likely to hide. It does not run the tests and it does not judge them; it authors the plan — and, just as importantly, it draws the boundary around what that plan does and does not cover.
Example¶
A SaaS platform ships with 8 optional plugins, 5 supported payment gateways, 4 regional data-residency modes, and 3 database backends. That is 8×5×4×3 = 480 fully-specified configurations before counting plugin-on/off subsets — far too many to certify for every customer. The release team has budget for roughly 40 integration runs. So instead of guessing, they build a covering array: a set of ~35 configurations engineered so that every pair of settings (e.g. "EU-residency mode + Stripe gateway", "analytics plugin + Postgres backend") co-occurs in at least one tested configuration.[1] Then they spend the remaining budget on five hand-picked high-risk combinations that domain knowledge flags — the new fraud plugin against the new gateway, the residency mode against the sharded backend.
The output is a 40-row test plan plus an explicit sentence: "Every pairwise setting combination is covered, plus five named higher-order risks; three-way combinations outside that list are untested and therefore unsupported." That boundary is what stops a single green test run from being marketed as "works with everything."
How it works¶
- Enumerate the factors and levels. List the independent choices (plugins, gateways, regions, backends) and how many values each can take — the dimensions of the space.
- Pick a coverage target. Decide the strength: cover all single settings (weak), all pairs (t=2, the common sweet spot), or selected triples where theory predicts interaction. Coverage strength is the main lever on cost.
- Generate a minimal satisfying set. Use a covering-array or orthogonal-array construction to produce the fewest configurations that hit the coverage target, so budget buys guaranteed reach rather than random luck.
- Overlay risk. Add specific high-consequence or high-coupling combinations the pure-coverage set might under-weight, and drop combinations that are impossible or forbidden by the contracts.
- Freeze the boundary. Record precisely which combinations are in the plan so untested regions default to "unsupported," not "assumed fine."
Tuning parameters¶
- Coverage strength (t) — pairwise, three-way, or mixed. Each step up multiplies the array size but catches deeper interactions; pairwise catches the majority of interaction defects for a fraction of exhaustive cost.
- Risk weighting — how heavily historical failures, coupling, and consequence bias the sample toward certain combinations. Heavy weighting finds known dangers fast but can starve the long tail of novel ones.
- Constraint handling — how aggressively impossible or prohibited combinations are pruned before generation. Tight pruning shrinks the array but risks excluding a "shouldn't happen" case that happens.
- Budget — the flat cap on number of runs. Lowering it forces weaker coverage or narrower scope; the strategy's job is to make that trade explicit rather than silent.
When it helps, and when it misleads¶
Its strength is that it converts combinatorial explosion into a bounded, auditable plan with a coverage guarantee attached — you can state exactly what fraction of the interaction space is covered and at what strength, which is the difference between reasoned sampling and testing theater. It is the mechanism that makes "we can't test everything" an engineering decision instead of an excuse.
Its failure mode is that coverage guarantees are only as honest as the space they are computed over. If a factor is missing from the model — an environment variable, a load level, a user locale nobody listed — the array covers a phantom space and reports full coverage while the real interaction hides in the unmodeled dimension. The classic misuse is treating "100% pairwise coverage" as "100% tested," when a defect that requires a specific three-way combination sails straight through a t=2 array. The guard is to state the coverage strength and the factor model alongside the percentage, and to feed observed field failures back as new factors so the model of the space keeps pace with reality.
How it implements the components¶
combinatorial_coverage_budget— it is the budget's allocator: it decides how the finite number of affordable runs is spent to maximize guaranteed reach.interaction_test_matrix— it authors the matrix's contents, selecting which pairwise, higher-order, and boundary cells the plan will contain.composability_claim_scope— the sampled region is the supported claim; combinations outside the array are explicitly scoped out as unsupported.
It designs the plan but renders no verdict — the invariant oracle that judges each combination belongs to Pairwise Interaction Probe and Property-Based Composition Testing, and running the plan in realistic settings belongs to Staged Integration Sandbox.
Related¶
- Instantiates: Composability Testing and Validation — the sampling strategy is how the archetype tames combinatorial explosion into a coverage plan.
- Sibling mechanisms: Pairwise Interaction Probe · Property-Based Composition Testing · Staged Integration Sandbox · Invariant Monitoring Dashboard · Fault-Injection Composition Probe · Incompatibility Root-Cause Analysis · Metamorphic Composition Test
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: The strategy deliberately selects a bounded set of factor combinations to run so pairwise or higher-order interaction coverage and risk evidence can be generated within a finite test budget, making it experimental test design.
Nearest alternative: Analysis, Modeling & Optimization — A covering-array calculation minimizes the configuration set, but that computation exists to choose active trials whose execution generates the evidence.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Design of experiments established factorial, covering-array, and risk-weighted sampling of interaction spaces under finite test budgets.
Related originating lineages:
- Computer Science & Software Engineering — Combinatorial testing operationalized t-way coverage for configurable software.
Review resolution: Both reviewers agree on statistics_experimental_design as primary. Reading the source mechanism confirms that its defining operation belongs to that lineage; the final record retains computer_science only where it materially formed the mechanism and keeps present-day application breadth separate from provenance.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
The pairwise-everything approach of Pairwise Interaction Probe is a special case of this strategy at t=2 with no budget cap. This mechanism generalizes it: when even all-pairs is unaffordable, or when higher-order coverage is worth buying, the sampling strategy is what decides the trade.
References¶
[1] Cohen, D. M., Dalal, S. R., Fredman, M. L., & Patton, G. C. "The AETG System: An Approach to Testing Based on Combinatorial Design". IEEE Transactions on Software Engineering 23(7), 437-444 (1997). AETG pairwise coverage requires every valid pair of parameter values to occur in at least one generated test. registry ↩