Complement Sensitivity Checklist¶
Review checklist — instantiates Complement Space Mapping
A short pre-flight list of questions that forces a team to look at whoever falls outside the focal set before shipping — who is omitted, who gets harmed, and where non-membership is being misread as the opposite.
Complement Sensitivity Checklist is a lightweight, human-run list of prompts that a team walks before committing a decision, model, or design — its entire purpose is to make the people think about the complement on purpose. Its defining move among its siblings is that it is a pre-flight prompt for humans, not a data artifact, a standing queue, or an enforced runtime rule: it asks a fixed set of questions ("who is outside the served set? what harm follows from omitting them? are we reading non-participation as disinterest?") and refuses sign-off until each is answered. Where the backlog tracks complement cases as ongoing work and the guardrail enforces interpretation limits in code, this checklist is the moment-of-decision conscience that surfaces the outside before anything ships.
Example¶
A product team is about to launch a recommendation model that ranks content for users. The focal subset the model optimizes is users who click; everyone else is complement. Before release they run the complement sensitivity checklist. Who is in the complement? Users who never click — but the list forces the split: some are genuinely uninterested, some never saw the item, some are new accounts with no history. What harm follows from omitting them? New users get starved of relevant content and churn, a real omission harm the click-optimized metric never shows. Are we treating non-membership as the opposite property? Yes — the model implicitly reads "did not click" as "not interested," when for the unseen-item group it only means "not shown."[n1] The checklist doesn't fix the model; it makes the team see three failure modes the success metric hid, and they add a cold-start path before launch rather than discovering the churn a quarter later.
How it works¶
- Fix a small, standing set of questions. The checklist is short and the same every time, so it is actually run rather than skipped; its power is completeness of prompting, not depth.
- Force an explicit answer per item. Each prompt — who is omitted, what harm, what is being misread — must be answered in writing before sign-off, converting a silent assumption into a stated one.
- Include the omission-harm prompt. One item asks specifically who is damaged by being left outside the focal set, so non-service is weighed at decision time.
- Include the opposite-property prompt. One item asks whether the design reads non-membership as the opposite property, catching the leap as a review question rather than a runtime failure.
Tuning parameters¶
- Checklist length — how many prompts; longer lists are more complete but get skimmed, shorter ones are run faithfully but may miss a case.
- Gate hardness — whether unanswered items block sign-off or merely advise; a hard gate guarantees the review but invites pro-forma answers.
- Trigger scope — which decisions require the checklist (every release vs. high-stakes only); universal use normalizes it but adds friction.
- Answer specificity bar — whether "considered, none found" is acceptable or evidence is required; a higher bar deters box-ticking.
- Reviewer independence — whether the author self-certifies or a second party runs it; independence catches blind spots the author shares.
When it helps, and when it misleads¶
Its strength is cheap, repeatable prompting at the exact moment blind spots do damage: it costs minutes, needs no tooling, and reliably surfaces the "who did we leave out?" question that success metrics are structurally built to hide. Because it is standing and identical each time, it catches the omission the team would otherwise be too close to see.
It misleads when it decays into ritual box-ticking — every item answered "N/A" or "considered, none found" so the form is complete and the thinking is not. A checklist can also breed false comfort: passing it feels like having handled the complement when it has only prompted about it, and a prompt is not a fix or a control. The classic misuse is treating a green checklist as evidence that the outside group is fine, when nothing was actually done for them. The guarding discipline is to require specific written answers rather than N/As, route anything the checklist surfaces into real follow-up work, and periodically audit past checklists against what later went wrong to keep the questions sharp.
How it implements the components¶
Complement Sensitivity Checklist realizes the decision-moment prompting side of the archetype — making the complement a deliberate review step:
omission_harm_review— a standing prompt forces the team to name who is harmed by being outside the focal set before sign-off, weighing omission at decision time.opposite_property_test— a standing prompt asks whether the design reads non-membership as the opposite property, catching the leap as a review question.
It does not maintain the outside cases as an owned, worked-down queue — that persistent tracking with a complement_update_rule is Residual Case Backlog — and it does not enforce interpretation limits on live downstream reasoning; that runtime downstream_use_constraint is Downstream Inference Guardrail.
Related¶
- Instantiates: Complement Space Mapping — it makes reviewing the complement a required step before a decision ships.
- Consumes: Universe–Subset–Complement Map supplies the declared frame the checklist asks its "who is outside?" questions against.
- Sibling mechanisms: Universe–Subset–Complement Map · Set-Difference Query · Membership Predicate Test · Inclusion/Exclusion Matrix · Disjointness and Exhaustiveness Review · Residual Case Backlog · Downstream Inference Guardrail · Universe Scope Change Log
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: A short pre-flight list of questions that forces a team to look at whoever falls outside the focal set before shipping — who is omitted, who gets harmed, and where non-membership is being misread as the opposite, making its operative form a bounded evaluation of existing evidence or work that produces a finding or disposition.
Independent corroboration: The frozen evidence defines Complement Sensitivity Checklist as 'A short pre-flight list of questions that forces a team to look at whoever falls outside the focal set before shipping — who is omitted, who gets harmed, and where non-membership is being misread as the opposite', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Statistical sampling and bias analysis made excluded cases, survivorship, and selection into the focal set central validity concerns.
Related originating lineages:
- Data Science & Analytics — Model-evaluation practice contributes checks on unobserved, unserved, and missing-label populations.
- Mathematics — Set complement supplies the exact formal partition the checklist forces reviewers to inspect.
Review resolution: Both reviewers agree on statistics_experimental_design as primary. Reading the source mechanism confirms that its defining operation belongs to that lineage; the final record retains data_science, mathematics only where it materially formed the mechanism and keeps present-day application breadth separate from provenance.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Survivorship bias — reasoning only from the cases that made it into the focal set, as in Abraham Wald's wartime analysis of returning aircraft: the bullet holes to reinforce were on the planes that didn't come back, the complement no one was looking at. The checklist's "who is outside?" prompt is a deliberate guard against exactly this. ↩