Random Sample Audit¶
Sampling audit — instantiates Accountable Gatekeeping Design
Pulls a random sample of gate decisions — including the rejected and demoted ones — and re-judges them to measure consistency and surface criteria nobody wrote down.
A Random Sample Audit draws a random slice of a gate's decisions and independently re-judges them to see how consistent the gate really is and what it is actually deciding on. Its defining move is twofold: the sample is random and spans all outcomes — passed, rejected, deferred, demoted — not just the cases that got through, and each sampled case is judged again by someone blind to the first call. Sampling the rejected pile is the load-bearing choice, because a gate's worst errors are the good cases it wrongly kept out, and those are invisible to anyone who only inspects what passed. Comparing first and second judgments exposes two things a live gate hides: how much the outcome depended on which reviewer caught the case, and whether an unwritten criterion is quietly driving decisions.
Example¶
A company screens thousands of résumés through a first-pass gate. To check it, an auditor pulls a random sample of a few hundred screened applications — deliberately including rejects, not just the advanced candidates — and has a second reviewer re-score each one blind to the original decision. Two findings fall out. First, the two reviewers agree only moderately, which means an applicant's fate depended partly on the luck of the draw rather than the stated criteria. Second, the disagreements cluster: strong candidates from less-familiar universities were being screened out even though "school prestige" appears nowhere in the published criteria — a shadow criterion operating unrecorded. Neither could have surfaced by looking only at who advanced; both point to a fix, one in calibration and one in the rubric.
How it works¶
- Sample across every outcome. The frame is random and includes rejected, deferred, and demoted cases, so wrongful exclusions — the gate's blind spot — can appear.
- Re-judge blind. A second reviewer decides each sampled case without seeing the original call, so the comparison measures the gate rather than deference to it.
- Measure consistency. Agreement between first and second judgments quantifies how reproducible the gate's decisions are.
- Read the disagreements for pattern. Where mismatches cluster on some feature, that feature is a candidate unwritten criterion or a source of drift.
Tuning parameters¶
- Sample size — larger samples detect smaller effects but cost more reviewer time; too small and nothing ever reaches significance, which is also how an audit is quietly defanged.
- Stratification — sample every outcome equally, or oversample rare and high-stakes classes; stratifying surfaces problems in thin slices a flat sample would miss.
- Blinding — whether the re-judge is truly blind to the original decision and decider; without it the audit measures agreement-with-authority, not accuracy.
- Action threshold — how large a disagreement rate or pattern triggers recalibration or a criteria change; set it in advance so findings cannot be explained away after the fact.
When it helps, and when it misleads¶
Its strength is unique reach: because it samples the rejected pile, it is the only mechanism here that can see the wrongly excluded, and by re-judging blind it turns "trust the reviewers" into a measured consistency rate and a list of unwritten rules to confront. It checks the gate against its own stated criteria.
Its failure mode is the audit built to reassure rather than to find. Sample only the cases that passed and you commit survivorship bias — inspecting the survivors while the informative failures, the good cases the gate rejected, sit unexamined — which is precisely the mistake this mechanism exists to avoid and precisely how it gets watered down.[n1] Undersize the sample and everything looks fine; skip the blinding and you measure conformity, not correctness; run it once for show and drift returns. The discipline is a random frame that includes rejects, genuinely blind re-judging, and action thresholds fixed before the results are seen.
How it implements the components¶
Random Sample Audit fills the retrospective consistency slice — the sampled re-judgment and what its disagreements reveal:
calibration_sample_set— the random, re-judged sample is the instrument that measures how consistent the gate's decisions are across reviewers and over time.shadow_criteria_probe— clustered disagreements expose criteria that are operating but unrecorded, dragging the gate's real basis into view.
It does not enforce a distribution — that is Quota or Portfolio Guardrail — nor hold the full population of decision records (that is Gatekeeping Decision Log), nor adjudicate a specific contested case (that is Independent Review Panel). For gates run by a ranking algorithm, the specialised counterpart is Algorithmic Ranking Audit.
Related¶
- Instantiates: Accountable Gatekeeping Design — it tests whether the gate's real decisions match its stated criteria and reveals where they do not.
- Consumes: Gatekeeping Decision Log supplies the population it samples from; Published Selection Criteria supplies the stated standard it checks actual decisions against.
- Sibling mechanisms: Independent Review Panel · Quota or Portfolio Guardrail · Algorithmic Ranking Audit · Gatekeeping Decision Log · Published Selection Criteria · Reasoned Decision Notice · Editorial Standards Board · Appeals or Reconsideration Workflow · Blind or Double-Blind Review · Transparency Report · Conflict-of-Interest Disclosure
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Random Sample Audit operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it pulls a random sample of gate decisions — including the rejected and demoted ones — and re-judges them to measure consistency and surface criteria nobody wrote down.
Independent corroboration: The frozen evidence defines Random Sample Audit as 'Pulls a random sample of gate decisions — including the rejected and demoted ones — and re-judges them to measure consistency and surface criteria nobody wrote down', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Accounting & Auditing
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Universal
Rationale: Selecting a random subset for reinspection is a canonical audit-sampling practice.
Related originating lineages:
- Law & Governance — Review of gate decisions supplied due-process and consistency concerns, especially for rejected cases.
- Public Administration & Policy — Review of gate decisions materially extends it to administrative discretion.
- Statistics & Experimental Design — Probability sampling supplied estimability and unbiased case selection.
Review resolution: Both blind reviewers agree on accounting_auditing as the primary origin. Explicit reconciliation resolves alternate_origin_disagreement, domain_reach_disagreement. The merged alternate lineages retain only domains the reviewers identified as materially formative; domain_reach=universal records later applicability separately from origin breadth.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
The audit measures the gate against its own stated criteria; it detects drift and unwritten rules but does not itself fix them — findings feed the criteria owner and the review panel. The one non-negotiable design choice is sampling the rejected cases: an audit that inspects only what passed can certify a gate that is quietly excluding exactly what it should let through.
[n1] Survivorship bias is the error of drawing conclusions from the cases that made it through a selection while ignoring those that did not — famously, Abraham Wald's wartime insight that the armour should go where the returning planes had no bullet holes, because the planes hit there were the ones that never came back. An audit of a gate that samples only accepted cases repeats the mistake, which is why sampling the rejected pile is the point. ↩