Nonrandom Sample Audit¶
Checklist — instantiates Perceived-Consensus Calibration
A checklist that interrogates who the visible sample actually is — and who it silently leaves out — before their agreement is read as the population's.
A strong impression of agreement is often just a strong impression of a skewed slice: the friends who reply, the enthusiasts who post, the high-status voices who dominate. Nonrandom Sample Audit is the structured checklist that interrogates that slice before its agreement is generalized. It walks through the mechanisms that make a visible sample unrepresentative — self-selection, homophily, channel bias, status weighting, survivorship — and holds them against a map of who the target population actually is, so the gap between "who spoke" and "who counts" becomes explicit. Its defining move is diagnosis of the existing sample: it does not gather new data, run a poll, or fix the bias — it exposes the bias in evidence you already have, and names which segments of the target population that evidence cannot speak for. Its whole output is a bias profile: here is how this sample is skewed, and here are the parts of the population it silently omits.
Example¶
A B2B software team is convinced that "customers love the new pricing" because the feedback has been overwhelmingly positive. Before that belief drives a broader rollout, an analyst runs the sample audit against the checklist. Where did the feedback come from? The customer advisory board, the top-tier accounts, and replies to an in-app survey. The audit maps these against the target population — all paying customers — and the skew lights up: the advisory board is self-selected superfans; top-tier accounts get discounts that blunt price pain; the in-app survey only reached customers active enough to see it, missing the churned and the disengaged entirely.
The audit's output is not "the pricing is bad." It is a bias profile: the positive signal comes from the most engaged, most subsidized, most survivorship-filtered 15% of customers; the price-sensitive small accounts and the recently churned are entirely absent from this evidence. That profile tells the team precisely which segment to go sample before believing the love is universal — and stops a confident rollout built on the cheers of the people who were never going to complain.
How it works¶
- Enumerate the visible source. The checklist first pins down exactly where the agreement was observed — which channel, which respondents, which selection process produced them.
- Test each bias pathway. It runs the sample against a fixed set of skew mechanisms — self-selection, homophily with the decision-makers, channel and status effects, non-response, survivorship — flagging each that plausibly applies.
- Overlay the target population. It holds the biased sample against a map of who actually counts for the decision, marking which real segments are over-weighted and which are missing entirely.
- Emit a bias profile, not a fix. The deliverable is an explicit account of how the sample skews and whom it cannot represent — the input a survey or interview then acts on.
Tuning parameters¶
- Checklist depth — a quick five-item pass versus a full bias inventory. Depth catches subtler skews but costs time and can over-flag on low-stakes calls.
- Population-map resolution — coarse segments versus fine ones. Finer segmentation exposes narrower omissions but demands you already know the population well.
- Evidence threshold for a flag — how much suspicion counts as "this bias is present." A sensitive threshold catches more but cries wolf; a strict one is cleaner but misses subtle skew.
- Scope of audit — a single feedback channel versus every source feeding the claim. Auditing all sources is thorough but heavy; one channel is fast but partial.
When it helps, and when it misleads¶
Its strength is that it attacks the specific failure other mechanisms assume away: that the sample itself is rotten. It is cheap — no new data collection — and it produces a targeted shopping list of missing segments, which is exactly what makes a follow-up survey efficient. The canonical lesson it operationalizes is survivorship bias[n1]: the loudest, most visible responses are systematically the ones that survived a selection filter, and reasoning from them alone is how a skewed slice gets mistaken for the whole.
Its failure mode is that an audit diagnoses but does not cure. It can tell you a sample is biased and even in which direction, but it cannot tell you what the unbiased answer is — mistaking "the sample is skewed positive" for "the truth is negative" is its own error. It is also only as good as the population map it audits against; a wrong or missing map lets real omissions pass unseen. And a determined team can audit endlessly as a way to avoid actually going and asking the missing people. The guarding discipline is to treat the audit as the setup for evidence-gathering, not a replacement for it: its output is a target list, and the target list is worthless until someone samples it.
How it implements the components¶
nonrandom_sample_exposure— the checklist's core work: it names the specific mechanisms (self-selection, homophily, channel, status, survivorship) by which the visible sample skews away from the population.target_population_map— it overlays the biased sample on an explicit map of who actually counts, marking the over-represented and the entirely-absent segments.
It does not meet the representative_evidence_requirement — the audit exposes bias but gathers no population-grade evidence to replace it, which is the Representative Consensus Survey — and it renders no belief_distribution_view of what people actually believe; that display is the Belief Distribution Dashboard.
Related¶
- Instantiates: Perceived-Consensus Calibration — supplies the bias profile and missing-segment list that tell the calibration where its evidence is untrustworthy.
- Sibling mechanisms: Anonymous Belief Pre-Poll · Belief Distribution Dashboard · Consensus Claim Evidence Log · False-Consensus Premortem · Minority Report Prompt · Outgroup or Edge-Case Interview · Representative Consensus Survey · Silent-Start Estimation Round
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Nonrandom Sample Audit operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it a checklist that interrogates who the visible sample actually is — and who it silently leaves out — before their agreement is read as the population's.
Independent corroboration: The frozen evidence defines Nonrandom Sample Audit as 'A checklist that interrogates who the visible sample actually is — and who it silently leaves out — before their agreement is read as the population's', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Universal
Rationale: Survey sampling and statistical inference developed audits of selection frames, coverage error, survivorship bias, and limits on generalizing from visible cases.
Related originating lineages:
- Communication & Media Studies — Public-opinion and media research materially developed cautions about visible voices and perceived consensus.
- Sociology & Anthropology — Social research contributes attention to systematically excluded groups and access pathways.
Review resolution: Both independent reviews agree on primary origin statistics_experimental_design; reconciliation resolves alternate_origin_disagreement, origin_mode_disagreement, domain_reach_disagreement, encyclopedia_synthesis_disagreement. Formative alternate lineages retained: communication_media_studies, sociology_anthropology. The broader reach of later applications is kept separate as domain_reach=universal; origin_mode=cross_disciplinary_synthesis describes the historical relationship among lineages. Confidence is conservatively reconciled to high, and encyclopedia_synthesis=true preserves the reviewers' boundary judgment.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Survivorship bias is the error of drawing conclusions from the cases that made it past a selection filter while the filtered-out cases stay invisible — Abraham Wald's WWII analysis of returning aircraft, which armored the sections not hit on survivors, is the canonical example. A sample audit exists to make that invisible filter explicit before the survivors' agreement is read as everyone's. ↩