Comparator Set Audit¶
Audit procedure — instantiates Dimensioned Comparison Framing
Interrogates who is in and out of the comparison set and why, hunting for opportunistically chosen peers, missing baselines, and self-serving inclusions — and records the membership decision for later challenge.
The Comparator Set Audit attacks the comparison at its most-exploited joint: set membership. Long before dimensions or scales matter, the choice of which items sit next to each other decides the answer — a favorable peer group flatters, an omitted baseline hides the real question, a self-selected roster stacks the deck. This mechanism's one job is to challenge that roster: for every included item, why is it in?; for every plausible item that's out, why is it excluded?; and what baseline or anchor is missing that would change the reading? Where the Comparison Basis Checklist asks whether a frame exists at all, the audit assumes a frame and stress-tests the specific population inside it, leaving a written trail of who was admitted and on what grounds.
Example¶
A board is told the CEO's pay is "right at the median" of a fourteen-company peer group and therefore reasonable. The compensation committee runs a comparator set audit. It asks, company by company, why each peer belongs: three are more than five times larger by revenue, and two operate in a higher-margin adjacent industry — all of which lift the median. It then asks what's missing: the two closest direct competitors by size and sector were left out, and both pay well below the quoted median. Finally it checks for a baseline the roster quietly dropped — the company's own pay three years ago, and the pay of the next-tier internal executives, either of which reframes "at the median" as "up sharply."
Reconstituted with the omitted true peers and an internal baseline restored, the same pay figure lands in the top quartile, not at the median. The audit changed no salary and scored no dimension; it exposed that the entire conclusion rode on an opportunistically assembled set. Its lasting output is a documented membership decision — inclusions, exclusions, and their justifications — that the board can point to when the number is challenged next year.
How it works¶
- List inclusions with reasons. For each item in the set, record the explicit criterion that admits it (size band, sector, task, population). Items that entered "because they were handy" are flagged.
- List the omissions. Enumerate plausible comparands that were left out and test each exclusion against the same criteria — asymmetry between the inclusion and exclusion rules is the tell for a rigged set.
- Restore the missing baseline. Check whether an anchor the question demands — a prior-period self, a control case, a market reference — is absent, and add it if its omission changes the reading.
- Record the membership decision. Write the final roster with its inclusion/exclusion rationale into an audit trail, so a future reviewer can see and contest exactly how the set was drawn.
Tuning parameters¶
- Inclusion-criterion strictness — narrow, near-identical peers vs. a broad set. Narrow sets are defensible but small and noisy; broad sets are stable but dilute the "like-for-like" claim.
- Omission scan breadth — how far afield to hunt for wrongly excluded comparands. Wider scans catch more cherry-picking but cost time and can drag in genuinely non-comparable items.
- Pre-registration — whether the set is fixed before results are seen. Pre-registering blocks post-hoc roster edits that chase a desired answer but sacrifices the flexibility to correct a genuinely wrong initial set.
- Baseline requirement — whether an anchor case is mandatory. Requiring one prevents free-floating "compared to nothing" claims but forces a choice of baseline that itself can be argued.
When it helps, and when it misleads¶
Its strength is that it targets the manipulation most other checks miss: even a flawless scoring grid returns a rigged answer if the rows were chosen to produce it, and the audit is the only sibling whose whole subject is that row selection. The written membership trail also makes the set contestable long after the analysis ships.
Its failure mode is reference-class drift dressed as diligence — an auditor can keep "adjusting" the set until it yields the wanted median, so the audit becomes the manipulation instead of its cure.[n1] The classic misuse is exclusion-by-technicality: dropping an inconvenient comparand on a criterion that would also disqualify several included ones. The guarding discipline is symmetry — the rule that admits an item must be the same rule that excludes one — and pre-registration of the set where stakes are high, so the roster cannot be quietly re-cut after the result is known.
How it implements the components¶
The audit realizes the set-legitimacy core of the archetype — the components that govern who is being compared and against what:
comparand_set— its central object; the audit is the mechanism that admits, rejects, and justifies each member of this set.shared_frame_of_reference— enforced as the membership rule: an item belongs only if it shares the frame, and exclusions are tested against that same frame.baseline_or_anchor_case— the audit checks for and restores a missing anchor (prior self, control, market reference) the question requires.comparison_audit_trail— the recorded inclusion/exclusion rationale, kept so the membership decision can be reviewed and contested later.
It does not choose the comparison dimensions, score cells, or read a ranking — the Dimensioned Comparison Matrix does that — and it does not annotate what the final relation means or which dimensions were incomparable, which is the Comparison Readout Annotation's role.
Related¶
- Instantiates: Dimensioned Comparison Framing — the audit guarantees the legitimacy of the set the archetype's later machinery operates on.
- Sibling mechanisms: Dimensioned Comparison Matrix · Comparison Basis Checklist · Pairwise Comparison Protocol · Dimension Weight Sensitivity Panel · Matched Case Comparison Sheet · Comparison Readout Annotation · Counterbalanced Comparison Display
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Interrogates who is in and out of the comparison set and why, hunting for opportunistically chosen peers, missing baselines, and self-serving inclusions — and records the membership decision for later challenge, making its operative form a bounded evaluation of existing evidence or work that produces a finding or disposition.
Independent corroboration: The frozen evidence defines Comparator Set Audit as 'Interrogates who is in and out of the comparison set and why, hunting for opportunistically chosen peers, missing baselines, and self-serving inclusions — and records the membership decision for later challenge', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Statistical reference-class practice established auditing inclusion, exclusion, and baseline choice because comparator membership drives inference.
Review resolution: The mechanism audits inclusion, exclusion, baseline, and reference-class selection before inference, a standard statistical-design validity concern. Peer benchmarking is an application rather than an independent origin, and the source does not add a distinct cross-disciplinary operation, so the single-lineage, non-synthetic classification is retained.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] A reference class is the set of cases an item is judged against; the choice of reference class often determines the verdict more than the item's own attributes. Because the class is chosen by the analyst, it is a prime site for both honest disagreement and deliberate gaming — which is exactly why the choice must be justified and, ideally, fixed in advance. ↩