Perverse Incentive Red Team¶
Test or assessment — instantiates Reinforcement Loop Design
Stress-tests the loop by asking how a rational, overloaded, fearful, or opportunistic actor might satisfy the reinforcement while violating the intent.
Every reinforcement loop can be satisfied more than one way, and the cheapest way is rarely the intended one. Perverse Incentive Red Team is the adversarial exercise that goes looking for the cheap way before the loop ships: a small group deliberately role-plays the reinforced actor — rational, rushed, scared, or opportunistic — and hunts for how to collect the reward or dodge the penalty while betraying the purpose. Its defining move is generative and hostile: it does not evaluate the loop for fairness or set limits, it manufactures attacks and then prescribes exactly what to watch so the predicted gaming can be caught in the wild. It is the loop's attacker, hired by the loop's designers.
Example¶
A software company plans to pay support engineers a quarterly bonus scaled to tickets closed — an attempt to reward responsiveness. Before it launches, a Perverse Incentive Red Team is convened for an afternoon. Playing the overloaded engineer, they find it in minutes: close the hard ticket as "resolved" and let the customer reopen a fresh one, splitting one problem into three closures. Playing the opportunistic engineer: cherry-pick trivial password resets and starve the thorny cases. Playing the fearful engineer: mark ambiguous tickets closed rather than escalate and look slow.
The team's output is not a verdict on whether the bonus is fair — that is someone else's job — but a ranked list of exploits and, for each, the divergence it would produce and the signal that would expose it: reopen-within-72-hours rate, closure-to-CSAT gap, and distribution of ticket difficulty per engineer. Those become the monitors the loop is required to carry from day one. The bonus still launches, but now with tripwires aimed at the exact behaviors the red team proved it would otherwise teach. When, months later, one engineer's reopen rate spikes, the monitor the red team specified is already there to catch it.
How it works¶
- Adopt hostile personas. Run the loop from the point of view of specific gaming archetypes — the rational maximizer, the overloaded corner-cutter, the fearful concealer, the opportunist — because each finds a different exploit.
- Attack the cheapest path. Ask what the least effortful way to satisfy the reinforcement is, then check whether it coincides with the intent; where it doesn't, you've found a perverse incentive.
- Rank by likelihood × harm. Not every exploit is worth guarding; prioritize the ones an ordinary actor would stumble into under real pressure.
- Convert each exploit into a monitor. For every surviving attack, specify the leading signal that would reveal it in production — the red team's deliverable is not a warning but an instrumented tripwire.
Tuning parameters¶
- Adversary strength — naive actor versus sophisticated maximizer. Modeling a clever adversary finds subtler exploits but can over-fit to attacks no real participant would attempt.
- Persona breadth — how many actor types are role-played. More personas surface more exploit classes but lengthen the exercise and blur focus.
- Scope — the reinforced behavior alone versus the whole surrounding system it interacts with. Wider scope catches cross-loop gaming but risks boiling the ocean.
- Monitor-conversion strictness — whether every ranked exploit must ship with a tripwire or only the top few. Full conversion is safest but instruments heavily; partial keeps the dashboard lean at some risk.
- Cadence — one pre-launch pass versus periodic re-teaming as actors adapt. Repeating catches learned gaming but costs recurring effort.
When it helps, and when it misleads¶
Its strength is that it finds the exploit on paper, cheaply, before it is trained into people at scale — and by handing off concrete monitors it makes its own predictions falsifiable rather than leaving vague worry. It is the structured antidote to Campbell's law, the observation that the more any indicator is used to allocate rewards, the more it will be gamed and the more it will distort what it measures.[n1]
Its failure mode is the theater of adversarial imagination: a red team can generate an intimidating list of exploits no real actor would bother with, and if that list isn't ranked by plausibility it produces paralysis or a bloated set of tripwires that no one watches. It can also breed a cynicism that treats every participant as a cheat, poisoning the trust a healthy loop needs. And it only predicts — it does not fix; a red team that never hands its findings to the mechanisms that redesign the reward or set the limits has done half a job. The guarding discipline is to rank by real-world likelihood, to route findings to the review that actually adjusts the loop, and to keep the exercise about the design's weak spots, not the participants' character.
How it implements the components¶
Perverse Incentive Red Team realizes the adversarial-safeguard side of the loop — the components that anticipate and instrument gaming, none that build the reward or set its limits:
perverse_incentive_check— its core: the deliberate, structured search for how the loop rewards the wrong action, punishes truth-telling, or invites metric gaming.outcome_monitor— it specifies the divergence signals (proxy-vs-outcome tripwires) that would catch each predicted exploit in production.
It examines but does not build the consequence structure: setting the reward_calibration, weighing autonomy_and_consent_boundary, and vetting alignment against the behavior_goal are the deliberative job of its nearest safeguard sibling, Consequence Design Review — the red team attacks; the review decides. It also does not run the standing dashboard that displays those monitors over time (Behavior Data Dashboard).
Related¶
- Instantiates: Reinforcement Loop Design — the red team hardens the loop by finding and instrumenting its exploits before launch.
- Consumes: Consequence Design Review — the red team's ranked exploits feed the review that adjusts the reward and its limits.
- Sibling mechanisms: Consequence Design Review · Behavior Data Dashboard · Reward or Recognition System · Reinforcement Schedule Design · Safety Reinforcement Protocol · Behavioral Prompting · Immediate Feedback Interface · Training Feedback Cycle · Habit Loop Mapping
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Perverse Incentive Red Team operates as an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation because it stress-tests the loop by asking how a rational, overloaded, fearful, or opportunistic actor might satisfy the reinforcement while violating the intent.
Independent corroboration: The frozen evidence defines Perverse Incentive Red Team as 'Stress-tests the loop by asking how a rational, overloaded, fearful, or opportunistic actor might satisfy the reinforcement while violating the intent', so its operative form is Experiment, Test & Rehearsal.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Organizational & Management Science
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Perverse Incentive Red Team is rooted in organizational and management science: Management control and Campbell's law motivate adversarially testing how measured incentives will be gamed.
Related originating lineages:
- Economics & Finance — Economics and finance materially shaped Perverse Incentive Red Team through incentives, contracts, markets, valuation, and strategic choice. Principal-agent and mechanism-design theory supplied the rational gaming and incentive-compatibility analysis.
- Military & Strategic Studies — Red-team practice supplied adversarial role-play and deliberate search for failure modes.
- Psychology — Psychology and behavioral science materially shaped Perverse Incentive Red Team through perception, judgment, learning, motivation, and behavioral bias.
Review resolution: Both blind reviewers agree that organizational and management practice is the primary origin. Reconciliation resolves alternate_origin_disagreement. Formative alternate lineages are retained as economics_finance, psychology, military_strategic_studies; later breadth of use is recorded separately as domain_reach=multi_domain, while origin_mode=cross_disciplinary_synthesis describes the relationship among origin lineages.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Campbell's law — "the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." The red team is the exercise that treats this as a design hazard to be probed rather than discovered in the field. ↩