Self Control¶
Core Idea¶
Self-control is the structural pattern in which an agent containing competing internal drives — a fast, salient, present-oriented impulse and a slower, goal-aligned, future-oriented evaluation — overrides the prepotent impulse in favor of the higher-order or longer-horizon objective, a dynamic Mischel (1989) first isolated experimentally in his delay-of-gratification paradigm. [1] It requires three things at once: a conflict between two valuations of the same action, a higher-order standard that ranks one valuation over the other, and an override capacity — limited, depletable, and trainable — that enforces the ranking against the pull of the immediate. [2] It is intrapersonal conflict resolved in favor of the represented goal over the felt impulse. The pattern presupposes a single agent divided against itself rather than two agents in dispute: the same system both generates the temptation and supplies the resistance. Without all three elements — conflict, standard, and override — there is no self-control to speak of, only preference (if there is no conflict), drift (if there is no standard), or wishing (if there is no override). The prime names the moment where a represented future wins out over a felt present inside one bounded controller.
How would you explain it like I'm…
Wait, don't grab
Beating Your Urges
Overriding impulse for goal
Structural Signature¶
Self-control encodes a structural pattern: dual-valuation conflict → higher-order standard → costly override → goal-aligned action. It separates two valuations of the same candidate action (the immediate appetitive value and the discounted long-horizon value) and names the work required for the second to defeat the first, a separation Ainslie (1992) formalized through hyperbolic discounting and the resulting preference reversals. [3] The signature is fundamentally one of an agent overruling a part of itself.
Recurring features:
- Override of a prepotent impulse in favor of a represented goal
- Conflict between immediate and delayed valuations of one action
- Higher-order standard that ranks competing internal drives
- Depletable, trainable override capacity enforcing the ranking
- Present self bound by the commitments of the future self
- Planner subsystem subordinating a doer subsystem
- Local gradient suppressed in service of a global objective
The structural insight is robust: a child resisting a marshmallow, a saver defeating present-biased preferences, a prefrontal circuit inhibiting a limbic reward signal, an AI agent suppressing a high-immediate-reward action, and a polity binding itself with a balanced-budget rule all exhibit the same override-against-impulse logic, a convergence Baumeister and colleagues (2007) trace across the self-regulation literature. [4] What travels is not the content of the temptation but the architecture: two valuations, a ranking, and an enforcer.
What It Is Not¶
Self-control is not the absence of desire. The cleanest misconception is to confuse not wanting something with wanting it but overriding the want. A person who feels no pull toward the dessert is not exercising self-control by declining it; self-control requires that the impulse be present and prepotent, and that it be defeated anyway. The everyday phrase "she has great self-control" often conflates these, crediting override where there was merely no temptation. [5] The prime applies only when there is a live conflict to be resolved.
Nor is self-control a claim that the overriding goal is correct. The pattern is silent on whether the higher-order standard is wise. An anorexic individual overriding hunger, a miser overriding the impulse to enjoy, or a zealot overriding compassion are all exercising self-control in the structural sense; the prime describes the mechanism, not the merit of the goal it serves. Self-control is not a virtue word here, even though ordinary usage treats it as one.
Self-control is also not unlimited or free. It is not the exercise of a faculty that can be summoned indefinitely on demand. The override capacity is depletable and exhaustible; relying on it as if it were costless predicts failure. A model that treats willpower as an inexhaustible resource will systematically mispredict when control collapses — late in the day, under stress, after prior exertion. The prime explicitly builds in cost.
Finally, self-control does not require conscious deliberation in the moment. Much of the most reliable override is not won by gritting one's teeth at the point of temptation; it is won upstream, by restructuring the situation so the conflict never fully arises (precommitment, removing the cue, automating the response). The prime covers both the in-the-moment override and the upstream maneuver that makes override unnecessary; it does not claim that control must look like a felt struggle.
Broad Use¶
Psychology: Resisting the marshmallow now to get two later; inhibiting a habitual or prepotent response in favor of a deliberate one; delay of gratification, ego-depletion phenomena, and trait conscientiousness, as Duckworth and Seligman (2005) document in their work linking self-discipline to outcomes that outpredict raw intelligence. [6]
Behavioral economics: Precommitment and self-binding to defeat present-biased preferences — Ulysses contracts, Christmas savings clubs, illiquid retirement accounts — and the planner-doer models that formalize an agent bargaining with its own future selves, as Thaler and Shefrin (1981) framed the economic-self-control problem. [7]
Neuroscience: Prefrontal top-down inhibition of limbic reward signals; the lateral prefrontal cortex modulating ventral striatal valuation during choice, a circuit-level account Heatherton and Wagner (2011) synthesize as the balance between top-down control and bottom-up reward drive. [8]
Artificial intelligence: An agent suppressing a high-immediate-reward action to maximize discounted long-run return; reward-hacking avoidance, where a system declines a proxy-maximizing shortcut in favor of the intended objective; specification-gaming resistance designed into the reward rather than trained into the policy.
Physiology and appetite: Satiety and inhibitory signaling overriding consummatory drives; the homeostatic and hedonic systems competing over a single eating decision, where the inhibitory pathway must defeat the appetitive one.
Public finance and institutions: Institutional rules — debt brakes, balanced-budget amendments, independent central banks — that bind a polity against its own short-term spending or inflationary impulses, the structural analogue of an individual tying themselves to the mast.
Clarity¶
Naming self-control as a structural pattern separates not wanting something from wanting it but overriding the want — a distinction collapsed in everyday talk and consequential for diagnosis. It lets practitioners locate failures precisely rather than attributing them globally to "weak will." A failure can sit in the standard (no goal ranks the impulse down, so there is nothing to enforce), in the conflict detection (the impulse is not recognized as conflicting with a goal until too late), or in the override capacity itself (the will and the goal are both present but the enforcer is depleted). [9] Each location implies a different remedy, and conflating them produces useless advice ("just try harder") aimed at the wrong component.
The prime also clarifies why self-control feels paradoxical: it is the experience of an agent acting against its own stated preference. By splitting the single agent into competing valuations, self-control dissolves the apparent contradiction — the agent is not irrational, it is divided, and the division is structural rather than a defect. This reframing turns a moral puzzle ("why can't I just do what I know is best?") into an engineering question ("which component is failing, and how is it loaded?").
Manages Complexity¶
Self-control organizes an agent's behavior into two interacting subsystems rather than a single coherent preference, bounding the apparent irrationality of acting against one's own stated goals. [7] Instead of modeling behavior as the output of one utility function, it admits that the same action can carry two valuations and that the agent contains machinery to adjudicate between them. This compresses a sprawling class of self-defeating behaviors — procrastination, addiction relapse, overspending, breaking a diet — into a single structural story: a prepotent impulse defeated an under-resourced override of a correctly-ranked goal.
The pattern explains self-defeating behavior as a structural feature of multi-system agents, not a mere character flaw, and it points to where intervention will pay off. Because it distinguishes changing the choice architecture from strengthening the override, it tells a practitioner whether to spend effort upstream (remove the temptation, add friction, precommit) or downstream (build the enforcing capacity, economize its use). It also makes visible the trade between the two: an agent that has cheap upstream restructuring available should not be spending scarce override on the same conflict repeatedly.
Abstract Reasoning¶
Recognizing the pattern licenses several non-obvious inferences. First, that override is costly and exhaustible, so it should be economized rather than relied upon — a system that must exercise control constantly is mis-designed. Second, that removing the conflict upstream (precommitment, situation selection) generically beats winning it repeatedly downstream, because each downstream win draws on the depletable resource while an upstream maneuver pays once. [4] Third, that the same agent can be productively modeled as a "planner" bargaining with a "doer," with the planner's leverage coming not from exhortation but from constraining the doer's future option set.
These inferences transfer beyond the human case. The pattern supports reasoning about any system that must subordinate a local, immediate gradient to a global objective: a greedy algorithm that must resist the locally optimal step, a control system that must not chase a transient setpoint deviation, a trading desk bound by risk limits against the temptation of a hot position. [10] Wherever a system is structured so that its immediate-reward signal diverges from its long-horizon objective, the self-control pattern predicts both the characteristic failure mode and the family of fixes.
Knowledge Transfer¶
The behavioral-economics insight that binding the future self outperforms willpower in the moment transfers cleanly to AI safety and to institution design. In AI safety, the lesson is to design the reward so as to remove the temptation rather than to train resistance to it — to make reward-hacking unrewarding by construction rather than hoping the policy learns to abstain, the structural analogue of building an illiquid account rather than relying on monthly restraint. [11] In fiscal institution design, the same logic yields the durable finding that rules beat discretion when present bias is structural: a constitutional debt brake binds the polity's future self where good intentions will not, exactly as a Ulysses contract binds the individual's.
The neuroscience of depletable inhibition transfers to operations and design. If override is a limited resource, like any limited resource it should be reserved for high-stakes conflicts and not squandered on routine ones; this reframes the design of environments (workplaces, diets, interfaces) around minimizing the number of override events demanded per day. [4] A practitioner who knows the prime in one domain can read it in another: the engineer who understands commitment devices recognizes the same shape in a central bank's independence, and the clinician who understands cue-removal in addiction recognizes it in a software default that pre-empts a bad choice.
Examples¶
Formal/abstract¶
Hyperbolic discounting and preference reversal: An agent prefers two units of reward at time t+1 over one unit at time t when both are distant, but reverses to prefer the one immediate unit when t arrives — the signature of hyperbolic rather than exponential discounting. Self-control is the structure that lets the earlier-self's ranking survive the reversal: a precommitment that removes the smaller-sooner option, or an override that enforces the larger-later choice at the point of temptation. The two valuations of the same action (now-discounted versus later-discounted) are explicit, the standard is the earlier preference, and the override (or the commitment device standing in for it) is what enforces it. [3] Mapped back: This is the prime in its barest form — conflict between two valuations of one action, a higher-order standard that ranks them, and an enforcer that makes the ranking stick against the pull of the immediate. The agent acting against its own momentary preference is not irrational; it is a divided system in which the represented future is being defended.
A reinforcement-learning agent with reward shaping: An agent is trained with a discount factor that values long-run return, but the environment offers a proxy that yields high immediate reward while subverting the intended objective (a reward-hacking shortcut). An agent that takes the shortcut maximizes the immediate signal; an agent exhibiting the self-control structure suppresses the prepotent high-reward action in favor of the discounted long-run objective. The most robust fix, however, is upstream: redesign the reward so the shortcut is no longer rewarding, removing the conflict rather than relying on a learned override. Mapped back: The structure is identical to the human case — a present-salient high-reward impulse, a long-horizon standard, and an override capacity — and it reproduces the same design lesson: binding the objective upstream beats training resistance downstream, because a downstream override is fragile and exhaustible while an upstream redesign holds by construction.
Applied/industry¶
Retirement saving and illiquid accounts: Workers who intend to save consistently fail to, because each month the present-biased doer reallocates the money to immediate consumption. Defined-contribution plans with automatic enrollment, payroll deduction before the money is ever seen, and early-withdrawal penalties function as a commitment device: they bind the future self by removing the smaller-sooner option from the doer's reach. The planner wins not by exhorting the doer monthly but by constraining the choice set in advance. Mapped back: Rather than spending depletable override every payday, the system relocates the conflict upstream and resolves it once — the canonical demonstration that situation design dominates repeated in-the-moment control.
Central-bank independence and fiscal rules: A government has a standing temptation to inflate or to deficit-spend for short-term gain, against its own long-run interest in price stability and solvency. Delegating monetary policy to an independent central bank, and adopting a constitutional debt brake, binds the polity's short-term impulse with an institutional override that the day-to-day political doer cannot easily reach. The rule is the higher-order standard given enforcement teeth. Mapped back: This is self-control scaled to a collective agent — the same conflict, standard, and override architecture, with the override embodied in an institution precisely because individual political will is known to be depletable and present-biased. The structural lesson (rules beat discretion when present bias is structural) is the institution-design face of binding the future self.
Structural Tensions¶
T1: Override strengthening versus conflict removal. The prime contains two genuinely different routes to the same goal-aligned outcome — build a stronger enforcer, or restructure the situation so the conflict never arises — and they pull against each other for finite attention and resources. Investing in willpower training treats the override as the lever; investing in commitment devices treats the choice architecture as the lever. A practitioner who fixates on one is blind to the other, and the two can even undercut each other: a person who removes all temptation never develops the override, while a person who relies only on override never builds the protective environment.
T2: Cost economy versus reliability. Because override is depletable, the prime says to economize it and reserve it for high-stakes conflicts. But a system that economizes too aggressively — automating and pre-committing everything — becomes brittle when a novel conflict arrives for which no upstream maneuver was prepared, and the under-exercised override turns out to be weak. The agent must spend enough control to keep the capacity trained while spending little enough to keep it available, and there is no general rule for where that balance sits.
T3: Whose standard counts. Self-control resolves conflict in favor of the higher-order standard, but the prime does not say which of the agent's valuations is the higher-order one. The present impulse and the future evaluation are both the agent's; privileging the future self over the present self is a substantive commitment, not a structural given. A framework that always sides with the patient long-horizon self can pathologize legitimate present enjoyment, and there is a real tension between honoring the planner and not tyrannizing the doer.
T4: The same act reads as discipline or as rigidity. An override that defeats a destructive impulse looks like admirable self-control; the identical structural act — overriding a felt drive in service of a represented standard — can be repression, compulsion, or self-denial when the standard is harsh or the impulse legitimate. Because the prime is silent on the merit of the goal, the same mechanism underwrites both the dieter's healthy restraint and the disorder's self-starvation, and an observer cannot tell which from the structure alone.
T5: Upstream binding can over-constrain the future. Precommitment defeats present bias by removing options from the future self, but the future self may face circumstances the binding agent did not foresee, in which the now-removed option was the right one. The illiquid account that protects against frivolous spending also blocks access during a genuine emergency; the rigid fiscal rule that prevents reckless deficits also prevents necessary counter-cyclical spending. Every Ulysses contract trades flexibility for protection, and the trade can go wrong.
T6: Modeling the agent as divided can become an excuse. Splitting the agent into planner and doer dissolves the moral puzzle of acting against one's own goals — but the same move can dissolve responsibility, letting the agent disown the impulse as if it belonged to someone else ("the doer did it"). The structural story that makes self-defeating behavior intelligible can, taken too far, make it seem inevitable, eroding the very sense of agency on which strengthening the override depends.
Structural–Framed Character¶
Self Control sits toward the structural side of the structural–framed spectrum, with some framing: it names the pattern in which an agent containing competing internal drives — a fast, present-oriented impulse and a slower, goal-aligned evaluation — overrides the prepotent impulse in favor of the higher-order or longer-horizon objective. It requires a conflict between two valuations and a standard that ranks one above the other.
The prime arose in psychology, where Mischel isolated it in delay-of-gratification experiments, and its framing in terms of a "higher-order standard" carries a mild normative flavor that brushes against the framed pole on origin, vocabulary, and evaluative weight. But the conflict-plus-override structure is formalized neutrally elsewhere: a control system that suppresses a high-gain transient to track a setpoint, or a reinforcement-learning agent discounting immediate reward for greater expected return, instantiates the same dynamic. Applying it recognizes a ranked-goal override already present rather than importing a stance. Several axes sit borderline while the underlying pattern reads recognizable and structural.
Substrate Independence¶
Self Control is a highly substrate-independent prime — composite 4 / 5 on the substrate-independence scale. Its structure — an agent with competing present-impulse and future-evaluation drives, plus a higher-order standard and an override capacity — is fairly substrate-agnostic and reaches across psychology, neuroscience (prefrontal inhibition of limbic reward), computation (an AI agent suppressing high-immediate-reward actions to avoid reward-hacking), and institutions (fiscal rules versus discretion). The 'bind the future self' insight transferring into AI safety and institution design is genuine cross-substrate evidence. It stops short of the top because it presupposes an agent with goals, so it never reaches purely physical substrates.
- Composite substrate independence — 4 / 5
- Domain breadth — 4 / 5
- Structural abstraction — 4 / 5
- Transfer evidence — 4 / 5
Relationships to Other Primes¶
Parents (1) — more general patterns this builds on
-
Self Control presupposes Temporal Inconsistency and Preference Reversals
Self-control presupposes temporal inconsistency because the conflict it resolves — between a salient present-oriented impulse and a slower future-oriented evaluation — is constituted by the prior structural pattern in which an agent's preference ordering reverses as the decision horizon approaches. Without that gap between distant-self and near-self valuations of the same action, there would be no impulse to override and no higher-order standard to enforce. Self-control inherits the temporal-inconsistency structure and supplies the intra-agent machinery — attention, commitment devices, depletable override capacity — that lets the represented goal win over the felt impulse.
Path to root: Self Control → Temporal Inconsistency and Preference Reversals → Time Preference (Discounting Future) → Preference
Neighborhood in Abstraction Space¶
Self Control sits among the more crowded primes in the catalog (8th percentile for distinctiveness): several abstractions describe nearly the same structure, so a description that fits it will tend to fit its neighbors too — transporting it usually means disambiguating within this family rather than landing on it exactly.
Family — Cooperation, Trust & Institutional Bonds (19 primes)
Nearest neighbors
- Cooperation — 0.84
- Commitment Device — 0.84
- Reputation — 0.83
- Decision — 0.83
- Conformity — 0.82
Computed from structural-signature embeddings · 2026-05-29
Not to Be Confused With¶
Self-control must first be distinguished from homeostasis, the nearest structural neighbor by surface similarity. Homeostasis holds a regulated variable near a setpoint through negative feedback: a deviation is sensed, and a corrective response drives the variable back toward the reference value. The crucial difference is that homeostasis involves no competing goal and no genuine conflict of valuation. The system is not torn between two rankings of the same action; it is simply correcting an error against a single fixed reference. Self-control, by contrast, is defined by a real conflict between two valuations of the same candidate action — the immediate appetitive value and the discounted long-horizon value — and by a higher-order standard that must adjudicate between them. A thermostat does not want the room hot while evaluating that it should be cool; it has one setpoint and corrects toward it. Where homeostasis has a setpoint and an error signal, self-control has a divided agent and a contested ranking. The two can co-occur — appetite regulation has both a homeostatic loop and a self-control conflict layered over it — but the homeostatic component is the error-correcting machinery while the self-control component is the override that fires when the hedonic valuation diverges from the homeostatic one. Confusing them collapses the very distinction the prime exists to make: that self-control is conflict resolved, not error corrected.
Self-control is also not self-handicapping, which is in many ways its structural opposite. Self-handicapping is a pre-emptive strategy in which an agent reduces its own effort or erects an obstacle to its own performance, so that a future failure can be attributed to the handicap rather than to lack of ability — protecting self-image at the cost of outcome. The student who parties the night before an exam so that a poor grade can be blamed on fatigue rather than incompetence is self-handicapping. Where self-control deploys an override to enforce the goal-aligned action against an impulse, self-handicapping deliberately withdraws effort and sabotages the goal-aligned action to manage the interpretation of its outcome. Self-control aims the agent at the represented future; self-handicapping aims the agent at protecting a present self-concept, even at the future's expense. The two share only the feature that both are intrapersonal maneuvers; in their direction — toward versus away from the higher-order goal — they are reversed.
Finally, self-control is distinct from self-efficacy, with which it is frequently conflated in everyday and even clinical talk. Self-efficacy is a belief — the agent's appraisal of its own capability to execute the behaviors required to attain a goal. Self-control is the enacted override itself: the actual subordination of a prepotent impulse to a higher-order standard. The distinction is sharp because the two can dissociate in both directions. An agent can have high self-efficacy ("I am confident I can resist") and still fail at self-control when the override capacity is depleted at the moment of temptation; the belief was sincere and even well-calibrated about ability-in-principle, yet the in-the-moment enforcement collapsed. Conversely, an agent low in self-efficacy can nonetheless succeed at control by relying on commitment devices that make the override unnecessary — the doubter who locks the cupboard exercises self-control through structure despite believing their willpower is weak. Self-efficacy is a forecast about the controller; self-control is the controller actually firing. A theory that predicts behavior from efficacy beliefs alone will mispredict exactly the cases the prime cares about most — the confident relapse and the doubtful success — because it has mistaken the belief about the capacity for the operation of the capacity.
Solution Archetypes¶
No catalogued solution archetypes reference this prime yet.
Notes¶
Self-control operates at multiple scales, and the structure recurs at each while the substrate of the override changes. At the individual scale the override is a depletable cognitive-neural capacity; at the institutional scale it is a rule with enforcement teeth (a debt brake, an independent central bank); at the computational scale it is a constraint built into a reward or a policy. Recognizing which scale is in play matters, because the depletion story that governs individual willpower does not transfer literally to an institutional rule, whose "fatigue" is political rather than metabolic.
A recurring confusion is between self-control and patience or time preference itself. Time preference is the discount structure — how an agent values the future relative to the present. Self-control is what an agent does when its discount structure produces a conflict it wishes to resolve in favor of the future. An agent with no present bias needs no self-control; the prime becomes relevant precisely when the discount function is hyperbolic enough to generate preference reversals. The prime was in fact surfaced while processing time-preference discounting, and the two are tightly coupled, but they are not the same: one describes the valuation, the other the machinery for overruling it.
The depletion model of override (ego depletion) has had a contested empirical history, with large replication efforts qualifying the strength of the effect. The prime does not stand or fall on any particular quantitative model of depletion; what is structural is that the override is finite and economizable, whatever its precise dynamics. Treating it as costless is the error the prime guards against, and that holds even under weaker depletion models.
Because the prime is silent on the merit of the higher-order standard, it must be paired with normative reasoning about goals. The same override architecture serves the recovering addict and the self-starving patient; the structure tells you the mechanism, not whether the goal it enforces is worth enforcing. Critical reasoning about which self deserves to win must accompany any technical reasoning about how to make it win.
References¶
[1] Mischel, W., Shoda, Y., & Rodriguez, M. L. (1989). Delay of gratification in children. Science, 244(4907), 933–938. The delay-of-gratification paradigm that experimentally isolated self-control as the override of a salient present impulse in favor of a larger delayed reward. ↩
[2] Carver, C. S., & Scheier, M. F. (1981). Attention and Self-Regulation: A Control-Theory Approach to Human Behavior. Springer-Verlag. Cybernetic feedback-loop account of self-regulation as a reference standard, a monitoring comparator, and an operating capacity that closes the discrepancy — the three constitutive elements of the override. ↩
[3] Ainslie, G. (1992). Picoeconomics: The Strategic Interaction of Successive Motivational States within the Person. Cambridge University Press. Formalizes hyperbolic discounting and the resulting preference reversals as the source of the dual-valuation self-control conflict and the bare minimal instance of conflict, standard, and enforcer. ↩
[4] Baumeister, R. F., Vohs, K. D., & Tice, D. M. (2007). The strength model of self-control. Current Directions in Psychological Science, 16(6), 351–355. Strength-model summary showing that the same depletion-recovery dynamics generalize across self-control, focus, and emotion-regulation domains despite domain-specific resources. ↩
[5] Metcalfe, J., & Mischel, W. (1999). A hot/cool-system analysis of delay of gratification: Dynamics of willpower. Psychological Review, 106(1), 3–19. Two-system (hot "go" / cool "know") account requiring a live, prepotent impulse that is recognized and overridden, distinguishing wanting-but-overriding from the mere absence of temptation. ↩
[6] Duckworth, A. L., & Seligman, M. E. P. (2005). Self-discipline outdoes IQ in predicting academic performance of adolescents. Psychological Science, 16(12), 939–944. Longitudinal evidence that trait self-discipline predicts academic outcomes more than twice as strongly as IQ, evidencing the behavioral reach of self-control. ↩
[7] Thaler, R. H., & Shefrin, H. M. (1981). An economic theory of self-control. Journal of Political Economy, 89(2), 392–406. Planner-doer model of intertemporal choice: explains why pension systems, withdrawal-penalty accounts, and automatic deductions are effective institutional remedies for present-biased savings reversals. ↩
[8] Heatherton, T. F., & Wagner, D. D. (2011). Cognitive neuroscience of self-regulation failure. Trends in Cognitive Sciences, 15(3), 132–139. Reviews prefrontal top-down control over subcortical reward and emotion regions as the neural realization of the override and the locus of its failure. ↩
[9] Baumeister, R. F., & Heatherton, T. F. (1996). Self-regulation failure: An overview. Psychological Inquiry, 7(1), 1–15. Localizes self-regulatory failure to deficient standards, inadequate monitoring, or inadequate strength — the three components into which the prime decomposes control failures. ↩
[10] Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press. Standard reference on the temporal credit-assignment problem: discounting and eligibility traces back-project credit for a delayed reward across the actions that produced it (850), the same backward propagation that, applied to incident review, resists stopping at the proximate actor (855). ↩
[11] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete problems in AI safety. arXiv preprint arXiv:1606.06565. Defines reward hacking and specification gaming, motivating the design lesson to remove the proxy-maximizing temptation in the reward rather than train resistance into the policy. ↩