Reinforcement Learning¶
Sutton, R. S., & Barto, A. G. (1998/2018). Reinforcement Learning: An Introduction. MIT Press.
Cited by¶
27 citations across 27 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Adaptation
- Systems that adapt to noise thrash; systems that ignore signals lag. Selection or learning pressure. Some mechanism preferentially retains variants that perform better under the new conditions — natural selection, reinforcement, deliberate choice, algorithmic update.
This sourceStandard reference on reinforcement learning; treats the reward signal as the explicit, transparent argument to the update rule that drives a learning system's adaptive modification.
Supported in partVerified against the work's full text
Sutton and Barto describe reward-driven retention of better-performing policies via learning and evolutionary methods, supporting the selection/learning-pressure mechanism but not the noise/lag framing.
“The policies that obtain the most reward, and random variations of them, are carried over to the next generation of policies, and the process repeats.”
- Systems that adapt to noise thrash; systems that ignore signals lag. Selection or learning pressure. Some mechanism preferentially retains variants that perform better under the new conditions — natural selection, reinforcement, deliberate choice, algorithmic update.
- Agency
- The goal-representation is the reward function \(R(s,a)\) — the agent is oriented toward states that yield reward.
This sourceStandard formalization of the reward function R(s,a), transition model, policy π, expected discounted return, value iteration, and discount factor γ — the RL instantiation of the goal/world-model/action-selection triad.
SupportedVerified against the work's full text
Sutton and Barto formalize the agent's goal as the reward signal and define r(s, a) as the expected immediate reward for a state-action pair.
“In reinforcement learning, the purpose or goal of the agent is formalized in terms of a special signal, called the reward, passing from the environment to the agent.”
- The goal-representation is the reward function \(R(s,a)\) — the agent is oriented toward states that yield reward.
- Binding Problem
- Reinforcement learning: a delayed reward and the action that caused it are features in separate time-slots, and eligibility traces are the binding mechanism for action–reward pairs in temporal credit assignment.
This sourceDevelops eligibility traces as the mechanism that binds delayed rewards to the actions that caused them in temporal credit assignment, with an explicit temporal decay window.
- Reinforcement learning: a delayed reward and the action that caused it are features in separate time-slots, and eligibility traces are the binding mechanism for action–reward pairs in temporal credit assignment.
- Cromwell's Rule
- Reinforcement learning — epsilon-greedy and Thompson sampling keep nonzero exploration probability on apparently dominated options, since pure-greedy zero-exploration is Cromwell-violating.
This sourceTreats the exploration–exploitation trade-off; epsilon-greedy and other methods keep nonzero exploration probability on apparently dominated actions, since a pure-greedy zero-exploration policy can never discover a better option.
Supported in partVerified against the work's full text
Sutton and Barto show pure greedy never samples apparently inferior actions while epsilon-greedy keeps a small exploration probability; Thompson sampling is only named briefly.
“Greedy action selection always exploits current knowledge to maximize immediate reward; it spends no time at all sampling apparently inferior actions to see if they might really be better.”
- Reinforcement learning — epsilon-greedy and Thompson sampling keep nonzero exploration probability on apparently dominated options, since pure-greedy zero-exploration is Cromwell-violating.
- Decision
- Artificial intelligence & operations research: Markov decision processes (MDPs), reinforcement learning (value iteration, policy gradient), bandit algorithms (exploration-exploitation trade-off), planning under partial observability (POMDPs), multi-agent decision-making, game theory (Nash equilibrium, strategic interaction), combinatorial optimization — a formal decision-under-uncertainty framework Sutton and Barto (2018) systematize in the canonical reinforcement-learning textbook.
This sourceSystematizes sequential decision-making under uncertainty via Markov decision processes, value iteration, policy gradients, and bandit exploration–exploitation.
Supported in partVerified against the work's full text
Sutton and Barto systematize MDPs, value iteration and bandit exploration as sequential decision-making under uncertainty; POMDPs, game theory, multi-agent and combinatorial optimization are not covered.
“Reinforcement learning uses the formal framework of Markov decision processes to”
- Artificial intelligence & operations research: Markov decision processes (MDPs), reinforcement learning (value iteration, policy gradient), bandit algorithms (exploration-exploitation trade-off), planning under partial observability (POMDPs), multi-agent decision-making, game theory (Nash equilibrium, strategic interaction), combinatorial optimization — a formal decision-under-uncertainty framework Sutton and Barto (2018) systematize in the canonical reinforcement-learning textbook.
- Dynamic Programming
- … — naive recursion would re-solve the same subproblems exponentially many times; (c) a well-defined state representation that captures all information needed to characterize a subproblem; and (d) a recurrence relation that expresses the optimal value at one state in terms of optimal values at related states.
This sourceCanonical RL textbook: derives value iteration, policy iteration, and temporal-difference methods as approximate DP inheriting Bellman's principle of optimality.
Supported in partVerified against the work's full text
Sutton and Barto support the recursive Bellman-equation and state framing of DP, but never name optimal substructure or overlapping subproblems.
“A fundamental property of value functions used throughout reinforcement learning and dynamic programming is that they satisfy recursive relationships”
- … — naive recursion would re-solve the same subproblems exponentially many times; (c) a well-defined state representation that captures all information needed to characterize a subproblem; and (d) a recurrence relation that expresses the optimal value at one state in terms of optimal values at related states.
- Learning
- Learning encodes a structural pattern: experiential input → agent with modifiable internal state → update mechanism → durable capability change — a four-role decomposition that Sutton and Barto (2018) formalize in the reinforcement-learning setting (state, signal, update rule, retained policy) and that Bishop (2006) develops in the supervised-learning setting (parameters, training data, loss-driven update, learned model).
This sourceStandard reference on the temporal credit-assignment problem: discounting and eligibility traces back-project credit for a delayed reward across the actions that produced it (850), the same backward propagation that, applied to incident review, resists stopping at the proximate actor (855).
Supported in partVerified against the work's full text
Sutton and Barto describe a policy altered by reward signals from experience, which fits the RL half of the claim; the four-role learning decomposition itself and the Bishop half are not in this book.
“The reward signal is the primary basis for altering the policy; if an action selected by the policy is followed by low reward, then the policy may be changed to select some other action in that situation in the future.”
- Learning encodes a structural pattern: experiential input → agent with modifiable internal state → update mechanism → durable capability change — a four-role decomposition that Sutton and Barto (2018) formalize in the reinforcement-learning setting (state, signal, update rule, retained policy) and that Bishop (2006) develops in the supervised-learning setting (parameters, training data, loss-driven update, learned model).
- Markov Decision Processes (MDPs)
- This example illustrates MDP deployment in a mid-size retail context (the kind of classical-OR-meets-modern-RL synthesis that Sutton and Barto (2018) frame as the modern reach of the MDP framework): substantial problem scale (10K SKU-stores) handled through per-SKU-store decomposition; MDP framework providing structure for a classical sequential-decision problem (inventory); meaningful operational and financial outcomes; and practical challenges in state representation, forecasting integration, and change management.
This sourceStandard reference on the temporal credit-assignment problem: discounting and eligibility traces back-project credit for a delayed reward across the actions that produced it (850), the same backward propagation that, applied to incident review, resists stopping at the proximate actor (855).
Supported in partVerified against the work's full text
Sutton and Barto support the framing that RL builds on MDPs, dynamic programming and operations research, but say nothing about retail inventory or the example's figures.
“Bellman (1957b) also introduced the discrete stochastic version of the optimal control problem known as Markov decision processes (MDPs).”
- This example illustrates MDP deployment in a mid-size retail context (the kind of classical-OR-meets-modern-RL synthesis that Sutton and Barto (2018) frame as the modern reach of the MDP framework): substantial problem scale (10K SKU-stores) handled through per-SKU-store decomposition; MDP framework providing structure for a classical sequential-decision problem (inventory); meaningful operational and financial outcomes; and practical challenges in state representation, forecasting integration, and change management.
- Optimization
- Machine learning is optimization-at-scale: empirical risk minimization for supervised learning, hyperparameter tuning over compositional spaces, reinforcement-learning policy optimization
This sourceStandard reference on the temporal credit-assignment problem: discounting and eligibility traces back-project credit for a delayed reward across the actions that produced it (850), the same backward propagation that, applied to incident review, resists stopping at the proximate actor (855).
- Machine learning is optimization-at-scale: empirical risk minimization for supervised learning, hyperparameter tuning over compositional spaces, reinforcement-learning policy optimization
- Progressive Refinement from Core Model
- The pattern makes explicit what would otherwise be implicit in much modeling work: the choice of baseline is a methodological commitment, and the smallness of corrections is an auditable assumption.
This sourceFormal algorithms for policy improvement and value iteration as iterative refinement of behavioral baselines.
- The pattern makes explicit what would otherwise be implicit in much modeling work: the choice of baseline is a methodological commitment, and the smallness of corrections is an auditable assumption.
- Reinforcement
- In machine learning, reinforcement learning names an entire field, where agents learn policies by maximizing cumulative reward through value updates and policy gradients.
This sourceStandard text defining RL, value updates, temporal-difference learning ($\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$), and the exploration–exploitation tradeoff.
SupportedVerified against the work's full text
Sutton and Barto present reinforcement learning as a field whose agents learn policies to maximize cumulative reward, using value functions and policy-gradient methods.
“is simultaneously a problem, a class of solution methods that work well on the problem, and the field that studies this problem and its solution methods.”
- In machine learning, reinforcement learning names an entire field, where agents learn policies by maximizing cumulative reward through value updates and policy gradients.
- Responsibility Attribution
- Eligibility traces, advantage estimation, and discounting are mechanisms for back-projecting credit through a chain of actions toward a delayed outcome.
This sourceStandard reference on the temporal credit-assignment problem: discounting and eligibility traces back-project credit for a delayed reward across the actions that produced it (850), the same backward propagation that, applied to incident review, resists stopping at the proximate actor (855).
Supported in partVerified against the work's full text
Names eligibility traces as a backward-view mechanism and defines discounting of delayed rewards; 'advantage estimation' and the Chapter 12 detail are not in the cached text.
“view idea of eligibility traces (now treated independently in Chapter 12)”
- Eligibility traces, advantage estimation, and discounting are mechanisms for back-projecting credit through a chain of actions toward a delayed outcome.
- Reward Prediction Error
- Midbrain dopamine responses and temporal-difference learning supply canonical biological and computational realizations of this role.
Supported in partVerified against the work's full text
Sutton and Barto note an uncanny similarity between TD algorithm behavior and dopamine neuron activity, but the chapter on the reward prediction error hypothesis is not in the cached text.
“an uncanny similarity between the behavior of temporal-di... algorithms and the activity of dopamine producing neurons in the brain”
- Midbrain dopamine responses and temporal-difference learning supply canonical biological and computational realizations of this role.
- Self Control
- The pattern supports reasoning about any system that must subordinate a local, immediate gradient to a global objective: a greedy algorithm that must resist the locally optimal step, a control system that must not chase a transient setpoint deviation, a trading desk bound by risk limits against the temptation of a hot position.
This sourceStandard reference on the temporal credit-assignment problem: discounting and eligibility traces back-project credit for a delayed reward across the actions that produced it (850), the same backward propagation that, applied to incident review, resists stopping at the proximate actor (855).
Supported in partVerified against the work's full text
Backs the greedy/myopic example: maximizing immediate reward can lower long-run return, so the greedy step must sometimes be resisted. Control-system and trading-desk examples are not covered.
“But in general, acting to maximize immediate reward can reduce access to future rewards so that the return is reduced.”
- The pattern supports reasoning about any system that must subordinate a local, immediate gradient to a global objective: a greedy algorithm that must resist the locally optimal step, a control system that must not chase a transient setpoint deviation, a trading desk bound by risk limits against the temptation of a hot position.
- Variance Bounds Selection Response
- In reinforcement learning and multi-armed bandits, the improvement rate of mean reward scales with the variance of the reward-estimate distribution, and exploration strategies (epsilon-greedy, UCB, Thompson sampling) are variance-regeneration mechanisms.
This sourceExploration strategies (epsilon-greedy, UCB, Thompson sampling) as variance-regeneration mechanisms; improvement tied to estimate variance.
Supported in partVerified against the work's full text
Sutton and Barto present epsilon-greedy, UCB and Thompson sampling as exploration methods and tie UCB to estimate uncertainty, but not as variance-regeneration or a variance-scaled improvement rate.
“the square-root term is a measure of the uncertainty or variance in the estimate of”
- In reinforcement learning and multi-armed bandits, the improvement rate of mean reward scales with the variance of the reward-estimate distribution, and exploration strategies (epsilon-greedy, UCB, Thompson sampling) are variance-regeneration mechanisms.
- Variation Strategies
- "Are we balancing exploration and exploitation appropriately?"—the kind of explicit parameterization Sutton and Barto (2018) recommend by framing exploration policies (ε-greedy, softmax, UCB) as tunable hyperparameters rather than vague intuitions.
This sourceStandard reference on the temporal credit-assignment problem: discounting and eligibility traces back-project credit for a delayed reward across the actions that produced it (850), the same backward propagation that, applied to incident review, resists stopping at the proximate actor (855).
Supported in partVerified against the work's full text
Sutton and Barto present epsilon-greedy, soft-max and UCB each with an explicit parameter and a parameter study, but never use or recommend a 'hyperparameter' framing.
“Note also the characteristic inverted-U shapes of each algorithm’s performance; all the algorithms perform best at an intermediate value of their parameter, neither too large nor too small.”
- "Are we balancing exploration and exploitation appropriately?"—the kind of explicit parameterization Sutton and Barto (2018) recommend by framing exploration policies (ε-greedy, softmax, UCB) as tunable hyperparameters rather than vague intuitions.
Domain-specific¶
Mechanisms¶
- Credit Assignment Trace
- Its strength is that it prevents the wrong cue from being reinforced — the root of shortcut learning, superstition, and spurious correlation — and it is indispensable wherever reward is delayed, where the naïve "blame the last thing" heuristic fails outright.
This sourceUses eligibility traces to assign credit across previously visited states or actions rather than only the most recent event.
- Its strength is that it prevents the wrong cue from being reinforced — the root of shortcut learning, superstition, and spurious correlation — and it is indispensable wherever reward is delayed, where the naïve "blame the last thing" heuristic fails outright.
- Exploration Capture Protocol
- The tension it manages is the explore-exploit tradeoff studied in the multi-armed bandit.
This sourcePresents the multi-armed bandit as the canonical setting for the exploration-exploitation tradeoff.
- The tension it manages is the explore-exploit tradeoff studied in the multi-armed bandit.
- Exploration Policy
- Over months the system stays calibrated to real conditions rather than freezing around an assumption that was true a year ago.
This sourceExplains how continued exploration and recency-weighted updating let a bandit policy track changing conditions instead of remaining tied to stale estimates.
- Over months the system stays calibrated to real conditions rather than freezing around an assumption that was true a year ago.
- Explore–Exploit Review Loop
- … core trap — resources coasting into a once-great approach on the strength of its history — and it embodies the exploration–exploitation trade-off familiar from bandit and reinforcement-learning settings, where every period spends some effort confirming the best option and some searching for a better one.
This sourceFrames repeated choice as an exploration–exploitation trade-off between using the currently best option and testing alternatives.
- … core trap — resources coasting into a once-great approach on the strength of its history — and it embodies the exploration–exploitation trade-off familiar from bandit and reinforcement-learning settings, where every period spends some effort confirming the best option and some searching for a better one.
- Parallel Pilot Trials
- Reading marginal response — not just totals — is what makes it a genuine explore-and-exploit instrument: it says where another unit of effort pays off most right now.
This sourceModels exploration and exploitation by estimating each option’s expected reward and allocating action accordingly.
- Reading marginal response — not just totals — is what makes it a genuine explore-and-exploit instrument: it says where another unit of effort pays off most right now.
- Reward Baseline Dashboard
- This mirrors the reinforcement-learning practice of learning on reward minus a baseline: an outcome equal to its baseline carries no signal.
This sourceUses reward minus a learned baseline as the update signal, so an outcome equal to the baseline contributes zero advantage.
- This mirrors the reinforcement-learning practice of learning on reward minus a baseline: an outcome equal to its baseline carries no signal.
- Search Space Diagram
- It makes the exploration–exploitation trade-off — chase the promising lead, or sweep the unknown — an explicit, tunable choice rather than an unexamined reflex.
This sourceFrames action selection as a tunable trade-off between exploiting high estimated value and exploring uncertain alternatives.
- It makes the exploration–exploitation trade-off — chase the promising lead, or sweep the unknown — an explicit, tunable choice rather than an unexamined reflex.
- Temporal-Difference Update
- The signed residual between successive expectations is the entire teacher.
This sourceDefines the temporal-difference error as the signed residual that drives the update from one expectation toward its successor.
- The signed residual between successive expectations is the entire teacher.
- Temporal-Difference Update Rule
- Its strength is learning from incomplete experience: it updates online, before outcomes resolve, and threads a late reward back to the earlier decisions that produced it — exactly the temporal credit problem raw outcome-magnitude learning botches.
This sourceExplains temporal-difference learning as updating estimates online from incomplete trajectories and using later rewards and bootstrapped estimates to assign credit to earlier states and actions.
- Its strength is learning from incomplete experience: it updates online, before outcomes resolve, and threads a late reward back to the earlier decisions that produced it — exactly the temporal credit problem raw outcome-magnitude learning botches.
Verification¶
Does it exist? Not checked yet. This entry carries no identifier to resolve. It was extracted from the citation as written in the article, normalized, and deduplicated against the rest of the registry.
Does it back the claim? Read against the text for 13 of 27 citations: 2 supported, 11 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Links previously used in the corpus¶
Before the registry existed this work was also linked 2 other ways.
- http://incompleteideas.net/book/the-book-2nd.html ×7
- http://incompleteideas.net/book/RLbook2020.pdf ×1
Registry ID ref:69955d045f54 · see in the full table