Reinforcement Learning

Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction. MIT Press.

Type

Book

Intellectual base

Review or monograph

Year

2018

Link

https://mitpress.mit.edu/9780262039246/reinforcement-learning/

Cited from

biology_ecology, cognitive_science, mathematics, neuroscience, operations_research, philosophy, psychology, statistics_experimental_design

Cited by

20 citations across 20 artifacts.

Each citation links to the sentence it supports in the citing article.

Primes

• Adaptation
  • Systems that adapt to noise thrash; systems that ignore signals lag. Selection or learning pressure. Some mechanism preferentially retains variants that perform better under the new conditions — natural selection, reinforcement, deliberate choice, algorithmic update.

    This sourceStandard reference on reinforcement learning; treats the reward signal as the explicit, transparent argument to the update rule that drives a learning system's adaptive modification.

• Agency
  • The goal-representation is the reward function \(R(s,a)\) — the agent is oriented toward states that yield reward.

    This sourceStandard formalization of the reward function R(s,a), transition model, policy π, expected discounted return, value iteration, and discount factor γ — the RL instantiation of the goal/world-model/action-selection triad.

• Binding Problem
  • Reinforcement learning: a delayed reward and the action that caused it are features in separate time-slots, and eligibility traces are the binding mechanism for action–reward pairs in temporal credit assignment.

    This sourceDevelops eligibility traces as the mechanism that binds delayed rewards to the actions that caused them in temporal credit assignment, with an explicit temporal decay window.

• Cromwell's Rule
  • Reinforcement learning — epsilon-greedy and Thompson sampling keep nonzero exploration probability on apparently dominated options, since pure-greedy zero-exploration is Cromwell-violating.

    This sourceTreats the exploration–exploitation trade-off; epsilon-greedy and other methods keep nonzero exploration probability on apparently dominated actions, since a pure-greedy zero-exploration policy can never discover a better option.

• Decision
  • Artificial intelligence & operations research: Markov decision processes (MDPs), reinforcement learning (value iteration, policy gradient), bandit algorithms (exploration-exploitation trade-off), planning under partial observability (POMDPs), multi-agent decision-making, game theory (Nash equilibrium, strategic interaction), combinatorial optimization — a formal decision-under-uncertainty framework Sutton and Barto (2018) systematize in the canonical reinforcement-learning textbook.

    This sourceSystematizes sequential decision-making under uncertainty via Markov decision processes, value iteration, policy gradients, and bandit exploration–exploitation.

• Dynamic Programming
  • … — naive recursion would re-solve the same subproblems exponentially many times; (c) a well-defined state representation that captures all information needed to characterize a subproblem; and (d) a recurrence relation that expresses the optimal value at one state in terms of optimal values at related states.

    This sourceCanonical RL textbook: derives value iteration, policy iteration, and temporal-difference methods as approximate DP inheriting Bellman's principle of optimality

• Learning
  • Learning encodes a structural pattern: experiential input → agent with modifiable internal state → update mechanism → durable capability change — a four-role decomposition that Sutton and Barto (2018) formalize in the reinforcement-learning setting (state, signal, update rule, retained policy) and that Bishop (2006) develops in the supervised-learning setting (parameters, training data, loss-driven update, learned model).

    This sourceStandard reference on the temporal credit-assignment problem: discounting and eligibility traces back-project credit for a delayed reward across the actions that produced it (850), the same backward propagation that, applied to incident review, resists stopping at the proximate actor (855).

• Markov Decision Processes (MDPs)

  • This sourceStandard reference on the temporal credit-assignment problem: discounting and eligibility traces back-project credit for a delayed reward across the actions that produced it (850), the same backward propagation that, applied to incident review, resists stopping at the proximate actor (855).

• Optimization
  • … market design, mechanism design (already FACT-resolved via mechanism_design #501 in DP-01), and dynamic pricing. Machine learning is optimization-at-scale: empirical risk minimization for supervised learning, hyperparameter tuning over compositional spaces, reinforcement-learning policy optimization

    This sourceStandard reference on the temporal credit-assignment problem: discounting and eligibility traces back-project credit for a delayed reward across the actions that produced it (850), the same backward propagation that, applied to incident review, resists stopping at the proximate actor (855).

• Progressive Refinement from Core Model

  • This sourceFormal algorithms for policy improvement and value iteration as iterative refinement of behavioral baselines.

• Reinforcement
  • In machine learning, reinforcement learning names an entire field, where agents learn policies by maximizing cumulative reward through value updates and policy gradients.

    This sourceStandard text defining RL, value updates, temporal-difference learning ($\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$), and the exploration–exploitation tradeoff.

• Responsibility Attribution
  • Eligibility traces, advantage estimation, and discounting are mechanisms for back-projecting credit through a chain of actions toward a delayed outcome.

    This sourceStandard reference on the temporal credit-assignment problem: discounting and eligibility traces back-project credit for a delayed reward across the actions that produced it (850), the same backward propagation that, applied to incident review, resists stopping at the proximate actor (855).

• Reward Prediction Error
  • Midbrain dopamine responses and temporal-difference learning supply canonical biological and computational realizations of this role.
• Self Control
  • The pattern supports reasoning about any system that must subordinate a local, immediate gradient to a global objective: a greedy algorithm that must resist the locally optimal step, a control system that must not chase a transient setpoint deviation, a trading desk bound by risk limits against the temptation of a hot position.

    This sourceStandard reference on the temporal credit-assignment problem: discounting and eligibility traces back-project credit for a delayed reward across the actions that produced it (850), the same backward propagation that, applied to incident review, resists stopping at the proximate actor (855).

• Variance Bounds Selection Response
  • In reinforcement learning and multi-armed bandits, the improvement rate of mean reward scales with the variance of the reward-estimate distribution, and exploration strategies (epsilon-greedy, UCB, Thompson sampling) are variance-regeneration mechanisms.

    This sourceExploration strategies (epsilon-greedy, UCB, Thompson sampling) as variance-regeneration mechanisms; improvement tied to estimate variance.

• Variation Strategies
  • "Are we balancing exploration and exploitation appropriately?"—the kind of explicit parameterization Sutton and Barto (2018) recommend by framing exploration policies (ε-greedy, softmax, UCB) as tunable hyperparameters rather than vague intuitions.

    This sourceStandard reference on the temporal credit-assignment problem: discounting and eligibility traces back-project credit for a delayed reward across the actions that produced it (850), the same backward propagation that, applied to incident review, resists stopping at the proximate actor (855).

Domain-specific

• Model-Free Reinforcement Learning
  • Sutton and Barto organize temporal-difference control and policy-gradient methods around this direct experience-to-value-or-policy route.
• Reinforcement learning
  • Its objective is not to imitate labeled examples but to improve a policy so as to maximize expected cumulative reward, commonly a discounted return \(G_t=\sum_{k\ge0}\gamma^kR_{t+k+1}\) or an average-reward criterion.

Mechanisms

• Explore–Exploit Review Loop
  • … core trap — resources coasting into a once-great approach on the strength of its history — and it embodies the exploration–exploitation trade-off familiar from bandit and reinforcement-learning settings, where every period spends some effort confirming the best option and some searching for a better one.

    This sourceFrames repeated choice as an exploration–exploitation trade-off between using the currently best option and testing alternatives.

• Parallel Pilot Trials
  • Reading marginal response — not just totals — is what makes it a genuine explore-and-exploit instrument: it says where another unit of effort pays off most right now.

    This sourceModels exploration and exploitation by estimating each option’s expected reward and allocating action accordingly.

Verification

This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.

Links previously used in the corpus

Before the registry existed this work was also linked 1 other way.

• http://incompleteideas.net/book/the-book-2nd.html ×6

Registry ID ref:69955d045f54 · see in the full table