Markov Decision Processes¶
Puterman, M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley.
Cited by¶
4 citations across 4 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Markov Decision Processes (MDPs)
- MDP applications span essentially every domain involving sequential decisions under uncertainty, as Puterman (1994) catalogues in the canonical OR reference.
This sourceCanonical operations-research reference on MDPs; comprehensive treatment of finite/infinite-horizon, discounted/average-reward, and structured-policy results, with applications across inventory, maintenance, and queueing.
- MDP applications span essentially every domain involving sequential decisions under uncertainty, as Puterman (1994) catalogues in the canonical OR reference.
- Optimal Stopping Rule
- Not the full sequential-control apparatus of `markov_decision_processes_mdps`. An MDP optimizes a policy over actions and rewards across many states; optimal stopping is the special case where the only action is "stop or continue."
This sourceCanonical treatment of MDPs as policy optimization over actions and rewards across states; optimal stopping is the degenerate case with a stop/continue action set.
- Not the full sequential-control apparatus of `markov_decision_processes_mdps`. An MDP optimizes a policy over actions and rewards across many states; optimal stopping is the special case where the only action is "stop or continue."
Domain-specific¶
- Bellman Equation
- Policy iteration is the move chosen when evaluation and improvement separate cleanly: alternate policy evaluation with greedy improvement, predicting monotone improvement to the optimum in finitely many steps for finite MDPs
This sourceTextbook treatment of value iteration and policy iteration for Markov decision processes, developing contraction mappings on the value space and analysing policy iteration as alternating evaluation and improvement steps.
Supported in partVerified against the work's full text
“case consists of an [improvement] and an [evaluation] step. In the [improvement] step, a decision”
- Policy iteration is the move chosen when evaluation and improvement separate cleanly: alternate policy evaluation with greedy improvement, predicting monotone improvement to the optimum in finitely many steps for finite MDPs
Mechanisms¶
- Dynamic Programming / Value Iteration
- The discount factor is not a detail here: it is what makes each backup a contraction and therefore guaranteed to converge to a unique optimum.
This sourceFor discounted Markov decision processes, establishes that the Bellman backup is a contraction with modulus equal to the discount factor, so iteration converges to the unique optimal value function.
- The discount factor is not a detail here: it is what makes each backup a contraction and therefore guaranteed to converge to a unique optimum.
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 1 of 4 citations: 1 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:430d6b73b15d · see in the full table