A reward prediction error is the signed difference between received reward and the reward or value that a learner expected. A positive error means the outcome was better than predicted, a negative error means it was worse, and a zero error means the outcome carried no new reward information. The learner uses that signed error to update value estimates, action policies, or reward-linked associations.
The reward/value restriction is constitutive. Forecast residuals, sensor innovations, reconstruction errors, and generic observed-minus-predicted differences belong to prediction_error; they are not reward prediction errors merely because they can drive an update.
How would you explain it like I'm…
The Surprise Teacher
Imagine you expect one cookie and you get one cookie — no surprise, nothing to learn. But if you expected one and got three, that happy surprise makes you remember whatever led to it. And if you expected one and got none, that letdown makes you trust it less next time. Surprise is the teacher; getting exactly what you expected teaches nothing.
Better Or Worse Than Expected
A reward prediction error is the gap between what you expected and what you actually got — and that gap, not the reward itself, is what teaches you. If the outcome matches your expectation, there's no error and you learn nothing. If it's better than expected, that's a positive error and it strengthens whatever predicted it. If it's worse, that's a negative error and it weakens those predictors. So the system keeps a guess, gets a result, and pays attention to (result minus guess). A neat side effect: as your guesses get better, the surprises shrink and learning naturally slows down — small errors mean you've about maxed out, not that you failed.
Surprise Is The Signal
A reward prediction error is the pattern where a system learns not from raw outcomes but from the gap between expected and received outcomes, using the sign and size of that gap — rather than the outcome itself — to update its model. Outcomes that match expectation make no error and produce no learning; outcomes that beat expectation make a positive error and reinforce whatever predicted them; outcomes that fall short make a negative error and weaken those predictors. The commitment is that the system carries a prediction, receives a signal, and computes a scalar error (signal minus prediction) that serves as the teaching signal for whatever updates the predictor. It's the dual of outcome-only learning: a dog that just salivates when food arrives is responding to the food, but a prediction-error learner that already expected the food learns nothing from it — only unexpected food (positive error) or unexpectedly absent food (negative error) teaches. As the predictor improves, errors shrink and learning slows on its own, so the absence of error signals a ceiling, not a failure.
A reward prediction error is the structural pattern in which a system learns not from raw outcomes but from the gap between expected and received outcomes, and uses the sign and size of that gap, rather than the outcome itself, to update its model. Outcomes matching expectation generate no error and produce no learning; outcomes exceeding expectation produce a positive error and reinforce whatever predicted them; outcomes falling short produce a negative error and weaken those predictors. The essential commitment is that the system carries a prediction (an expectation, forecast, or value estimate), receives a signal (an outcome, reward, or measurement), and computes a scalar error (signal minus prediction) that serves as the teaching signal for whatever process updates the predictor. Every instance specifies four parameters: the predictor (the model issuing expectations), the prediction (its output on a particular trial), the observed outcome, and the learning rate (how strongly the error updates the predictor). The error is the load-bearing currency of learning — a system without prediction errors keeps no record of surprise and does not improve — and the pattern lets a reasoner ask crisp questions raw-outcome accounts cannot: whose prediction error, against what predictor, with what learning rate. It is the dual of outcome-only learning: a Pavlovian organism that salivates when food arrives is responding to the stimulus, not its mismatch with expectation, whereas a prediction-error learner that already expected the food learns nothing from its arrival, while unexpected food (positive error) or unexpectedly absent food (negative error) teaches. A structural consequence is the baseline-shift phenomenon — as the predictor improves, the errors shrink and learning slows of its own accord, so the absence of further error signals that the system has reached its current ceiling, not that effort has failed.
The abstraction separates reward magnitude from reward surprise. A large fully expected reward can generate no update, while a small unexpected reward can generate a strong positive teaching signal. It also distinguishes an outcome from the learner's model-relative error about that outcome.
Value learning compresses into predicted value, received reward, signed error, learning rate, and the policy or value estimate being updated. The sign diagnoses whether the system should strengthen or weaken the expectation or action.
Reason from the learner's expectation, not the observer's hindsight. The same outcome can be positive error for one learner, zero for another, and negative error for a third because their predictions differ.
The temporal-difference equation transfers directly between computational neuroscience and reinforcement learning. The generic observed-minus-predicted operation transfers through the parent prediction_error; reward-specific claims should not be exported to every residual-producing system.
Reward prediction error is prediction error specialized to reward or value predictions and value-policy updating.
Children (2) — more specific cases that build on this
Crespi EffectDomain-specificis part ofReward Prediction Error
Reward Prediction Error is a constituent of the Crespi Effect because the signed gap between received and expected reward drives both the transient response and recalibration of the expectation.
ReinforcementPrimeis part of, typicalReward Prediction Error
Reward prediction error is a common internal teaching signal in reinforcement, updating value or action strength from expected-versus-received reward.
Condition / exception Reinforcement may be specified through direct consequence contingencies or schedules without an explicit reward predictor and computed temporal-difference error.