Temporal-Difference Update Rule¶
Online update rule — instantiates Prediction-Error Learning Calibration
Updates an estimate from the gap between successive predictions — bootstrapping off the next step rather than waiting for the final outcome — and propagates that error back across the delay.
Temporal-Difference Update Rule is the learning step for situations where the final outcome arrives late but partial evidence arrives continuously. Its defining move is bootstrapping: instead of waiting to see how things turn out, it treats its own next prediction as a stand-in target and learns from the change between one estimate and the next — the temporal-difference error. When a later estimate revises an earlier one, that revision is itself the signal, and the rule pushes the earlier estimate toward it and lets the correction flow back to the steps that led there. This is what lets a system learn before the reward lands, and assign a surprise to the moment that actually earned it rather than only to the end.
Example¶
A navigation app estimates a trip at 40 minutes. Ten minutes in, at the first junction, its model — now seeing real traffic — implies 38 minutes still to go, when the original plan had budgeted 30 for the remainder. That 8-minute jump between successive estimates is a temporal-difference error, and the rule acts on it immediately: it raises the value estimate for that whole route without waiting for the car to arrive, and propagates the correction back to the earlier legs that fed into the too-rosy original. By the time the driver reaches the destination, the route's estimates have already been re-learned mid-trip. A rule that waited for the final arrival time would have learned the same lesson hours of trips later.
How it works¶
The distinguishing method is the bootstrapped target and its reach back through time. Each step forms a target from the immediate signal plus the discounted next estimate; the difference between that target and the current estimate is the error to learn from; and an eligibility mechanism spreads that error back over recent steps so credit lands across the delay, not just on the last one. The horizon dial governs how many steps ahead the target reaches — from one-step bootstrapping to nearly-full-outcome — trading bias against variance. The rule takes the size of each correction as an input; its own contribution is the target, the horizon, and the temporal routing of the error.
Tuning parameters¶
- Bootstrap horizon — how many steps ahead the target looks (one-step, n-step, or λ-weighted); short horizons are low-variance but lean hard on possibly-wrong estimates, long horizons are less biased but noisier.
- Discount factor — how much a delayed payoff counts toward the current estimate; lower is myopic, higher values distant consequences and lengthens the credit chain.
- Trace decay — how far back an error is allowed to spread; wider traces reach the true cause faster but risk crediting irrelevant intervening steps.
- Step-size (external) — the magnitude of each correction, supplied by the learning-rate mechanism rather than set here; the rule applies it but does not schedule it.
- On- vs off-policy target — whether the bootstrap target follows the behavior actually taken or the best available, which changes what the estimate converges to.
When it helps, and when it misleads¶
Its strength is learning from incomplete experience: it updates online, before outcomes resolve, and threads a late reward back to the earlier decisions that produced it — exactly the temporal credit problem raw outcome-magnitude learning botches.[n1] That makes it powerful anywhere feedback is delayed and episodes are long.
Its failure mode is intrinsic to bootstrapping: because the target is partly the system's own estimate, errors in that estimate feed back into the update, and under aggressive settings the estimates can bias or diverge rather than converge. Too long a trace smears a surprise across steps that had nothing to do with it. The classic misuse is cranking the step-size to "learn faster," which turns transient noise into permanent swings — precisely why the gain is factored out to a separate, deliberately-scheduled dial. The discipline is to match horizon and step-size to the noise level and to check the bootstrapped estimates against realized outcomes periodically rather than trusting them indefinitely.
How it implements the components¶
temporal_difference_horizon— it sets how far ahead the bootstrap target reaches (one-step through λ-weighted), the dial that trades bias against variance.update_target— it constructs the target the estimate is moved toward: the immediate signal plus the discounted next estimate.feedback_timing_alignment— its eligibility traces route the error back across the delay to the steps that earned it, aligning a late signal with the moment of the action.
It does not set the size of each correction — that step-size schedule is Learning Rate Schedule — nor does it decide which specific cue among several deserves the credit; Credit Assignment Trace owns that attribution.
Related¶
- Instantiates: Prediction-Error Learning Calibration — it is the update engine that turns a signed, in-flight error into a revised estimate.
- Consumes: Learning Rate Schedule supplies the step-size; Prediction–Outcome Delta Log supplies the realized errors that anchor the bootstrap.
- Sibling mechanisms: Learning Rate Schedule · Credit Assignment Trace · Reward Baseline Dashboard · Surprise Threshold Alert · Prediction–Outcome Delta Log
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Temporal-Difference Update Rule operates as an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution because it updates an estimate from the gap between successive predictions — bootstrapping off the next step rather than waiting for the final outcome — and propagates that error back across the delay.
Independent corroboration: The frozen evidence defines Temporal-Difference Update Rule as 'Updates an estimate from the gap between successive predictions — bootstrapping off the next step rather than waiting for the final outcome — and propagates that error back across the delay', so its operative form is Analysis, Modeling & Optimization.
Nearest alternative: Rule, Policy & Commitment — Temporal-Difference Update Rule includes features of a standing rule, threshold, contractual commitment, or policy constraint governing future conduct, but its defining operation is an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Single lineage
Present-day reach: Specialized
Rationale: The defining operation is: Updates an estimate from the gap between successive predictions — bootstrapping off the next step rather than waiting for the final outcome — and propagates that error back across the delay. In the computer_science lineage, that operation is specifically evidenced by authoritative or primary work that introduces bootstrapped prediction updates driven by the difference between successive predictions before a final outcome. This makes computer_science the best historical origin, while the retained alternates document contributing methods and later applications rather than being mistaken for coequal origins.
Related originating lineages:
- Engineering & Design — Engineering design, reliability, and systems-safety practice supplies a parallel or contributing lineage for the mechanism's defining operation: updates an estimate from the gap between successive predictions — bootstrapping off the next step rather than waiting for the final outcome — and propagates that error back across….
- Organizational & Management Science — organizational_management supplies a historically relevant parallel or contributing practice for the defining operation—Updates an estimate from the gap between successive predictions — bootstrapping off the next step rather than waiting for the final outcome — and propagates that error back across the delay—but the evidence does not make it the best primary lineage.
- Psychology — Experimental, clinical, and behavioral psychology supplies a parallel or contributing lineage for the mechanism's defining operation: updates an estimate from the gap between successive predictions — bootstrapping off the next step rather than waiting for the final outcome — and propagates that error back across….
- Statistics & Experimental Design — Statistics, experimental design, and measurement theory supplies a parallel or contributing lineage for the mechanism's defining operation: updates an estimate from the gap between successive predictions — bootstrapping off the next step rather than waiting for the final outcome — and propagates that error back across….
- Systems Thinking & Cybernetics — Systems science's feedback, stock-flow, boundary, and regulation tradition provides a formative adjacent lineage for the same temporal difference update rule operation.
Review resolution: The blind reviewers disagree on primary lineage (organizational_management versus computer_science), so I adjudicated the mechanism rather than inheriting either label. The defining operation is: Updates an estimate from the gap between successive predictions — bootstrapping off the next step rather than waiting for the final outcome — and propagates that error back across the delay. In the computer_science lineage, that operation is specifically evidenced by authoritative or primary work that introduces bootstrapped prediction updates driven by the difference between successive predictions before a final outcome. This makes computer_science the best historical origin, while the retained alternates document contributing methods and later applications rather than being mistaken for coequal origins. The cited Sutton, Learning to Predict by the Methods of Temporal Differences directly supports the mechanism-specific operation and its disciplinary lineage. I retain all independently explained historical alternates without a numeric cap. origin_mode=single_lineage records how the mechanism arose; domain_reach=specialized separately records how broadly it can now be applied.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
Notes¶
This rule assigns credit across time — which past moment to revise — which is not the same as assigning it across cues: when several signals coincide, deciding which one earned the outcome is Credit Assignment Trace's job. Keeping the temporal update and the cue attribution separate stops a delayed error from being blamed on whatever happened to be present at the wrong step.
[n1] Temporal-difference learning updates a prediction from the difference between successive predictions rather than from the final outcome, using eligibility traces to distribute the error over recently-visited states. It is a foundational reinforcement-learning method, and the same signed prediction-error idea is used to model reward-prediction-error signaling in the brain. ↩