Almost-Reward Annotation¶
Data label — instantiates Counterfactual Proximity Signal Calibration
Attaches a bounded partial-credit label to an almost-successful case so a learner is nudged toward the missing step without being paid the full reward.
An Almost-Reward Annotation lives on a single data point. When a case did not succeed but came close, the annotation marks it "almost — and this is how much better success would have been, and along what gap", then converts that gap into a bounded partial-credit signal a learner can absorb. Its defining move is that it operates at the level of the individual example and shapes the magnitude of a signal, not the downstream consequences. It answers one question — how much learning pressure should this near-success carry — and deliberately caps the answer strictly below the credit reserved for a real win, so that "almost" never quietly becomes "won."
Example¶
A programming course uses an autograder. A student's submission passes nine of ten hidden tests and fails only the empty-input edge case. Binary grading has two bad options: award zero because not all tests passed, or (if the rubric is lenient) full marks that erase the missing case. Almost-Reward Annotation tags the submission differently: near-success, value gap ≈ "one guard clause from correct," partial signal ≈ 0.7 of full credit routed only into the practice recommender, capped below the marks a fully-correct solution earns. The signal is pointed — it names the missing edge-case handling rather than a diffuse "close." The outcome: the student's next recommended exercise targets input validation, while the gradebook still shows the assignment as not-yet-passed. The near-miss taught something without being scored as a pass.
How it works¶
The annotation is a two-step transform on one case. First it fixes the value delta: in the grader's own value frame, how much better the near-success would have been than what happened, with a sign (upward — success was within reach). Then it applies a bounded transform — monotonic in the delta and in closeness, but saturating well short of the full-reward ceiling — and emits the resulting partial signal attached to the example. The two guarantees that make it this mechanism and not a bonus hack: the signal is derived from a stated value gap rather than hand-tuned, and it is hard-capped below success so no accumulation of almosts can equal a win.
Tuning parameters¶
- Partial-credit ceiling — the fraction of full reward an almost may reach. Lower keeps the incentive to actually finish sharp; higher speeds learning from sparse near-successes but risks the learner settling for "close."
- Delta scaling — whether signal grows linearly or concavely with the value gap. Concave scaling stops a single large-delta case from dominating the batch.
- Value frame — whose "better" defines the delta (the grader's, the user's, the platform's). Changing it re-prices every annotation.
- Annotation threshold — how close a case must be before it earns any label at all. Tightening it suppresses noise; loosening it labels more cases but dilutes the signal.
When it helps, and when it misleads¶
Its strength is in sparse-reward settings where genuine successes are rare but near-successes are frequent — it turns a wall of zeros into graded feedback that points at the missing step. Its classic misuse is reward hacking: if the partial signal is not strictly bounded and tied to a real value gap, a learner optimizes the proxy (rack up almosts) instead of the target (succeed). The literature on potential-based reward shaping is precise about this danger — a shaping signal is safe only when it cannot change which behavior is optimal.[1] The guarding discipline is to keep the cap firmly below full reward and to derive the label from the stated value delta, never from a discretionary bonus, so the annotation can never be farmed into a substitute for the outcome it approximates.
How it implements the components¶
value_delta_frame— the annotation's first field is the signed magnitude of how much better the near-success would have been, in the grader's value frame.signal_transformation_rule— its second act is the bounded, saturating transform that turns that delta into a capped partial-credit signal on the example.
It does NOT implement learning_update_boundary — deciding what a signal is allowed to change downstream (retrain, redesign, escalate) is Near-Miss Response Tier's job; this label only sets the signal's size on one case. It also leaves proximity_metric_or_ordering to Near-Miss Distance Scorecard.
Related¶
- Instantiates: Counterfactual Proximity Signal Calibration — supplies the bounded, example-level learning signal the archetype's transformation step calls for.
- Consumes: Counterfactual Value-Delta Table supplies the value-gap magnitudes the label prices.
- Sibling mechanisms: Near-Miss Response Tier · Close-Call Review Protocol · Counterfactual Plausibility Filter · Counterfactual Value-Delta Table · Near-Miss Distance Scorecard · Proximity Signal Backtest · Regret-Weighted Decision Log · Salience Overweighting Check · Threshold Band Map
Editorial Notes¶
Form Classification¶
Form family: Interface, Display & Cue
Rationale: The mechanism attaches a bounded partial-credit label to an almost-successful case so the learner perceives proximity and attends to the missing step, making it a behavioral cue.
Nearest alternative: Representation, Specification & Plan — The annotation is an information artifact, but its operative value is the signal presented to and acted on by the learner rather than static reference.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Reinforcement learning and machine-learning reward shaping formalize auxiliary graded signals, policy-invariance constraints, bounded rewards, and the danger of proxy optimization.
Related originating lineages:
- Cognitive Science — Learning signals and error-driven updating provide cognitive models.
- Education & Pedagogy — Partial credit, formative assessment, and targeted practice recommendations supply the worked instructional form.
- Psychology — Operant conditioning and shaping by successive approximations provide an older behavioral account of reinforcing near-target behavior.
Review resolution: The machine-readable partial reward is computer-science primary, while pedagogy, behavioral psychology, and cognitive learning theory materially shape the bounded near-success signal. The synthesized cross-domain artifact remains high-confidence despite differing reviewer labels for the learning-science alternate.
Attribution caveat: The specific annotation is synthesized from RL reward shaping and educational partial credit rather than a standard named label.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
References¶
[1] Potential-based reward shaping (Ng, Harada & Russell, 1999) shows that an auxiliary reward derived from a potential function leaves the optimal policy unchanged, whereas an arbitrary bonus can be gamed. It is the formal reason an almost-reward must be bounded and value-anchored rather than a free-floating incentive. withdrawn registry ↩