Overcoming Catastrophic Forgetting in Neural Networks.¶
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., et al. (2017). Overcoming Catastrophic Forgetting in Neural Networks. Proceedings of the National Academy of Sciences, 114(13), 3521-3526.
Cited by¶
4 citations across 4 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Cognitive Flexibility
- In machine learning it shows up as meta-learning, mixture-of-experts gating, and continual learning, with catastrophic forgetting as the pathological loss of the repertoire.
This sourceCharacterizes catastrophic forgetting as loss of a learned repertoire and proposes elastic weight consolidation to preserve it.
Supported in partVerified against the work's full text
Defines catastrophic forgetting as abrupt loss of earlier-task knowledge in continual learning; does not discuss meta-learning or mixture-of-experts gating.
“the tendency for knowledge of the previously learned task(s) (e.g., task A ) to be abruptly lost as information relevant to the current task”
- In machine learning it shows up as meta-learning, mixture-of-experts gating, and continual learning, with catastrophic forgetting as the pathological loss of the repertoire.
- Memory Consolidation
- The diagnostic payoff is concrete — continual-learning methods (elastic weight consolidation) treat "catastrophic forgetting" not as a learning failure but as a consolidation failure (the window was disrupted by un-interleaved new tasks) and intervene in the stabilization stage, not in encoding.
This sourceIntroduces Elastic Weight Consolidation, engineering a consolidation stage that protects weights important to prior tasks.
Supported in partVerified against the work's full text
The paper introduces EWC, inspired by synaptic consolidation, and says forgetting arises under sequential (non-interleaved) training; it does not call forgetting a 'consolidation failure' or place EWC in a separate stabilization stage.
“By interleaving data from multiple tasks during learning, forgetting does not occur because the weights of the network can be jointly optimized for performance on all tasks.”
- The diagnostic payoff is concrete — continual-learning methods (elastic weight consolidation) treat "catastrophic forgetting" not as a learning failure but as a consolidation failure (the window was disrupted by un-interleaved new tasks) and intervene in the stabilization stage, not in encoding.
- Two-Store Architecture
- In machine learning, replay buffers and online weights pair with target networks and consolidated weights, as in DQN, elastic weight consolidation, and Dyna-style designs; without the slow store, online updates produce catastrophic forgetting.
This sourceIntroduces elastic weight consolidation; demonstrates catastrophic forgetting absent a consolidation/slow-store mechanism.
Supported in partVerified against the work's full text
Kirkpatrick et al. support catastrophic forgetting under sequential online training and EWC as a consolidation remedy, with DQN replay and target networks used. Dyna-style designs are absent.
“occurs specifically when the network is trained sequentially on multiple tasks because the weights in the network that are important for task A are changed to meet the objectives of task B”
- In machine learning, replay buffers and online weights pair with target networks and consolidated weights, as in DQN, elastic weight consolidation, and Dyna-style designs; without the slow store, online updates produce catastrophic forgetting.
Mechanisms¶
- Selective Parameter Freezing
- Its strength is that it lets several context-maps coexist on one substrate without keeping a full copy per context, directly countering catastrophic forgetting
This sourceShows that elastic weight consolidation can retain several sequentially learned tasks in one network by selectively protecting weights important to earlier tasks.
- Its strength is that it lets several context-maps coexist on one substrate without keeping a full copy per context, directly countering catastrophic forgetting
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 3 of 4 citations: 3 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:849082368a44 · see in the full table