Adam: A Method for Stochastic Optimization.¶
Kingma, D. P., & Ba, J. (2015). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations (ICLR).
Cited by¶
3 citations across 3 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Gain Control
- A structurally identical applied instance is adaptive optimisers in machine learning, where per-parameter gradients are divided by a running estimate of their variance, gain-controlling each parameter update.
This sourceThe Adam optimizer divides each parameter's gradient by a running estimate of its (second-moment) variance — per-parameter gain control on gradient updates.
- A structurally identical applied instance is adaptive optimisers in machine learning, where per-parameter gradients are divided by a running estimate of their variance, gain-controlling each parameter update.
- Higher Order Function
- A machine-learning training pipeline runs the same structure mechanically: the first-order layer is a learning algorithm mapping datasets to models;
optimizer = Adam(beta1=0.9)is a higher-order function — a factory returning a configured update rule — and hyperparameter optimization is a higher-order rule taking learning algorithms as input and returning tuned ones.This sourceIn Proceedings of the 3rd International Conference on Learning Representations (ICLR). 2015. Defines the Adam optimizer; a configured optimizer is a factory returning a parameter-update rule, the higher-order layer over the learning algorithm.
- A machine-learning training pipeline runs the same structure mechanically: the first-order layer is a learning algorithm mapping datasets to models;
- Metaplasticity
- The ML parallel is exact and was a documented two-way port: an adaptive optimizer (Adam, learning-rate warmup-and-decay) maintains per-parameter learning rates that themselves update on the history of gradients — a slow controller wrapped around the fast weight-update rule, raising or lowering how much each weight moves per step based on accumulated gradient statistics.
This sourceAdaptive optimizer maintaining per-parameter learning rates that update on accumulated first- and second-moment gradient statistics—a slow controller over the fast weight-update rule.
- The ML parallel is exact and was a documented two-way port: an adaptive optimizer (Adam, learning-rate warmup-and-decay) maintains per-parameter learning rates that themselves update on the history of gradients — a slow controller wrapped around the fast weight-update rule, raising or lowering how much each weight moves per step based on accumulated gradient statistics.
Verification¶
This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.
Registry ID ref:671e2d495ef9 · see in the full table