Batch Normalization¶
Ioffe, S., & Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Proceedings of the 32nd International Conference on Machine Learning (ICML), 448-456.
Cited by¶
2 citations across 2 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Gain Control
- In machine learning batch, layer, and group normalisation measure recent activation statistics and divisively rescale, return and advantage normalisation do the same in reinforcement learning, and adaptive optimisers apply per-parameter gain control to gradients by dividing by running variance.
This sourceIntroduces batch normalization, which measures recent activation statistics and divisively rescales — gain control on a network's signal path.
Supported in partVerified against the work's full text
The source is the batch normalization paper, supporting the batch-normalization clause, but it does not cover layer/group normalization, RL return/advantage normalization, or adaptive optimisers.
“Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”
- In machine learning batch, layer, and group normalisation measure recent activation statistics and divisively rescale, return and advantage normalisation do the same in reinforcement learning, and adaptive optimisers apply per-parameter gain control to gradients by dividing by running variance.
- Neuromodulation
- The per-example or per-feature gain-and-bias pattern of batch and layer normalization moved from vision networks into transformer attention design.
This sourceIntroduces the learned per-feature gain-and-bias normalization pattern that moved from vision networks into transformer attention.
Supported in partVerified against the publisher's abstract
This is the 2015 paper that introduced batch normalization of layer inputs for deep vision networks; it supplies the antecedent of the claim, not the asserted transfer of that pattern into transformer attention design.
“Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Proceedings of the 32nd International Conference on Machine Learning , PMLR 37:448-456, 2015. Training Deep Neural Networks is complicated by the fact that the distribution of each layer’s inputs changes during training, as the parameters of the previous layers change.”
- The per-example or per-feature gain-and-bias pattern of batch and layer normalization moved from vision networks into transformer attention design.
Verification¶
Does it exist? Not checked yet. This entry carries no identifier to resolve. It was extracted from the citation as written in the article, normalized, and deduplicated against the rest of the registry.
Does it back the claim? Read against the text for 2 of 2 citations: 2 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:50dd153d38ca · see in the full table