Outrageously Large Neural Networks¶
Shazeer, N., Mirhoseini, Maziarz, Davis, Le, Hinton, & Dean, J. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. International Conference on Learning Representations.
Cited by¶
3 citations across 3 artifacts.
Each citation links to the sentence it supports in the citing article.
Domain-specific¶
Mechanisms¶
- Gated Expert Router
- The discipline is to monitor expert utilization, audit the routing from outside with a confusion matrix, and keep a fallback for cues the gate has never credibly seen.
This sourceIntroduces load-balancing objectives that monitor and discourage highly unequal utilization of mixture-of-experts subnetworks.
- The discipline is to monitor expert utilization, audit the routing from outside with a confusion matrix, and keep a fallback for cues the gate has never credibly seen.
- Sparse Attention Mask
- Its characteristic failure mode is a router that learns to send almost everything to a few experts, leaving the rest untrained and the capacity wasted — the load-imbalance / expert-collapse problem that mixture-of-experts systems guard against with explicit balancing.
This sourceDocuments self-reinforcing gate imbalance in which a few experts dominate and prescribes explicit importance and load-balancing losses.
- Its characteristic failure mode is a router that learns to send almost everything to a few experts, leaving the rest untrained and the capacity wasted — the load-imbalance / expert-collapse problem that mixture-of-experts systems guard against with explicit balancing.
Verification¶
Does it exist? Not checked yet. This entry carries no identifier to resolve. It was extracted from the citation as written in the article, normalized, and deduplicated against the rest of the registry.
Does it back the claim? Not recorded. Neither this nor any other of the 3 citations of this work carries a recorded support check.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Links previously used in the corpus¶
Before the registry existed this work was also linked 2 other ways.
Registry ID ref:697380114f11 · see in the full table