Training Compute-Optimal Large Language Models

Hoffmann, J., Borgeaud, S., & Mensch, A. (2022). Training Compute-Optimal Large Language Models. Advances in Neural Information Processing Systems 35.

Type

Preprint

Intellectual base

Grey literature

Year

2022

Link

https://arxiv.org/abs/2203.15556

Cited from

biology_ecology, general

Cited by

2 citations across 2 artifacts.

Each citation links to the sentence it supports in the citing article.

Primes

• Diminishing Incremental Gains
  • This experimental approach is increasingly standard in data-driven organizations: Hoffmann et al. (2022), in the Chinchilla scaling-laws study, ran a sweep of compute-matched training runs to fit the actual concave loss-versus-compute curve and overturn earlier extrapolations about optimal model-versus-data ratios.
• Liebig's Law of the Minimum
  • In AI training, model performance caps on whichever of data quantity, data quality, compute, parameters, or inference budget is binding at the current scale, and the compute-optimal literature is essentially a search for the Liebig configuration.

    This sourceThe Chinchilla result showing prior large models were parameter-rich but data-starved, a Liebig diagnosis over training inputs.

Verification

This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.

Registry ID ref:a4d07aab7655 · see in the full table