Training Compute-Optimal Large Language Models

Hoffmann, J., Borgeaud, S., & Mensch, A. (2022). Training Compute-Optimal Large Language Models. Advances in Neural Information Processing Systems 35.

Type

Preprint

Intellectual base

Grey literature

Year

2022

Link

https://arxiv.org/abs/2203.15556

Cited from

biology_ecology, synthesized

Cited by

2 citations across 2 artifacts.

Each citation links to the sentence it supports in the citing article.

Primes

• Diminishing Incremental Gains
  • This experimental approach is increasingly standard in data-driven organizations: Hoffmann et al. (2022), in the Chinchilla scaling-laws study, ran a sweep of compute-matched training runs to fit the actual concave loss-versus-compute curve and overturn earlier extrapolations about optimal model-versus-data ratios.
• Liebig's Law of the Minimum
  • In AI training, model performance caps on whichever of data quantity, data quality, compute, parameters, or inference budget is binding at the current scale, and the compute-optimal literature is essentially a search for the Liebig configuration.

    This sourceThe Chinchilla result showing prior large models were parameter-rich but data-starved, a Liebig diagnosis over training inputs.

Verification

Does it exist? Not checked yet. This entry carries no identifier to resolve. It was extracted from the citation as written in the article, normalized, and deduplicated against the rest of the registry.

Does it back the claim? Not recorded. Neither this nor any other of the 2 citations of this work carries a recorded support check.

Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.

See how references were verified.

Registry ID ref:a4d07aab7655 · see in the full table