Distilling the knowledge in a neural network

Hinton, G., Vinyals, & Dean, J. (2014/2015). Distilling the knowledge in a neural network. arXiv preprint.

Type

Preprint

Intellectual base

Grey literature

Year

2015 this edition · first published 2014

Link

https://arxiv.org/abs/1503.02531

Cited from

education_pedagogy

Cited by

3 citations across 2 artifacts.

Each citation links to the sentence it supports in the citing article.

Primes

• Pedagogy
  • The same teach-to-induce-capability structure runs without a human teacher and without a conscious learner, which is the cleanest available demonstration that the pattern is substrate-independent rather than a property of human schoolrooms, a transfer Hinton, Vinyals, and Dean (2015) operationalized in their knowledge-distillation framework that explicitly casts a high-capacity teacher model as instructing a compact student through softened targets.

Mechanisms

• Model Distillation
  • A classic misuse is distilling for task A and then quietly using the student for task B, outside the scope where it was ever validated — knowledge distillation guarantees adequacy only on the task and distribution it was checked against.

    This sourceCompact task models such as DistilBERT follow the same recipe; the guarantee they carry extends only to the task and distribution on which the student was validated.

  • This is the shape of knowledge distillation as introduced by Hinton, Vinyals, and Dean, and of small task models such as DistilBERT trained to reproduce a larger model's behavior.

    This sourceIntroduces knowledge distillation as transferring knowledge from a cumbersome ensemble or model into a smaller model that is easier to deploy.

Verification

Does it exist? Not checked yet. This entry carries no identifier to resolve. It was extracted from the citation as written in the article, normalized, and deduplicated against the rest of the registry.

Does it back the claim? Not recorded. Neither this nor any other of the 3 citations of this work carries a recorded support check.

Was it audited? Yes. A second, independent pass read the citation against the article text and recorded a verdict.

Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.

See how references were verified.

Registry ID ref:59d367fc18ca · see in the full table