Distilling the knowledge in a neural network

Hinton, G., Vinyals, & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint.

Type

Preprint

Intellectual base

Grey literature

Year

2015

Link

no authoritative link yet

Cited from

education_pedagogy

Cited by

2 citations across 2 artifacts.

Each citation links to the sentence it supports in the citing article.

Primes

• Pedagogy
  • The same teach-to-induce-capability structure runs without a human teacher and without a conscious learner, which is the cleanest available demonstration that the pattern is substrate-independent rather than a property of human schoolrooms, a transfer Hinton, Vinyals, and Dean (2015) operationalized in their knowledge-distillation framework that explicitly casts a high-capacity teacher model as instructing a compact student through softened targets.

Mechanisms

• Model Distillation
  • This is the shape of knowledge distillation as introduced by Hinton, Vinyals, and Dean, and of small task models such as DistilBERT trained to reproduce a larger model's behavior.

    This sourceCompact task models such as DistilBERT follow the same recipe; the guarantee they carry extends only to the task and distribution on which the student was validated.

Verification

This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.

Registry ID ref:59d367fc18ca · see in the full table