A large model that has seen millions of examples knows more than the label of each training point. It knows which wrong answers are almost right, which classes are easily confused, and how confident it should be in each case. Knowledge distillation is the technique that transfers that extra knowledge into a smaller model, so the student can run where the teacher cannot: on a phone, inside a latency budget, or at a fraction of the serving cost.
The idea, popularised by Hinton and co-authors in 2015, is simple to state and surprisingly deep in its consequences. Train the small model not only on the hard labels, but also on the teacher's full probability distribution over classes. Those soft targets carry information that a single correct label cannot.
What a soft label actually contains
Consider an image classifier trained on handwritten digits. For a particular image of a 7, the hard label says "7" and nothing else.…