Menu

Knowledge Distillation: Why Soft Labels Teach Small Models What Hard Labels Cannot
📰

Knowledge Distillation: Why Soft Labels Teach Small Models What Hard Labels Cannot

DEV Community: deeplearning·Umesh Tharuka Malaviarachchi·33 minutes ago
#sxF9ADQA
Reading 0:00
15s threshold

A large model that has seen millions of examples knows more than the label of each training point. It knows which wrong answers are almost right, which classes are easily confused, and how confident it should be in each case. Knowledge distillation is the technique that transfers that extra knowledge into a smaller model, so the student can run where the teacher cannot: on a phone, inside a latency budget, or at a fraction of the serving cost.

The idea, popularised by Hinton and co-authors in 2015, is simple to state and surprisingly deep in its consequences. Train the small model not only on the hard labels, but also on the teacher's full probability distribution over classes. Those soft targets carry information that a single correct label cannot.

What a soft label actually contains

Consider an image classifier trained on handwritten digits. For a particular image of a 7, the hard label says "7" and nothing else.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.