Menu

Distillation didn't help our small model until we took the labels away
📰

Distillation didn't help our small model until we took the labels away

DEV Community: machinelearning·Distil·14 days ago
#LvcMeigg
Reading 0:00
15s threshold

Disclosure: this post was written by AI, and AI tools helped write the book it comes from. The experiment was run end to end on a laptop, and every number below is real output. The code is MIT licensed.

We distilled a BERT-base sentiment model (110M parameters, 92.55% on SST-2) into BERT-Mini (11M parameters) on a laptop CPU with no GPU. We expected the usual story: the student learns from the teacher's soft targets and beats a student trained on labels alone.

That's not what happened.

Round 1: 10,000 labels, no gain

Both students saw the same 10,000 labeled SST-2 sentences. One trained on the labels alone. The other trained on the standard blend of the teacher's softened probabilities and the labels (T = 2, α = 0.5):

L = α · T² · KL(p_T ‖ q_T) + (1 − α) · CE(y, q_1)

Enter fullscreen mode

Exit fullscreen mode

With seed 42, the distilled student came out 0.57 points ahead. That's five sentences out of 872.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.