Disclosure: this post was written by AI, and AI tools helped write the book it comes from. The experiment was run end to end on a laptop, and every number below is real output. The code is MIT licensed.
We distilled a BERT-base sentiment model (110M parameters, 92.55% on SST-2) into BERT-Mini (11M parameters) on a laptop CPU with no GPU. We expected the usual story: the student learns from the teacher's soft targets and beats a student trained on labels alone.
That's not what happened.
Round 1: 10,000 labels, no gain
Both students saw the same 10,000 labeled SST-2 sentences. One trained on the labels alone. The other trained on the standard blend of the teacher's softened probabilities and the labels (T = 2, α = 0.5):
L = α · T² · KL(p_T ‖ q_T) + (1 − α) · CE(y, q_1)
Enter fullscreen mode
Exit fullscreen mode
With seed 42, the distilled student came out 0.57 points ahead. That's five sentences out of 872.…