Knowledge distillation trains a small student model to copy a large teacher, either by matching the teacher's full probability distribution over next tokens or by fine-tuning on text the teacher wrote. On Nodus, the managed nodus/distill runtime (Beta) trains a student on a teacher's distributions from one manifest, with a dollar cap and a price estimate before the run starts. Which method you pick decides how often you pay for the teacher, and in our worked plan that swings the bill by almost 2x.
There are four families of LLM distillation in common use. Each one answers a different question: what signal does the student learn from, and whose text is it learning on?
What does the student actually learn from the teacher?
In the original method, the student learns the teacher's soft labels: its whole probability distribution, not just its top answer. Geoffrey Hinton, Oriol Vinyals and Jeff Dean described it in Distilling the Knowledge in a Neural Network (2015).…