Menu

Synthetic high‑value questions boost distillation efficiency
📰

Synthetic high‑value questions boost distillation efficiency

DEV Community: machinelearning·Papers Mache·19 days ago
#PZJwy2Cz
Long article (355 words) — no AI summary yet.
Reading 0:00
15s threshold

Generating high‑value synthetic questions injects dense teacher supervision into on‑policy distillation, substantially reducing the number of required training questions. The twist is that these queries need no reference answers; their value comes from how much they expose the teacher’s knowledge when a strong model answers them.

Before this work, practitioners relied on real user questions or unoptimized student‑generated prompts, often filtering them with KL divergence to approximate relevance. Vanilla on‑policy distillation (OPD) and KL‑filtered real queries have been the default baselines for math reasoning, code generation, and agentic tasks.

A2D lifts average test scores to the mid‑40s and 60s, outpacing vanilla OPD by several points.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.