Distillation without data — how it works
You can train a fast student model without collecting a dataset — the teacher generates its own supervision.
No dataset required
Instead of an expensive labeled corpus, a frozen teacher synthesizes targets on the fly from prompts alone. Cheaper, and surprisingly, a better regularizer.