Distillation: teaching a smaller model from a larger one

The teaching relationship

A teacher model can supply outputs or other signals for training a student. This may help the student perform a defined task with fewer resources. It does not transfer every capability of the larger model, and the quality of the teaching material matters.

Inspect the examples

Teacher mistakes can become training examples too. Check representative samples and keep evaluation data separate. A support classifier should see ambiguous and out-of-scope requests, not only clean examples with obvious answers. Decide what the student should do when evidence is insufficient.

Evaluate the result directly

Compare the student with a simple baseline on task success, latency and resource use. Look specifically for cases the teacher handled but the student missed. Document training-data permissions and version the dataset. A distilled model can be useful for a narrow workflow without being suitable for general conversation. The useful outcome is dependable behaviour under real operating conditions, not a smaller system that merely imitates the teacher’s confident writing style.

Sources & notes

AI-assisted editorial content checked against the linked sources.

arxiv.org — research paper

Sources reviewed for the September 2026 launch edition.