Small language models: when a smaller model is enough

Match capability to the job

A model used to route short requests does not necessarily need the same breadth as one used for open-ended research. Smaller models can offer practical advantages in memory use and deployment, but size alone does not establish speed, cost or accuracy in a particular system.

Create a bounded comparison

Use the same representative inputs and success criteria for a smaller and a larger candidate. Include ambiguous cases and a route to human review. For an illustrative classification task, inspect errors for rare categories instead of relying only on overall accuracy.

Keep the surrounding system simple

Clear labels, concise context and well-defined outputs may matter more than adding model capacity. Measure actual end-to-end latency and resource use on the hardware you will operate. A smaller model that needs repeated retries may be a poor fit; a larger model that adds no useful accuracy may be unnecessary. The goal is the least complex system that meets the task’s quality and operating requirements, with a documented way to handle cases outside its competence.

Sources & notes

AI-assisted editorial content checked against the linked sources.

OpenAI: Working with evals

Sources reviewed for the September 2026 launch edition.