The quick read
- Hold some examples back while improving prompts or settings.
- Group failures by cause rather than hiding them inside one average.
Start with real work
Collect representative tasks before comparing models. Include common cases, difficult exceptions and examples where the correct response is to ask for more information. Define what counts as success for each: factual accuracy, correct formatting, a completed action or an appropriate refusal.
Keep the test honest
Hold some examples back while improving prompts or settings. Otherwise it is easy to optimise for the test cases you already know. Record the model version, instructions and relevant tool configuration. A score without that context may be impossible to reproduce after a product update.
Use results to decide
Group failures by cause rather than hiding them inside one average. A system that performs well overall may still fail consistently for one document type or language. In an illustrative support workflow, compare incorrect answers separately from unnecessary escalations because the consequences differ. Evaluation is useful when it changes a decision: what to deploy, what to review manually and what to improve next. A leaderboard position cannot replace a test built around your own work.
Sources & notes
AI-assisted editorial content checked against the linked sources.
Sources reviewed for the September 2026 launch edition.