The quick read
- For factual tasks, evaluate support, completeness and appropriate abstention.
- Combine measures when the product needs both correctness and usability, but show what each result means.
Define the quality claim
Accuracy tests compare outputs with an established answer or acceptance rule. Preference tests ask which output people favour. A response can be polished and preferred while factually wrong; a correct response can still be hard to use. Neither measure should silently stand in for the other.
Design separate checks
For factual tasks, evaluate support, completeness and appropriate abstention. For experience quality, use representative raters and clear criteria such as readability or usefulness. Randomise presentation where practical and record disagreements rather than hiding them inside one score.
Report the limits
Combine measures when the product needs both correctness and usability, but show what each result means. Keep examples of important failures and test changes against the same baseline. Avoid declaring a universal winner from a small task set or a preference vote alone.
Sources & notes
An editorial decision framework, not a scored benchmark or hands-on test.
developers.openai.com — official reference
Sources reviewed for the September 2026 launch edition.