The quick read
- Compare the diagnosis, patch and validation.
- Measure the effort needed to understand and accept the change.
Create a fair task
Use the same repository state, reproduction steps and acceptance criteria. Pick work that is substantial enough to require inspection but small enough to review. Record the tools and permissions available to each agent.
Follow the evidence
Compare the diagnosis, patch and validation. Check whether commands actually ran and whether their results support the conclusion. A passing test that simply mirrors the implementation may miss the original problem. Inspect unexpected changes to unrelated files.
Judge the full handoff
Measure the effort needed to understand and accept the change. Note how each agent handles missing information or a blocked check. Current product interfaces differ, so do not generalise from one setup to every environment. Choose based on correct, maintainable work and a clear account of limitations. The useful result is a patch a developer can review confidently, not merely a persuasive explanation of what the agent attempted.
Sources & notes
An editorial decision framework, not a scored benchmark or hands-on test.
anthropic.com — official reference
openai.com — official reference
Sources reviewed for the September 2026 launch edition.