A serious agentic-coding benchmark (DeepSWE: contamination-free, behavior-verified, long-horizon tasks) showed something public leaderboards hide: frontier agents sitting in a 30-point band on SWE-Bench Pro spread across a ~70-point gap. Rankings compress real differences.
The hidden variable is the verifier. Inherited test suites produce 8.5% false positives and 24% false negatives. And scores are harness-relative: the same model inside different harnesses is effectively a different system.
What I do with this, in practice
- I evaluate models on my own work, with my own scenarios. Leaderboards I read out of curiosity.
- I own the harness, the tools and the verification. That's where quality lives.
- I treat the model as a swappable component. When something better ships, I switch in an afternoon.
If you're a decision-maker: ask your team where the evals live. If you're an engineer: if the answer is "in the prompt", you've found the problem.
// happy shipping. or happy debugging, depends on the harness.