Model benchmarks
Also called SWE-bench, Leaderboard
A fixed set of tasks that models are scored on so they can be compared. SWE-bench is the best known for coding: given a real bug report from GitHub and the project's code, the model has to produce a change, and tests decide whether it worked.
A score tells you how a model did on those tasks under those tests. The tests can be wrong, the answers may have been in the training text, and your code base isn't in it. Treat scores as a shortlist, then test on your own work.
Say it to your agent
“Don't pick the model from a leaderboard. Take 10 tasks we have already solved from git log, run each candidate on them in its own worktree, score with our test suite plus a test it can't see, and show me pass count and cost.”