Guides
What is an AI model benchmark, and how do you read the score
The three parts of every test, the five kinds you will meet, and the questions that separate a real result from a launch-day claim.
An AI model benchmark is a fixed set of tasks, run under stated settings and graded by a set rule, so that different models can be scored the same way. A score tells you how one model did on that test under those settings and nothing more, so the fine print around a number matters as much as the number itself.
The three parts of any benchmark
- The tasks. Questions, bug reports, puzzles or chat prompts. MMLU, an early example, covers 57 tasks, from elementary mathematics to law.
- The settings. The prompt, whether tools are allowed, the reasoning effort, the agent wrapped around the model and how many tries it gets. "Pass@1" means one attempt per task counts.
- The metric. Accuracy, share of tasks resolved, or a rating built from comparisons.
Change any one of the three and the score changes with it. That is why a lab's launch chart and an outside leaderboard can report different numbers for the same model.
Five kinds you will run into
| Kind | Examples | A high score suggests |
|---|---|---|
| Knowledge questions | GPQA, Humanity's Last Exam | Strong recall and reasoning on expert material |
| Coding agents | SWE-bench Verified, Terminal-Bench 4.0 | It can fix real code or finish terminal tasks on its own |
| New-skill puzzles | ARC-AGI | It adapts to problems it has not seen before |
| Human preference | Arena | People liked its answers more in blind votes |
| Composite index | Artificial Analysis Intelligence Index, Epoch Capabilities Index | Broad ability across many tests on one scale |
GPQA's 448 questions show how hard these can be: experts holding or studying for a PhD in the field scored 65%, while skilled non-experts with the open web scored 34%. Humanity's Last Exam goes further, with 2,500 questions from close to 1,000 subject experts and a private held-out set kept back to catch overfitting.
Five questions to ask before you trust a score
- Who ran it? A lab testing its own model is not the same as an outside group. Artificial Analysis, for one, says it evaluates every model under identical conditions.
- With what settings? Look for "with tools", effort levels or a custom agent. SWE-bench offers a bash-only view that puts every model in the same minimal agent for a fair comparison.
- Is the test still hard? Once top models crowd near the ceiling, gaps stop meaning much. HLE released a Diamond set in September 2026 for that moment.
- Could the model have seen the questions? LiveCodeBench dates each problem so it can test models only on ones newer than their training data.
- When was it measured? Standings shift within weeks. Write the date next to any number you quote.
Where TechTwitter IO fits
Those five questions are what tech Twitter argues about after every launch, and TechTwitter IO turns the arguments into a weekly satire. Its Trust Me Bro board, for instance, scores models by how many unconfirmed claims and footnoted chart results they rack up against outside receipts. Other boards track who switched to what, vibes, hype and model "crimes", all counted from posts on X and linked to them.
It is the most entertaining way to learn to spot a weak benchmark claim, and the paper is free. It is also satire by its own footer, so use it to sharpen your questions and take the actual numbers from the benchmark's own page.
Short answer
An AI model benchmark is a repeatable test with a scoring rule. Read the score together with who ran it, the settings, the date and whether the test still has headroom. For a broad view start with an index such as Artificial Analysis, and for a specific job use the benchmark built for it.