Benchmarks - public reference
Benchmarks (SWE-bench, GAIA, τ-bench, HumanEval) give a shared reference: you run the agent on the same dataset others published and the number speaks the same language. Classic risk: optimizing for the benchmark and losing in the real world.
What it is
A benchmark is a frozen dataset plus an execution protocol plus a published number. Someone assembles 500 tasks, fixes the metric, runs several models on the same hardware and publishes a table. You repeat the experiment, get a number and compare it with the table. It is the same model as classic SDLC: the test is stable, the subject varies, the verdict is comparable.
For agents the mechanics change a bit. Public benchmarks tend to be too static (single answer, no interaction) or synthetic (simulated customers that look human without having nuance). That is why a benchmark is a starting point, never a finish line.
The four that matter
(Verified) is the de facto standard for coding agents. About 500 real GitHub issues, each with the bug description and the test that validates the fix. The agent receives the repository, reads the code, proposes a patch and runs the tests. An agent that resolves patches resolves N% of the benchmark.
GAIA measures general-purpose agents on multi-hop tasks (cross-referencing sources, reading PDFs, summing values). It mixes tools, reasoning and retrieval. It is the most used benchmark for comparing agents that browse the web.
τ-bench (tau-bench) focuses on dialogue between agent and simulated user, with tools and policies. It measures whether the agent completes the customer's task without blowing the tool call budget. It is a recent reference (2024) and has gained traction in customer support.
HumanEval is the parent of them all: 164 code problems in Python, function generates function, asserts validate. It remains useful as a sanity check for a plain model, but any modern agent passes without breaking a sweat.
Running a local benchmark
Most benchmarks publish a Python harness that standardizes execution. The skeleton below shows the mechanics of iterating over the cases, exposing the agent as a pure function so the harness can compare output with the answer key:
import json
from datasets import load_dataset
from agent import solve # your agent's function
CASES = load_dataset("princeton-nlp/SWE-bench_Verified", split="test")
def run(limit: int | None = None):
passed = 0
total = 0
failures = []
for case in CASES:
if limit and total >= limit:
break
total += 1
try:
patch = solve(case["repo"], case["problem_statement"])
ok = apply_and_test(case["instance_id"], patch, case["test_patch"])
if ok:
passed += 1
else:
failures.append((case["instance_id"], "tests_failed"))
except Exception as e:
failures.append((case["instance_id"], f"exception:{type(e).__name__}"))
return {"passed": passed, "total": total, "failures": failures}
if __name__ == "__main__":
print(json.dumps(run(limit=50), indent=2))The risk of overfitting
When the dataset is public and the result becomes a marketing metric, someone learns to hack the benchmark. The most didactic case is contamination: the model is trained on text that contains the benchmark's cases, and the 'answer' becomes memorization disguised as reasoning. Another trick is cherry-picking: running the agent 10 times, picking the best run and publishing it.
Practical defense. The first is to run the benchmark blinded: never let the agent read the expected solution, and truncate the dataset if it would leak through the context. The second is to cross-check with a private dataset you keep confidential, of the same format, and require that the two move together. The third is to monitor the gap between benchmark and production, and investigate whenever it diverges.
Benchmark versus production
A benchmark is static, finite and public. Production is infinite, noisy and full of long tail (frustrated user, , ambiguous query). An agent can score 80% on GAIA and still be bad for the real user.
The common-sense reading treats the benchmark as triage and the private dataset as the source of truth. Publishing the number in the README is honest marketing. Trusting it to close a release is foolish.
How to choose the right benchmark
Start with the workload. If your agent reads code, SWE-bench. If it talks to customers, τ-bench. If it does web research, GAIA. If the work is pure generation, HumanEval does the job.
Then look at the version. SWE-bench Verified is the human-validated subset of the original SWE-bench. Always prefer the verified version when it exists, because it reduces the noise of ambiguous cases.
Finally, measure the cost. Each case has a budget of tokens and tool calls. Compare cost per solved case, not just the hit rate. An agent that solves 60% with half the tokens is usually better than one that solves 80% with double.
Strengths
- direct comparison between models and between vendors, on the same yardstick
- obvious regressions (the vendor regressed the model between versions) jump out
- predictable cost per case, useful for API budgeting
Limitations
- contamination: the benchmark may leak into the training of the evaluated model
- the long tail of production does not show up: a rare case that takes down 1% of users is not in the 500
- gaming: time spent hacking the benchmark rarely improves the real agent
connections on the map
- Benchmarks - public referencesupplies dataEvals - quality measured in production
updated on 14/10/2026 · sources to go deeper · report an error
Study before
test what you learned
Loading questions…