Agentic Coding 2.0
A frontier benchmark for evaluating whether coding agents can plan, execute, verify, and recover across complex terminal-native engineering tasks.
overview
Agentic Coding 2.0 is the next generation of Snorkel’s original Agentic Coding benchmark. Built from a representative frontier subset of our Terminal-Bench+ dataset, it expands the evaluation from difficult coding problems to autonomous engineering across software engineering, debugging, systems, security, data processing, machine learning, scientific computing, games, and build and dependency management.
Each task runs in an isolated, air-gapped environment. Agents must use tools, manage intermediate state, validate their work, and recover from errors. Every task includes a reference solution, deterministic tests, and rubrics that evaluate both the final result and the agent’s trajectory.
At a glance
200
frontier tasks
9
tasks types
9
target languages
Leaderboard
| Rank | Model | Pass@1 | Pass@5 | Cost / Trial | Cost / Task |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra |
47.6%
|
58.7%
|
$1.16 | $5.79 |
| 2 | Fable 5.1 |
39.6%
|
54.8%
|
$3.86 | $19.28 |
| 3 | Claude Opus 5 |
38.9%
|
62%
|
$2.91 | $14.55 |
| 4 | Grok 4.6 |
33.6%
|
53.1%
|
$1.11 | $5.57 |
| 5 | Gemini Flash 3.8 |
32.6%
|
53.1%
|
$1.56 | $7.8 |
| 6 | Kimi K3 |
26.1%
|
52.4%
|
$1.58 | $7.9 |
| 7 | Muse Spark 1.3 |
25%
|
40.3%
|
$1.83 | $9.13 |
| 8 | GLM 5.3 |
24.9%
|
46.3%
|
$1.2 | $6.01 |
| 9 | DeepSeek V4 Pro |
14.8%
|
35.6%
|
$1.66 | $8.29 |
| 10 | Qwen 3.8 Max |
14.7%
|
26.2%
|
$0.9 | $4.49 |
| 11 | Nemotron 3 Ultra 550B |
5.1%
|
10.6%
|
$0.92 | $4.6 |
Frontier performance
Methodology
Evaluator
Tasks use the Harbor Terminal-Bench format with Docker environments, packaged dependencies, reference solutions, supporting files, and no network access.
scoring note
Five attempts per task support pass-rate calibration and repeatability. A task must satisfy its deterministic tests and required rubric criteria to receive credit.
Behind the benchmark
The original Agentic Coding benchmark established a focused evaluation of multi-step coding tasks. Agentic Coding 2.0 broadens that foundation into a more demanding test of autonomous engineering.
The benchmark evaluates the complete agent loop: understanding the assignment, exploring an unfamiliar environment, selecting and sequencing tools, executing a solution, inspecting the result, correcting mistakes, and producing a verifiable final state.
