Terminal-Bench 4.0
A benchmark to measure and evolve with the frontier of agent work: real terminal environments, real software engineering tasks, and a rolling task set that is revised as agents catch up to it. Version 4.0 trims the 3.0 task set from 74 to 66 tasks, removing eight and revising 20.
At a glance
66
8
20
Key takeaways
GPT-6 Astra (Codex, max effort) leads at 58.2% ± 2.8% resolution, with Fable 5.1 (Claude Code, max effort) close behind at 57.9% — a gap of just 0.3 points. Grok 4.5 and Sonnet 5 tie near the bottom of the ranked field at 12.4%, with Gemini 3.7 Flash trailing further at 11.2%, a spread of roughly 47 points across the fourteen evaluated model and agent configurations.
Leaderboard
| Rank | Model | Effort | Agent | Resolution Rate | Release Date | Tokens | Cost |
|---|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra | max | Codex |
58.2%
±2.8
|
2026-09-03 | 1.5B | $3.3k |
| 2 | Fable 5.1 | max | Claude Code |
57.9%
±3.8
|
2026-09-01 | 2.7B | $6.2k |
| 3 | Opus 5 | xhigh | Claude Code |
53.9%
±3.2
|
2026-07-24 | 6.9B | $6.1k |
| 4 | Fable 5 | max | Claude Code |
44.5%
±3.8
|
2026-06-09 | 3.8B | $7.3k |
| 5 | GLM 5.3 | max | Claude Code |
41.8%
±3.2
|
2026-08-14 | 8.7B | $2.7k |
| 6 | GPT-5.6 Sol | max | Codex |
37.3%
±3.8
|
2026-06-26 | 4.4B | $2.5k |
| 7 | Opus 4.8 | max | Claude Code |
23.6%
±3.6
|
2026-05-28 | 6.4B | $6.5k |
| 8 | GPT-5.6 Terra | max | Codex |
21.5%
±3.3
|
2026-06-26 | 5.7B | $1.7k |
| 9 | Grok 4.6 | high | Grok Build |
20.3%
±3.1
|
2026-08-12 | 4.0B | $3.6k |
| 10 | Gemini Flash 3.8 | high | mini-SWE-agent |
19.1%
±3.4
|
2026-09-02 | 17.2B | $1.8k |
| 11 | GPT-5.6 Luna | max | Codex |
17.3%
±2.8
|
2026-06-26 | 11.6B | $0.3k |
| 12 | Grok 4.5 | high | Grok Build |
12.4%
±2.6
|
2026-07-16 | 3.4B | $2.1k |
| 13 | Sonnet 5 | max | Claude Code |
12.4%
±3.1
|
2026-06-30 | 21.6B | $9.6k |
| 14 | Gemini 3.7 Flash | high | mini-SWE-agent |
11.2%
±2.4
|
2026-08-13 | 11.1B | $1.3k |
How Terminal-Bench 4.0 compares
Several recent benchmarks make progress on behavioral testing and instruction realism. The following table provides a brief comparison.
Benchmark
Released
Tasks
Change
Terminal-Bench 4.0
Aug 2026
66
8 tasks removed, 20 revised, 0 added
Terminal-Bench 3.0
Jul 2026
74
New rolling task set (formerly branded Frontier-Bench); initial 74 tasks added
Terminal-Bench 2.1
May 2026
89
Revision fixing 28 tasks; task count unchanged
Terminal-Bench 2.0
Nov 2025
89
Harder, curated task set; introduced Harbor, the agent eval and optimization framework
Terminal-Bench 1.0
May 2025
80
Initial release (Terminal-Bench-Core), led by Stanford and the Laude Institute
Behind the benchmark
Terminal-Bench measures whether an AI agent can operate a real terminal to complete real software engineering and systems tasks: writing and debugging code, configuring environments, and recovering from failure, all without a GUI. Version 3.0 introduced the current 74 task families; version 4.0 is a maintenance release that removes eight tasks and revises twenty, netting 66 tasks, with no new tasks added in this cycle.
Methodology
Resolution rate
The public Terminal-Bench site marks its task and leaderboard data with a canary string, requesting that benchmark data not appear in model training corpora.
Run command
Evaluated via the Harbor evaluation harness: harbor run -d terminal-bench/terminal-bench@4.0.0. Dataset pinned at hub.harborframework.com.
From the blog
Resources
Acknowledgments
Terminal-Bench is hosted by Harbor and the Laude Institute, and supported by Snorkel AI via the Open Benchmarks Grants program. The v4.0.0 release was authored by RyanMarten and Snorkel's Justin Bauer is among the benchmark's senior reviewers.
