Back to benchmarks
Version 4.0
View all models
Agentic Coding
Open Benchmarks Grants

Terminal-Bench 4.0

A benchmark to measure and evolve with the frontier of agent work: real terminal environments, real software engineering tasks, and a rolling task set that is revised as agents catch up to it. Version 4.0 trims the 3.0 task set from 74 to 66 tasks, removing eight and revising 20.

Built with
Image
Image
Image
Image
Resources
Release Notes
Harbor Hub
Website

Overview

Terminal-Bench measures whether an AI agent can operate a real terminal to complete real software engineering and systems tasks: writing and debugging code, configuring environments, and recovering from failure, all without a GUI. Version 3.0 introduced the current 74 task families; version 4.0 is a maintenance release that removes eight tasks and revises twenty, netting 66 tasks, with no new tasks added in this cycle.

Headline finding, in this evaluation: Claude Opus 5 (Claude Code, max effort) leads at 51.8% ± 3.4% resolution, roughly 7 points ahead of Claude Fable 5 in second place. The bottom of the ranked field, Grok 4.5 and Claude Sonnet 5, both land at 12.4%, a spread of nearly 40 points across the ten evaluated model and agent configurations

At a glance

66

tasks in v4.0.0, down from 74 in v3.0.0

8

tasks removed in v4.0.0

20

tasks revised in v4.0.0

Leaderboard

Rank Model Effort Agent Resolution Rate Release Date Tokens Cost
1 GPT-6 Astra max Codex
58.2% ±2.8
2026-09-03 1.5B $3.3k
2 Fable 5.1 max Claude Code
57.9% ±3.8
2026-09-01 2.7B $6.2k
3 Opus 5 xhigh Claude Code
53.9% ±3.2
2026-07-24 6.9B $6.1k
4 Fable 5 max Claude Code
44.5% ±3.8
2026-06-09 3.8B $7.3k
5 GLM-5.3 max Claude Code
41.8% ±3.2
2026-08-14 8.7B $2.7k
6 GPT-5.6 Sol max Codex
37.3% ±3.8
2026-06-26 4.4B $2.5k
7 Opus 4.8 max Claude Code
23.6% ±3.6
2026-05-28 6.4B $6.5k
8 GPT-5.6 Terra max Codex
21.5% ±3.3
2026-06-26 5.7B $1.7k
9 Grok 4.6 high Grok Build
20.3% ±3.1
2026-08-12 4.0B $3.6k
10 Gemini 3.8 Flash high mini-SWE-agent
19.1% ±3.4
2026-09-02 17.2B $1.8k
11 GPT-5.6 Luna max Codex
17.3% ±2.8
2026-06-26 11.6B $0.3k
12 Grok 4.5 high Grok Build
12.4% ±2.6
2026-07-16 3.4B $2.1k
13 Sonnet 5 max Claude Code
12.4% ±3.1
2026-06-30 21.6B $9.6k
14 Gemini 3.7 Flash high mini-SWE-agent
11.2% ±2.4
2026-08-13 11.1B $1.3k

Terminal-Bench 4.0 Pareto Frontier

Loading chart data...
Chart data visualization

How Terminal-Bench 4.0 compares

Several recent benchmarks make progress on behavioral testing and instruction realism. The following table provides a brief comparison.

Benchmark

Released

Tasks

Change

Terminal-Bench 4.0

Aug 2026

66

8 tasks removed, 20 revised, 0 added

Terminal-Bench 3.0

Jul 2026

74

New rolling task set (formerly branded Frontier-Bench); initial 74 tasks added

Terminal-Bench 2.1

May 2026

89

Revision fixing 28 tasks; task count unchanged

Terminal-Bench 2.0

Nov 2025

89

Harder, curated task set; introduced Harbor, the agent eval and optimization framework

Terminal-Bench 1.0

May 2025

80

Initial release (Terminal-Bench-Core), led by Stanford and the Laude Institute

Methodology

Resolution rate

Share of the 66 v4.0.0 tasks each agent and model configuration resolves, reported with a margin reflecting run—to—run variance. “Effort” (max, high, and similar) is the harness’s own reasoning or agent effort setting for that run, not a Terminal-Bench parameter.
Cost and tokens
Totals are for the full evaluation run across all 66 tasks, not per task; they are not normalized for effort setting, so runs with a higher effort setting (like Sonnet 5’s) show substantially higher token and cost totals without a proportional gain in resolution rate.
Contamination control

The public Terminal-Bench site marks its task and leaderboard data with a canary string, requesting that benchmark data not appear in model training corpora.

Run command

Evaluated via the Harbor evaluation harness: harbor run -d terminal-bench/terminal-bench@4.0.0. Dataset pinned at hub.harborframework.com.

From the blog

Image for Terminal-Bench 4.0: Why Continuous Benchmarks Require Continuous QA
Research

Terminal-Bench 4.0: Why Continuous Benchmarks Require Continuous QA

The speed of new frontier model releases keeps accelerating. Meanwhile benchmarks struggle to keep up and saturate quickly, often being...
Justin Bauer Headshot
Justin Bauer
August 27, 2026

Resources

Release Notes
GitHub

Discord

Website

Acknowledgments

Terminal-Bench is hosted by Harbor and the Laude Institute, and supported by Snorkel AI via the Open Benchmarks Grants program. The v4.0.0 release was authored by RyanMarten and Snorkel's Justin Bauer is among the benchmark's senior reviewers. 

FAQs

Get notified when we launch a new benchmark

Share this benchmark

More benchmarks

Explore all
Agentic Coding

Agentic Coding 2.0

Evaluating whether coding agents can plan, execute, verify, and recover across complex terminal-native engineering tasks.

By pass@1
1
Image
GPT-6 Astra
47.6%
2
Image
Fable 5.1
39.6%
3
Image
Opus 5
38.9%
Software Engineering

SWE-bench CLI

Tests whether an AI coding agent can diagnose and deliver a validated, multi-file change in a real open-source repository, navigating code, tests, dependencies, and tooling through the command line.

By pass@1
1
Image
Fable 5.1
14.5%
2
Image
Opus 5
14.0%
3
Image
GPT-6 Astra
12.4%
Enterprise Environments

SnorkelUnderwrite 2.0

Measures an agent’s ability to turn incomplete, distributed insurance evidence into an auditable underwriting decision while respecting authority and policy constraints.

By pass@1
1
Image
GLM 5.3
30.4%
2
Image
DeepSeek V4 Pro
28.5%
3
Image
Kimi K3
27.9%
Enterprise Environments

SnorkelManufacturing

Measures how well AI agents can turn fragmented plant-floor, engineering, and supplier evidence into safe, technically defensible decisions and actions.

By pass@1
1
Image
Grok 4.6
16.4%
2
Image
GLM 5.3
11.9%
3
Image
Fable 5.1
11.3%
Enterprise Environments

SnorkelRevOps

Tests whether AI agents can reconcile revenue systems, enforce hard commercial controls, and carry a decision through to the correct business state.

By pass@1
1
Image
Grok 4.6
15.8%
2
Image
Fable 5.1
14.0%
3
Image
Kimi K3
13.1%
Enterprise Environments

SnorkelFinance 2.0

Scores how agents gather evidence, perform financial analysis, follow compliance constraints, and complete required state updates in a simulated environment.

By pass@1
1
Image
Grok 4.6
25.1%
2
Image
GLM 5.3
21.2%
3
Image
Fable 5.1
19.6%
Enterprise Environments

SnorkelLegal

Evaluates AI agents on whether they can advance a legal matter while preserving the chain from controlling evidence to procedure, authority, and action.

By pass@1
1
Image
Grok 4.6
40.7%
2
Image
GLM 5.3
37.8%
3
Image
Muse Spark 1.3
36.5%
Knowledge Work

WorkplaceAgents

Evaluates AI agents on economically valuable professional work and verifiable workplace deliverables.

By pass@1
1
Image
Muse Spark 1.3
17.7%
2
Image
Grok 4.6
17.1%
3
Image
Fable 5.1
15.8%
Scientific & Research Workflows

Terminal-Bench-Science

A benchmark for evaluating AI agents on workflows from researchers’ own work. Scientists, not model developers or vendors, set the bar for scientific capability in AI.

By Resolution Rate
1
Image
Fable 5.1
40.0%
2
Image
Opus 5
30.0%
3
Image
GPT-5.6 Sol
22.4%
Agentic Coding

Terminal-Bench 3.0

The next frontier benchmark for agent work. A harder, more domain-diverse successor to Terminal-Bench 2.1 — built in the open, task by task, under continuous adversarial review.

By Resolution Rate
1
Image
Opus 5
42.7%
2
Image
GPT-5.6 Sol
34.6%
3
Image
Fable 5
34.1%
Computer Use

OSWorld 2.0

Long-horizon professional workflows with verifiable outcomes across 55 sub-industries. 147 public tasks of a 1,500+ task corpus, sourced and validated by 300+ industry experts.

By binary accuracy (500 steps)
1
Image
Opus 5 · max
44.33%
2
Image
Opus 5 · xhigh
36.89%
3
Image
Opus 5 · high
33.33%
Software Engineering

Senior SWE-Bench

Evaluating coding agents on senior-level engineering work.

Tasteful Solve Rate
1
Image
Fable 5.1
34.7%
2
Image
Fable 5
34.7%
3
Image
Opus 5
34.7%
Knowledge Work

Agents’ Last Exam

Evaluating AI agents on long-horizon, economically valuable professional workflows with verifiable outcomes.

By Binary Accuracy
1
Image
GPT-6 Astra
34.2
2
Image
Muse Spark 1.3
32.2%
3
Image
Opus 5
31.6%
Software Engineering

SlopCode Bench

Measures code quality degradation in AI-assisted codebases. Tracks checkpoint solve rates, erosion (code bloat), and verbosity under realistic repo conditions.

Top Models by Iso Solve
1
Image
GPT-5.5
28.06%
2
Image
GPT-5.3-Codex
26.02%
3
Image
GPT-5.4
23.47%
Capability/Efficiency

Continual Learning Bench

Evaluates whether AI systems improve from prior experience across sequential, stateful tasks, measuring real in-context learning, not just raw capability.

Top Systems (Agg. Reward)
1
Image
Sonnet 4.6 · ICL
+0.196
2
Image
GPT-5.4 · ICL
+0.189
3
Image
Sonnet 4.6 · Claude Code
+0.185
of

For models that need to be right. Not just good enough.

Request dataset samples
Explore research