Science & Research

Open Benchmarks Grants

Terminal-Bench-Science

Terminal-Bench-Science evaluates AI agents on workflows from researchers' own work. Scientists, not model developers or vendors, set the bar for scientific capability in AI.

Built with

Resources

At a glance

70

tasks in the v0.1 release

5

domain areas covered

40%

top resolution rate (Fable 5.1)

Key takeaways

Fable 5.1 leads Terminal-Bench-Science at 40.0% resolution — the strongest result in this release — with Opus 5 second at 30.0%. Both scores fall well below the 50–80% range frontier models reach on general coding benchmarks, underscoring that authentic scientific workflows remain considerably harder for agents than general software engineering.

Leaderboard

Rank	Model	Agent	Effort	Resolution Rate	Release Date	Tokens	Cost
1	GPT-6 Astra	Codex	max	68.1% ±3.2	2026-09-03	2.3B	$4,997
2	Fable 5.1	Claude Code	max	40% ±3.4	2026-09-01	3.8B	$6.25K
3	Opus 5	Claude Code	max	30% ±3.2	2026-07-24	7.3B	$6.99K
4	GPT-5.6 Sol	Codex	max	22.4% ±2.9	2026-07-09	8.4B	$4.22K
5	Fable 5	Claude Code	max	21.4% ±2.8	2026-06-09	6.4B	$14.18K
6	DeepSeek V4.1 Flash	Codex	max	15.7% ±2.5	2026-09-10	14.9B	$385.93
7	Grok 4.7	Grok Build	xhigh	14.3% ±2.4	2026-09-21	4.2B	$3,071
8	Gemini 3.8 Flash	mini-SWE-agent	high	12.4% ±2.3	2026-09-02	10.5B	$1.12K
9	Opus 4.8	Claude Code	max	10.5% ±2.1	2026-05-28	6.8B	$5.84K
10	GPT-5.6 Terra	Codex	max	8.6% ±1.9	2026-07-09	7.6B	$1.98K
11	GLM 5.3	Claude Code	max	8.1% ±1.9	2026-08-14	8.5B	$2.73K
12	Kimi K3	Claude Code	max	7.1% ±1.8	2026-07-16	3.2B	$1.52K
13	Grok 4.6	Grok Build	xhigh	7.1% ±1.8	2026-08-12	3.7B	$3.34K
14	Gemini 3.7 Flash	mini-SWE-agent	high	5.7% ±1.6	2026-08-13	12.0B	$1.30K
15	DeepSeek V4 Pro	Codex	max	3.8% ±1.3	2026-08-13	11.2B	$1,120.15
16	GPT-5.6 Luna	Codex	max	3.3% ±1.2	2026-07-09	14.2B	$383.20

Rank	Model	Agent	Effort	Resolution Rate	Release Date	Tokens	Cost
1	GPT-6 Astra	Codex	max	59.6% ±6.5	2026-09-03	499.9M	$1.03K
2	Fable 5.1	Claude Code	max	38.6% ±6.4	2026-09-01	751.4M	$1.11K
3	Opus 5	Claude Code	max	29.8% ±6.1	2026-07-24	1.3B	$1.32K
4	GPT-5.6 Sol	Codex	max	17.5% ±5	2026-07-09	911.2M	$513.19
5	Fable 5	Claude Code	max	17.5% ±5	2026-06-09	864.3M	$1.96K
6	Gemini 3.8 Flash	mini-SWE-agent	high	14% ±4.6	2026-09-02	947.5M	$129.49
7	Grok 4.7	Grok Build	xhigh	12.3% ±4.3	2026-09-21	516.7M	$377.22
8	GLM 5.3	Claude Code	max	10.5% ±4.1	2026-08-14	1.5B	$1.31K
9	DeepSeek V4.1 Flash	Codex	max	10.5% ±4.1	2026-09-10	2.5B	$64.23
10	Gemini 3.7 Flash	mini-SWE-agent	high	8.8% ±3.7	2026-08-13	704.8M	$100.54
11	Grok 4.6	Grok Build	xhigh	7% ±3.4	2026-08-12	367.5M	$281.65
12	GPT-5.6 Terra	Codex	max	7% ±3.4	2026-07-09	1B	$297.96
13	Kimi K3	Claude Code	max	7% ±3.4	2026-07-16	612.8M	$591.34
14	Opus 4.8	Claude Code	max	7% ±3.4	2026-05-28	780.5M	$806.35
15	GPT-5.6 Luna	Codex	max	1.8% ±1.7	2026-07-09	2.9B	$83.51
16	DeepSeek V4 Pro	Codex	max	0%	2026-08-13	2.1B	$107.43

Rank	Model	Agent	Effort	Resolution Rate	Release Date	Tokens	Cost
1	GPT-6 Astra	Codex	max	58.8% ±6.9	2026-09-03	558.1M	$1.22K
2	Fable 5.1	Claude Code	max	37.3% ±6.8	2026-09-01	1B	$1.82K
3	Opus 5	Claude Code	max	27.5% ±6.2	2026-07-24	2.1B	$1.85K
4	DeepSeek V4.1 Flash	Codex	max	27.5% ±6.2	2026-09-10	4.6B	$118.82
5	Gemini 3.8 Flash	mini-SWE-agent	high	23.5% ±5.9	2026-09-02	2.8B	$288.67
6	GPT-5.6 Sol	Codex	max	23.5% ±5.9	2026-07-09	2.4B	$1.14K
7	Opus 4.8	Claude Code	max	21.6% ±5.8	2026-05-28	2B	$1.6K
8	Fable 5	Claude Code	max	17.6% ±5.3	2026-06-09	1.5B	$3.58K
9	Grok 4.7	Grok Build	xhigh	15.7% ±5.1	2026-09-21	913.1M	$680.14
10	Kimi K3	Claude Code	max	11.8% ±4.5	2026-07-16	799.3M	$536.23
11	GPT-5.6 Terra	Codex	max	9.8% ±4.2	2026-07-09	3.3B	$791.88
12	GLM 5.3	Claude Code	max	9.8% ±4.2	2026-08-14	2.3B	$1.73K
13	Gemini 3.7 Flash	mini-SWE-agent	high	7.8% ±3.8	2026-08-13	4.7B	$490.74
14	Grok 4.6	Grok Build	xhigh	5.9% ±3.3	2026-08-12	646.9M	$539.59
15	DeepSeek V4 Pro	Codex	max	3.9% ±2.7	2026-08-13	2.8B	$140.45
16	GPT-5.6 Luna	Codex	max	0%	2026-07-09	4.8B	$116.84

Rank	Model	Agent	Effort	Resolution Rate	Release Date	Tokens	Cost
1	GPT-6 Astra	Codex	max	79.2% ±8.3	2026-09-03	293.3M	$595.78
2	Opus 5	Claude Code	max	45.8% ±10.2	2026-07-24	897.5M	$1.15K
3	Fable 5.1	Claude Code	max	37.5% ±9.9	2026-09-01	431.2M	$746.48
4	Grok 4.7	Grok Build	xhigh	25% ±8.8	2026-09-21	437.3M	$314.11
5	Fable 5	Claude Code	max	25% ±8.8	2026-06-09	602.6M	$1.62K
6	GPT-5.6 Sol	Codex	max	20.8% ±8.3	2026-07-09	1.2B	$584.56
7	Gemini 3.8 Flash	mini-SWE-agent	high	12.5% ±6.8	2026-09-02	1.1B	$117.58
8	Kimi K3	Claude Code	max	12.5% ±6.8	2026-07-16	380.1M	$342.60
9	GPT-5.6 Terra	Codex	max	8.3% ±5.6	2026-07-09	1.2B	$302.41
10	DeepSeek V4.1 Flash	Codex	max	8.3% ±5.6	2026-09-10	1.8B	$39.17
11	Grok 4.6	Grok Build	xhigh	4.2% ±4.1	2026-08-12	244.9M	$217.33
12	Opus 4.8	Claude Code	max	4.2% ±4.1	2026-05-28	753.7M	$694.26
13	GLM 5.3	Claude Code	max	4.2% ±4.1	2026-08-14	1.6B	$1.4K
14	GPT-5.6 Luna	Codex	max	0%	2026-07-09	1.6B	$41.41
15	Gemini 3.7 Flash	mini-SWE-agent	high	0%	2026-08-13	503.6M	$63.51
16	DeepSeek V4 Pro	Codex	max	0%	2026-08-13	1.5B	$71.82

Rank	Model	Agent	Effort	Resolution Rate	Release Date	Tokens	Cost
1	GPT-6 Astra	Codex	max	70.4% ±8.8	2026-09-03	275.5M	$698.22
2	Fable 5.1	Claude Code	max	48.1% ±9.6	2026-09-01	633.4M	$976.65
3	Opus 5	Claude Code	max	29.6% ±8.8	2026-07-24	1.2B	$1.04K
4	Grok 4.7	Grok Build	xhigh	25.9% ±8.4	2026-09-21	464.6M	$343.58
5	DeepSeek V4.1 Flash	Codex	max	18.5% ±7.5	2026-09-10	3B	$69.75
6	Grok 4.6	Grok Build	xhigh	14.8% ±6.8	2026-08-12	276.7M	$231.57
7	GPT-5.6 Sol	Codex	max	14.8% ±6.8	2026-07-09	569.2M	$315.06
8	Gemini 3.8 Flash	mini-SWE-agent	high	11.1% ±6	2026-09-02	520.2M	$70.26
9	DeepSeek V4 Pro	Codex	max	11.1% ±6	2026-08-13	2.2B	$108.40
10	Fable 5	Claude Code	max	11.1% ±6	2026-06-09	1.3B	$2.39K
11	GPT-5.6 Luna	Codex	max	7.4% ±5	2026-07-09	1.2B	$35.82
12	Gemini 3.7 Flash	mini-SWE-agent	high	7.4% ±5	2026-08-13	377.2M	$56.41
13	GPT-5.6 Terra	Codex	max	7.4% ±5	2026-07-09	464.1M	$141.00
14	GLM 5.3	Claude Code	max	7.4% ±5	2026-08-14	992.1M	$788.27
15	Opus 4.8	Claude Code	max	7.4% ±5	2026-05-28	1.1B	$831.49
16	Kimi K3	Claude Code	max	3.7% ±3.6	2026-07-16	536.3M	$354.73

Rank	Model	Agent	Effort	Resolution Rate	Release Date	Tokens	Cost
1	GPT-6 Astra	Codex	max	80.4% ±5.6	2026-09-03	661.6M	$1.45K
2	Fable 5.1	Claude Code	max	41.2% ±6.9	2026-09-01	891.4M	$1.6K
3	Fable 5	Claude Code	max	33.3% ±6.6	2026-06-09	2.1B	$4.63K
4	GPT-5.6 Sol	Codex	max	31.4% ±6.5	2026-07-09	3.3B	$1.66K
5	Opus 5	Claude Code	max	25.5% ±6.1	2026-07-24	1.7B	$1.63K
6	DeepSeek V4.1 Flash	Codex	max	11.8% ±4.5	2026-09-10	2.9B	$93.97
7	GPT-5.6 Terra	Codex	max	9.8% ±4.2	2026-07-09	1.6B	$450.66
8	GPT-5.6 Luna	Codex	max	7.8% ±3.8	2026-07-09	3.7B	$105.62
9	Opus 4.8	Claude Code	max	7.8% ±3.8	2026-05-28	2.2B	$1.9K
10	DeepSeek V4 Pro	Codex	max	5.9% ±3.3	2026-08-13	2.7B	$146.13
11	GLM 5.3	Claude Code	max	5.9% ±3.3	2026-08-14	2.1B	$1.57K
12	Grok 4.6	Grok Build	xhigh	5.9% ±3.3	2026-08-12	2.2B	$2.07K
13	Grok 4.7	Grok Build	xhigh	3.9% ±2.7	2026-09-21	1.9B	$1.36K
14	Gemini 3.7 Flash	mini-SWE-agent	high	2% ±1.9	2026-08-13	5.7B	$585.39
15	Kimi K3	Claude Code	max	2% ±1.9	2026-07-16	863.2M	$714.76
16	Gemini 3.8 Flash	mini-SWE-agent	high	0%	2026-09-02	5.2B	$510.98

TAXONOMY

Domain areas

The v0.1 release spans five domain areas, unevenly weighted toward life and physical sciences. Resolution rates above are aggregated across all 70 tasks.

19

Life sciences

17

Physical sciences

8

Earth sciences

17

Mathematical sciences

9

Engineering sciences

Sample tasks

A sample of task titles from the v0.1 release. Browse the full contribution pipeline on the task dashboard →

Applied Mathematics / Physics

GRMHD primitive variable recovery

Recover physical (primitive) variables from conserved quantities in ideal general-relativistic magnetohydrodynamics under a strict double-precision arithmetic policy, graded against a hidden verifier corpus beyond the public sample cases.

(opens in a new tab)

Operations Research

Matrix-Free Computation of the Smallest Symplectic Eigenpairs

Implement a matrix-free solver for the invariant subspace of the smallest symplectic eigenvalues of a large implicit positive-definite operator, graded on symplectic feasibility, first-order stationarity, and certified-subspace accuracy under oracle, time, and memory limits.

(opens in a new tab)

Statistics / Ocean Sciences

Honest posterior intervals for a mixed-layer ocean model

Produce honest Bayesian posterior intervals from correlated, instrument-biased ocean temperature-profile residuals, where a naive independent-Gaussian likelihood passes standard convergence diagnostics while returning intervals several times too narrow.

(opens in a new tab)

Biology

Add task: crispr-mhcii-screen

Analyze deposited CRISPR-Cas9 pooled-screen count matrices from unequal-depth melanoma MHC-II FACS replicates to produce a defensible gene hit list, with no processed reference hit list provided.

(opens in a new tab)

Biology

Genome-to-phenotype prediction under multi-cohort lineage shift

Identify genuine causal genome-to-phenotype markers across six family-disjoint cohorts where decoy modules are statistically identical to genuine ones in labeled data but independently rewired per deployment cohort, defeating ordinary cross-validation.

(opens in a new tab)

Physics

Self-Consistent Magnetic Phase Boundaries in Correlated Moiré Models

Solve for self-consistent magnetic phase boundaries in spin-orbit-coupled moiré Hamiltonians, requiring simultaneous convergence of coupled impurity self-energies and structural relaxation with general spin mixing.

(opens in a new tab)

Chemistry

Hydration free energy calculation of a solute model

Compute a solute's hydration free energy despite a slow conformational degree of freedom orthogonal to the alchemical variable, where a routine calculation gets trapped in one metastable state and reports a confidently wrong, deceptively low-uncertainty result.

(opens in a new tab)

Astronomy

Fast Radio Burst Reconstruction

Detect and reconstruct a fast radio burst from noisy time-series data under strict timing and precision requirements, verified against dispersion-measure and pulse-width tolerances.

(opens in a new tab)

Review pipeline

How tasks are graded

Every task in Terminal-Bench-Science goes through a review pipeline before it is eligible to be merged into a release, and every evaluation run separates the agent’s working environment from the environment that grades it.

Proposal

A domain expert or contributor proposes a task drawn from real research work.

LLM Judge

An automated judge screens the proposal for basic feasibility and completeness.

Reviewer

A first-pass human reviewer checks instruction clarity and task scope.

Senior Reviewer

A senior reviewer checks instruction and verifier alignment, task realism, and under or over specification risk.

Pull Request

The task is opened as a pull request against the public task repository.

Static Checks

Automated checks validate task structure, environment definitions, and verifier code.

Agent Judge

Frontier, oracle, and cheating-agent trial runs confirm the task is solvable, gradeable, and resistant to shortcuts.

Agent Trials

Additional agent trial runs stress test the task before it is accepted.

Merge

The task is merged into the public task set and becomes eligible for a future release.

Resources

Discord

Behind the benchmark

Terminal-Bench-Science evaluates agents on real scientific research workflows rather than software repository tasks. Version 0.1, the benchmark's first public release, covers 70 tasks across five domain areas, each sourced from work a practicing researcher actually performed.

Acknowledgments

Terminal-Bench-Science is a collaboration between Stanford University, the Laude Institute, and domain experts across scientific disciplines and research institutions worldwide, led by Steven Dillmann (Stanford University), Sanmi Koyejo (Stanford University), and Ludwig Schmidt (Stanford University, Anthropic)

Snorkel supports Terminal-Bench Science through the Open Benchmarks Grants program.

FAQs

Terminal-Bench-Science is a benchmark that evaluates AI agents on real computational workflows from scientific research, extending Terminal-Bench (adopted by frontier labs including Anthropic, OpenAI, and Google DeepMind for software engineering evaluation) into the natural sciences. Tasks run in containerized environments and are checked through programmatic, task-specific verification rather than open-ended judgment.

Each task starts as a proposal, which an LLM judge screens against a task-proposal rubric before human reviewers approve it for implementation. The implementation is submitted as a pull request, evaluated against a task-implementation rubric, and assessed by domain and technical reviewers for scientific validity, realism, and verification quality. Before merge, frontier, oracle, and cheating-agent trial runs confirm the task is solvable, gradeable, and resistant to shortcuts.

Snorkel supports Terminal-Bench-Science through its Open Benchmarks Grants program, which funds open-source AI research and reproducible evaluation, and helped surface the original call for scientist contributors ahead of this release. Terminal-Bench-Science is otherwise an open academic collaboration hosted by Stanford University and the Laude Institute, built by the Terminal-Bench and Harbor Framework team together with scientific contributors.