Back to benchmarks
View all models
Software Engineering

SWE-bench CLI

Tests whether an AI coding agent can diagnose and deliver a validated, multi-file change in a real open-source repository, navigating code, tests, dependencies, and tooling through the command line.

overview

SWE-bench CLI evaluates whether an agent can complete substantive software engineering work inside an unfamiliar repository. Tasks require agents to inspect the codebase, understand a change request, modify multiple files, run tests, diagnose failures, and leave the repository in a verified state.

The benchmark spans backend systems, developer tooling, front-end interfaces, language runtimes, networking, storage, infrastructure, machine learning, scientific computing, and security.

At a glance

200

frontier tasks

11

languages

10

technical domains

8

task types

Leaderboard

Rank Model Pass@1 Pass@5 Cost / Trial Cost / Task
1 Fable 5.1
14.5%
26.2%
$12.99 $64.93
2 Claude Opus 5
14%
24.7%
$23.73 $118.64
3 GPT-6 Astra
12.4%
18.8%
$2.16 $10.8
4 Gemini Flash 3.8
6.3%
14.1%
$4.61 $23.06
5 Grok 4.6
5.7%
13%
$3.3 $16.5
6 Qwen 3.8 Max
4.9%
10.5%
$3.15 $15.77
7 Kimi K3
4.1%
11.9%
$6.09 $30.45
8 GLM 5.3
3.5%
10.8%
$8.19 $40.96
9 DeepSeek V4 Pro
2.3%
5.2%
$2.2 $10.98
10 Muse Spark 1.3
2.2%
5.7%
$5.09 $25.47
11 Nemotron 3 Ultra 550B
0.5%
1.1%
$4.62 $23.11
icon bulb

Want to evaluate your model against SWE-bench CLI? Talk to our team

Frontier performance

Loading chart data...
Chart data visualization

Methodology

Evaluator

Harbor runs repository-level fail-to-pass and pass-to-pass test suites against the agent’s resulting code. Task metadata and rubric checks provide additional requirements for correctness.
timeout
Task-specific agent and verifier timeouts bound each CLI trace. All evaluations run in reproducible, network-disabled Docker environments.
integration

Each task includes a containerized repository, natural-language problem statement, reference patch, test configuration, and CLI-accessible development environment.

scoring note

The task reward is binary. A task receives credit only when all required tests pass. Golden changes must represent substantive engineering work spanning at least two files.

Behind the benchmark

Many coding evaluations reduce software engineering to generating a plausible patch. Real repository work requires navigating unfamiliar code, tracing behavior across files, selecting the right tests, diagnosing failures, and preserving existing functionality.

SWE-bench CLI focuses on that complete workflow. It measures whether an agent can operate as a software engineer inside a real codebase, not merely produce code that looks correct in isolation.

More benchmarks

Explore all
Agentic Coding

Agentic Coding 2.0

Evaluating whether coding agents can plan, execute, verify, and recover across complex terminal-native engineering tasks.

By pass@1
1
Image
GPT-6 Astra
47.6%
2
Image
Fable 5.1
39.6%
3
Image
Opus 5
38.9%
Enterprise Environments

SnorkelUnderwrite 2.0

Measures an agent’s ability to turn incomplete, distributed insurance evidence into an auditable underwriting decision while respecting authority and policy constraints.

By pass@1
1
Image
GLM 5.3
30.4%
2
Image
DeepSeek V4 Pro
28.5%
3
Image
Kimi K3
27.9%
Enterprise Environments

SnorkelManufacturing

Measures how well AI agents can turn fragmented plant-floor, engineering, and supplier evidence into safe, technically defensible decisions and actions.

By pass@1
1
Image
Grok 4.6
16.4%
2
Image
GLM 5.3
11.9%
3
Image
Fable 5.1
11.3%
Enterprise Environments

SnorkelRevOps

Tests whether AI agents can reconcile revenue systems, enforce hard commercial controls, and carry a decision through to the correct business state.

By pass@1
1
Image
Grok 4.6
15.8%
2
Image
Fable 5.1
14.0%
3
Image
Kimi K3
13.1%
Enterprise Environments

SnorkelFinance 2.0

Scores how agents gather evidence, perform financial analysis, follow compliance constraints, and complete required state updates in a simulated environment.

By pass@1
1
Image
Grok 4.6
25.1%
2
Image
GLM 5.3
21.2%
3
Image
Fable 5.1
19.6%
Enterprise Environments

SnorkelLegal

Evaluates AI agents on whether they can advance a legal matter while preserving the chain from controlling evidence to procedure, authority, and action.

By pass@1
1
Image
Grok 4.6
40.7%
2
Image
GLM 5.3
37.8%
3
Image
Muse Spark 1.3
36.5%
Knowledge Work

WorkplaceAgents

Evaluates AI agents on economically valuable professional work and verifiable workplace deliverables.

By pass@1
1
Image
Muse Spark 1.3
17.7%
2
Image
Grok 4.6
17.1%
3
Image
Fable 5.1
15.8%
Agentic Coding

Terminal-Bench 4.0

A benchmark to measure and evolve with the frontier of agent work: real terminal environments, real software engineering tasks, and a rolling task set that is revised as agents catch up to it.

By Resolution Rate
1
Image
GPT-6 Astra
58.2%
2
Image
Fable 5.1
57.9%
3
Image
Opus 5
53.9%
Scientific & Research Workflows

Terminal-Bench-Science

A benchmark for evaluating AI agents on workflows from researchers’ own work. Scientists, not model developers or vendors, set the bar for scientific capability in AI.

By Resolution Rate
1
Image
Fable 5.1
40.0%
2
Image
Opus 5
30.0%
3
Image
GPT-5.6 Sol
22.4%
Agentic Coding

Terminal-Bench 3.0

The next frontier benchmark for agent work. A harder, more domain-diverse successor to Terminal-Bench 2.1 — built in the open, task by task, under continuous adversarial review.

By Resolution Rate
1
Image
Opus 5
42.7%
2
Image
GPT-5.6 Sol
34.6%
3
Image
Fable 5
34.1%
Computer Use

OSWorld 2.0

Long-horizon professional workflows with verifiable outcomes across 55 sub-industries. 147 public tasks of a 1,500+ task corpus, sourced and validated by 300+ industry experts.

By binary accuracy (500 steps)
1
Image
Opus 5 · max
44.33%
2
Image
Opus 5 · xhigh
36.89%
3
Image
Opus 5 · high
33.33%
Software Engineering

Senior SWE-Bench

Evaluating coding agents on senior-level engineering work.

Tasteful Solve Rate
1
Image
Fable 5.1
34.7%
2
Image
Fable 5
34.7%
3
Image
Opus 5
34.7%
Knowledge Work

Agents’ Last Exam

Evaluating AI agents on long-horizon, economically valuable professional workflows with verifiable outcomes.

By Binary Accuracy
1
Image
GPT-6 Astra
34.2
2
Image
Muse Spark 1.3
32.2%
3
Image
Opus 5
31.6%
Software Engineering

SlopCode Bench

Measures code quality degradation in AI-assisted codebases. Tracks checkpoint solve rates, erosion (code bloat), and verbosity under realistic repo conditions.

Top Models by Iso Solve
1
Image
GPT-5.5
28.06%
2
Image
GPT-5.3-Codex
26.02%
3
Image
GPT-5.4
23.47%
Capability/Efficiency

Continual Learning Bench

Evaluates whether AI systems improve from prior experience across sequential, stateful tasks, measuring real in-context learning, not just raw capability.

Top Systems (Agg. Reward)
1
Image
Sonnet 4.6 · ICL
+0.196
2
Image
GPT-5.4 · ICL
+0.189
3
Image
Sonnet 4.6 · Claude Code
+0.185
of

For models that need to be right. Not just good enough.

Request dataset samples
Explore research