Senior SWE-Bench
A benchmark for evaluating coding agents on senior-level engineering work: building features from realistic instructions, investigating bugs that require runtime investigation, and shipping code that aligns to existing codebase conventions.
# Set env vars required for your model # export ANTHROPIC_API_KEY=sk-ant-... # export OPENAI_API_KEY=sk-proj-... # export MY_PROVIDERS_API_KEY=... # [Optional] Set the models for the test stage (defaults below) # export SSB_OVERRIDE_VA_HARNESS=miniswebench # export SSB_OVERRIDE_VA_MODEL=anthropic/claude-sonnet-4-6 # export SSB_OVERRIDE_ALL_JUDGE_MODEL=anthropic/claude-sonnet-4-6 # export SSB_OVERRIDE_CLASSIFIER_MODEL=anthropic/claude-haiku-4-5 # Set depending on what you want to run MODEL=anthropic/claude-opus-4-8 AGENT=mini-swe-agent # Run Harbor harbor run --repo snorkel-ai/senior-swe-bench -a $AGENT -m $MODEL
At a glance
100
tasks (50 public/ 50 private)
12
open-source repositories
6
evaluation gates per task
15+
agents evaluated
100+
PR-author commits required
Frontier performance
Key takeaways
Fable 5.1, Fable 5, Opus 5, and GPT-5.6 Sol — are tied at the top of Senior SWE-Bench with a 34.7% tasteful solve rate, well above the sub-25% ceiling reported in earlier evaluations. Even so, top-performing frontier models still fail to complete tasks with senior-level correctness and taste more than 65% of the time.
Leaderboard
|
Rank
|
Model
|
Harness
|
Effort
|
Tasteful Solve Rate (pass@1)
|
Basic Solve Rate (pass@1)
|
Avg Steps
|
Avg Tokens
|
|---|---|---|---|---|---|---|---|
| 1 | Fable 5.1 | Mini-SWE-Agent | medium |
34.7%
|
57.9%
|
77 | 37.9K |
| 2 | Fable 5 | Mini-SWE-Agent | high |
34.7%
|
53.7%
|
119 | 58.4K |
| 3 | Opus 5 | Mini-SWE-Agent | high |
34.7%
|
62.1%
|
141 | 71.0K |
| 4 | GPT-5.6 Sol | Mini-SWE-Agent | xhigh |
34.7%
|
53.7%
|
50 | 32.7K |
| 5 | Opus 4.8 | Mini-SWE-Agent | max |
30.5%
|
44.2%
|
138 | 134.2K |
| 6 | GPT-5.5 | Mini-SWE-Agent | xhigh |
29.5%
|
52.6%
|
85 | 36.1K |
| 7 | Sonnet 5 | Mini-SWE-Agent | max |
28%
|
50.5%
|
318 | 180.7K |
| 8 | Grok 4.7 | Mini-SWE-Agent | xhigh |
27.4%
|
49.5%
|
150 | 39.4K |
| 9 | GPT-5.6 Terra | Mini-SWE-Agent | high |
27.4%
|
36.8%
|
36 | 18.2K |
| 10 | Grok 4.6 | Mini-SWE-Agent | high |
26.3%
|
51.6%
|
125 | 23.9K |
| 11 | Grok 4.5 | Mini-SWE-Agent | medium |
23.2%
|
50.5%
|
72 | 21.2K |
| 12 | MiniMax M3 | Mini-SWE-Agent | adaptive |
21.1%
|
41.1%
|
335 | 75.8K |
| 13 | Kimi K3 | Mini-SWE-Agent | max |
20.2%
|
40.4%
|
224 | 54.8K |
| 14 | Opus 4.7 | Mini-SWE-Agent | max |
18.1%
|
46.8%
|
285 | 100.9K |
| 15 | GLM-5.2 | Mini-SWE-Agent | max |
17.9%
|
35.8%
|
194 | 66.2K |
| 16 | GPT-5.4 | Mini-SWE-Agent | xhigh |
16.8%
|
41.1%
|
86 | 53.9K |
| 17 | Gemini 3.7 Flash | Mini-SWE-Agent | high |
14.7%
|
44.2%
|
254 | 47.3K |
| 18 | Gemini 3.5 Flash | Mini-SWE-Agent | high |
6.3%
|
22.1%
|
231 | 70.4K |
| 19 | Gemini 3.1 Pro | Mini-SWE-Agent | high |
2.1%
|
9.5%
|
108 | 17.3K |
Basic solve rate is the share of an agent's runs that pass every pre-written verifier and automated validation test. Tasteful solve rate requires all of that and clears every additional quality gate: rubric, bloat, codebase practice, and relative taste vs. an expert reference.
Source repositories
Twelve open-source projects sampled across libraries, tools, services, and full applications. Most Senior SWE-Bench tasks are based on PRs authored by engineers with 100 commits in the respective repository, with maintainer-authored PRs oversampled.
| Repository | Languages | Type | Description | LOC | Started | Stars |
|---|---|---|---|---|---|---|
| electric-sql/electric | Elixir, TypeScript | Service | Postgres real-time sync | 345k | 2022 | 10.2k |
| go-gitea/gitea | Go | Application | Self-hosted Git forge | 397k | 2016 | 56.3k |
| PostHog/posthog | Python, TypeScript | Application | Product analytics platform | 3.8M | 2020 | 35.1k |
| PrefectHQ/prefect | Python | Library | Workflow orchestration | 664k | 2018 | 22.6k |
| better-auth/better-auth | TypeScript | Library | Authentication framework | 289k | 2024 | 28.7k |
| gravitational/teleport | Go, TypeScript | Application | Infrastructure access platform | 2.8M | 2015 | 20.5k |
| vercel/turborepo | Rust, TypeScript | Tool | Monorepo build system | 215k | 2021 | 30.6k |
| plausible/analytics | Elixir | Application | Privacy-friendly web analytics | 228k | 2018 | 27.2k |
| firezone/firezone | Elixir, Rust | Application | Zero-trust access platform | 247k | 2020 | 8.7k |
| paperless-ngx/paperless-ngx | Python, TypeScript | Application | Document management system | 148k | 2022 | 42.2k |
| immich-app/immich | TypeScript | Application | Self-hosted photo backup | 542k | 2022 | 103.6k |
| harbor-framework/harbor | Python | Tool | Agent evaluation harness | 219k | 2025 | 2.5k |
Sample tasks
Realistic vs. over-specified instructions
For comparison, here are two bug-task instructions from real-world source PRs. Senior SWE-Bench frames bugs as natural-language behavioral reports; SWE-Bench Pro spells out full reproduction steps and expected behavior. Behavioral testing lets the realistic version stay short without sacrificing reliable grading.
549 chars
~0 code symbols
5,888 chars
~32 code symbols
Illustrative samples. Senior SWE-Bench instructions read like an issue report on Slack; verifier-driven benchmarks lean on rigid, over-specified reproduction steps. Note: instructions do not represent the same task.
Comparison to other benchmarks
Several recent benchmarks make progress on behavioral testing and instruction realism. The following table provides a brief comparison.
Benchmark
Task style and source
Instruction realism
Reward mechanisms
Open source
Senior SWE-Bench
Real-world PRs
High (natural language message)
- Verifiers (behavioral)
- Validation agent
- Task rubrics
- Taste judge
Yes
SWE-Bench Pro
Real-world PRs
Low (full specs)
- Verifiers (implementation-specific)
- Rubric
Yes
DeepSWE
Invented tasks in real repos
Mixed (some full specs)
- Verifiers (behavioral)
Yes
FrontierCode
Real-world PRs
Unknown (examples are mixed)
- Verifiers (behavioral)
- LLM-adapted verifiers
- Agent-written tests (reverse)
- Code quality judge
No
ProgramBench
Full program recreation
—
- Verifiers (behavioral)
No
Methodology
Metrics
Pass@1 on the Mini-SWE-Agent harness (Harbor-compatible). Tasteful Solve Rate requires all six gates (verifiers + validation + rubric + bloat + practice + relative taste) to pass simultaneously; Basic Solve Rate removes the taste-related gates and measures correctness only.
For feature tasks, an agent (Mini-SWE-Agent with Claude Sonnet 4.6) writes behavioral tests adapted to each submitted solution using an expert-authored recipe. Each task is calibrated by running 3× on the oracle patch and 3× on no-op — rejected if pass³ < 1 on oracle or pass³ > 0 on no-op. Wall-clock time overhead 6–20% (median 11%); token cost overhead 2–16% (median 6%). Measured on Claude Opus 4.8 trials. In practice, less than 5% of trials are discarded.
An LLM judge grades each patch against the expert reference solution along two axes: relative code quality (minimality, approach, hygiene, fluency, craftsmanship) and codebase practice alignment (style consistency, pattern adherence, library usage, abstraction level, documentation fit). Thresholds set conservatively (any score > 2/5) and calibrated against human reviewers.
QUALITY CONTROL
Every task passes three layers of review: automated LLM-based checks, research-team review for overall design and implementation quality, and SWE-expert review via the Snorkel AI expert network using an extensive rubric. Each task includes a "guided" variant whose instruction adds optional hints — useful for performance diagnosis or curriculum learning — without prescribing the solution.
What the results show
Three patterns emerge from the leaderboard. None depend on the absolute scores — all are about the gap between correctness and senior-level taste.
Taste opens a 3–4× gap.
Frontier agents pass basic correctness (verifiers + validation tests) on 19–55% of tasks but earn a Tasteful Solve on only 4–24%. Every model loses 41–79% of its basic-solve credit when the taste, bloat, and codebase-practice gates are applied.
Correctness and taste are different skills.
GPT-5.5 wins on Basic Solve at 55%, but Claude Opus 4.8 wins on Tasteful at 24% — the rankings flip between the two metrics. Models that write the most runtime-passing patches don't always write the most senior-grade ones.
Even the top model misses senior taste 3 out of 4 times
Claude Opus 4.8 leads at 24% Tasteful Solve — meaning an agent that passes runtime tests is still failing the bar a senior engineer would hold 76% of the time.
Resources
Blog
Github
Task viewer
# Set env vars required for your model # export ANTHROPIC_API_KEY=sk-ant-... # export OPENAI_API_KEY=sk-proj-... # export MY_PROVIDERS_API_KEY=... # [Optional] Set the models for the test stage (defaults below) # export SSB_OVERRIDE_VA_HARNESS=miniswebench # export SSB_OVERRIDE_VA_MODEL=anthropic/claude-sonnet-4-6 # export SSB_OVERRIDE_ALL_JUDGE_MODEL=anthropic/claude-sonnet-4-6 # export SSB_OVERRIDE_CLASSIFIER_MODEL=anthropic/claude-haiku-4-5 # Set depending on what you want to run MODEL=anthropic/claude-opus-4-8 AGENT=mini-swe-agent # Run Harbor harbor run --repo snorkel-ai/senior-swe-bench -a $AGENT -m $MODEL
Behind the benchmark
Most software-engineering benchmarks evaluate AI agents like junior engineers — over-specified requirements graded against a fixed test suite. Senior SWE-Bench reframes the problem around the senior-level work deployed coding agents are actually expected to do, sourced from real pull requests across twelve open-source projects.
Acknowledgments
Senior SWE-Bench is led by Henry Kiss Ehrenberg with contributions from Vincent S. Chen at Snorkel AI; Austin W. Hanjie and Karthik Narasimhan at Princeton University; and Gabriel Orlanski and Frederic Sala at the University of Wisconsin–Madison.
All tasks are created and reviewed by contributing research staff and software engineers from the Snorkel AI expert network, in concert with specialized coding and evaluation agents. The benchmark is open source and Harbor-compatible.
