SWE-Bench-CLI+
Stress-test & train frontier coding agents in production-grade codebases
SWE-Bench-CLI+ captures real-world software engineering tasks drawn from production-grade open-source repositories. Each task drops an agent into a live codebase and asks it to make meaningful changes with terminal access, repository context, and deterministic tests in place.
Developed by Snorkel’s AI Data Research Lab, SWE-Bench-CLI+ provides thousands of Harbor-format engineering tasks — built to provide stronger evaluation and training signal than patch-only benchmarks.
Coding task categories include
Fix
Tasks that focus on fixing known bugs or faults within the codebaseFeature
Introducing new internal or user-oriented featuresRefactor
Restructuring of the existing codebase with a focus on maintainabilityBuild
Tasks that focus on changes to build configurationsPerformance
Tasks that require performance to be improved (e.g., reducing memory consumption)Chore
Tasks designed around project-wide housekeeping needs, including dependency bumps, version increments, and miscellaneous cleanup.
SWE-Bench-CLI+ is intentionally calibrated to stress-test state-of-the-art coding agents
Built for frontier model evaluation:
- Tiered difficulty across Core, Advanced and Frontier
- Frontier subset where leading models score ≤20% accuracy
- Evaluated against GPT-5 and Claude Sonnet 4.5 with the Terminus-2 agent in Harbor
If your agent succeeds here, it performs in production.
Why the Snorkel Data Series
Expert-led validation
Every task is built and validated through a multi-layer quality pipeline.
Expert review
Expert contributors independently review solution correctness, prompt clarity, test reliability, and tagging.
Quality control checks
Snorkel's proprietary quality control models validate every component of a task.
Difficulty validation
Task difficulty is measured empirically against frontier models and agents.
Distribution guardrails
Diversity across language, task type, and difficulty is controlled through task metadata.
