Terminal-Bench 3.0

Frontier-Bench (formerly Terminal-Bench 3.0) is the next version of Terminal-Bench, a benchmark measuring whether AI agents can perform valuable work in containerized terminal environments. It is currently being built in the open: anyone can propose and submit tasks, which go through automated and human review before being added to the suite.

Every task PR runs through static checks (canary, Dockerfile sanity, metadata), an automated 28-criteria implementation rubric, Docker build + oracle + no-op validation, then human maintainer review. Once approved, it's also run through agent trials and adversarial “cheat” trials, and can be hardened further with an iterative hacker–fixer loop.

Every task is classified into one of seven fixed top-level domains (Science, Software, ML, Operations, Security, Hardware, Media) and a more specific subcategory within that domain, chosen by the primary skill the task exercises rather than incidental tooling. The domain list is closed; subcategories can grow as new task types arrive.

Tasks default to a “separate verifier” mode so the grading container can't inherit anything the agent modified. Maintainers can also trigger adversarial cheat trials, where an agent is explicitly prompted to hack the verifier instead of solving the task, and an iterative hacker–fixer loop that hardens a task until an adversarial agent can no longer find an exploit.

Yes — Frontier-Bench is actively accepting task submissions from the community. See the contributing guide and task-proposal rubric on GitHub before starting, and consider getting feedback on your task idea before investing time in a full submission.

Frontier-Bench is hosted by Harbor and the Laude Institute, led by Ryan Marten, Alex Shaw, Andy Konwinski, and Ludwig Schmidt, with data and research contributions from Snorkel AI led by Justin Bauer — who also serves as a task reviewer for the benchmark — and Vincent Sunn Chen, and additional support through Snorkel's Open Benchmarks Grants program.

Frontier-Bench is the official successor to Terminal-Bench, built by the same team. Where Terminal-Bench tasks have become saturated — top agents already clearing 75–84% on 2.1 — Frontier-Bench introduces a harder, more domain-diverse task set to keep discriminating between frontier and sub-frontier models.