OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
A first-of-its-kind scalable, real computer environment for multimodal agents — supporting task setup, execution-based evaluation, and interactive learning across operating systems.
What is OSWorld?
OSWorld is a benchmark and environment for evaluating AI agents on real computer tasks. Built by researchers at the University of Hong Kong, Salesforce Research, Carnegie Mellon, and the University of Waterloo, it provides a scalable, executable environment where multimodal agents operate real desktop and web applications across Ubuntu, Windows, and macOS — the same software people use every day.
Unlike benchmarks confined to a single app or domain, OSWorld treats the whole operating system as the playground. Agents click, type, navigate file systems, and chain workflows across multiple applications. Each of its 369 tasks is drawn from real-world computer use and ships with a detailed initial-state configuration and a custom, execution-based evaluation script for reliable, reproducible scoring.
369
Real computer
tasks
3
Operating
Systems
134
Execution-based
eval functions
72%
Human success
rate
Why OSWorld matters
Autonomous agents that complete complex computer tasks with minimal human intervention could transform human-computer interaction — boosting accessibility and productivity for everyone. But measuring real progress requires environments that reflect the messy, open-ended nature of actual computer use, not narrow toy settings.
Most prior benchmarks either lacked an interactive environment or were locked to a single application or domain. OSWorld closes that gap: it is a controllable, executable environment with high scalability for adding new tasks involving arbitrary apps, native multimodal support, and cross-application workflows that can even begin from an intermediate initial state.
KEY INSIGHT
The headline finding: humans complete over 72% of OSWorld tasks, while at the benchmark’s 2024 release the best AI agent reached just 12.24% — exposing deep gaps in GUI grounding and operational knowledge. Those gaps have driven rapid progress: the OSWorld-Verified leaderboard now tracks frontier agents climbing well past those early numbers.
How the environment works
Every OSWorld task is defined by a configuration file that drives four stages: initializing the task state, agent interaction with the live environment, post-processing once the agent signals completion, and execution-based evaluation that retrieves files and information and runs a verification function to score the result.
Environments run in parallel on a single host machine for evaluation or interactive learning, and headless operation is fully supported. As of the OSWorld-Verified upgrade, AWS support brings a full evaluation run down to roughly one hour.Environments run in parallel on a single host machine for evaluation or interactive learning, and headless operation is fully supported. As of the OSWorld-Verified upgrade, AWS support brings a full evaluation run down to roughly one hour.
Initial state setup
Each task starts from a precisely configured environment so runs are reproducible across machines and providers.
Execution-based scoring
Custom verification scripts inspect the real end state — files, app state, outputs — rather than matching against a fixed answer.
Cross-OS support
Tasks run on Ubuntu, Windows, and macOS via VMware, VirtualBox, Docker, or cloud providers like AWS.
Parallel & headless
Multiple environments run side-by-side on one host, enabling fast evaluation and interactive learning at scale.
Task & domain coverage
OSWorld’s 369 tasks span real web and desktop applications across open domains, OS-level file I/O, and multi-application workflows — plus an additional 43 Windows-based tasks reserved for analysis. The table below shows the exact distribution of the 369 Ubuntu tasks by application domain, taken directly from the benchmark’s official task index (evaluation_examples/test_all.json).
| Application domain | Tasks |
|---|---|
| Multi-app workflows | 101 |
| LibreOffice Calc (spreadsheets) | 47 |
| LibreOffice Impress (presentations) | 47 |
| Chrome (web browser) | 46 |
| GIMP (image editing) | 26 |
| OS (file system & settings) | 24 |
| LibreOffice Writer (documents) | 23 |
| VS Code (code editor) | 23 |
| VLC (media player) | 17 |
| Thunderbird (email) | 15 |
| Total | 369 |
NOTE ON GOOGLE DRIVE TASKS
8 of the 369 tasks depend on Google Drive and may require manual configuration due to network factors. Excluding them to run 361 tasks is officially permitted and accepted for leaderboard submission.
The benchmark & leaderboard
OSWorld evaluates a broad mix of agent paradigms: general-purpose models (prompted for computer use), specialized computer-use models trained for the job, and agentic frameworks that combine a planner with a grounder. Baselines span open-source representatives such as UI-TARS, Agent-S, Qwen, Mixtral, and CogAgent, alongside closed-source families including Operator, GPT, Gemini, and Claude.
The July 2025 OSWorld-Verified upgrade fixed community-reported issues, added AWS support, and refreshed all results under unified evaluation settings, so scores are directly comparable. Verified trajectories are published openly for community analysis, and the project actively welcomes pull requests adding new models and methods.
WHAT THE ANALYSIS REVEALS
- Resolution: Higher screenshot resolution typically leads to improved agent performance.
- History: Longer text-based trajectory history helps — but screenshot-only history doesn’t, and long context strains efficiency.
- Robustness: Today’s VLM agents are not robust to UI layout changes and visual noise.
- Cross-OS: Performance correlates strongly across operating systems — insights transfer reliably from one OS to another.