Long-Horizon Workflows Expose Agent Reliability Gaps

Computer-use agents are often evaluated on short tasks where success depends on operating a browser, clicking the right control, or writing a small script. OSWorld 2.0 targets a harder failure mode: work that unfolds across applications, documents, emails, user state, and changing conditions. The benchmark contains 108 long-horizon workflows that take humans a median of about 1.6 hours, compared with roughly two minutes for OSWorld 1.0, and the paper uses those tasks to ask whether current agents can manage professional-style computer work rather than isolated UI actions.

The answer is mostly no. The authors report that the best configuration, Claude Opus 4.8 with maximum thinking and batched tool calls, completes 20.6% of tasks under the primary 500-step binary metric, with a 54.8% partial score. GPT-5.5 is more token-efficient but plateaus near 13% completion. That pattern matters because the failures are not mainly low-level GUI misses. The paper’s analysis points to agents losing constraints, missing mid-task updates, guessing when user clarification is required, and skipping verification.

Core Contribution

The main contribution is a benchmark design that makes long-horizon computer use measurable without reducing it to toy browsing. Tasks span everyday and professional workflows, but their distinguishing feature is state: receipts, emails, financial records, prior reports, hidden profile information, and dynamic updates can all affect the correct final action. Figure 1 captures the intended difficulty through an ExpenseFlow reimbursement example, where the agent must follow a tutorial, reconcile evidence across GMail and ChaseBank, handle a new email, and identify an inconsistency before final submission.

This is new relative to earlier computer-use benchmarks in two ways. First, task length is much higher: the abstract reports an average of 318 tool calls for Claude Opus 4.7 at maximum thinking, versus about 30 in OSWorld 1.0. Second, the benchmark is organized around challenge phenomena such as streaming interaction, dynamic environments, cross-source reasoning, implicit-state inference, and visual-spatial precision, rather than around app categories alone.

Technical Approach

OSWorld 2.0 is built as an executable benchmark rather than a static question set. The task construction pipeline starts from team brainstorming, interviews, questionnaires, and synthetic proposals, then filters ideas by complexity, diversity, and feasibility. The final tasks configure self-hosted web services, applications, initial and final workspace states, simulated user channels, and dynamic-update hooks. The authors also describe iterative quality assurance: unit-test generation, human cross-checking, trajectory rollouts, and targeted review for feasibility, partial rewards, and reward-hacking risks.

Figure 2 is useful because it shows the benchmark as infrastructure, not just annotation. The testbed has to set up the initial state, expose the right files and services, inject changes at the right time, and judge outcomes with both strict completion and partial credit.

The paper also tries to measure economic coverage rather than treating all workflows as equally representative. Its occupation-family analysis and GDP-proxy table do not prove that the benchmark is a full sample of economic activity, but they are a practical attempt to avoid a benchmark made only of familiar office tasks.

Results and Analysis

The headline result is the gap between partial progress and full completion. Under the primary metric, Claude Opus 4.8 reaches 20.6% binary completion at 500 steps and 54.8% partial score. GPT-5.5 uses fewer tokens but levels off near 13% completion. Earlier model-by-model analysis in the paper’s figures reports completion shares in the 3% to 14% range across five evaluated models, with partial progress dominating all systems. Figure 7 also shows a strategy split: GPT-5.5 is the most programmatic solver at 78% Code/API/file actions, while Opus 4.7 splits Code/API/file and GUI at 37% each.

The cost-performance plots reinforce the same reading. More reasoning effort and larger step budgets can improve partial progress, but they do not close the gap to reliable task completion. Figure 5 compares binary completion against average turns and partial reward against output tokens, showing that extra compute buys some progress but leaves a steep reliability frontier.

The paper’s evidence supports a conservative conclusion: current agents can navigate pieces of long workflows, but they still fail as accountable operators. The most informative failures are procedural rather than perceptual. Agents miss information that arrives after the task starts, fail to recover hidden state, neglect user clarification when evidence conflicts, and end without checking that the assembled output is consistent. For teams evaluating computer-use agents, OSWorld 2.0 is therefore less a leaderboard than a diagnostic: it separates “can act on a computer” from “can finish a multi-source workflow without losing the thread.”

Limitations

The benchmark is broad for its category, but it is still a curated set of 108 tasks. Its realism depends on the fidelity of simulated services, user channels, and workspace states. The paper’s strongest claims are about evaluated frontier agents under the specified step and reasoning settings, not about all possible scaffolds, human-agent collaboration patterns, or future systems trained directly on OSWorld-style workflows.