Agents’ Last Exam Reports Below-1% Average Full-Pass Rates on Its Hardest Tier

Benchmark scores can suggest broad competence while saying little about whether an agent can complete a paid workflow end to end. Agents’ Last Exam (ALE) targets that discrepancy: it evaluates long-horizon, economically relevant tasks whose outcomes can be checked, rather than treating isolated answers or short tool-use episodes as a proxy for deployment. The central empirical message is sobering. Across the reported mainstream harness and backbone configurations, average full-pass performance on the hardest tier remains below 1%.

Core Contribution

ALE is a benchmark construction effort as much as an evaluation set. The authors organize non-physical work around the U.S. O*NET/SOC 2018 occupational taxonomy, then translate professional activities into runnable tasks with verifiable outcomes. Its 13 industry clusters, 55 subfields, and more than 1,000 tasks are intended to cover work that has recognizable economic value rather than the narrow capabilities usually isolated by agent benchmarks.

That design changes what a score means. A full pass requires an agent to carry a workflow through to its specified endpoint, so partial progress does not become evidence of task completion. The authors’ argument is that this is the missing measurement layer between strong benchmark behavior and GDP-relevant deployment. The benchmark is also designed to grow as experts contribute new workflows, making coverage an ongoing process instead of a fixed test set.

Figure 1 places the benchmark’s taxonomy across professional task families and realistic workflows. It makes clear that ALE is not framed as a collection of coding tasks with a few domain additions; breadth across occupations is part of the contribution.

Technical Approach

The paper builds tasks with more than 250 industry experts and arranges them hierarchically from industry clusters to subfields and individual task instances. The current runnable instances target Linux or Windows virtual machines, which permits controlled execution while retaining tool use and multi-step workflow requirements. Outcomes are meant to be verifiable, a necessary property if a benchmark is to distinguish a plausible-looking trace from a completed assignment.

The task-distribution figure also separates fully implemented instances from expert submissions still awaiting a quality-control process. That distinction matters: a large task pipeline is not equivalent to an equally large evaluated benchmark. Every displayed subdomain has non-zero coverage, but the visual accounting shows that some of the claimed breadth is still moving through validation rather than already available for scoring.

Results and Analysis

The reported result is not a close leaderboard race. The hardest ALE tier has an average full pass rate below 1% across reported mainstream combinations of agent harnesses and backbone models. Under ALE’s all-or-nothing completion criterion, this average indicates that full completion is rare in the reported evaluations. This supports the paper’s narrower claim that benchmark success elsewhere should not be read as evidence of reliable autonomous performance on extended, economically meaningful tasks.

The result is useful precisely because it is strict. A low full-pass rate could reflect weak planning, brittle computer use, domain knowledge gaps, poor recovery from tool failures, or difficulty meeting the verifier’s final condition; ALE does not make those causes interchangeable. Its value is to expose the compound problem. For practitioners, that means the benchmark is more informative for deciding where human supervision remains necessary than for selecting a model on a single capability dimension.

Limits of the Evidence

ALE’s scope is intentionally limited to non-physical industries, so it does not establish performance on embodied work or physical-world operations. Its present execution environment is also confined to Linux and Windows virtual machines. More fundamentally, a below-1% average identifies a large capability gap but does not by itself identify which component of an agent stack is responsible. The living-benchmark model is promising, yet it makes stable longitudinal comparisons dependent on clear versioning and on consistent quality control as new expert-authored tasks enter the pool.