Professional GUI Agents Stall on Long-Horizon Workflows

GUI-agent benchmarks often test whether a model can click through common apps or finish short web and desktop tasks. Workflow-GYM asks a harder question: can an agent operate specialized professional software long enough to produce a verifiable work product? The benchmark covers real workflows such as QGIS vector analysis, FreeCAD modeling, KNIME dashboard construction, Money Manager accounting, CapCut video editing, and Blender asset creation. The paper’s answer is blunt: current agents can interact with these interfaces, but they rarely finish the whole job.

Core Contribution

The paper’s main contribution is a benchmark design centered on workflow rather than isolated UI actions. Each task starts from a natural-language instruction, a preconfigured virtual-machine environment, required input artifacts, and a final success criterion. The agent receives no step-by-step help in the main setting; it must plan, operate the GUI, recover from errors, and produce either a final artifact or a target software state.

The dataset is larger and more domain-specific than a typical GUI demo suite: 338 tasks across 6 primary domains and 23 fine-grained subdomains. The authors also report 56 reusable software-level virtual machines, which matters because professional workflows depend on installed applications, versioned menus, file paths, rendering behavior, and system-level state. Task length is part of the benchmark definition: 129 tasks require 30–44 expert steps, 159 require 45–60, and 50 require 61–110.

Technical Approach

Workflow-GYM is built through a four-stage pipeline. Domain experts first propose tasks that meet four filters: realism, domain specificity, multi-step complexity, and verifiability. The environments are then standardized as reusable virtual machines with required software preinstalled; task-specific inputs are injected at runtime. For each task, experts provide the user-facing instruction, expected outcome, evaluation criteria, and a hidden expert procedure used for validation rather than as agent guidance.

The evaluation combines rule checks and model-based judging. For artifact tasks, the output files can be verified directly against the task criteria. For GUI-state tasks without explicit files, the system captures screenshots after execution and asks a vision-language judge to score success against the predefined rubric. The main experiments run each model for three trials per task, giving 338 × 3 = 1014 trials, with a maximum interaction budget of 400 rounds.

Figure 7 illustrates one reason this setup is difficult: a GRASS GIS task fails because the agent does not know the software’s interface and repeatedly searches the wrong menu path for an OGR import tool.

Results and Analysis

The headline result is that even the best systems stay near one-third success. Gemini-3.1-pro reaches 30.67% average pass rate and 41.12% pass@3; Kimi-k2.6 is close at 29.68% average pass rate and 41.42% pass@3. The remaining models trail: Seed-2.0-lite at 18.24%, GPT-5.4 at 17.85%, GPT-5.4-mini at 15.98%, and Gemini-3-flash at 7.89% average pass rate.

Difficulty scaling is the most useful signal. Gemini-3.1-pro falls from 39.28% on easy tasks to 26.62% on medium and 21.33% on hard tasks. Kimi-k2.6 drops from 34.63% to 27.04% and 25.33%. That pattern supports the authors’ claim that failures are not only about visual grounding or a missing click. The longer the chain, the more likely an early wrong state, skipped stage, or local repair attempt invalidates later work.

Domain results also separate structured GUIs from visually open-ended work. Kimi-k2.6 is strongest on Data Analysis at 45.24% and Multimedia & Creative at 27.64%, while Gemini-3.1-pro leads Engineering & Design at 24.44%, Finance & Management at 37.44%, Geography & Environment at 27.93%, and Scientific Computing at 31.97%. Across all models, average pass rates range from 26.09% in Data Analysis to 14.38% in Engineering & Design, suggesting that discrete, form-like workflows remain easier than spatial manipulation and design tasks.

The failure analysis is more informative than the leaderboard. The paper separates failures into workflow incompletion and final-state-but-incorrect outcomes. Workflow incompletion has a strong negative correlation with overall success, Pearson r = -0.97 with p = 0.0010. Weaker models mostly fail before reaching the final stage; stronger models more often complete something, but the result is wrong.

What the Benchmark Exposes

The ablations show that procedural guidance helps but does not solve the benchmark. Textual step-by-step tutorials improve Seed-2.0-lite from 18.24% to 30.47% and GPT-5.4 from 17.85% to 28.32%. On a 100-task subset, video tutorials raise Seed-2.0-lite from 19% to 31% and Gemini-3.1-pro from 28% to 35%. The cost is longer failed trajectories: for Gemini-3.1-pro, failed median steps rise from 100 without tutorials to 187 with video.

Figure 9 captures the paper’s strongest systems argument. In tasks requiring continuous visual feedback, such as drawing a line of exact length in design software, snapshot-only observation forces trial and error. The agent cannot monitor the action as a human would during the drag, so it repeatedly retries after seeing only the post-action state.

The evidence supports Workflow-GYM as a demanding benchmark, not as a complete measure of professional automation. It tests agents under a particular interaction protocol, with a fixed round budget and partially automated judging. Still, the gap between 30.67% success and end-to-end reliability is large enough that the main conclusion is well supported: professional GUI workflows remain beyond current general-purpose agents.