On-Policy Distillation Reduces Cross-Platform GUI Forgetting

GUI agents are moving from single-device demonstrations toward agents that can operate across desktop and mobile environments. That shift creates a training problem: desktop and phone interfaces use different action conventions, screen layouts, and task rhythms, so a policy trained jointly or sequentially can mix behaviors or lose earlier skills. This paper addresses that problem with two linked contributions: a cross-platform interaction dataset called Uni-GUI and a continual-learning method called UI-MOPD.

Core Contribution

The paper’s central claim is that cross-platform GUI learning should not treat all demonstrations as one homogeneous behavior stream. A desktop agent and a mobile agent may both click, type, scroll, and answer the user, but their priors differ: coordinates, navigation habits, app structures, and failure recovery patterns are platform-specific. UI-MOPD keeps those priors explicit by using multiple platform-specific teachers during on-policy training.

That is the part that is genuinely different from a plain supervised fine-tuning or joint-training recipe. Instead of distilling from one teacher or replaying a mixed dataset, the method selects the teacher according to the active environment and transfers that teacher’s behavior into a shared policy. The goal is a single agent that adapts to a new platform while retaining performance on platforms it has already learned.

Technical Approach

UI-MOPD combines continual learning with multi-teacher on-policy distillation. The shared policy interacts with the current GUI environment, and a platform-conditioned distillation signal aligns its behavior with the teacher assigned to that platform. The paper frames this as a way to reduce behavioral pattern mixing: the model is not merely told that all GUI actions are examples of the same task, but is guided toward the platform convention that matches the current environment.

The attached appendix page gives a concrete view of the mobile data-collection interface. The mobile trajectory prompt assigns Gemini-3.1-Pro the role of an Android phone operator and constrains outputs to a JSON action space: click, double_tap, long_press, drag, input_text, answer, navigate_home, navigate_back, scroll, status, wait, ask_user, and keyboard_enter. Coordinates are normalized to $[0,1000]$, with the origin at the top-left corner, and the prompt requires one tool call per action. The execution rules also encode practical mobile-agent behavior: use wait rather than answer for loading, try alternatives after repeated failures, reverse scroll direction when needed, and ask the user only when task information is insufficient.

Those details matter because they show that Uni-GUI is not just a set of screenshots or text instructions. The authors are trying to collect executable trajectories with platform-specific action semantics. For continual GUI learning, that distinction is material: a model that learns the wrong action format or navigation convention can fail even when it understands the user’s intent.

Results and Analysis

The paper reports evaluations on OSWorld and MobileWorld, two benchmarks that test GUI task completion in desktop and mobile settings. UI-MOPD reaches 38.2% task success on OSWorld and 12.0% on MobileWorld. The authors present these numbers as evidence that platform-conditioned on-policy distillation can balance retention of existing platform skills with adaptation to a new platform.

The supported result is narrower than the claim but still useful. The method is evaluated on real GUI benchmarks rather than only offline imitation loss, and task success is the right metric for this setting because execution errors compound across steps. A 38.2% desktop success rate suggests the shared policy can preserve meaningful desktop capability, while the 12.0% MobileWorld result indicates that mobile transfer remains much harder. The paper’s own framing treats this as a balance between retention and adaptation, not as solved cross-platform autonomy.

The main caveat is that the available metadata and appendix page do not expose the full comparison table, ablations, Uni-GUI scale, or teacher-model baselines. Without those, the headline result supports the direction of the method but does not by itself isolate which component matters most: the dataset, the on-policy training loop, the platform-conditioned teacher choice, or the distillation objective. The evidence is therefore best read as a promising benchmark-backed method for continual GUI-agent learning, with open questions about component-level attribution and mobile reliability.