Alibaba has put AI through "military-style training" on 100 real devices, making it fully capable of handling operations on various screens. So what is the difference between this and an AI phone?
On August 20, Alibaba launched a GUI intelligent agent base model for digital device execution — Qwen-UI-Agent. This model is mainly oriented to mobile, PC, web, deep search and other scenarios, which can operate devices through virtual clicks, CLI execution and other methods to complete complex tasks.
Among the 6 core GUI benchmark tests displayed on the project homepage, Qwen-UI-Agent ranked first in 5 of them, scoring higher than models such as GPT-5.6 Sol and Claude Opus 4.8, and also outperforming Seed 2.1 Pro released by ByteDance in June this year. It is worth mentioning that the main version of this model has only 27B parameters, which is built on the Qwen 3.5 series foundation model.
Alibaba has demonstrated multiple application scenarios of Qwen-UI-Agent. Users can ask the model to open the map on the mobile phone through simulated clicks, find nearby cafes, and search for reviews of the cafes on social media platforms to assist users in decision-making.
When performing tasks on a PC, Qwen-UI-Agent will perform more CLI actions, and output multiple actions in batches in a single decision, to complete tasks such as cross-site product research, price checking, and generating price comparison schemes.
In the deep search scenario, the GUI capability and CLI capability of this model can cooperate to complete tasks: GUI is responsible for web retrieval and data source positioning, while CLI is responsible for data download and analysis.
In addition, this model can also be used for cross-device execution between mobile phones and computers, PPT report generation and other tasks. It can also actively pay attention to information such as message notifications on the device, and provide users with recommendations, schemes and other content.
Alibaba simultaneously released the technical report and project homepage of the Qwen-UI-Agent model. In the technical report, Alibaba revealed that in order to train this model, they built a real machine mobile environment covering more than 100 real mobile phones and over 150 applications for task construction, trajectory collection, model training and evaluation. In addition to the 27B version, this model also provides versions with 35B-A3B and 4B parameters. They believe that GUI agents are expected to become general actuators on existing digital devices.
Technical Report: https://arxiv.org/abs/2607.28227
Project Homepage: https://tongyi-mai.github.io/Qwen-UI-Agent
GitHub: https://github.com/Tongyi-MAI/MAI-UI
01. The GUI agent faces the sim-to-real gap, and the unique ecosystem of Chinese applications brings additional challenges
The ideal scenario for a GUI agent is that as long as there is an interface, it can understand, click, and input like a human, to complete complex tasks for you. However, the Alibaba team found that the vast majority of current GUI agents are trapped in simulators: they achieve excellent scores in a clean sandbox environment, but once placed on a real mobile phone, their performance will drop sharply when facing changing interfaces, pop-up advertisements, expired login status, and network jitter.
This "simulated environment to real environment" gap (sim-to-real gap) is particularly serious in China's mobile ecosystem, because the super apps in the Chinese market have deep functions, scattered entrances and numerous pop-ups, which cannot be reproduced in the laboratory environment at all.
At the same time, the demands of real users are no longer limited to a few clicks in a single App. A complete office or life workflow often requires jumping back and forth between mobile phones, computers and browsers, and even requires the agent to take the initiative to perceive, for example, after seeing a flight cancellation notification, automatically retrieve alternative solutions, check schedule conflicts, and generate a recovery plan.
Most of the existing agents are in a passive question-and-answer mode, and will not act until they receive instructions from users. They can neither collaborate across devices nor have the ability to plan and correct errors for long-term tasks.
The more underlying problem is training efficiency. The data collection of traditional GUI models is highly dependent on manual work: manually writing tasks, building environments, labeling results, analyzing failures, and so on. As the capability boundary of agents expands, this workflow is no longer sustainable.
Alibaba believes that the next-generation GUI foundation model must solve three major contradictions at the same time: the contradiction between simulation and reality, the contradiction between single-point capability and complete workflow, and the contradiction between manual drive and large-scale scaling. This is exactly the starting point for the establishment of the Qwen-UI-Agent project.
02. Hundreds of real machines build mobile runtime, multi-round RL improves task success rate
In order to build Qwen-UI-Agent, Alibaba designed a full-stack framework covering the underlying environment, action space, training paradigm and upper-layer Harness.
First of all, the real machine training environment. The research team built a real mobile runtime composed of more than 100 real mobile phones and more than 150 applications.
This system is equipped with a health-aware scheduler, which can monitor the application status, account login status and network condition of each device in real time, and automatically switch in case of failure. It also supports virtual screen technology, allowing a single mobile phone to run multiple application sessions at the same time, increasing the concurrent rollout efficiency by about 20 times.
The model is directly exposed to real login status, dynamic content and random pop-ups during the training phase, which alleviates the sim-to-real gap to a certain extent.
Secondly, the unified action space. Qwen-UI-Agent integrates GUI operations, CLI commands (such as bash scripts) and API calls into the same action system. For structured tasks such as table processing and batch file operations, the model can directly write code to execute.
For operations that require visual confirmation, such as filling in forms, searching within pages, zooming maps, etc., it switches back to GUI clicks.
In a single inference, Qwen-UI-Agent can also output batch processing actions to complete multiple consecutive operations at one time, greatly reducing the number of interaction rounds.
Statistically, in the OSWorld desktop tasks, CLI actions account for more than 40%, and batch processing actions account for about 40%.
At the training level, the team designed a set of "agent-driven data flywheel". The whole process is automatically completed by AI, including task synthesis, environment setup, result verification, failure diagnosis and planning of the next round of training data, while humans are only responsible for supervision and correction.
The training is divided into three stages: supervised fine-tuning (SFT) builds basic capabilities; Action RL specially corrects common action errors such as positioning errors, repeated loops, and premature termination.
Online RL optimizes the end-to-end success rate in long-term trajectories of more than 100 steps, and schedules nearly 10,000 concurrent environments to accelerate rollout. For example, after RL training, the model will actively verify the results and repair errors after finding them, instead of directly announcing success.
The top layer is the Harness layer, which is responsible for active services and cross-device collaboration. It can read mobile phone notifications, actively identify events such as flight cancellations and generate execution plans.
It can also string cross-device tasks into workflows, so that the information on the mobile phone can be transferred to the computer for further processing.
In terms of security, in the face of illegal or high-risk requests, Qwen-UI-Agent will not perform any interface operations, and directly reject and terminate the task; in sensitive scenarios such as payment, data deletion, and privacy authorization, it will actively stop at key steps, explain the situation to the user, and continue after obtaining explicit confirmation.
03. The model has superior parameter efficiency, and pop-up ads and other interferences affect execution performance
Among the 6 core benchmark tests covering mobile, desktop, web and GUI positioning scenarios, Qwen-UI-Agent won 5 first places.
This model performs relatively well on the mobile end. On the real device evaluation set MobileWorld-Real, it leads GPT-5.6 Sol, Claude Opus 4.8 and ByteDance Seed 2.1 Pro by a large margin with a success rate of 92.2%, and reaches 97.5% on the AndroidDaily daily task set, which is close to full score.
On the desktop end, it achieves 79.5% on OSWorld-Verified, second only to Claude Opus 4.8.
The paper also reveals several noteworthy quantitative results and behavioral insights.
The first is the parameter efficiency of the model. The 27B version of Qwen-UI-Agent outperforms the flagship models of multiple manufacturers, while the 35B-A3B version can still reach 87.4% on MobileWorld-Real, indicating that this training method enables small and medium-sized models to obtain stronger task completion rate and deployment cost-effectiveness through real environment data.