Does Embodied Intelligence Enter the Era of Agentic Robotics? Tsinghua University and UC Berkeley Open Source the Unified RUA Framework

Robots are ushering in their own Computer Use moment.

On the 10th, Anthropic's newly released Fable 5.2 brought new progress. Compared with previous models, its physical robot control capability has been significantly improved, and it can even complete tasks that have never received special training before.

In the RoboDojo Sim simulation evaluation, Fable 5.2 scored 64.9%, far exceeding GPT-6 Astra (29.8%) in Anthropic's internal tests with the same configuration.

Back in September, MIT researcher Phillip Isola proposed in Robot-Use Agents that in the future, robots connected to the Internet and open to sensor and actuator interfaces may all become tools that general-purpose Agents can call.

This also aligns with the division of labor idea of System 1 and System 2 in the field of embodied intelligence: general models are responsible for high-level reasoning and task planning, while low-level policies are responsible for specific action execution. Models such as GPT-6 Astra and Fable 5.2 have initially verified the feasibility of this route.

However, to make it work smoothly, there are still two key problems to be solved urgently.

The first is the model. Cutting-edge closed-source models have demonstrated good robot operation capabilities, but researchers cannot directly access their weights, let alone use robot interaction data to conduct continuous training on them.

The second is the system. Different robots have significant differences in hardware structures, sensors and control interfaces, and the simulation environment is also different from the real environment. If the tool system needs to be redeveloped every time a new type of robot is connected, the cost of cross-environment migration and large-scale experiments will remain high.

Therefore, RUA requires a unified interaction interface to shield the underlying differences between different hardware and environments, so that the model can obtain perceptual information, call control tools and receive execution feedback in a consistent manner.

Recently, the research team of DING Ning, LIU Zhiyuan, XU Mengdi, ZHOU Bowen from Tsinghua University, in collaboration with the team of Professor Pieter Abbeel, co-director of the Berkeley Artificial Intelligence Research Laboratory (BAIR) at the University of California, Berkeley, jointly launched and open-sourced uni-robot-agent, trying to establish a unified and modular Harness for Robot-Use Agent (RUA), so that language models can manipulate different robots and simulation environments through standardized tool interfaces.

Paper link: https://arxiv.org/pdf/2611.01560

Blog link: https://openrua.ai/uni-robot-agent

Code link: https://github.com/Open-RUA/uni-robot-agent

This framework performs well in actual evaluations. On the RoboDojo benchmark, the open-weight model Qwen3.8-Flash-Next paired with this framework scored 31.39 points, exceeding the 28.97 points of GPT-6 Astra + RoboProbe previously announced by the RoboDojo official. Under the same framework, GPT-6 Astra further scored 51.90 points.

In the evaluation of 19 real-machine tasks, Qwen, which has not been specially fine-tuned for tasks, scored 53.50 points, leading the fine-tuned π₀.₅ and zero-shot Dream-1 by 5.75 points and 10 points respectively. GPT-6 Astra paired with this framework scored 80.50 points.

Beyond benchmark scores, this work also tested the maximum extent to which the model can operate robots, and sorted out the failure causes, capability shortcomings and operation costs.

Furthermore, the team aims to build a general base that allows open-source models to connect to different robots, paving the way for subsequent training and local deployment.

Why is the unified interface the foundation of cross-environment generalization?

In the field of Coding Agent, the same set of tools can be used for different programming tasks. The model reads files, executes codes and checks running results through similar interfaces, and can reuse existing tool use experience in different environments.

Robot operation involves more hardware differences. Even for the same task of grasping objects, the action interfaces of single-arm robots and dual-arm robots may be different. The model needs to understand new parameters, coordinate systems and control rules, and the original tool calling experience is difficult to migrate directly.

uni-robot-agent hopes to bridge these differences through a unified tool interface.

For example, the Franka single-arm robot needs to locate, grasp and stack building blocks, the ALOHA dual-arm robot needs to coordinate two robotic arms to complete object handover, and the real YuMi dual-arm robot needs to control the alignment and insertion of pins and jacks. Despite the differences in hardware structure, action space and task objectives, the model can call the unified perception, motion, skill and task control interfaces, and the underlying adapter handles the control differences of different robots.

In other words, the model can follow consistent tool calling rules. When connecting to a new environment, developers only need to implement the corresponding adapter, reducing modifications to the upper-level Agent logic.

Reusability of tools in different environments.

The unified interface has a deeper meaning.

If different environments use the same tool semantics, parameter units and feedback formats, the interaction trajectories generated by robots executing tasks will be easier to aggregate, and the experience accumulated by the model in single-arm stacking, dual-arm handover and real-machine insertion can be integrated into the same data system.

This design creates conditions for cross-environment Agent training, giving researchers the opportunity to train the same model using interaction data from different robots to learn transferable tool use and operation capabilities.

How is uni-robot-agent designed?

The design of uni-robot-agent revolves around two goals. The first is to provide a unified foundation for cross-environment Agent training, and the second is to evaluate the existing robot operation capabilities of general models as objectively as possible.

To this end, the research team divides the system into five layers: Agent, Orchestration, Tools, Services and Backends.

The Agent layer is responsible for interacting with the model, managing context and task decision-making; the Orchestration layer is responsible for scheduling tool requests; the Tools layer provides a unified operation interface for the model; the Services layer manages operating resources; the Backends layer connects specific robots, simulators and model implementations.

This modular structure supports independent replacement of models, tools and robot backends. Researchers can compare the performance of different models and accumulate interaction data with consistent structures without rewriting the entire system.

Objective evaluation requires being as close to the real deployment conditions as possible. The simulation environment can directly provide the precise 3D position of objects, CAD models, hidden scene states, and even the judgment of task success. These information are usually not directly available on real robots. If the model completes tasks by relying on these privileged information, it will be difficult to judge how much visual understanding, spatial reasoning and operation capabilities it actually has.

Therefore, in the main simulation experiments, uni-robot-agent only provides the model with RGB images, camera calibration, robot body status and task instructions, and does not provide depth ground truth, object CAD assets, hidden scene annotations or task validator feedback, nor does it allow external network access. At the same time, the research team also restricts shortcuts specially customized for evaluation tasks, to prevent the model from getting high scores through pre-written task answers.

Then, without these additional information, how to make general models complete complex robot operations? The key lies in reasonably organizing the information obtained by the model and ensuring reliable tool execution.

Key design choices. Each column shows two coordinated designs, which are used to ensure physical state consistency, action execution accuracy and context management respectively.

For example, when a robot grasps an object, the model needs to calculate the grasping coordinates based on the image and the position of the robotic arm. If the image comes from the previous moment while the status of the robotic arm has been updated, the calculation result may be deviated.

uni-robot-agent binds the camera image, calibration information and robot status to the same moment, reducing positioning deviation caused by state desynchronization. When executing actions, it avoids repeated operations through request deduplication, and monitors the motion status of the robotic arm to identify action stagnation or execution exceptions.

After the action is completed, the system feeds back the execution result and new environmental observations to the model, helping it judge whether the task is successful and whether it is necessary to adjust the next operation.

To solve the problem of long action time of robots, the framework also introduces asynchronous execution. The model can continue to process other work while some tools are running, and motion requests on the same robotic arm are queued in order to avoid instruction conflicts.

These designs integrate the engineering experience in the robotics field into the Harness, reducing the interference of information misalignment, repeated actions and missing feedback on model performance.

By reasonably organizing information and providing reliable control tools, uni-robot-agent can still enable general language models to demonstrate competitive robot operation capabilities without relying on simulation privileged information and task-specific shortcuts.

Experimental verification: What can open models achieve?

To test the actual effect of the unified Harness, the research team first carried out simulation experiments on RoboDojo.

This benchmark contains 42 robot operation tasks, covering five dimensions: generalization, fine operation, long-term execution, memory and open-vocabulary instructions. Each group of main experiments was evaluated 252 times.

The impact of the unified Harness is very obvious.

RoboDojo benchmark test results. Avg. represents the unweighted average of each indicator under the five capability dimensions. †GAP (Hybrid) means that this solution integrates π₀.₅.

Under the same Qwen3.8-Flash-Next model, RoboProbe only scored 1.90 points, RPent scored 7.75 points, and GAP Hybrid combined with π₀.₅ reached 21.01 points.

After switching to uni-robot-agent, Qwen's score rose to 31.39 points, with a success rate of 25%. Compared with RoboProbe, the comprehensive score increased by 29.49 points.

This set of experiments shows that even if the model weights remain unchanged, the design of the tool system can significantly affect the robot operation performance.

The framework is also adapted to other models. DeepSeek-V4.1-Flash scored 27.11 points, and GPT-6 Astra reached 51.90 points with a success rate of 46.67%.

Different models have different advantages. Qwen and DeepSeek perform better on memory and open-vocabulary tasks than on fine operation and long-term execution; some dedicated VLA and WAM models have more advantages in long-term tasks.

The real-machine test further verifies the model's capabilities in the physical environment.

The research team used a dual-arm robot to carry out 19 operation evaluations, covering five dimensions: spatial understanding, object generalization, instruction following, combinatorial generalization and unseen tasks. Specific tasks include test tube rack insertion, dual-arm bottle transfer, stacking blocks in color order, whiteboard erasing, and completing the Tower of Hanoi puzzle.

Representative real robot tasks under the five dimensions.

GPT-6 Astra scored 80.50 points, including a full score of 20 points in the combinatorial generalization dimension; Qwen scored 53.50 points, leading the task fine-tuned π₀.₅ and zero-shot Dream-1.

Real-machine evaluation results. The full score for each capability is 20 points, and the total full score is 100 points.

However, there are input condition differences in this set of comparisons. Qwen and Astra use RGB images and depth information