Members from Tsinghua University's Yao Class and top talents from Harvard have enabled robots to write code and hone their skills independently, finally eliminating the need for humans to guide them step by step manually.

According to Robot Insight's report on October 19, research teams from Harvard University, NVIDIA, University of California, Berkeley, Massachusetts Institute of Technology and Stanford University recently launched the open-source framework RAX (Robot Agent eXploration).

The core idea of RAX is to scale up the autonomous exploration of robots. Agents acquire experience online, convert valid experience into executable programs, and continuously improve their own capabilities.

Simply put, RAX enables a general-purpose agent that can write code to "learn by doing" in real or simulated environments, and turn the effective practices it tries into programs that can run repeatedly.

▲ Scaling Online Exploration for Robot Agents

For example, when a robot is tasked with opening an air fryer, the agent initially has to process data from the depth camera, write analysis scripts and test actions at the same time. It took 44 minutes and 23 analysis scripts to complete the first grasping.

But once the agent figures out the effective decision sequence of "how to locate the handle, how to grasp, how to pull the drawer open step by step, how to place the banana, and how to close it", it can compile this process into a modular program, after which the robot can execute it quickly and automatically.

Based on RAX, the team studied how to scale up online exploration, as well as the performance of this method on different robot bodies and tasks.

Experimental results show that as the exploration scale expands, the generalization ability of robots in unseen environments continues to improve. In simulation experiments, the average performance of RAX is 37.4% higher than the best baseline model.

It is reported that Huang Runhan, co-first author of the paper, graduated from the Yao Class of Tsinghua University with a bachelor's degree in engineering in computer science in June this year, and joined Harvard University in August of the same year to pursue his doctorate in computer science. His research mainly focuses on robotics, generative models and agents.

Another co-first author, Xiangcheng Zhang, is also pursuing a doctorate in computer science at Harvard University.

01.

Agents directly write programs to enable robots to learn to operate air fryers autonomously

The VLA model and WAM model fine-tune pre-trained VLMs and video generation models with large-scale teleoperation data respectively. However, deploying the system to a new robot or making it perform a new task usually still requires dedicated collection of teleoperation data.

Such strategies are essentially limited by the demonstrated data they imitate. Even if the robot can interact and has additional computing power during testing, it is difficult to achieve generalization and self-improvement.

This limitation raises a core question: Can robots achieve self-improvement and learn to discover new skills through autonomous online experience and active exploration?

The research team proposed a new idea: let agents write programs directly for robots, so that the entire process does not require manual design of action primitives.

The newly proposed RAX is a cognitive architecture for robotic agents. In RAX, the general coding agent receives natural language instructions, at most a few target images or visual demonstrations, plus an exploration episode budget.

Subsequently, the agent interacts with the robot directly through the robot's perception and control interface. The entire process does not require manual teleoperation, nor does it rely on value function guidance.

The parameters of the underlying language model remain fixed all the time, and the system does not perform policy training. At the same time, the system achieves learning by continuously updating a long-term persistent workspace, which contains programs, notes and verified skills.

The agent will convert the successful interaction process into an autonomously executable code program, and save knowledge that can be reused for subsequent experiments and tasks.

As the exploration scale expands, the research team also observed that the system gradually demonstrates multiple capabilities.

1. Multimodal Perception

To build a model that conforms to the real physical world, the team provides agents with multimodal sensor inputs, including RGB images, depth maps, camera parameters and force signals.

With the help of this information, the agent can use depth data to build a 3D world model, and control the contact process through force feedback.

In the robot air fryer opening task, the agent opens the air fryer drawer, puts a banana into the frying basket, and then closes the drawer.

The agent first converts the observation data collected by the RGB-D camera into point clouds and unifies them into the robot base coordinate system. Then, it uses the data measured by itself to build the geometric model of the air fryer.

When facing a new scene layout, the agent uses a robust estimation method to fit the axis and handle direction of the air fryer, so as to determine its position and posture.

Since the drawer is movable, the model needs to represent the articulated structure of the object. The model consists of a fixed body and a drawer.

Based on this model, the agent plans every action, including grasping, pulling and releasing, and checks whether each motion trajectory will collide.

2. Planning and Control

After building an accurate world model, the agent can directly plan robot actions in 3D space, and use the robot control software development kit (SDK) and kinematic solver to execute the plan.

To connect the agent and the physical robot, the team encapsulates the robot's control SDK as a Model Context Protocol (MCP) tool, and provides auxiliary tools such as inverse kinematics (IK) solvers and robot URDF models.

With the help of these tools, the agent can plan each action and search for feasible motion trajectories. During the planning process, it will consider the robot's posture, collision risk and joint limits.

Taking the air fryer task as an example, there is a short horizontal bar at the top of the drawer handle, and the end of the bar extends downward to form a grasping part.

If the agent grabs the handle from the front like a human, the joint near the root of the horizontal bar will hardly have enough movement margin when pulling the drawer.

Therefore, as shown in actions 1 and 2 in the figure, the agent chooses to grab the short horizontal bar from above and pull the drawer open step by step with a step length of 3 centimeters each time.

The next step is the operation of putting the banana in. The banana is 18 centimeters long, while the frying basket is 19 centimeters long, leaving almost no extra space.

If the banana is kept horizontal and released directly from the hand, it will hit the top of the air fryer. At the same time, the spread fingers may also hit the body of the device.

As shown in action 3 in the figure, to solve this problem, the agent tilts its hand 60° around the axis of the banana, so that the concave side of the banana faces the air fryer. Then, it releases the banana at the position where the middle of the banana just passes the center of the frying basket, and puts the banana into the basket.

All these actions are discovered by the agent itself through searching and trying different control tools during the exploration process, without receiving human suggestions or using human demonstrations.

3. From Exploration to Autonomous Robot Programs

Each action relies on large language model for reasoning, which is costly and slow in execution.

To achieve high-frequency closed-loop planning, the agent will automatically convert the decision-making process into an autonomous robot program, which can execute tasks directly through code.

The RAX agent can directly control the robot without relying on pre-trained policies or manually written action primitives.

During the exploration process, the control right is gradually transferred from high-level reasoning to autonomously executed code programs. Eventually, the agent realizes autonomous decision-making through the code generated by itself.

In the initial stage, the agent controls the robot by calling tools and relies on the large language model to perceive the scene.

When the task is successfully completed, the agent will compile these decisions into a code program. After that, the program can run independently without waiting for the main agent to reason at every step.

For example, after identifying the shape and geometric features of an object, the agent will generate a 3D object model. When encountering a new object layout, the program can use the point cloud generated by depth data to automatically estimate the position and posture of the object.

In this way, the main agent does not need to process the original image repeatedly, which greatly reduces the time overhead and computing cost of the perception process.

The agent will also split the task into multiple stages, and implement modular skills through programs. Each skill includes pre-execution check, action execution and result verification.

The data measured during the exploration process, such as moving distance and grasping direction, will also be converted into program parameters.

The agent will repeatedly test different versions of the program during the exploration process, and judge which specific skill needs to be modified according to the failure situation of each stage.

This design can directly map each failure to a specific stage, making it easy to locate and correct problems.

When the program reaches sufficient reliability, the agent will let it conduct unattended tests in new scenarios. The program has an exception handling mechanism, and the agent will monitor the logs generated by the program to intervene in emergency situations.

If a large change occurs in the open environment that leads to perception or planning failure, the program will throw an exception. At this point, the agent can intervene again, perceive the current environment, and make necessary adjustments.

By testing and debugging programs on a large scale during the exploration phase, the system can eventually generate a complete automated program. Even when facing unseen scenarios, the program can independently complete perception, decision-making and action execution.

4. Memory-based Continual Learning

Robotic agents, like humans, follow a similar cognitive process: repeatedly practicing individual skills, mastering them proficiently, and then combining these skills to fully rehearse complex long-sequence tasks.

The research team lets the agent generate and improve skills through repeated practice, and gradually reuse the existing capabilities in the skill library to complete increasingly complex tasks.

Taking the home scenario as an example, the team lets the agent practice different subtasks, such as grasping and placing boxes and buckets, or sorting and organizing items on the desktop.

During the practice process, the agent will develop autonomous execution programs and improve the robustness of the programs through end-to-end repeated drills. Eventually, the verified programs will be stored in the multi-task skill library.

When executing a long-sequence task composed of multiple subtasks, the agent can directly reuse these skills. At the same time, it will also check the execution records and applicable prerequisites of the skills to cope with environmental changes that have not been encountered during the practice phase.

This skill acquisition and practice process can also be carried out in parallel between multiple robots and multiple tasks, so as to realize collaborative learning between multi-robot systems.

02.

Front-end and back-end separation design: one architecture adapts to multiple robots

RAX uses a unified agent architecture that supports multiple robot bodies in simulation environments and the real world, including single-arm manipulators, dual-arm robots, dexterous hands, mobile manipulation robots and humanoid robots.

To achieve this goal, in terms of design, RAX adopts the common front-end and back-end separation design in software engineering, separating the front end from the back end that provides services for it.

In RAX, the back end is responsible for the physical system of the robot, including hardware drivers, simulators, underlying controllers, sensor processing workflows, and related safety protection mechanisms.

It runs the drivers and real-time controllers provided by the manufacturer, outputs calibrated RGB-D images, as well as the internal parameters and poses of the corresponding cameras. The back end also continuously records the robot's sensor data and status. In this way, the agent can view the actual results after the action is executed.

The front end receives only two types of content from the system: tools and skills.

Tools open up the functions provided by the back end as much as possible, including environment observation, underlying control, and model-based auxiliary functions.

For example, forward kinematics and inverse kinematics solving, motion planning supported by the platform, and preview functions that execute candidate actions only on the robot model. This preview does not include the actual environment scene.

The tools also provide basic geometric functions required to build the perception system, including back-projecting image pixels into 3D points with actual scales, obtaining the pose of the camera in the robot coordinate system, and obtaining the kinematic chain from each camera to the end effector.

Skills are described in the form of documents on how to use these tools. The content includes the robot's coordinate system, action space, control frequency, various constraints and failure modes, as well as perception specifications, exploration processes and memory mechanisms.

These skills are organized around four dimensions: task strategy, collaborative control, perception, and robot-specific control.

The system will combine the corresponding skills into the independent workspace used in each experiment according to the robot body and observation mode, where the agent can observe the environment, make decisions and execute actions, and write programs to implement decision logic at the same time.

The front end and the back end are only connected through the tool interface. This front-end and back-end separation design gives RAX the ability to support multiple robot bodies from the architectural level.

The same front end can be connected to back ends of different platforms, including the real Franka and Rainbow RB-Y1 robots, as well as various simulation benchmark environments.

03.

Six types of benchmark evaluations: an average lead of 37.4 points in five groups of comparisons

In terms of simulation tests, the team evaluated RAX on six benchmarks: RoboDojo, BEHAVIOR-1K, HumanoidArena, DexJoCo, LIBERO and LIBERO-Pro,