Science Robotics: It is no longer a novelty for robots to perform somersaults, and teams from UC Berkeley and Stanford have figured out the critical problem of "when a robot should perform somersaults"

For humanoid robots to truly enter environments designed for humans, walking is only the starting point. They also need to complete highly dynamic actions such as running, jumping, rolling, turning, and obstacle avoidance, and naturally switch skills when tasks change. What is more challenging is that these actions must be stable while having human-like coordination and rhythm.

Traditional methods usually rely on simplified dynamic models, hierarchical planners or reward functions designed for a single task, which are prone to unnatural movements such as "continuous stepping", "long-term knee bending" and "severe body impact"; although Reinforcement Learning (RL) has enabled humanoid robots to master flat-ground walking, stair climbing, running and complex terrain traversal, adding each new behavior often requires new reward design, parameter adjustment, or even training from scratch.

Moreover, these methods are usually limited to specific actions or specific goals, and lack the generality of combining diverse skills, especially when dealing with unseen tasks.

Nowadays, teams from the University of California, Berkeley and Stanford University have achieved new breakthroughs:

Their reinforcement learning-based motion tracking framework BeyondMimic enables humanoid robots to first learn from human motions, and then perform skill combination and task adaptation through a guided diffusion model.

Based on this framework, humanoid robots can learn hundreds of actions under a unified training setup, including aerial cartwheels, spinning kicks, hook kicks, sprints and dances, and transfer these skills to real hardware. Moreover, once deployed, the robot can also handle path navigation, joystick teleoperation, obstacle avoidance and action insertion tasks that were not seen in the training phase.

The research team stated that this work realizes the scalable acquisition of human-like motion skills from human actions, as well as motion synthesis that can generalize and adapt beyond the training setup. Future iterative versions of BeyondMimic are expected to develop models that can learn directly from human demonstrations, and enable robots to perform diverse actions in different environments.

The relevant research paper was recently published in the authoritative scientific journal Science Robotics.

Paper link: https://www.science.org/doi/10.1126/scirobotics.adx8924

Not Only Learning, But Also Proficient in Skill Combination

The goal of BeyondMimic is to enable humanoid robots to synthesize diverse actions in downstream tasks unseen during the training phase, while maintaining continuous agility and human-like naturalness.

As shown in the figure below, the BeyondMimic framework includes the following two stages: in the first stage, scalable reinforcement learning motion tracking is used to train a set of motion skills with high dynamic capabilities from diverse human actions; in the second stage, a unified latent state-action diffusion model is used to integrate these skills into the same motion distribution, and generate and combine actions according to new goals during the inference phase through classifier guidance.

Fig. | Overview of BeyondMimic. A. Through motion tracking, scalable and robust learning is achieved from human motions to obtain agile human-like behavioral capabilities; B. With the help of guided diffusion, the learned diverse motion skills are utilized to realize multi-functional control for unseen downstream tasks.

Stage 1: Learn Human Motions with a Unified Recipe

The task of the first stage is to learn a set of transferable motion skills from human demonstrations. Instead of designing independent rewards and training processes for each action, the research team made different reference actions share the same set of motion tracking formulas, reward design, observation space, action space and hyperparameters.

Previous methods usually face two choices. Training a single multi-skill policy has good scalability, but when reinforcement learning exploration is insufficient, it is easy to produce unnatural actions. Training specialized policies separately for each action can improve naturalness, but it requires parameter tuning for each action one by one, making it difficult to scale to a large-scale skill library. BeyondMimic adopts a unified motion tracking process and shared hyperparameters, achieving a balance between scalability and action naturalness, enabling it to learn different types of human actions while retaining human-level agility.

This stage also combines precise actuator modeling, less domain randomization, continuous orientation representation, no historical observation and low-latency deployment. The research team believes that the sim-to-real transfer capability does not depend on a single component, but comes from the overall coordination of these designs.

The training data contains about 2.5 hours of human motions. The robot verified all actions in high-fidelity simulation, and deployed 21 representative action segments on the real robot, with a total duration of about 15 minutes. These actions cover balancing behaviors such as standing on one leg and getting up from different postures, as well as highly dynamic actions such as one-legged jumps, turning kicks, forward jumps with 180° or 360° rotation, and aerial cartwheels. In addition, the system also learned skills with different styles and contact modes, such as elderly-style walking, dancing, tennis, badminton, ground crawling and getting up from the ground.

Some reference actions last more than 3 minutes and are co-trained with many other skills. Even so, the policy still retains the agility and style details of the actions.

Stage 2: Combine Skills with Latent Diffusion Model

In this stage, the research team trained a unified latent state-action diffusion model to integrate the multiple motion skills obtained in the first stage.

Different from diffusion policies that only generate actions, the latent state-action diffusion model not only predicts actions, but also implicitly captures the impact of actions on future states. This enables the model to predict future trajectories during the inference process, providing a basis for task adaptation and skill switching.

The key capability of the model comes from classifier guidance. It is a gradient-based guidance process that can use task-related cost signals during the generation process to guide unconditional action generation to goal-oriented conditional generation.

During the predictive control process, the model first predicts the states and actions for a period of time in the future, and then gradually corrects the prediction results through multiple rounds of denoising to make the trajectory converge to the specified target. The target can be the desired speed, waypoints, key postures or obstacle distances.

This makes planning and control no longer completely separated. Traditional decoupled solutions usually use a kinematic diffusion planner to generate reference trajectories offline, and then hand them over to an independent physical tracker for execution. Such a process is prone to reintroduce mismatches between the planner and the tracker, and the action quality is limited by the capability of the tracker.

BeyondMimic integrates planning and control into the same latent state-action model, allowing classifier guidance to be adjusted during testing directly on real hardware. The model has learned a set of feasible human motion skills, so new tasks only need to provide relatively simple target costs to trigger appropriate action combinations.

Under this mechanism, the robot can switch between walking and running according to speed commands, and can also smoothly transition from the joystick-controlled walking state to rolling, spinning kick or hook kick, and then return to the original motion mode. By inputting sparse future keyframes, the system can also complete the intermediate actions between keyframes to realize motion inpainting.

The deployment of both stages relies on a real-time C++ framework. Even with only CPU and a moderately performing mobile GPU, the system still achieves strong sim-to-real transfer.

Achieve Both Agility and Naturalness

The research team evaluated BeyondMimic from three levels: first, whether the robot can restore diverse and highly dynamic human actions in the real environment; second, through user studies, to see if the postures of walking and running are natural enough; third, to test whether the unified controller can flexibly switch actions and combine tasks during the inference phase based on targets such as speed, waypoints, keyframes and obstacles.

The results show that BeyondMimic can not only stably transfer multiple motion skills to real hardware, but also maintain coherent and smooth actions with obvious human-like features when facing unseen tasks.

1. Diverse Actions and Human-Level Agility

The research team tested the robot on outdoor soft soil, fallen leaves and uneven ground. These contact conditions did not appear in the training data, but the robot still completed aerial cartwheels, continuous cartwheels, ground crawling, jumping up from the ground and martial arts-inspired actions.

During the aerial cartwheel, the robot's peak acceleration reached 31 m/s², the maximum pelvic angular velocity reached 15.7 rad/s, and the average angular velocity was 7.01 rad/s. The research team pointed out that skilled human aerial movements have been reported to have an average angular velocity of about 7.75 rad/s. The robot maintained good posture control when landing, and hardly needed additional recovery actions.

In terms of continuous actions, the robot can complete five consecutive Cristiano Ronaldo-style celebration jumps without obvious loss of stability or action style. These results show that the motion skills learned in the first stage can not only be transferred to real hardware, but also maintain high dynamic performance in complex, contact-rich outdoor environments.

2. Naturalness of Walking and Running

To evaluate the naturalness of actions, the research team organized 77 participants for a user study. Participants watched 20 groups of walking and running clips from BeyondMimic and Unitree's native controller, and selected which group looked more human-like and more natural.

The research team finally obtained 1539 valid choices. Overall, BeyondMimic received 70.8% of the preferences. In terms of gait, BeyondMimic had a preference rate of 57.0% for walking actions; for running actions, BeyondMimic's preference rate reached 84.7%.

When the robot is slightly supported artificially during walking, it will maintain a certain degree of compliance, pause and stabilize in place; after the external force is released, it will smoothly resume walking. This response avoids stiff or exaggerated reactions, and also reflects the controller's ability to adapt to external disturbances beyond natural movements.

3. Motion Switching and Skill Combination in Unseen Tasks

This work also focuses on whether BeyondMimic can organize the learned skills according to different targets during the inference phase. The research team tested tasks such as speed commands, waypoint navigation, keyframe action insertion, and obstacle avoidance.

In the commanded conditional motion experiment, the system receives linear velocity, yaw velocity or waypoint targets. Facing different initial positions, the robot can generate smooth trajectories to reach the target point. When input via a joystick, it can track omnidirectional speed commands and continue moving towards the target after large external disturbances such as kicks.

In longer-distance tests, the robot ran continuously for more than 50 meters along the track. Simulation evaluation shows that the average speed tracking errors for walking and running are 12.14% and 13.65% respectively. At low speed commands, the policy can generate stable walking, or light jogging; as the target speed changes, the robot can smoothly transition from walking to running without manually switching policies.

The research team also tested the action insertion capability using motion inpainting. The robot was originally in a joystick-controlled walking state, and the system inserted the target keyframe of the rolling action every 0.2 seconds. The diffusion policy then generates a continuous trajectory, allowing the robot to smoothly transition from walking to aerial cartwheel, and return to the command-controlled walking state after completing the action. The same method also supports transitioning from walking to lying down and then standing up, as well as switching from walking to highly dynamic actions such as spinning kicks and hook kicks.

In the task combination experiment, the research team combined the waypoint tracking cost with the obstacle avoidance cost to realize simple scene-aware navigation. When facing an obstacle directly in front, the robot uses obstacle distance information to adjust the future trajectory during the diffusion denoising process, and finally bypasses the obstacle to reach the target point. When the waypoint cost is replaced by the joystick tracking cost, the obstacle avoidance mechanism still works, that is, even if the user input direction is not completely consistent with the safe path, the robot can reduce the collision risk to a certain extent.

These results prove that BeyondMimic is not only able to reproduce the complete actions in the training data. Through classifier guidance in the inference phase, the unified controller can switch between speed, waypoint, keyframe and obstacle avoidance targets, and combine multiple learned skills into new continuous behaviors.

Limitations and Future Directions

Of course, the BeyondMimic framework still has many areas to be improved.

The research team stated that the diffusion model of BeyondMimic is highly dependent on the quality of the underlying state estimation system. If there is an error in proprioception, the error information will be directly propagated to the generated trajectory. The latent diffusion model has a certain degree of robustness to noise, but further combining sensor fusion or learning-based state estimation is still an important direction to improve performance.

The current system's prediction horizon is sufficient to support reactive control and local obstacle avoidance, but it is not enough to complete long-term planning that requires judging distant targets or obstacles in advance.

Historical information helps to stabilize future predictions, but it may also cause the model to fall into repetitive action patterns during the guidance process. The research team mitigates this by increasing the guidance weight, but excessive weight may make the denoising process unstable during mode switching or high-variance states. Therefore, the robot is still more prone to tripping at the beginning and end stages of actions.

In addition, the current guidance-based optimization is more suitable for coarse-grained targets, and the effect on fine-grained control is relatively limited, and light adjustment of the guidance weight is still required. The research team believes that future work can explore supervised fine-tuning and adapter-based control layers to support more fine-grained and composable trajectory control, reducing the need for manual parameter tuning.

The value of BeyondMimic lies not only in making robots complete a certain roll or a certain run, but in integrating high-quality action learning, skill switching, task planning and real-time control into the same framework.

When new human actions can continuously expand the skill library, and new tasks only need to provide target costs instead of retraining for each task, humanoid robots will likely evolve from "executing according to preset actions" to "organizing behaviors according to the environment".

This is also the long-term direction that BeyondMimic points to: let robots learn directly from human demonstrations, and maintain adaptability across different actions, environments and targets.

This article is from the WeChat Official Account "Academic Headline" (ID: SciTouTiao), author: Academic Headline, published by 36Kr with authorization.