General-purpose robots face an "Everest-level" test: Humans have reached the summit with full marks, while the most advanced AI is still at the foot of the mountain

In the Era of Embodied Intelligence, How Far Are General Robot Manipulation Policies From Reaching the Summit? Existing benchmarks still cannot fully answer this question.

Many existing benchmarks rely on simple, short-horizon tasks or tasks with narrow skill coverage; simulation-based evaluation boasts high efficiency and strong scalability, but struggles to reflect real-world physical deployment capabilities; real-world evaluation is closer to actual deployment scenarios, yet comes with high costs, long cycles, and poor reproducibility.

To address this issue, a joint research team from the University of Hong Kong, University of California, Berkeley, Tsinghua University, Peking University and other institutions proposes a unified evaluation benchmark RoboDojo spanning simulation and the real world, covering diverse, long-horizon, high-precision, memory-dependent and open-ended manipulation scenarios.

Paper Link: https://arxiv.org/abs/2607.04434

Experimental results show that current general robot manipulation policies are far from reliable: even the state-of-the-art (SOTA) models struggle to stably complete complex tasks in RoboDojo, achieving only a 12.8% success rate in real-world deployment, which is drastically lower than the full-score performance of human teleoperation.

Based on these results, the research team established an open leaderboard, systematically analyzed the limitations of current policies in both simulation and the real world, and outlined key directions for future general manipulation policies.

Figure | Overview of RoboDojo.

RoboDojo: A Unified Robot Evaluation Benchmark

Overall, RoboDojo defines its simulation benchmark around five core manipulation capabilities, and the simulation evaluation platform, real-world evaluation platform, and XPolicyLab jointly complete task construction, policy training, deployment, and evaluation.

Figure | Overview of RoboDojo Tasks.

The simulation benchmark covers five core manipulation capabilities: generalization, memory, long-horizon execution, fine-grained control, and open instruction understanding, and systematically diagnoses the weaknesses of policies across different manipulation capabilities through 42 tasks.

• Generalization tasks test the policy's adaptability to unseen backgrounds, lighting conditions, targets, and clutter;
• Memory tasks examine the utilization of historical information;
• Long-horizon tasks evaluate the policy's ability to complete multi-step manipulation sequences;
• Fine-grained control tasks focus on high-precision contact manipulations such as narrow-hole insertion;
• Open tasks test whether policies can transfer learned skills to unseen language instructions and new targets.

In terms of system composition, RoboDojo consists of three parts: the simulation evaluation platform, the real-world evaluation platform, and XPolicyLab, as detailed below:

Simulation Platform: Responsible for task construction, asset generation, parallel evaluation, and data collection. Tasks and scenarios are defined via configuration files, with different objects constructed as digital twin assets; during evaluation, the platform can run multiple scenarios in parallel to improve efficiency; for data collection, trajectories can be automatically generated for simple tasks, while high-quality demonstrations for complex tasks are obtained through VR teleoperation.

Figure | Skill Diversity in RoboDojo.

Real-World Evaluation: Emphasizes reproducibility, reducing random factors in real robot experiments by unifying hardware, workspace, lighting, scenario resetting, and evaluation protocols; RoboDojo-RealEval standardizes task reset and execution processes via a touchscreen interface and layout overlay mechanism.

Figure | Highlights of Real-World Tasks.

XPolicyLab: Unifies the policy development, training, evaluation, and deployment workflow, reducing policy integration costs through standardized data conversion, training templates, deployment processes, and observation-action interfaces.

Additionally, RoboDojo features dual leaderboards for simulation and the real world: the simulation leaderboard repeatedly evaluates policies across the five core capabilities, with experienced VR teleoperators serving as the human reference; the real-world leaderboard, built on RoboDojo-RealEval, evaluates 10 representative policies across 3 robot platforms and 18 tasks, with unified reset, deployment, and scoring protocols.

Figure | Overview of the RoboDojo-RealEval System.

What Are the Evaluation Results of RoboDojo?

The research team found that current general robot manipulation policies are overall unreliable: even the leading models today show significant gaps compared to human experts, and advantages demonstrated in simulation do not necessarily translate to real-world deployment. That said, RoboDojo itself delivers strong evaluation efficiency and reproducibility. The detailed results are as follows:

1. Simulation Results

Current leading policies still fall far short of human performance: Top-performing models include Hy-Embodied-0.5-VLA, Spatial Forcing, π0.5, X-VLA, etc., yet the best policy achieves only an average success rate of 8.80% and a mean score of 13.07, drastically lower than the 76.03% success rate and 80.42 score posted by human experts, indicating a massive gap between current policies and the ability to stably complete tasks.

Figure | RoboDojo Simulation Benchmark Leaderboard.

Capability development across different models is uneven: Spatial Forcing excels at generalization, X-VLA leads in fine manipulation, π0.5 performs relatively better on open tasks, and Hy-Embodied-0.5-VLA delivers the best performance in long-horizon execution, memory, and overall results; however, these advantages are mostly limited to single dimensions and cannot cover the full task set. Even in the relatively strong generalization and long-horizon dimensions, the maximum success rate is below 15%, with fine-grained control, memory, and open instruction understanding remaining persistent weaknesses.

In terms of generalization, scenario randomization drastically degrades policy performance: Hy-Embodied-0.5-VLA leads in standard scenarios, but its performance drops by 92.9% after randomization; Spatial Forcing is relatively more stable in randomized scenarios, yet its absolute performance remains very low. Current policies are highly sensitive to changes in visual and spatial distributions, and reliable generalization still requires more robust control, closed-loop correction, and failure recovery capabilities.

Figure | Policy Performance Under Standard and Randomized Visual Settings.

In terms of long-horizon execution: Improvements brought by action priors, embodied pre-training, and spatial grounding remain limited. The success rates of Hy-Embodied-0.5-VLA, π0.5, and Spatial Forcing are close to 15%, showing that models can advance some multi-step tasks; however, the peak success rate still stays below 15%, and scores are higher than success rates, meaning models often make progress on early steps but struggle to stably reach the final goal, with skill composition, phase decision-making, and error recovery capabilities still insufficient.

In terms of fine-grained control: X-VLA delivers the best performance, with a 12.00% success rate and a score of 18.32, which is still far from reaching a reliable level. Hy-Embodied-0.5-VLA, despite its top overall performance, lags behind X-VLA and Spatial Forcing in this dimension, indicating that overall performance cannot be directly translated into local fine-grained control; 3D localization, smooth action prediction, and closed-loop contact control remain persistent weaknesses.

In terms of memory: Current policies still struggle to effectively leverage historical information to complete subsequent operations. Hy-Embodied-0.5-VLA performs best, with embodied pre-training and action priors helping to utilize historical information; however, the explicit memory design of EventVLA and the implicit temporal memory of X-WAM fail to deliver stable success, demonstrating that memory capabilities still cannot be reliably converted into controllable performance.

In terms of open semantic manipulation: Current policies still struggle to handle open instructions and new targets. Even the top-performing π0.5 achieves only a 1.67% success rate and a score of 1.98. While strong vision-language backbones can improve perception and language understanding, they still cannot stably align open instructions, visual affordances, and executable actions, making the translation from semantic goals to robot behaviors a critical remaining weakness.

2. Real-World Results

In terms of real-world deployment, current general robot manipulation policies are still unreliable. The top-performing policy π0.5 achieves only a 12.8% success rate and a score of 22.9 across 18 real-world tasks, far below the full-score performance of human teleoperation. Scores are generally higher than success rates, indicating that models often complete partial steps but fail to finish the full tasks.

Figure | RoboDojo Real-World Benchmark Leaderboard.

Simulation performance is not fully aligned with real-world deployability. π0.5 performs strongly in both simulation and real-world evaluations, but the real-world ranking does not fully follow the simulation results: InternVLA-A1 and GalaxeaVLA outperform expectations in real-world tests, while some top simulation models show reduced advantages on physical robots. Simulation is more suitable for capability diagnosis, while real-world evaluation verifies policies' adaptability to physical factors such as perception noise, calibration errors, execution latency, and unstable contact.

Real-world evaluation exposes execution and safety risks beyond success rates. Real-world deployment may encounter action jitter, invalid repetitions, unstable contact, and even safety-critical behaviors, with models like DM0 still requiring manual monitoring or intervention. This proves that real-world evaluation cannot only focus on whether tasks are completed, but also needs to verify the control stability and failure recovery capabilities of policies on physical robots.

3. Evaluation Efficiency and Stability

RoboDojo delivers excellent evaluation efficiency and stability. Heterogeneous parallel simulation improves the throughput of large-scale testing; in real-world evaluation, π0.5 completes 180 trials in only 202 minutes, with an average of ~11.2 minutes for 10 trials per task, reflecting the efficiency gains brought by standardized resetting, unified interfaces, and local deployment.

Figure | Real-World Evaluation Time per Task.

Repeated evaluation results show that the maximum standard deviation of success rates in simulation and the real world is only 1.1 and 1.3 percentage points respectively, making the leaderboard results stable and reproducible.

Figure | Full Stability of Real-World Evaluation for Each Task Across Multiple Repeated Runs.

Limitations and Future Development

The