The Tsinghua research team has constructed the HandsOn dexterous manipulation benchmark, which comprehensively evaluates the bimanual capabilities of robots across 66 tasks.
The "test paper" for robotic manipulation needs to be redesigned specifically for dexterous hands.
Most past robotic manipulation benchmarks were built around grippers. With the continuous accumulation of first-person (ego) human manipulation data, how humans grasp, use tools, and complete fine contact operations is being recorded on an increasingly large scale. With a structure similar to the human hand, dexterous hands are expected to achieve larger-scale learning with the help of these data, becoming an important direction for robotic manipulation.
However, as dexterous hands keep evolving, the corresponding "test paper" has not kept pace.
Dexterous hands have a large number of active joints, their tasks are full of contact interactions, they often require bimanual collaboration, and they need to preserve previous results in long action sequences. It is difficult to judge whether the task is successful, and it is also hard to locate at which step the failure occurs. The benchmarks from the gripper era cannot directly solve these problems, while a systematic and reliable evaluation system for dexterous hands is still absent at present.
While data and models are advancing, there is no proper ruler to measure the progress.
Recently, the team led by Ning Ding from Tsinghua University, in collaboration with researchers from Beijing University of Posts and Telecommunications, Peking University and Nature Will, jointly built a composable simulation platform and evaluation benchmark HandsOn for cross-embodiment dexterous manipulation, aiming to fill this gap.
HandsOn contains 66 human-validated tasks and 18,000 human teleoperation demonstration records in simulation, with evaluations centered on tool use, fine contact, bimanual collaboration and task transfer. The platform supports 5 types of robotic arms, 6 types of dexterous hands, and about 200 assets of rigid bodies, articulated objects and deformable objects, which can be combined in randomized scenarios.
HandsOn contains 18,000 carefully selected simulation teleoperation demonstrations, and the tasks are divided into Core-50 and Transfer-16.
The report card given by this benchmark is not easy to achieve: the team evaluated different technical routes including diffusion policies, vision-language-action models, and world-action models. The results show that in the Core-50 tasks with provided training demonstrations, the best-performing trained multi-task policy achieves a success rate of 29.1%. For the Transfer-16 tasks without target task training demonstrations, the average task success rate of all 8 trained multi-task policies is lower than 1%.
In addition, GPT-6, without fine-tuning on the benchmark data, controls the palm and fingers through the move_eef interface. In 660 tests, its success rates on Core-50 and Transfer-16 are 5.2% and 1.9% respectively.
The team further tracked the links including grasping, contact, bimanual coordination and target state maintenance, and found that robots can often complete the first half of the action sequence, but make mistakes at critical moments.
Relevant papers, codes and data resources have been made public.
arXiv Paper: https://arxiv.org/abs/2610.26765
GitHub Code: https://github.com/TsinghuaC3I/HandsOn
Hugging Face Dataset: https://huggingface.co/datasets/Cyan666/HandsOn-Datasets
Why do dexterous hands need a new set of evaluation benchmarks?
The difficulty of dexterous hands is not just that they have more fingers than grippers, but that they need to control the constantly changing contact relationships during the operation process.
After picking up a mouse, the robot needs to avoid triggering a click in advance; when adjusting two knobs, it needs to keep both of them at the target positions at the same time; when placing a box cover, it also needs to maintain the original alignment state after releasing the hand. All these tasks require the robot to coordinate finger movements, judge contact states, and ensure that completed operations will not be damaged by subsequent actions.
This also changes the evaluation method for robotic manipulation. For grippers, whether the grasping is successful and whether the object is placed at the specified position are usually important judgment criteria. For dexterous hands, simply grasping the object is far from enough. The robot also needs to actually use tools, complete fine operations, and maintain the correct object state in multi-step tasks. Progress in one action does not mean that the entire task has been successfully completed.
The accumulation of first-person human operation data provides new opportunities for dexterous hands to learn these skills. However, after the number of demonstration data increases, two questions still need to be answered: can the model stably reproduce the learned operations? Can it transfer existing skills when facing new objects, task combinations or deformable objects?
Existing dexterous manipulation benchmarks have covered different robot embodiments, functional operations and bimanual collaboration. However, there is still room for further improvement in related research in terms of the scale of human demonstration data, dedicated unseen tasks for evaluation, and diagnosis of operation failure processes.
The design of HandsOn starts from this point, separating the measurement of execution ability and generalization ability, and dividing the 66 tasks into two evaluation sets: Core-50 and Transfer-16.
Core-50 provides demonstration data for policy learning, and evaluates policy performance through a new round of independent execution tests; Transfer-16 is specially designed for generalization evaluation, and the demonstration data of target tasks will not be included in the training set.
HandsOn Suite: Core-50 contains demonstration content; Transfer-16 is for evaluation only.
This design not only verifies whether the robot can complete tasks with demonstrations, but also further examines whether existing skills can be transferred to new operating conditions. Combined with phased diagnosis, researchers can also analyze whether the failure occurs in contact establishment, functional operation, bimanual coordination, target state maintenance, or subsequent step connection.
Core-50: Can it perform stably after learning the tasks?
Core-50 contains 50 tasks with provided training demonstrations, covering capabilities such as function understanding, numerical input, spatial relationship, articulated operation, precise contact and bimanual collaboration.
These categories correspond to specific real-life scenarios. When operating scissors, the robot needs to lift them off the tabletop and complete three opening and closing movements while continuously grasping them; when setting a timer, it needs to input the specified number of minutes and seconds, and withdraw both hands after starting; when unfolding a folding fan, it requires both hands to hold the outer fan ribs, unfold it to the target angle and maintain the state. The coffee-making task further connects steps including taking a cup, closing the door, receiving coffee and placing the cup, to examine the connection between actions.
Coffee making: multiple sub-goals in series,
Bottle brushing: sequence and bimanual division of labor,
Unfolding a folding fan: bimanual coordination of joint movements,
Setting a timer: numerical target and keypad operation,
Opening and closing scissors in the air: grasping and functional actions,
Activating an electric drill: contact and function activation,
Six task examples from Core-50. Each task has clear goals, action constraints and completion conditions; all pictures are original images of the simulation scenarios.
The task design is developed along three dimensions.
The first dimension is interaction type, including basic operations, function usage and contact-intensive operations.
The second dimension is task structure, including single-stage operations, repeated operations and long-duration tasks.
The third dimension is object type, covering rigid bodies, objects with rotating or sliding joints, and deformable objects.
In Core-50, the three types of interactions include 17, 18 and 15 tasks respectively. The cross-combination of the three dimensions enables the benchmark to cover operation scenarios of different difficulty levels.
Transfer-16: Can the tasks still be completed when the objects and scenarios change?
Transfer-16 contains 16 evaluation tasks without target task training demonstrations, further examining whether the model can transfer existing skills. Among them, 7 tasks test asset or relationship transfer, 6 tasks test skill combination, and the other 3 tasks involve deformable object manipulation.
Asset or relationship transfer requires the model to face new objects or target relationships, find the correct operation target and complete the task. Skill combination requires the model to connect the learned actions in the correct order. Deformable object manipulation introduces towels, sponges and pillows, requiring the model to cope with the shape changes of objects during the operation process.
For example, in the task of closing the box after passing a cube, the robot needs to first complete the bimanual handover, then put the cube into the box, and finally align and close the box cover. Each local action may have appeared in other tasks, but the model still needs to judge whether the current step is completed and whether the result obtained in the previous step can be maintained.
The requirement for continuously folding a towel is more complex. The robot needs to complete two consecutive folds, and keep the towel stably on the tabletop after releasing it. Flexible objects will change their shape with the grasping position and the way of applying force, making the operation process more difficult to predict.
Mixer knobs: asset transfer,
Closing the box after passing a cube: skill combination,
Continuously folding a towel: flexible object manipulation,
Compressing a pillow into a box: deformation and bimanual handling,
Representative tasks in Transfer-16, demonstrating asset transfer, skill combination and deformable object manipulation.
It can be seen that Core-50 mainly examines the learning and execution ability on tasks with demonstrations, while Transfer-16 further tests whether existing capabilities can be transferred and combined.
Enable failures to be accurately recorded
To ensure the reliability of evaluation, HandsOn has formulated a task card for each task, specifying the goal, standard initial state, expected final state, allowed and prohibited actions, and the evidence for judging whether the task is completed.
The completion conditions can be judged based on information such as object pose, joint state or images. The research team also carried out manual operation verification on the tasks, and set up an independent review process to check whether the success mark is consistent with the actual physical state.
In multi-step tasks, the evaluation will also record the phased progress completed in sequence. For example, the cube handover task will distinguish between the state where both hands are in contact with the object and the state where the receiving hand can independently support the object. The skill combination task will check whether the previous steps are completed and whether the existing results are maintained.
This allows researchers to distinguish two situations: the model does not complete the task at all, or it has completed part of it but fails in subsequent steps.
After 18,000 demonstrations, what has the robot learned?
D