Robots Learn Manipulation by Watching Videos: UC Berkeley First Establishes the Deployment Pipeline from Internet Videos to Real Dexterous Hands

[Introduction] Using only monocular RGB videos, "Do as I Do" transforms daily human operations into executable trajectories for the Sharpa Wave, bridging the crucial link from video to robot data in human - like dexterous operations.

Humans often start learning dexterous operations by "watching".

Children watch others crack eggs, pour water, and hammer nails, and gradually learn these actions through imitation. However, robots are different. Today, robot learning still relies more on "doing", such as high - cost teleoperation, a large number of simulation executions, or collecting real - machine data in carefully arranged scenarios.

In fact, data available for robots to "watch" already exists. YouTube, first - person datasets, and generative videos already contain a vast amount of materials of human hands interacting with objects. The real bottleneck is not the lack of data , but whether data conversion can be completed: how to transform these noisy monocular RGB videos into action trajectories that a multi - fingered dexterous hand can execute?

The end - to - end process proposed by the UC Berkeley team aims to solve this problem. The research team has established the first complete link that can generate real - machine execution trajectories for a real dexterous hand from online videos: first, reconstruct the 4D hand - object interaction process from monocular RGB videos in real scenarios, and then redirect these interaction trajectories to the Sharpa Wave dexterous hand with 22 degrees of freedom.

Paper link: https://arxiv.org/abs/2606.19333 

Project link: https://do-as-i-do.com/

The entire path generated 500 verified trajectories for 20 types of operation actions and deployed 10 real tasks at 50Hz on the dual UR3e robotic arm + dual Sharpa Wave platform.

Problem: "Seeing" ≠ "Doing"

To scale up dexterous robot data, three structural problems still need to be addressed:

It is still difficult to stably reconstruct hand - object interactions in monocular RGB videos

Videos in real scenarios often have problems such as motion blur, occlusion, and depth ambiguity, and the types of objects are not fixed. Tracking methods like FoundationPose may lose pose locking under slight blur. Some joint reconstruction methods are more dependent on laboratory environments or can only handle pre - set object categories.

Without stable 4D hand - object reconstruction, human videos are difficult to use in robot learning.

Noisy reference trajectories can invalidate action redirection

Previous dynamics - aware action redirection methods, such as SPIDER or tracking methods based on reinforcement learning (RL), usually assume that the input is clean MoCap ground - truth data. However, in fact, the reference trajectories reconstructed from online videos may not be clean. They may have temporal discontinuities, misaligned contact relationships, and even contain physically impossible initial states.

These issues will directly affect subsequent optimization. The paper's experiments show that the failure rate of directly using sampling - based optimization methods on such noisy reference trajectories can reach 75%.

Teleoperation itself is difficult to scale up

Teleoperation can provide real robot data, but it is costly. It relies on professional operators, dedicated equipment, and requires individual data collection for specific tasks. Relying solely on teleoperation, it is difficult to cover the rich operations in an hour of human cooking videos, let alone the vast amount of human videos on the entire Internet.

Therefore, the question that "Do as I Do" wants to answer is: Can a robot go from "seeing" to "doing" using only monocular RGB videos, without presetting grasping priors and without limiting the types of rigid objects?

Solution

The process of "Do as I Do" is divided into two stages:

Stage 1: Stably track objects using guided diffusion

SAM 3D can generate object meshes for single - frame images. However, if each frame is processed independently, the generated results are prone to drift and it is difficult to maintain temporal continuity.

Therefore, "Do as I Do" tries to first select an anchor frame and fix the object's shape in this frame. During the flow - matching denoising process of subsequent frames, the system will make the pose sampling results of the current frame approach the pose of the previous frame, so as to obtain more continuous pose trajectories while keeping the object's shape consistent. At the same time, the system will also adaptively adjust the pose according to the object's rotation speed estimated by 2D point tracking. This can avoid overly rigid tracking and reduce incorrect flips.

In the manual comparison and evaluation of 150 real - scenario videos, evaluators considered the tracking results of "Do as I Do" better than FoundationPose in 67% of the samples. In many samples, multiple evaluators gave consistent judgments.

Stage 2: Robust action redirection for noisy reference trajectories

Based on the sampling/MPPI optimization framework of SPIDER, "Do as I Do" further adds three designs to handle the noisy reference trajectories reconstructed from online videos:

After these improvements, "Do as I Do" increased the success rate of action redirection from 25% to 71% on noisy real - scenario reference trajectories.

Experimental Results

Reconstruction ability benchmark test (SOTA)

Action redirection benchmark test

Data sources of 500 verified trajectories

This method ultimately covers 20 types of operation actions. These actions are not simple picking or placing, but more complex operations closer to human daily life, including placing, picking, scrubbing, smearing, squeezing, ironing, brushing, dusting, digging, erasing, pouring, writing, whipping, stirring, stabbing, compacting, drilling, hammering, cutting, and brushing sauce.

Real - Machine Deployment

These trajectories are not limited to simulations. The research team selected 10 representative actions from them and deployed them on the dual UR3e robotic arm + dual Sharpa Wave dexterous hand platform, and completed real - machine execution at a control frequency of 50Hz.

The deployed actions cover different object shapes and various grasping methods, including three - finger writing - style grasping, power grasping, palmar grasping, and parallel extension grasping.

The Sharpa Wave has 22 degrees of freedom and is close to the size of a human hand, so it is more suitable as the target body for migrating human hand actions. Actions such as whipping, stirring, and hammering require the cooperation of both hands, which are difficult to achieve with traditional parallel grippers. The gesture switching frequency of the Wave exceeds 4Hz and the fingertip force is 50N, which can meet the force and speed requirements of these actions.

From reconstruction, simulation (MuJoCo Warp, 200Hz) to real - world deployment, the research team uses the Sharpa Wave as the target hand shape for action redirection and migrates the operation trajectories in human videos to this body.

EgoScale also redirects human hand key points to this hand shape, and CAIP conducts evaluation and verification on the Dexmate Vega + dual Wave platform. Since the target hand shape is closer to a human hand, the morphological difference that the system needs to cross when migrating human actions to robot execution is smaller.

Screening Manual: Why 95% of Online Videos Cannot Be Used Directly

For teams hoping to scale up the use of human video data, including research teams in directions like EgoScale, "Do as I Do" also gives a very practical reminder: more videos are not always better. The ability to screen out usable data is equally important.

The research team analyzed 2000 10 - second video clips from the 100DOH dataset (which have been screened for hand - object interactions):

The result is straightforward: if the original videos are not pre - processed and are directly used for robot learning, only about one - twentieth of the data may be truly usable. Therefore, "Do as I Do" has also summarized a set of data screening points: Check whether the hand and the object are always within the frame, confirm whether the action spans camera switches, exclude clips with excessive camera movement, and identify situations where SAM 3D may fail. For any team hoping to establish the "human video to robot execution" process on dexterous hands, this screening process will be an essential basic step.

Conclusion: Human Videos Are Becoming Robot Data

For a long time in the past, "Do as I Do" was more like an ideal in the field of artificial intelligence (AI): to enable robots to understand human demonstrations and migrate these actions to their own bodies. The research by UC Berkeley is turning this ideal into reality: input a video link, and the system can reconstruct the hand - object interaction process in it and transform it into an executable action trajectory for the Sharpa Wave.

In a sense, the world's largest operation dataset already exists - it is hidden in the videos that people shoot, upload, and share every day. What "Do as I Do" needs to do is to transform these videos into 22 - degree - of - freedom joint trajectories that a dexterous hand can execute.

Watch, reconstruct, redirect, and then execute on a real robot.

Reference materials: https://do-as-i-do.com/

This article is from the WeChat official account "New Intelligence Yuan", edited by LRST, and published by 36Kr with authorization.