RCIM 2026: Xi'an Jiaotong University participates in the release of Video2Knowledge: a single mobile phone video is enough to teach robots new skills.

Over the past two years, the most intractable problem in the embodied intelligence industry has been the data for machine learning.

For a robot to master skills such as grasping, handling, and assembly, it often requires a team of engineers to repeatedly demonstrate with teleoperation equipment, or generate massive trajectories in a simulation environment. A smooth sequence of picking up, moving, and placing may take months of data collection and training. Worse still, once the scenario changes or the parts are replaced, most of the data will become invalid directly.

Everyone hopes to find a lower-cost path. Humans can learn new movements by watching a video once, so why can't robots? The team of Li Feifei and Figure AI are both exploring this direction, and Tesla also announced that it will let Optimus conduct training with the help of employees' operation videos. But this path has an inherent problem: the body structure of humans and robots is completely different. The fine operations completed by five fingers cannot be directly replicated by a two-finger gripper.

To solve this problem, the LISMS Lab of the University of Auckland and Xi'an Jiaotong University jointly proposed a framework named Video2Knowledge. Its core idea is not to imitate human action trajectories, but to enable the machine to understand "what task is happening in the video" and then plan how to complete it on its own. The entire framework requires no depth sensors, no multi-view camera arrays, and no CAD drawings of objects as input, only an ordinary monocular RGB video shot with a mobile phone.

However, monocular video has a fatal defect: no depth information. The position of the same object will "drift" when viewed from different perspectives, and the rotation angle is also prone to "jump". Feeding such biased data to the robot will inevitably lead to inaccurate movements.

To solve this problem, the system first extracts the motion information of hands and objects from the video. For hand pose estimation, the WiLoR model is used to restore dense 3D hand meshes from a single RGB image. The accuracy of left hand tracking reaches 95.1%, 89.2% and 96.0% from three perspectives respectively, and that of the right hand is 87.5%, 95.6% and 97.5%. For object tracking, methods such as OnePoseViaGen are adopted, which do not require CAD models, directly reconstruct the 3D shape of the object from the video, and then track the 6D pose of each frame.

The poses obtained through feature matching often have unreasonable rotation jumps, so the team designed a "physics-aware correction module" for this. The correction is carried out in four stages: first, eliminate position outliers, then detect and correct rotation jumps at the 90° and 180° levels, and finally verify through projection consistency to ensure that the corrected pose matches the image.

The correction effect is very obvious. From the front left perspective, the pose accuracy of the operated object increases from 52.5% to 59.0%, and that of the target object increases from 84.5% to 89.0%. In specific perspectives and tasks, the shape accuracy of the operated object can even rise directly from 18.2% to 95.5%.

However, the paper also points out that in scenarios with fast motion or severe occlusion, the correction module will sometimes amplify the error instead, which is also the problem the team will solve in the next step. Compared with traditional smoothing methods such as Kalman filter and moving average filter, none of those filters can improve the accuracy, and only this correction module is on par with or exceeds the original input in all indicators. Ablation experiments show that after removing the single-axis rotation detection, the shape and pose accuracy of the operated object each drop by about 18%; after removing the projection verification, the rotation accuracy drops directly to the lowest level.

With stable hand and object trajectories, the system starts to analyze semantics. Through clues such as hand-object distance, object motion state, and grasping pose, the continuous video is divided into meaningful action segments, such as approaching, grasping, moving, holding, and placing, while recording the final position and orientation change of the object.

On two public assembly datasets HA-ViD and Assembly101, the extracted task knowledge is compared with the real data reference, and the task sequence consistency reaches 79.1%, and the final geometric state consistency reaches 72.5%.

Finally, key frames are extracted from the action segments as path points, gripper opening and closing instructions are generated, and exported in JSON format to be directly fed to the robot.

After this process was successfully run in the simulation environment, the team verified it on a real production line. The operator demonstrated how to place the detection probe on the workpiece, and the Omron TM5-900 collaborative robot learned by watching the video. After one-time manual configurations such as anchor frame selection, hand-eye coordinate mapping and grasping pose calibration are completed, the system can convert the content into a robot action sequence in about 18 minutes, while traditional manual programming takes about 35 minutes, and teach pendant programming takes about 25 minutes.

If the position of the workpiece or probe changes, just shoot a new video, and the system will automatically recalculate the path without reprogramming. Through two brand-new tasks of loop nesting and multi-waypoint transfer, the system can also successfully extract the task structure and generate executable sequences.

Of course, it also has shortcomings. Under the conditions of small objects, severe occlusion, and fast motion, the quality of pose estimation decreases significantly, and the correction module sometimes amplifies the error. The team said that in the next step, multi-scale detection and time-series perception architecture will be used to improve robustness, and combined task representation will be explored, so that the learned operation primitives can be reorganized into new tasks to achieve longer-range autonomous planning.

In general, from teleoperation to simulation training, from large model fine-tuning to video learning, the ways for robots to acquire skills are being gradually expanded. What Video2Knowledge provides is exactly a low-threshold path: an ordinary video allows the robot to understand the task and then try to execute it. When a workshop worker shoots a gesture with a mobile phone and the robot can learn to do the corresponding work, machine learning may no longer require huge, laborious efforts.

This article is from the WeChat official account "Machine Frontline", the author is a content creator focused on robotics, and published with authorization from 36Kr.