Fei-Fei Li teaches robots to touch mahjong tiles and squeeze toothpaste, and AI finally gains a "sense of touch".

In 2009, Li Fei-Fei's team released ImageNet, giving artificial intelligence the first opportunity to recognize the world through massive amounts of images. Fifteen years later, she proposed "spatial intelligence" in a speech, arguing that the next step for AI is not just to identify what is in an image, but to understand where objects are, how they are related to each other, and how the world will change after humans take actions in it.

This year, this research line has taken another step forward. In June, T-Rex, a robotics research project involving Li Fei-Fei, was released. The researchers added high-frequency tactile feedback to the robot, enabling it to complete tasks such as turning book pages, picking up eggs, squeezing toothpaste, pulling out thin cards, and screwing in light bulbs.

Why did a scientist renowned for computer vision start talking repeatedly about tactile sensing, simulators and robots? This does not mean that vision is obsolete. On the contrary, today's embodied intelligence manufacturers are still installing more cameras, more precise depth sensors and LiDAR on robots.

The real change that has taken place is that the industry has gradually realized that "seeing" is only the first step for intelligence to enter the physical world, and an image is only the surface left by the world.

AI May Not Be Looking At All

In March this year, a study involving Li Fei-Fei conducted an experiment: researchers asked a multimodal model a visual question, but deliberately did not actually upload the image. Theoretically, the model should answer "I did not see the image and cannot make a judgment".

However, many cutting-edge models still began to describe the content of the image seriously, give a complete analysis process, and even achieve high scores in some visual question answering tests. For example, in a chest X-ray question answering benchmark, the model had never seen any X-ray film, but its performance still ranked at the top.

Researchers call this phenomenon "phantom reasoning". The model does not necessarily answer questions based on visual evidence, but may use text clues in the questions and common collocations in training data to guess what an image "that usually appears with this question" should look like.

Similar to the fact that large models do not necessarily know what they are talking about, but are predicting "the next most likely word to appear". If the question asks "What signal is the vehicle in the picture waiting for", even without the image, people may guess based on common sense that the answer is nothing more than a traffic light.

After the model has processed enough image captions, diagnostic reports and question-and-answer data, it may also bypass real visual observation and jump directly from the question to the answer. The fact that it sounds very much like it has seen does not mean it has really seen.

In the chat window, as long as the answer sounds reasonable, users may not care whether the model gets it by looking at the picture or guessing through language. But after entering the physical world, the difference between the two will be exposed immediately.

Is the cup on the left or right of the table? Is it ten centimeters or twenty centimeters away from the arm? These are not problems that can be muddled through by linguistic common sense. Especially for robots, a difference of a few centimeters will directly lead to a failed grasp.

This is why increasingly complex perception systems are appearing on today's robots. Unitree G1 uses depth cameras and 3D LiDAR, one responsible for judging the distance of objects, and the other for building a distance map of the surrounding space.

In addition to RGB-D cameras and LiDAR, Agibot Lingxi X2 also uses front-facing binocular cameras, rear-facing cameras and touch sensors to supplement environmental information from different directions.

These configurations look like another arms race for sensors, but they are actually making up for different blind spots in vision. The head camera is suitable for observing the entire room, but it is easily blocked by its own hand when the robotic arm is close to the object; LiDAR can measure distance and build maps, but it cannot tell whether the thing on the table is a cup or an apple.

For humans, turning their heads, squinting their eyes or reaching out to test are actions that hardly require thinking. Robots need to piece together information from different cameras, radars and joint sensors to confirm that they are facing the same, continuously existing physical space.

Therefore, the first problem encountered by embodied intelligence is not whether the model can recognize a cup, but whether it can confirm that such a cup is really there.

Most World Models Are Essentially Just Drawing the World

In the past two years, "model" has become one of the most popular buzzwords in the AI industry. A video generation model can generate a sequence where a robot walks into the living room, picks up a basket, leans close to a toy, and then puts the toy into the basket based on a single prompt.

The light, motion and camera angles in these scenes can be very natural, and viewers can easily get the feeling that since the model can generate this process, it probably already understands how the environment and objects work. But just looking at the video, details like how heavy the cup is, how much force the cabinet door hinge can withstand, and whether the robotic hand will slip when it touches the cup wall do not need to be answered at all. As long as the next frame looks reasonable, the task is completed.

In fact, for robots, these details that cannot be fully demonstrated in the footage precisely determine whether the task can be successful.

World Labs categorized the technologies collectively referred to as "world models" into three types this year. The first type is the renderer, which is responsible for generating pixels for human viewing; the second type is the simulator, which is responsible for representing geometric structures, physical properties and dynamic changes; the third type is the planner, which determines what the robot should do next based on the target.

This classification dimension precisely illustrates the problem we raised earlier: if a model can only draw a world, it does not mean it can work inside it.

A car can drive smoothly through the city in a generated video, but the model may not know whether the streets outside the camera continue to exist, whether the buildings have a stable 3D structure, and whether there is real friction between the wheels and the ground. For the audience, looking realistic is enough; for robots, the world must be computable, collidable, and must continue to exist after it turns around.

Marble launched by World Labs can already generate explorable 3D environments from text, images or videos. In addition to the images for human viewing, it can also output collision meshes, allowing the physics engine to judge whether an object can pass through a wall and where it will stop after falling to the ground.

NVIDIA's Cosmos integrates world generation, video prediction and robot training into the same Physical AI system, hoping to fill the gap of insufficient real robot data with a large number of synthetic scenarios.

This line of research sounds grand, but when implemented into the R&D process of robotics companies, it is actually very simple: let the robot fail repeatedly in the virtual world first.

However, to teach a bipedal robot to walk, it is impossible to rely solely on engineers pushing the real machine thousands of times every day. ExDynamics' TRON 1 supports simulation platforms such as NVIDIA Isaac, MuJoCo and Gazebo, allowing developers to test different ground conditions, slopes and collision scenarios in the virtual environment first. Agibot has released AgiBot Digital World, which uses the simulation environment to generate operational data, train models and evaluate robot policies.

Training platforms supported by ExDynamics

Letting a robot fall in reality may mean component wear, maintenance costs and safety risks; in the simulator, thousands of virtual robots can try the same action at the same time. The height of the table, the friction coefficient of the ground, and the position of objects can be constantly changed. Robots do not have to break ten thousand cups to learn how to hold one steadily.

However, the simulator is not the answer itself. The ground in the virtual world is always cleaner than the real ground, and the friction force set by the model cannot cover every kind of material. No matter how proficient the robot practices in the simulator, after entering factories, shopping malls and homes, it will still encounter unexpected situations that are not written into the parameter table. This is the "sim-to-real gap" that the robotics industry has always faced.

The significance of World Labs' acquisition of SceniX on July 21 lies precisely here. It is no longer satisfied with generating a 3D space that can be viewed and explored, but begins to connect the simulated world with the action feedback of real robots.

Tactile Sensing Is Not Just Another Pair of Eyes

Even if vision and simulation are sufficiently mature, after the robot actually touches an object, it will still encounter a type of problem that cameras are difficult to solve independently.

When picking up an egg, how does the robot know that the force of its fingers is too large? When pulling out a thin card, how does it judge that it is holding the card instead of just touching the tabletop? When screwing in a light bulb, how does it sense that the thread is stuck and further rotation may damage the object? The camera can see that the hand is approaching the target, but it is difficult to judge in time what is happening on the contact surface.

What tactile sensing solves is not "where the object is", but "what happens after contact". Whether the pressure increases, whether the object starts to slide, whether the material deforms, and whether the grasp is stable, all this information occurs under the visual frame.

Helix 02 released by Figure this year integrates the head camera, palm camera, fingertip tactile sensing and whole-body proprioception into the same control system. When the head camera is blocked by the robotic arm, the camera in the palm continues to provide close-up images; the fingertip sensor can sense forces as low as 3 grams, allowing the robot to take out a single pill from a medicine box, control the syringe to push out an accurate dose, and separate very small targets from cluttered objects.

Domestic manufacturers are also rapidly supplementing this perception capability. Upstream manufacturers like Percipio are directly forming a product chain of tactile sensors, dexterous hands, humanoid robots, data acquisition and multimodal models, hoping to turn tactile sensing from a component-level problem into part of robot training data.

Image source: Percipio

But installing sensors on a robot does not mean that it will naturally know how to use tactile sensing. Many abilities called "common sense" do not come from how many images we have seen, but from the body learning to adapt actions to the world through long-term contact.

The T-Rex research involving Li Fei-Fei demonstrates why this step is