When embodied intelligence has evolved to the world model stage, there are unexpectedly far more "problems" than before.

Embodied intelligence is at a critical point where technical routes are accelerating differentiation, with growing explorations and an increasing number of new models and new methods emerging.

"Speaking of the technical routes for embodied intelligence, they are indeed a bit dazzling," an industry practitioner in the technology sector told Tencent Technology. "Two years ago, everyone was still talking about VLA, but in the past year, the industry has started frequent discussions on new directions such as world models and latent spaces. Some teams are also continuously integrating reinforcement learning, tactile perception, and online feedback into models. Every company has its own set of arguments, and the more you listen, the harder it is to judge which route is more likely to lead to the future."

Xi Ning, Director of the Robotics Branch of the Chinese Institute of Electronics, Chair Professor and Head of the Department of Robotics and Automation at the University of Hong Kong, and Director of the Institute of Emerging Technologies, told Tencent Technology that the current embodied technical routes are still in a stage of "a hundred flowers blooming". Different teams are experimenting with their own methods, and which route is truly effective still needs to be tested in practice. Before the routes converge, new ideas and methods will continue to emerge.

The differentiation of routes continues to extend further. Even if they are all called "world models", different teams have different answers about what exactly they should learn and what problems they should solve.

On September 1 local time in the United States, the team led by Professor Li Feifei from Stanford University released Atlas, the world's first multimodal world model, whose core capabilities revolve around "understanding and simulating the 3D world", aiming to enable the model to understand the world through the position, shape, distance and spatial relationships of objects.

A few months ago, Li Feifei published an article titled A Functional Taxonomy of World Models to sort out her thoughts on "world models". She divided world models into Renderer, Simulator and Planner according to their functions: some are responsible for generating a seemingly realistic world, some try to simulate how the world operates, and others ultimately help agents determine what to do next. All things currently called "world models" are essentially projections of this cycle in three different directions.

In Li Feifei's classification, these three capabilities may eventually converge, but today's world models are clearly still in the stage of parallel exploration of multiple paths.

On August 24, Turing Award winner Yann LeCun and other scholars published the paper LpWM: A Case for Sparse Representations in World Models, further studying how world models should represent the physical world internally. The paper proposes to use sparse representations to replace the dense representations previously adopted by LeWM, making it easier to predict changes in the world state after an action occurs, thus reducing the complexity of the predictor required for planning.

A few days later on August 27, Yann LeCun and his team released LeVJEPA, further extending the previous learning method of LeJEPA to videos, to continue researching how AI can understand the world from videos. It aims to use less computing power to learn how objects move, how the environment changes, and the connections between previous and subsequent states from a large number of videos.

From industrial companies to cutting-edge researchers such as Li Feifei and Yann LeCun, the core focus of the industry is actually the same problem: how exactly models can learn to understand and represent the real world, and further convert this understanding into capabilities of prediction, planning and action.

Around this problem, domestic embodied intelligence teams have also begun to give their own more specific technical answers, and different teams are advancing from directions such as VLA, world models, and reinforcement learning respectively.

"This is also the core goal of our physics-native model: to truly understand the laws of the world at the underlying level," Zhang Tao, Founder and CEO of LightX Tech, stated at a recent media exchange meeting.

LightX Tech, with a Tsinghua University background, recently jointly released the first-generation physics-native world model Phi-WM 1.0 ActEffect with the research group of Professor Li Shengbo from Tsinghua University. This route focuses on physical states, state transitions and action impacts, hoping that the model can understand how the world changes and what results robot actions will bring, ultimately aiming to improve the generalization ability of robots.

Zhang Tao believes that what embodied intelligence really needs to verify next mainly falls into two aspects: one is whether robots can understand the underlying laws of the physical world, so as to reduce repeated trial and error when facing new objects and new tasks; the other is whether this capability can enter real scenarios, solve problems that traditional automation has long been difficult to cover, and create quantifiable production value.

In the view of Lv Yao, Chief Scientist of LightX Lab, "understanding" in the physical world must ultimately be implemented on what exactly the model learns from data: whether it continues to fit the statistical relationship between input and output, or can identify how states change, why these actions lead to different results, and which of this information can be transferred to new tasks and scenarios.

"Relying only on successful demonstrations for imitation learning can only bring limited gaps between models. What truly raises the upper limit of capabilities is counterfactual data formed by different actions and their success or failure results under the same state." Lv Yao said.

The data scale of the embodied industry will continue to grow, and models will continue to iterate, but what truly widens the gap may increasingly fall on the matter of "what exactly the model has learned".

Zhang Tao, Founder and CEO of LightX Tech, and Lv Yao, Chief Scientist of LightX Tech. The image is AI-generated.

The following is the selected transcript of the media exchange meeting with Zhang Tao, Founder and CEO of LightX Tech, and Lv Yao, Chief Scientist of LightX Lab:

01

Let Robots Figure Out "Why" First

Q: LightX Tech has released the physics-native world model Phi-WM 1.0. What exactly does the term "physics-native" refer to here?

Lv Yao: We mainly have two layers of understanding.

First, the Phi system is designed from the very beginning for the interaction between robots and the real physical world.

Second, in terms of model architecture and training methods, we hope to learn the truly native information of the physical world from interaction data, the core of which is objective physical laws and the causal relationship of state changes driven by actions.

Q: At present, the mainstream of the industry is still VLA, while more and more companies are starting to develop world models. Why does LightX Tech think it is necessary to switch to a different route?

Lv Yao: Based on our judgment, VLA has begun to touch some capability boundaries.

VLA is good at learning the mapping between input and action output, so it performs well in imitating human demonstrations, but it has limitations in environmental understanding and long-horizon task reasoning.

In addition, many current VLAs rely on language models at the front end, and then map language tokens to the action space at the back end. We believe there is a relatively large modal gap in this process.

Therefore, relying only on expanding the data scale to train VLA to obtain robot policies with truly universal generalization capabilities is very difficult in our opinion. This is also why more and more teams are turning to world models.

The world model we understand should be able to represent real world states in the latent space, learn how these states transition, and especially understand the impact of robot actions on state transitions.

Q: How does LightX Tech judge the current embodied intelligence route that focuses mainly on prediction? What other problems need to be solved for this route?

Lv Yao: The pure video prediction route is not ineffective. It can well benefit from the Scaling Law, as more data usually leads to better performance.

But the real difficulty lies in moving from 99% to close to 100%. If the model mainly relies on statistical fitting to learn the mapping between input and output, without physical laws and structural constraints, problems similar to the "hallucination" of large language models will be very difficult to completely solve.

Robots ultimately need to perform actions in the physical world, which have higher requirements for accuracy, stability and safety. Therefore, we believe that models need to learn physical laws and action causality, and truly understand how an action will change the world.

Zhang Tao: Let's take a simple example. Throwing objects of different weights and shapes into a basket can be learned by the model after a lot of training. But when encountering a new object that has never been seen before, if it completely relies on data fitting, it may still need to go through many attempts again. If the model understands the underlying laws such as gravity, mass and shape, it can adapt faster when facing new objects. This is also the core goal of our development of physics-native intelligence.

Q: Why do you think the states in the latent space are more suitable for robots than directly predicting future videos? Why do you split physical states into explicit and implicit parts?

Lv Yao: The advantage of pixel space prediction is obvious, as it can make good use of video data and the Scaling Law. But we think it has two problems:

The first is reliability. Video generation models themselves have hallucination problems, while robots have very high requirements for success rates.

The second is reasoning efficiency. If you need to continuously predict future frames during deployment and then generate actions based on the results, the entire link will be relatively long, which is not conducive to industrial real-time control.

More importantly, for a robot to complete a specific task, it does not need to fully restore all the visual details of the future world. We prefer the model to extract state information directly related to the task, learn how these states change in the feature space, and then use this state transition to help the policy generate actions.

On this basis, we further split the states into two parts: explicit and implicit.

Many current methods will uniformly encode different inputs such as videos and ontology states, and then hand them over to neural networks for learning. But we believe that some physical states can already be clearly described by humans, such as position, speed, force, pose, etc., and their change laws can be characterized by dynamics or kinematics. There is no need for this part to completely rely on neural networks to learn from scratch, and existing physical knowledge can be directly utilized.

For information that is difficult to fully describe through explicit formulas, such as images and 3D point clouds, we then use neural networks to learn the states and change laws within them. This is what we call the explicit-implicit state decoupled representation: parts that can be explicitly modeled directly use physical knowledge, and parts that are difficult to explicitly modeled are handed over to data-driven methods for learning.

Q: Many teams are currently working on latent space world models. What is the biggest difference between LightX Tech's "physics-native world model" and theirs?

Lv Yao: Many current implicit world models mainly solve the representation problem, that is, they no longer reconstruct the entire pixel space, but perform action reasoning in the latent space.

For us, the latent space is just one of the features. We pay more attention to whether we can extract physical signals truly related to the task in the latent space, further learn physical laws, and the impact of actions on future results.

The key here is not just "whether it is a latent space", but what the model has learned in it.

02

With More Data, You Still Need to Know What to Learn

Q: From the perspective of physics-native intelligence, what kind of data determines the upper limit of robot capabilities?

Lv Yao: We believe that a very critical data source is counterfactual data generated by a large number of robot trials and errors in the simulation environment.

If you only rely on imitation learning to fit successful demonstrations, the gaps between models of different companies may not be particularly large.

More importantly, we need to let the model know that changing an action under the same state will change the world, and which actions will succeed and which will fail. So what is really scarce is not just more successful demonstrations, but data that can help the model complete counterfactual reasoning and contrastive learning.

Q: After entering the physical world, does the Scaling Law still hold? Can continuously expanding the data scale continue to improve the model capabilities of robots?

Lv Yao: I think it still holds, and we need to stick to it. The amount of data required to fully describe the physical world may be even larger than the scale of language data.

The current problem is that the data scale of the entire embodied intelligence industry is still far from this level, especially if it mainly relies on real robot data, the quantity is even more limited. So we do not oppose the Scaling Law, nor do we oppose data-driven approaches. Phi is also data-driven.

What we rethink is the same question: as the data scale increases, what exactly should the model learn from these data.

Q: What roles do Internet videos, Ego data, UMI data, real robot data and simulation data play in training respectively?

Lv Yao: We emphasize the concept of "data functions".

Internet videos and Ego data without action annotations are low-cost and large-scale, which can help the model establish the most basic world prior. Ego, UMI, real robot and simulation data with action annotations further help the model learn the relationship between actions and state changes. At the stage of specific robot configurations and tasks, we will add more real robot teleoperation and simulation data.

A very critical stage is to carry out large-scale counterfactual sampling in simulation, so that the model can see what results different actions will produce under the same state. Finally, fine-tuning is carried out on real robots and in real scenarios.

Q: Will mixing data from multiple sources interfere with each other?

Lv Yao: The key is not to simply mix all data together and train them with the same objective. Different data have different functions. For example, Internet videos establish world prior, action data help understand state transitions, and simulation data undertake counterfactual exploration. We pay more attention to what capabilities different data can bring to the model, and at which training stage they should be used.

Q: Internet videos do not have unified standards and action annotations, are they of great practical value?

Lv Yao: Our practical experience is that the processing difficulty is not that high, just clean out the data with particularly poor quality.

Internet data is mainly used for general pre-training, similar to the cold start of the model. But for us, the more critical parts are mid-training and post-training, especially counterfactual correction of state changes caused by actions.

Q: So in your training system, simulation data will account for a large proportion?

Lv Yao: At least at the current stage, we believe that simulation data should account for the absolute majority. Because learning state transitions requires a large amount of data of "what results different actions will produce under the same state". It is difficult for the real world to keep returning to the same state to let robots try different actions repeatedly, while simulation can generate such data at low cost and on a large scale.

We judge that the demand for this type of data may be ten times or even a hundred times that of real successful demonstration data.

Q: If you enter a new workstation or task, how much real robot data do you usually need to supplement to complete model adaptation?

Zhang Tao: This first depends on what technical path is adopted.

If you completely rely on real data for fitting, the data volume may need to reach millions or even higher. But we are not completely following this route at present. Our approach is that we first complete pre-training through simulation and real robot data to give the model basic capabilities, and then enter real robots and real scenarios for post-training.

Based on current experience, the real robot post-training data that a specific task really needs to supplement is about tens of hours to dozens of hours, generally no more than 100 hours. The pre-training and simulation learning in the early stage can significantly reduce the subsequent demand for real robot data.

Q: Compared with large language models, the parameter scale of current embodied models is generally much smaller. What are the main limiting factors?

Lv Yao: At present, we focus on solving rigid body objects and related operations in industrial scenarios. Within this scope, the existing model scale is basically sufficient. Another limiting factor is data. If there is not enough data to "feed" the model, simply continuing to expand the parameter scale does not make much sense.

In the long run, if we really want to build