What exactly is the world model? How do Li Feifei, Zhu Jun and others interpret it?

In 2026, "World Model" is one of the most ambiguous terms with no universal consensus in the AI industry.

A continuously generated video is called a world model; a digital environment that changes in real time with keyboard and mouse operations is also called a world model; systems that predict future states in latent space, and policies that directly output robot actions, all use this same name.

All these models process information about the world, but their capability boundaries are vastly different. The frames in generated videos can be realistic enough, yet they violate the laws of real physics; a robot can complete a single grasp, but it may only be familiar with the objects, camera positions and motion trajectories in the laboratory. It is very difficult to tell whether a system has learned visual correlation, physical structure, or the relationship between action and result only through demos.

Conceptual confusion raises the difficulty of technical judgment. Image quality, geometric accuracy, prediction capability and task success rate are mixed for comparison. Different teams seem to compete on the same track, but in fact they may be answering different questions.

Therefore, world model research has returned to a fundamental question: what capabilities must models have to move from generating content to understanding and changing the world?

Li Fei-Fei and World Labs divide world models into renderer, simulator and planner by function, which output pixels, world states and actions respectively.

Link: https://www.worldlabs.ai/blog/taxonomy-of-world-models

Yann LeCun advocates learning predictable world structures in abstract latent space, so that the model retains information useful for understanding, reasoning and planning.

Link: https://ai.meta.com/research/vjepa/

Jun Zhu from Tsinghua University puts forward the third perspective. As early as last December when Motus was released, he publicly elaborated the overall concept of the "General World Model". In March this year, he further positioned it as the foundation connecting the digital world and the physical world. A few months later, Jun Zhu and his team further improved this framework in their paper General World Models from First-Principles, defining core capabilities from first principles and presenting a five-level evolution roadmap.

Paper link: https://www.shengshu.com/assets/gwm-principles-and-roadmap.pdf

Technical report: https://www.shengshu.com/zh/general-world-model/

The paper summarizes the core capabilities of the general world model as Understanding, Imagination and Action. The model needs to form judgments about the current world from historical observations, deduce the possible future brought by different choices, take actions, and correct itself with new observations.

Following this framework, video generation, real-time interaction, latent space prediction and robot control can be regarded as different stages on the same capability roadmap.

How far the world model is from general intelligence has also been transformed into more specific questions: Can it maintain a coherent world? Can it predict the consequences of interventions? Can it learn from real feedback? Can it gradually form goals and organize more complex actions?

First Principles of World Models

Many people have the experience of learning to ride a bicycle.

At the beginning, the body always wobbles. When you see the handlebar leaning to one side and feel the center of gravity of your body starting to sink, you will adjust the direction immediately, and continue to make corrections according to new feedback. As practice increases, the brain gradually learns to predict what result a certain action will bring.

This is the most plain implication of the world model.

In 1943, Kenneth Craik proposed that humans form "small-scale models" of reality in their minds, so as to deduce the possible results of different actions. POMDP in reinforcement learning further describes this process: the agent receives local observations, estimates the world state, takes actions, and updates judgments based on new observations.

Jun Zhu's team summarizes this logic into three core capabilities of the general world model, which are connected to each other to form a continuously updated closed loop:

Understanding: Integrate information such as vision, language, sound, touch or robot sensors into internal judgments of the current world;

Imagination: Deduce multiple possible futures from the current state, especially "what will happen if I take action A instead of action B";

Action: Turn predictions into interventions on the digital or physical environment, and let new observations after actions test and correct the model in turn.

The three core capabilities of the general world model form a closed loop: Understanding is responsible for inferring the current state, Imagination is responsible for predicting possible futures, and Action changes the world and provides new evidence for the next round of understanding.

This is also why video generation is not equal to the whole of the world model.

Video models can answer what the next frame may look like, but to move towards a general world model, it also needs to answer questions like "If I change this condition, move this object, apply this action, how will the world be different?" Such predictions with action conditions touch on causality, counterfactuals and executable decisions.

How do Understanding, Imagination and Action form a closed loop?

To describe how world models evolve, Jun Zhu's team further proposes a five-level roadmap, from L1 Generating the World, L2 Interacting with the World, L3 Acting in the World, gradually moving to L4 Autonomous World Agent and L5 World Organizer.

It is more intuitive to understand this with a glass pushed to the edge of the table.

The L1 model can generate continuous frames of the glass slipping, hitting the ground and shattering; the L2 model can continue to evolve this world after the user changes the perspective and the pushing direction; at L3, the model needs to judge whether the glass is about to fall, predict whether it is too late to reach out, and output the grasping action that the robot can actually execute. Only after taking action, will it know whether its judgment on the weight, friction and grasping timing of the glass is correct.

Going further, L4 no longer waits for instructions from humans, it needs to be able to discover risks, observe actively, supplement information, and correct strategies after failure. L5 further faces multi-agent collaboration, such as who grabs the glass, who avoids obstacles, who deploys tools, and how to rearrange tasks after the environment changes.

The significance of this roadmap is that it turns the term "world model" into a set of problems that can be tested level by level.

L1 checks whether the generated trajectory is coherent, L2 checks whether the world can respond continuously, L3 checks whether the model can change the physical environment, L4 checks whether the system can actively explore and learn continuously, and L5 checks whether it can organize multi-agent collaboration in an open environment.

Formal modeling objectives and acquired capabilities of the five-level roadmap for General World Models (GWM).

According to the judgment given in the paper, existing systems have touched the first three levels, L4 and L5 are still open problems, and the main gaps include causality and physical grounding, persistent memory, online learning, efficient deployment and safety control. The evaluation also needs to compare predictions with real results, and investigate the model's transfer ability on unfamiliar tasks, environments and agents.

From Video to Embodiment: Two Verifications of the Same Hypothesis

The difficulty of world models does not only lie in algorithms.

Internet videos are huge in scale, recording a wide range of objects, people, scenes and movements, from which models can learn spatial structure, object persistence, human behavior and how events usually develop, but videos rarely synchronously record the actions, forces and intentions that cause changes. Robot trajectories can connect observations, actions and results, but the collection cost is higher, and they are easily bound to specific hardware and tasks.

The path proposed by Jun Zhu's team is to put these two types of data into a data pyramid that moves from observation to action.

The data pyramid of General World Models (GWM) and its relationship with the five-level capability roadmap.

The bottom layer is internet-scale videos, used to obtain world knowledge and dynamic priors. Upwards are domain videos, first-view human videos, human demonstrations with action records, and the top layer is real robot interaction data. The higher it goes, the scarcer and more expensive the data is, but the corresponding relationship between actions, tasks and robot agents is also clearer.

This explains why Shengshu Technology is advancing both video generation and embodied intelligence at the same time: Videos provide the breadth of world knowledge, and robot data completes action grounding.

On the digital world side, the Vidu series continuously explores the generation and interaction of the visual world. Video is not the end point of the world model, but an important starting point, which allows the model to learn how objects, scenes, movements and events evolve in large-scale temporal and spatial changes. Vidu S1 further pushes one-time generation to real-time response, allowing user input to continuously change subsequent content, and testing whether the model can maintain continuous state during the interaction process.

On the physical world side, Motus and Motubrain connect environment understanding, state prediction and robot actions into the same link. The model needs to see the environment clearly, predict the consequences of interventions, then turn judgments into executable actions, and accept the test of real results.

Existing representative systems that implement L1 to L3.

The two product lines thus form two verifications for the same world model hypothesis. The digital environment tests whether the model can understand and imagine the world, and the physical environment tests whether this knowledge can support actions and be corrected in feedback.

To make the two types of verification share underlying capabilities, images, videos, languages and robot actions need to perform collaborative calculations around the same world state. Traditional modular systems transfer information step by step, which easily loses geometric, temporal and uncertainty information at the interface. If all modalities share parameters completely, it may also cause training conflicts, making massive video data overwhelm scarce robot signals.

The MoT (Mixture-of-Transformers) architecture proposed by Jun Zhu's team attempts to strike a balance between the two. Different modalities retain their own expert parameters, and then exchange context through a shared attention mechanism. Vision tells the model what is happening in the environment, language provides goals and constraints, actions bring interventions to the environment, and the three types of information jointly update the model's judgment of the world.

MoT provides the foundation for unified computing, and physical laws, cross-agent transfer, real-time reasoning and safety control still need to be solved by data, training and system engineering. Its goal is very clear: the general world model needs a world state that can be read and written jointly by different modalities and continuously updated.

Motus2:

Let the robot run a trial in its own world before taking real actions

If Motus and Motubrain pushed Shengshu's world model exploration to L3, enabling robots to generate actions based on environment understanding and future prediction, then the recently proposed Motus2 further adds result evaluation and strategy feedback, trying to make the action closed loop have the ability of self-improvement.

Motus2 does not make action generation, future simulation and result evaluation three independent systems. On the contrary, it uses a set of video-action models with shared parameters to expose three control interfaces:

Policy / World-Action Model: Propose executable candidate actions;

Simulator / Action-Conditioned World Model: Predict the possible consequences of these actions in the visual world;

Evaluator / Value Model: Judge which result is closer to the task goal, and use this judgment for selection and improvement.

You can think of it as a real process of "being thoughtful first, then dexterous".

The robot no longer just gives an action directly after seeing the object, but first proposes several candidate schemes, rehearses the frames and task progress that each will bring in the model, and then chooses the better step. After execution, successful, failed and less ideal results are no longer just data waste, but will become evidence for the next round of dynamic modeling and value learning.

Actions, predictions, evaluations, feedback and re-actions thus enter the same decision-making and learning loop, which is also the closed-loop self-evolution emphasized by Motus2.

On the data side, Motus2 adopts a hierarchical training path, expanding from monocular first-view videos to synchronized binocular first-view data, and then completing agent adaptation through robot data. Its training uses about 130,000 hours of human first-view records, and more than 100 hours of robot trajectories and human-robot alignment data.

This path is to let the model accumulate a common experience about how humans interact with the world first, and then let it learn how this body turns understanding into actions through robot data.