The world model that everyone is raving about is becoming the biggest "concept basket" of the AI industry.

The "world model" has arguably been one of the most hyped yet ambiguous terms in the AI industry over the past two years.

At GTC, Jensen Huang framed "Physical AI" and world models as the core of the next generation of artificial intelligence; autonomous driving companies view world models as an unavoidable path to high-level intelligent driving; robotics manufacturers claim their world models can endow robotic arms with general manipulation capabilities; even video generation teams have rebranded their latest models as world models, emphasizing that the ability to generate video equates to the ability to model the world.

World models are rapidly becoming a new "universal label" in the AI field, much like the metaverse in previous years and large language models last year. Slapping this label on any product is seen as a ticket to the center of capital attention and public discourse.

Behind this phenomenon lies the industry's collective anxiety over diminishing marginal returns from large language models: as internet text data is fully mined and the novelty of generative content fades quickly, AI urgently needs to find the next trillion-yuan application scenario, which requires moving from the world of bits to the world of atoms, and from processing information to manipulating physical entities.

Precisely because world models hold such enormous potential capabilities and use cases, they still face significant gaps and dilemmas before reaching true practical usability. This is not a problem of isolated technological breakthroughs, but a systematic puzzle spanning three dimensions: concepts, data, and architecture. Beneath each layer of difficulty lies a cognitive chasm that AI must cross to transition into the physical world.

Concept Precedes Consensus: A Label Carnival With Diverse Hidden Agendas

The first major dilemma facing world models is precisely where the hype is most intense—the concept itself.

Numerous video generation models, 3D reconstruction tools, and multimodal large models are all rushing to attach this label, yet the industry has never reached a consensus on the definition, technical routes, and evaluation standards for world models. The same term refers to completely different technologies in the mouths of different companies.

This is by no means accidental. The essence of conceptual chaos is that players across different tracks are leveraging the world model narrative to seize technical discourse power for the next generation of AI.

For content generation companies, repackaging video generation as a world model transforms the old AIGC story into a new narrative of interactive world generation with greater imaginative potential, directly elevating their valuation space.

The world model referred to by robotics enterprises specifically describes the modeling capability of robotic arms for the physical properties, spatial positions, and interaction feedback of operational objects, aiming to build a technical barrier in the homogenized hardware red ocean.

For autonomous driving companies, world models enable real-time prediction of the motion trajectories of traffic participants and environmental changes, serving as the core driver for upgrading from passive perception to active anticipation, and an essential element in the narrative of high-level intelligent driving.

For computing power vendors like NVIDIA, the world model is a foundational model built on a simulation infrastructure similar to Omniverse that connects the full "perception-simulation-planning" pipeline, delivering general modeling capabilities to the entire industry.

Diverse commercial demands have caused the same term to carry completely different technical connotations.

To clarify this concept, Fei-Fei Li attempted to establish an analytical framework for this chaotic situation in a long article published this June. She categorized existing world models into three types:

Renderers only focus on "looking right," generating beautiful pixels and videos without guaranteeing physical or geometric correctness. Typical examples are Google's Genie and OpenAI's Sora, with core metrics being visual realism and spatiotemporal coherence.

Simulators pursue structural precision, outputting information like geometric data, material parameters, and collision meshes rather than visuals. For instance, NVIDIA's Omniverse physics simulation module and Unity's PhysX AI serve as the core foundation for digital twins and industrial simulation.

Planners act as bridges between perception and action, enabling agents to anticipate world changes before taking action—such as trajectory prediction networks in autonomous driving and robot motion planning models, which directly support agent decision-making.

Yet this classification itself reveals the problem: when a technical concept requires lengthy explanations to define its boundaries, it means it is far from reaching the stage of technological convergence.

Before 2012, deep learning also went through a phase of multi-route competition, but that earlier debate was ultimately resolved by data and computing power. To this day, there is no unified benchmark for world models: video generation solutions use FVD (Frechet Video Distance) and CLIP scores to measure performance; robotics solutions verify capabilities through grasping success rates and task completion rates; autonomous driving solutions evaluate value via trajectory prediction errors and takeover rates. Without a unified measuring stick, there can be no coordinates for technological iteration. This conceptual melee will likely continue for a very long time.

The Rules of the Bit World Do Not Apply to the Atom World

When training large language models, data is nearly inexhaustible: web pages, books, papers, and posts from the internet can be scraped and used with extremely low labeling costs. But training a model that understands the physical world requires a completely different type of data—multimodal interaction data with precise geometric annotations, physical parameters, and action labels.

The root of this gap lies in the fundamental difference in information dimensions. Text consists of discrete, standardized, unimodal symbols, with relatively fixed word meanings that make labeling simple. The physical world, by contrast, is continuous, high-dimensional, and multi-causally coupled. Take the simplest action of "picking up a paper cup" as an example: it involves dozens of physical quantities including visual texture, spatial depth, finger force, cup deformation, friction coefficient, and motion acceleration. All data must be time-aligned at the microsecond level, or it loses its training value.

More intractably, even if you spend enormous sums to collect data, it may not be the right data.

The cost of collecting real physical data is staggering. Take autonomous driving as an example: the hardware cost of LiDARs, cameras, and IMUs on each test vehicle exceeds one million yuan. When factoring in labeling and vehicle operation costs, the comprehensive cost of collecting just one hour of real road test data can reach thousands of yuan. To cover long-tail scenarios like rain, snow, nighttime conditions, construction zones, and irregular obstacles, millions of kilometers of road test data are required, resulting in astronomical total costs. The robotics field is even more extreme: Figure AI once disclosed that the collection cost of one hour of real operational data for humanoid robots is thousands of times higher than that of text data for large language models, with additional risks of hardware wear and safety incidents.

Current world model application scenarios are largely limited to specific domains like autonomous driving and video games, where data scale and diversity cannot support a general model. The scenarios in the real physical world are infinite: objects under different lighting conditions, varying friction levels from different degrees of wear, collision effects at different angles... these long-tail scenarios, which are key to testing a model's generalization ability, can never be fully covered through collection alone.

Synthetic data was once seen as a silver bullet to break this deadlock. Using physics simulation engines and game engines to generate virtual data in batches is far cheaper than real-world collection. But the pitfalls on this path are deeper than many people expected.

Synthetic data was once seen as a silver bullet to break this deadlock. Using physics simulation engines and game engines to generate virtual data in batches is far cheaper than real-world collection.

The industry has now formed three mainstream solutions:

The first is to generate standardized dynamics data based on classic physics engines such as MuJoCo, Bullet, and PhysX—examples include DeepMind's DM Control Suite and OpenAI's Gym.

The second is domain randomization technology, which improves model generalization by randomly changing lighting, textures, and physical parameters in the simulation environment. The most iconic case is OpenAI's Dactyl robotic hand, which was trained in simulation through randomized friction coefficients, lighting, and other parameters, and ultimately completed the task of solving a Rubik's cube in the real world.

The third is generative AI completion, which uses diffusion models to generate realistic textures and narrow the visual sim-to-real gap—for example, NVIDIA's Drive Sim uses this technology to enrich details in simulation scenarios.

But the pitfalls on this path are deeper than many people expected.

Many people assume the sim-to-real gap is caused by insufficiently realistic visuals, but that is not the case at all. The real gap lies in the shift in physical distributions: friction coefficients, elastic moduli, and air resistance in simulation engines are all artificially set ideal values, but physical parameters in the real world change continuously and influence each other. For example, the friction of a piece of rubber varies with different temperatures and levels of wear, and these subtle differences cannot be 100% replicated by simulation.

Public industry tests have shown that a robot model with a 98% grasping success rate in simulation often sees its success rate drop directly below 60% when transferred to the real world. Even with domain randomization optimization, it is difficult to break through the 85% bottleneck, and the remaining gap must be filled through fine-tuning with real data.

Current popular industry solutions, such as real-world data closed loops and the hybrid "synthetic pre-training + real fine-tuning" approach, only alleviate the contradiction and cannot fundamentally solve the problem.

The growth law of the bit world is that marginal cost tends toward zero, while the cost law of the atom world is that every step forward requires significant real investment. This fundamental logical conflict is an unavoidable chasm for world models.

Three Routes, Three Philosophies for Understanding the World

Even with a unified definition and sufficient data, a deeper problem remains: what architecture should we use to build a world model?

The essence of this question is at which level AI should "represent the world"—pixels, geometric structures, or abstract states? Different answers point to completely different technical paths, underpinned by entirely distinct underlying philosophies.

The first path is the pixel interaction route represented by Google's Genie 3, with the underlying logic that "vision equals existence."

The underlying architectures of such models are mostly built on spatiotemporal diffusion models, adding temporal dimension attention mechanisms to traditional image diffusion to let the model learn motion continuity between video frames. Take Genie 3 as an example: it supports multi-condition inputs such as text, images, and action commands to generate 1080P resolution interactive videos. Users can control character movement and interactions in the frame via keyboard and mouse, and the model can generate logically consistent subsequent frames in real time, delivering extremely high immersion.

The advantage of this path is "fast monetization": it can be directly applied in scenarios like games, content generation, and digital humans, with low training data barriers—massive videos on the internet can serve as training materials, enabling rapid model iteration.

But its disadvantages are also fatal: pixel-level fitting does not equate to physical correctness. It can generate footage of a cup shattering, but it does not understand why the fragments fly in specific directions, much less how different floor surfaces would produce different bouncing effects. Industry tests show that videos generated by Sora often contain physical errors such as objects passing through solid surfaces, non-conservation of momentum, and inconsistent lighting and shadows. Using it to guide robot operations is like asking someone who has only watched movies to operate a machine tool—it may look the part, but accidents can happen at any time.

The second path is the spatial structure route represented by Fei-Fei Li's World Labs' Marble model, with the underlying assumption that "structure precedes physics."

The core of this route is to reconstruct precise 3D spatial structures from visual inputs, using geometric representations instead of pixel representations. The Marble model can generate exportable 3D mesh environments with semantic labels from multi-view image inputs, supporting agents to navigate and plan interactions within them. The output 3D assets can be directly imported into game engines for use.

In the autonomous driving domain, this route has already achieved large-scale mass production and deployment. Occupancy Networks have become a standard feature in high-level intelligent driving systems: they do not rely on high-precision maps, but use visual inputs from vehicle-mounted multi-cameras to construct a 3D voxel space of the surrounding environment in real time, identifying which areas are occupied by obstacles and which are drivable. The latest intelligent driving systems from Tesla, Xpeng, and Li Auto are all equipped with spatial world models based on Occupancy Networks, which can effectively identify targets that traditional perception solutions often miss, such as irregular obstacles and construction barriers.

But its problems are also obvious: 3D structures are merely the static skeleton of the physical world, while the core value of world models lies in predicting dynamic changes. Although the spatial route can tell you the angle of a slope, it cannot directly calculate the acceleration of a rolling ball; it can restore the shape of an object, but it struggles to simulate the deformation of soft bodies and fluids. There is still a long engineering road ahead from static structures to dynamic physics.

The third path is the cognitive representation route represented by Yann LeCun's JEPA architecture, with the underlying logic that "abstraction equals cognition."

It does not generate pixels, but only predicts abstract world states, making it theoretically the closest to the human brain's "mental model." When we walk, we do not render images in our heads, but only anticipate road conditions under our feet; when we throw an object, we do not calculate the motion of every pixel, but only judge the force and landing point.

In terms of technical implementation, JEPA uses an encoder to map input images to a high-dimensional latent space, then uses a predictor to predict future latent space representations based on historical states. The entire process generates no pixels, drastically reducing computational costs and allowing the model to focus more on semantic-level causal rules.

The DreamerV3 algorithm, built on a similar line of thinking, is already one of the mainstream solutions for robot reinforcement learning today: it constructs a world model in latent space, predicts environmental feedback under different actions, and then makes decision plans based on the internal model without needing to interact with the real environment every time, greatly improving sample efficiency.

This path is the closest to the essence of general intelligence, but also the most "distant." The latent space representation is a black box—you cannot know exactly what the model understands, and it is difficult to pinpoint the cause when errors occur. More critically, there is still no mature general solution for converting abstract states into precise motor control commands or interfacing with underlying motion planning systems. Most implementations can only perform end-to-end training on specific tasks, limiting their generalization capabilities.

All three routes have their own theoretical foundations and unavoidable shortcomings. A growing number of people in the industry believe that the eventual general world model will most likely be a fusion of the three: using 3D structures to build the skeleton, adding constraints via physics engines, and using abstract representations for decision-making. NVIDIA's World Foundation Model and Tesla's FSD end-to-end system are already attempting multi-route integration. However, pixels, geometry, and latent spaces are three completely different representation systems, and enabling their precise alignment and efficient collaboration is itself a world-class challenge.

Vertical Scenarios Take the Lead, General Intelligence Remains at the Foot of the Mountain

How far is the world model from the real world? There is no simple answer to this question.

At least for the next three to five years, world models will remain in a phase of continuous evolution and iteration. Current world models are like deep learning around 2012—back then, data silos were severe, technical routes were undecided, benchmarks were still being contested, and the ChatGPT moment had not yet arrived.

But the challenges facing world models may be even greater than those faced by deep learning back then. Deep learning deals with pattern recognition, extracting statistical rules from data. Its victory was a resonance of computing power, data, and architecture, essentially a triumph of statistical fitting. World models, by contrast, must handle causal reasoning, understanding why objects move in certain ways and why events unfold as they do.

The gap between these two capabilities is not just a few years of technical iteration, but a fundamental shift in cognitive paradigm. Statistical fitting can achieve breakthroughs by piling on computing power and data, but causal reasoning requires underlying abstractions of how the world operates—something closer to the core of human intelligence, and thus far more difficult to realize.

Of course, we do not need to dismiss the value of general world models just because they are distant.