Fei-Fei Li's New Interview: LLMs Predict Tokens, and World Models Predict "New Perspectives"

This time, Li Fei-Fei is fully determined to make Atlas crystal clear.

Right after the launch of Atlas, the world's first-ever multimodal world model, she joined the other two co-founders of World Labs as guests on a16z, diving deep into what Atlas actually is, the core technologies behind it, and what it truly means for the industry.

From left to right: co-founders Justin Johnson and Ben Mildenhall, Li Fei-Fei, host Martin Casado

There is a huge amount of valuable information, let's start with the key takeaways first:

The core basic behavior of Atlas is to move to a new position and continue observing. LLMs predict the next token, video models predict the next frame, while Atlas predicts new perspectives. Feed it several photos, and it will understand the spatial relationship between these scenes, then tell you what the view would look like if you look at the scene from another position.

Generation and reconstruction are integrated into the same model for the first time. In the past, generation was responsible for creating imaginary non-existent scenes, while reconstruction was responsible for restoring real spaces, with the two research fields developing independently for decades. Atlas connects the two via camera poses, and processes text, images, videos and 3D content simultaneously.

The 3D acquisition workload could be reduced by two orders of magnitude overnight. Previously, reconstructing a single space required shooting hundreds or even thousands of photos around it; Atlas aims to cut this number down to just a few or dozens of photos. With three iPhones, you can even shoot *The Matrix*-style bullet time effects.

Its target use cases are not limited to film, gaming and 3D creation. In Li Fei-Fei's view, the biggest bottleneck for current robotics is data rather than chips. Atlas can import real environments into the simulation world much faster, and generate a large number of training scenarios through randomization.

The end goal is not just more realistic scenes. World Labs will add dynamic capabilities, editing functions and interaction features to Atlas next, enabling the model not only to generate the world, but also to understand how the world responds to actions. Going one step further, the simulator itself could even evolve into a planner.

The most ambitious judgment comes at the very end.

They place "novel view prediction" on the same level as "next token prediction", and believe it could be the fundamental primitive leading to AGI.

Whether this judgment holds up remains to be verified by subsequent iterations of the model, but if you are concerned about how world models and spatial intelligence will evolve, this conversation is well worth reading.

The full edited transcript is as follows, enjoy.

What Atlas is and why it matters

Host: Who would like to start by introducing what was released yesterday, and why it is important?

Justin: Atlas is our new generation world model with three core capabilities: generating, reconstructing and simulating the world. Under these three core capabilities, there are several important specific functions. It excels at camera-conditioned generation. You can input an image and a camera motion trajectory, use them to control the model, and let it generate video frames along any perspective you want.

It also performs extremely well in sparse 3D reconstruction. You can input one or more real-world images, up to 100 frames, and use these views from different angles to reconstruct the real world. The reconstruction output can be either a video that walks through the space, or an explicit 3D reconstruction of the space.

Finally, it can also be used for simulation. We have demonstrated some stunning bullet time videos that have attracted widespread attention online, and we have also showcased robotics simulation use cases.

Host: What is a bullet time video?

Justin: The term comes from *The Matrix*. There is a famous scene in the first film where Neo falls backward, do you remember that? That scene is in slow motion, with the camera orbiting all the way around him. To shoot that scene, they arranged hundreds of cameras in a circle. Neo fell backward in the studio, hundreds of cameras shot from different angles in front of a green screen, then all the footage from these cameras was combined to create that classic shot in the movie.

Now with Atlas, you only need three cameras. No studio, no green screen, no expensive calibration required; we can literally set up three iPhones on tripods and start shooting right away.

You can shoot a person shooting a basketball, or a strawberry falling into a bowl of milk. After getting these three clips from the iPhones, we can redesign the shot: for example, freeze time, let the camera fly into the scene the moment the milk splashes up, and get a stunning frozen-time visual effect. All of this can be done with just a few cameras.

Host: Can you explain what Atlas actually does in the simplest terms possible? What are its inputs, and what are its outputs?

Justin: The most core, fundamental point of Atlas is novel view prediction. We think this is an exciting, very basic new primitive, a foundational model primitive that no one has ever implemented in this way before. We all know that LLMs are built on next token prediction; video models are built on next frame prediction.

What Atlas actually does is novel view prediction. Give it several views of a scene, or a description of the scene, and these inputs will go into what we call the spatial context, which implicitly describes the world we are referring to. Then you can place a virtual camera anywhere in time and space, and Atlas will understand what the world should look like when viewed from that specific time and spatial position.

Host: Ben, there are countless video models on the market right now, all of which claim to be world models and say they can generate new views. Can you break down specifically how it is different from all the various models that came before it?

Ben: The spatial context Justin mentioned earlier is especially critical here. Many video models first became famous for their ability to generate videos from a single image, or interpolate between the first and last frame; now we are starting to see some models that support multiple reference image inputs, for example 20, 30, 50 images.

But the key difference of Atlas is that every frame input has a clear spatial meaning. It is not just an image for the model to interpret arbitrarily, and you do not have to struggle repeatedly with text prompts to force the model to behave in a specific way. In Atlas, each image is associated with a 3D camera pose. This means you can complete reconstruction tasks with extremely high precision.

For example, if we take a photo at each of the four corners of this room, input these four views into the model, it can accurately reproduce everything you see in the room. It will not guess what is in the other corner or the relationship between objects, but accurately reproduce the content you provided.

Likewise, you can also use it for creation and imagination. Take two photos from different AI generation results, or from different real locations, you can arrange their positions, lay out the scene, and construct a walkthrough shot with clear creative intent. These shots are truly controlled by where you place the content precisely, and where the camera is pointing and moving.

This is very different from video models. The latter usually only has higher-level text control, you have to regenerate over and over again, and the experience is more like playing a slot machine.

A scaled-up video model, or a brand new architecture

Host: Is this just a natural, larger-scale traditional video model, or a new architecture?

Justin: I think it is something quite new, for several reasons. First of all, as we have been discussing, it does generation and reconstruction simultaneously in the same model. Just like Ben said, if you give it several views of this room, it can accurately reconstruct everything in the room exactly as you see it.

Historically, reconstruction has always been an independent subfield in computer vision, with its own dedicated tasks and specialized models. Generation is what all text-to-video models are good at, and what those large diffusion models have excelled at in the past few years. They are very suitable for creative applications: I imagine something that has never existed before.

Now with Atlas, for the first time we bring these two parts of visual intelligence into the same model. It can perform 3D reconstruction and generation at the same time within the same architecture; to achieve this, we had to make a number of changes.

First of all, we had to design it as a multimodal model from the very beginning. It natively processes text, images and videos, and also takes camera poses as a native input of the model; I do not think anyone has done this at the pre-training stage before. It also treats 3D as a native modality. So it was designed from the start to be natively multimodal in a way that I believe has never been done before.

Host: Sorry, I am not that familiar with this field. When you say 3D, are you referring to depth, a model, or something else?

Justin: The representation we currently adopt is a depth map. Suppose there is a frame of footage, the corresponding virtual camera tells us its position in 3D space; the camera's position and parameters are the native input of the model. Associated with this camera position can be both RGB images and depth maps. RGB tells you what the view looks like when viewed from that position in space; the depth map tells you what the 3D spatial structure you see from that position looks like. So text, images, videos, 3D, cameras, all these modalities are processed by it jointly in a multimodal manner.

Li Fei-Fei: I want to add a point. What Justin just said, and what Ben mentioned, are extremely important, but people have not paid enough attention to them: for the first time in the field of computer vision, we have achieved the unification of pixel generation and pixel reconstruction.

This field has existed for more than half a century. I have been working in it for decades, and I have lost count of how many PhD theses have studied reconstruction problems, or novel view synthesis problems. Moreover, our field has traditionally had many different research lines. If you attend a computer vision conference, you will see directions for pixel generation, recognition, and 3D reconstruction.

This is an elegant model. It uses views and view estimation as anchors to combine and unify the two problems of reconstruction and generation, which is extremely powerful.

Spatial intelligence, and the importance of novel view prediction

Host: Can we step back and fill in this background? I remember when you first founded the company, you said you wanted to solve the problem of spatial intelligence, and now you have this new model. As a layman, it feels very general-purpose to me. On one side is next token prediction, on the other side is novel view prediction. Can you outline why this is such an important step toward the general problem of spatial intelligence? Maybe you can start by explaining what spatial intelligence is.

Li Fei-Fei: Ultimately, spatial intelligence must enable us to generate a space, reason within it, as well as edit and interact in it. We can debate whether it is 3D or 4D. Ultimately it must be 4D, because there is also the time dimension; but even if we only talk about 3D, the aforementioned are the basic tasks that must be completed, the capabilities that spatial intelligence must have. With these, we can render, simulate and plan actions.

But to do all this, there is a fundamental problem that must be solved: understanding the geometry, structure and physics of space. I do believe that Atlas has taken an important step forward. Because now for every frame, you can generate and estimate a critical piece of information, that is the view, the camera pose.

This is the most critical information needed to understand spatial geometry. It can lead to all kinds of emergent behaviors, which are the capabilities we see in downstream applications of the model and demonstrated in our blog posts. So on the path to spatial intelligence, generating pixels is of course a relatively early step, which we have already seen in the countless models you mentioned.

But generating pixels that truly have spatial context and are grounded in space is definitely another important step, and Atlas has taken this very difficult step. We can certainly continue moving forward along this path. The fourth dimension is time, which brings dynamics; there is also higher-fidelity simulation, and more fine-grained characterization of space, all of which are part of the spatial intelligence roadmap.

Host: I really want to dive into where this will lead in the future, but first let's talk about how we got here. How long has World Labs been established?

Others: Two and a half years.

Host: You actually released models before, so why didn't you jump straight to Atlas back then?

Li Fei-Fei: Great question.