It is never the chips that trap robots, but the desert of physical data. Li Feifei has reduced the tedious two-hour street scanning work to only three photos.

Martin Casado, Partner at a16z, invited three co-founders of World Labs — Fei-Fei Li, computer vision scholar Justin Johnson, and Ben Mildenhall, the pioneer of NeRF (Neural Radiance Fields) — to record a podcast. Recently, the company co-founded by Fei-Fei Li just released its new model Atlas, which has drawn extremely high attention across the industry.

Everyone is actually familiar with the pain points of existing video models: Although the visuals are realistic, they are essentially "drawing cards by writing prompts". Once the camera pans a few more times, the space and objects will no longer align; while traditional 3D scanning is realistic, it requires you to walk around the room with your phone to shoot hundreds of photos, which is extremely laborious.

Atlas is designed to solve exactly this problem: integrating "content generation" and "3D reconstruction" into the same architecture. Its core primitive is called "New View Prediction" — no matter how many daily photos you feed it, as long as you specify a camera coordinate, it can infer what the world should look like when viewed from that physical position.

When talking about application implementation, the topic inevitably turns to robotics and embodied intelligence. Fei-Fei Li stated bluntly in the conversation that the most fatal bottleneck hindering robot implementation right now is not computing power or chips, but the data desert of the physical world. Generating real data is too expensive and too slow, and the extremely fast "Real-to-Sim" capability demonstrated by Atlas is exactly the shortest path to solve the data hunger problem.

The most thought-provoking point of the entire conversation is that they bring spatial intelligence back to the perspective of evolution. Large language models have brought emergent intelligence through "predicting the next Token", and the primitive of 3D world models may be exactly this "new view prediction". Just as the brilliant metaphor Fei-Fei Li mentioned at the end: nature gave animals eyes, but did not give eyes to trees, because once you start moving in space, you must learn to predict new perspectives.

Quick Overview of Key Points

• Say goodbye to "slot machine-style" card-drawing generation:  "In Atlas, every frame is bound to a rigorous 3D camera pose, and the system never lets the model make up the visuals arbitrarily. The content layout of the picture, camera orientation and motion trajectory are all precisely controlled, which is essentially different from the experience of traditional video models that rely on repeated luck."
• From shooting hundreds of photos to shooting three:  In the past, dense 3D reconstruction required ordinary people to take hundreds of photos or even walk around for two hours, but Atlas reduces the perspective demand by dozens of times. Even with just a few old photos, the dead angles that the camera has not scanned can be seamlessly filled by the spatial generation capability.
• The data desert of the physical world:  "Looking at it from a higher perspective, the biggest obstacle facing the current robotics field is data scarcity. The future bottleneck may be chips, but the current absolute bottleneck is the extreme scarcity of physical interaction data."
• Purify space from dynamic chaos:  To create extremely high-precision static 3D geometry, you cannot only feed the model static data. Only by exposing the model to a massive amount of dynamic world during pre-training can it truly learn to naturally decouple time flow and spatial structure.
• The completeness of spatial intelligence:  "Nature gave animals eyes, but did not give eyes to trees. Why? Because as long as you move, you will see a brand new perspective. New view prediction is the next Token prediction in the era of spatial intelligence."

The following is the full organized transcript of this conversation:

1 / Nail every frame of pixels into the real space

Host: Recently you released a cutting-edge model, which received extremely warm responses, and feedback from all parties keeps pouring in. I think our conversation today can start with the content released yesterday, then review the R&D background and expand step by step. Justin, can you first talk about what was released yesterday and why it is so critical?

Justin Johnson: Alright. Atlas is our new-generation world model, and its core capabilities are divided into three items: generating, reconstructing and simulating the world. Around these three points, it has several main functions.

The first is camera-conditioned generation. You can input an image and a camera motion trajectory to the model, and guide it to generate continuous video frames from the perspective you specified.

The second is sparse 3D reconstruction. Input one or more — up to 100 — photos from real perspectives, and it can reconstruct the real space. This kind of reconstruction can either be a new perspective roaming video that shuttles through the space, or directly output an explicit 3D space model.

The last part is simulation. We demonstrated the "bullet-time" video that has attracted a lot of attention online, and it can also be used for robot simulation.

Host: What is a "bullet-time" video?

Justin Johnson: Bullet time originated from the movie *The Matrix*. There is a famous scene in the first part: Neo (the protagonist played by Keanu Reeves) is leaning backward to dodge bullets...

Host: Oh, that's the scene!

Justin Johnson: Exactly that scene: he falls backward in slow motion, and the camera circles around him in all directions. Back then, to shoot this scene, hundreds of cameras were arranged in a circle in front of the green screen in the studio. When he fell backward, hundreds of cameras captured all angles at the same time, and finally synthesized this classic scene.

But now with Atlas, we only need a minimum of three cameras. No professional studio, no green screen, no expensive camera calibration required. Just set up three cameras — even three iPhones on tripods — to shoot what is happening, for example, someone shooting a basketball, or throwing a strawberry into a bowl of milk.

With only these three iPhone videos, we can recompose the frame: freeze time, let the virtual camera shuttle forward the moment the milk splashes, and get an amazing time-freeze perspective. All of this can be done with just a few ordinary daily cameras.

Host: Can you summarize the function of Atlas in the simplest words? That is, what is the input, and what is the output?

Justin Johnson: The most core and underlying logic of Atlas is "new view prediction". We believe this is a brand new basic primitive in the foundation model that no one has ever done before.

We know that large language models are based on "next-token prediction", video models are based on "next-frame prediction", and the core of Atlas is exactly "new view prediction".

Input several perspectives of a scene or a scene description, which will be converted into the so-called "spatial context" to implicitly describe the world we set. Then, as long as you place a virtual camera at any position in time and space, Atlas can infer what the world should look like when viewed from that spatiotemporal position.

Host: Ben, there are a huge number of video models on the market, all claiming to be world models and claiming to be able to generate new perspectives. Can you specifically talk about what the fundamental difference between Atlas and these previous models is?

Ben Mildenhall: Right, I think the "spatial context" that Justin just emphasized is very critical here. There are many video models on the market, most of which first became popular by generating videos from a single image or interpolating between the first and last frames. Now models that support omnidirectional reference are also emerging, which can receive 20, 30 or even 50 images.

But the key point of Atlas is that every frame of the input image has a definite spatially grounded physical position. It is by no means an ordinary picture that allows the model to make up arbitrarily, and you don't need to struggle with the prompt words to force the model to obey.

In Atlas, each image is bound to the corresponding 3D camera pose, so the accuracy of spatial reconstruction is extremely high. For example, if we take a photo at each of the four corners of the current room and input them into the model, it can accurately reproduce everything in this room; it will not blindly guess what is in the other corner, nor will it confuse the relative positions between objects, and will only faithfully restore the content you give it.

Similarly, this capability can also be used for creative conception. If you pick one photo from different AI-generated images or real scenes respectively, you can design a fully controlled shuttle camera movement by precisely setting their relative positions and camera layout in space. Where the picture content is placed, where the camera looks, and where it moves are all precisely controlled.

This is very different from traditional video models — those models only provide higher-level text control, and the generation process is like pulling a slot machine, which relies entirely on repeated retries to get lucky.

2 / The visual gap of half a century: the unification of breaking barriers between generation and reconstruction

Host: Is this just a simple scaled-up version of the traditional video model, or does it use a completely new architecture?

Justin Johnson: I think it is really new, for several reasons.

Host: Okay, tell me the details.

Justin Johnson: First of all, as we said earlier, it jointly implements "generation" and "reconstruction" in the same model. Just as Ben said, input several perspectives of this room to it, and it can reconstruct everything you see without any error.

But in the history of computer vision, reconstruction has always been an independent subfield, with dedicated research tasks and exclusive models; while generation is the direction that text-to-video models and various large diffusion models in recent years are good at. Generative models are very suitable for creative applications, for example, I want to imagine something that has never existed before.

But now, Atlas, for the first time, puts these two completely different branches of visual intelligence into the same model, and uses a single architecture to support 3D reconstruction and generation at the same time.

To achieve this, we have made several fundamental changes. First, it must be natively multimodal from the very beginning. It not only natively supports text, images and videos, but also feeds camera poses as native inputs directly to the model — as far as I know, no one has ever introduced this input during the pre-training stage before — and it also processes 3D itself as a native modality. So it has been designed to be truly natively multimodal from day one, which was not possible before...

Host: Sorry to interrupt, I don't know much about this field. The 3D you mentioned, does it specifically refer to depth maps, 3D models, or something else?

Justin Johnson: Right, the representation form we currently adopt is depth maps.

Specifically, when a frame of picture has 3D spatial coordinates marked by a virtual camera, the spatial position, internal and external parameters of the camera will be fed to the model as native inputs. Corresponding to this camera position, on the one hand, there is an RGB image showing the appearance of this position, and on the other hand, there is a depth map giving the 3D spatial structure under this perspective.

In this way, text, images, videos and 3D camera poses are all processed jointly by the same multimodal model.

Fei-Fei Li: I want to add a few words, because the point Justin and Ben just talked about is so critical, but the outside world is likely to underestimate its weight: This is the first time we have truly unified "pixel generation" and "pixel reconstruction" into the same model.

The computer vision field has been developing for more than half a century. I have been deeply engaged in this field for decades, and countless PhD theses are working on the problem of reconstruction or novel view synthesis. Traditionally, our field has always been divided into several tracks: when you attend top computer vision conferences, there will be a pixel generation track, an image recognition track, and an independent 3D reconstruction track.

But Atlas is a very elegant model, which takes view estimation as the anchor point, integrates the two major problems of reconstruction and generation, and this capability is extremely powerful.

Host: Can we step back and talk through some of the underlying logic? I remember when you first founded the company, you said that the goal of the company was to tackle "spatial intelligence". Now with this new model, as a layman, I think it is very general. Large language models are "next Token prediction", and what you do is "new view prediction": give one or a set of perspectives, and you can predict a brand new perspective. Can you sort out why this is an important step towards the general problem of spatial intelligence? Maybe we can start by talking about what spatial intelligence is.

Fei-Fei Li: Spatial intelligence must ultimately allow us to do three things: generate space, reason within space, and edit and interact in it.

Of course we can discuss whether it is 3D or 4D — adding the time dimension will definitely move towards 4D eventually — but even only in the 3D category, these are all core tasks that spatial intelligence must support. On this basis, you can do rendering, simulation and action planning. But to achieve all this, the first fundamental problem to solve is to understand the geometric shape, 3D structure and physical laws of space.

I think Atlas has taken a key step forward, because now when processing each frame of picture, the model can generate and estimate a core piece of information: the view, that is, the camera pose. This is the most critical information for understanding spatial geometry, which directly brings various downstream emergent capabilities we demonstrated in the blog post.

On the road to spatial intelligence, purely generating pixels is only the first step, and countless models on the market are doing this; but generating pixels that truly have spatial context and are strictly grounded with physical space geometry is a qualitative leap. And this is exactly the hard bone that Atlas has gnawed down.

Of course we can continue to move forward: for example, introduce time as the fourth dimension to bring dynamic changes, as well as higher-fidelity spatial simulation and detail depiction. These are all core components of the spatial intelligence roadmap.

3 / From scanning a whole house in two hours to shooting only three photos

Host: Okay. I definitely want to talk deeply about the future direction later, but we can first talk about how you got to this point. How long has World Labs been established?

Fei-Fei Li: Uh, two and a half months?

Ben Mildenhall / Justin Johnson: Right, almost two and a half months.

Fei-Fei Li: Two and a half months, yes.

Host: In fact, you have released models before this, so why didn't you just go straight to Atlas back then?

Justin Johnson: Good question!

Host: After all, the effect looks so amazing, right?

Fei-Fei Li: Because Justin's team needs a lot of chips.

Justin Johnson: Exactly, to truly scale up the model, a large amount of GPU computing power is required. Last year we released the Marble world model, which was our first main world model and also the underlying foundation of the existing Marble product.

Marble is very cool, it can accept image, video and text prompt inputs, and generate 3D worlds. But the core