Three smartphones recreate the "bullet time", and the new model Atlas from Li Feifei's team reconstructs a three-dimensional world with just a few photos.
In the latest episode of a16z, Fei-Fei Li, Justin Johnson and Ben Mildenhall, co-founders of World Labs, shared with Martin Casado, General Partner of a16z, their core judgment on spatial intelligence: AI should not only generate photorealistic images, but also understand what the same world looks like when viewed from different positions and at different times. The new model Atlas, built on "novel view prediction", integrates generation and 3D reconstruction, making it possible to realize applications such as reconstructing a scene with a small number of photos and capturing "bullet time" with three mobile phones. In their view, this path is expected to further connect creation, design and robot training; however, more mature dynamic simulation, editability and interactive capabilities still need to be developed, and the main bottleneck for continuing to expand the model scale is training computing power.
This article is compiled from a16z, the original link is https://www.youtube.com/watch?v=qn1QDDBnTA0, the following is the compiled content.
Atlas: Generate a 3D World from a Few Photos
Martin Casado: Yesterday was a big day for you. You released a new cutting-edge model, which received extremely enthusiastic feedback, and public attention is still growing. I think we might as well first clarify what exactly was released this time, and then talk about how it got to where it is today. Justin, why don't you start by introducing what was released yesterday and why it matters?
Justin Johnson: Atlas is our new-generation world model, which can basically do three things: generate the world, reconstruct the world, and simulate the world. Specifically, it has several core capabilities. First of all, it is extremely good at generation based on camera conditions. You can input an image, provide a camera motion trajectory, use this trajectory to guide the model, and make it generate video frames along any viewing angle you want. Secondly, it excels at 3D reconstruction using a small number of images. You can input one or more real-world images (supporting up to 100 frames) and let it reconstruct the real scene accordingly. The reconstruction result can be either a video that travels through this space, or an explicit 3D representation of this space. Finally, it can also be used for simulation. We demonstrated a very cool "bullet time" video that attracted a lot of attention online, and also showed its applications in robot simulation.
Martin Casado: What do you mean by a "bullet time" video?
Justin Johnson: The term comes from The Matrix. There is a famous shot in the first movie where Neo falls backward in slow motion while the camera circles around him. At that time, the shooting method was to arrange hundreds of cameras in a circle, let him complete the action in the studio in front of a green screen, and then use the footage captured by these cameras from different angles to create that classic shot. But now with Atlas, similar effects can be achieved with only three cameras. No studio, no green screen, no expensive calibration process. We can even directly mount three iPhones on tripods to capture what is happening, such as someone shooting a basketball, or a strawberry falling into a bowl of milk. Then, using only these three iPhone videos, we can redesign the camera movement: for example, freeze the moment when the milk splashes, and then push the camera forward to get a stunning time-freeze frame. The whole process only requires a few cameras.
Martin Casado: Can you explain in the simplest terms what Atlas is doing? What is the input, and what is the output?
Justin Johnson: The most core and fundamental thing Atlas does is "novel view prediction". We believe this is a very exciting fundamental task, and it is a new approach that no one has ever applied to foundation models before. Large language models are built on "predicting the next token", video models can be understood as being built on "predicting the next frame", and what Atlas does is "predicting novel views". Specifically, you give it several views of a scene, or a description of a scene, and these inputs will enter what we call the "spatial context", implicitly describing what kind of world we are talking about. Then, you can place a virtual camera at any spatial position and any time point, and let Atlas determine what the world should look like when viewed from that time and that position.
Martin Casado: Ben, there are a large number of video models now claiming to be world models, and claiming that they can generate novel views. Can you break down more specifically what makes Atlas different from all those previous models?
Ben Mildenhall: The "spatial context" Justin just mentioned is a very important difference. Many video models first gained attention because they can generate videos from a single image, or fill in intermediate frames between a given first frame and last frame. Now we are also starting to see some models that can comprehensively reference 20, 30, or even 50 images. But the key point of Atlas is that every frame you input has a clear spatial meaning. It does not arbitrarily interpret an image after receiving it, nor does it require you to repeatedly adjust text prompts and negotiate back and forth with the model to get it to meet a specific requirement. In Atlas, each image corresponds to a 3D camera pose, that is, the position and orientation of the camera. This allows it to complete reconstruction with extremely high accuracy. For example, if we take a photo at each of the four corners of this room and input these four views into the model, it can accurately reproduce what you see in the room. It will not arbitrarily guess what is in another corner or what the relationship between different objects is, but restore the scene based on the information you provide.
This can also be used for creation and imagination. For example, you can take two photos from different AI generation results or different real locations, and actually arrange them at specified positions to decorate the scene, so as to design a traversal shot with clear creative intent. You can precisely control where the content you want is placed, where the camera looks, and what path it moves along. I think this is very different from video models. The latter usually only provides relatively general text control, and you have to regenerate over and over again, a bit like repeatedly pulling a slot machine to wait for a satisfactory result.
Martin Casado: So is it just a natural, scaled-up version of traditional video models, or does it adopt a new architecture?
Justin Johnson: I think it is something quite new in several aspects. One difference we often mention is that it integrates generation and reconstruction into the same model to complete them together. Just as Ben just said, if you give it several views of this room, it can reconstruct everything in the room exactly as you actually see it. In the past, reconstruction has always been an independent research branch in computer vision, with its own dedicated tasks and specialized models. Generation is what various text-to-video models are good at, such as the large diffusion models that have emerged in the past few years. They are very suitable for creation: I imagine something that never existed before, and let the model generate it. Now, Atlas for the first time brings these two different parts of visual intelligence into the same model, allowing 3D reconstruction and generation to be completed together in the same architecture.
Justin Johnson: To achieve this, we have made some adjustments. First of all, the model must be multimodal from the very beginning. It natively supports text, images and videos, and also takes camera poses as a native input. As far as I know, no one has done this in the pre-training stage before. At the same time, it also processes 3D information as a native modality. Therefore, from the very beginning of its design, it is a native multimodal model, and as far as I know, no one else has adopted this approach.
Martin Casado: Sorry, I am not very familiar with this field. Does the "3D information" here refer to depth, 3D models, or something else?
Justin Johnson: The representation we currently adopt is depth map. That is to say, for a frame of image, you can have a virtual camera that indicates its position in 3D space. The position of the camera and camera parameters are all native inputs of the model. Corresponding to this camera position, there can be an RGB color image that tells the model what the world looks like viewed from this position; or there can be a depth map that tells the model the 3D spatial structure corresponding to this view. Therefore, modalities such as text, images, videos, 3D information and camera information are all processed together in the same model.
Fei-Fei Li: I would like to add a point, because what Justin and Ben just said is actually very important, but its importance has not been fully recognized. This is the first time in the field of computer vision that pixel generation and pixel reconstruction have been unified. This field has existed for more than half a century, and I have been working in it for decades. There are countless doctoral papers written around reconstruction or novel view synthesis. Traditionally, our field is also divided into different research directions. When you attend computer vision conferences, you will see sessions on pixel generation, recognition, and 3D reconstruction. And this model unifies reconstruction and generation in a very concise way, based on views and view estimation. This is an extremely powerful capability.
Martin Casado: Can we take a step back and clarify this issue? I remember when you first founded the company, you said you wanted to solve "spatial intelligence". Now with this new model, to a layman like me, it seems very general. Large language models predict the next token, while you predict novel views: given one or a set of views, you get another view. Can you explain why this is an important step towards spatial intelligence? Maybe we can start with what spatial intelligence is.
Fei-Fei Li: Ultimately, spatial intelligence must enable us to generate spaces, reason in spaces, edit spaces and interact with them. We can discuss whether it is 3D or 4D, but in the end it is definitely 4D including the time dimension. Even if we only consider 3D, all the above are basic tasks that spatial intelligence must support. With these capabilities, we can further perform rendering, simulation and action planning. But to achieve these, we must first solve a fundamental problem: understanding the geometric shape, structure and physical laws of space.
I do think Atlas has taken an important step forward. Because now, for every frame, you can generate and estimate a piece of important information, that is, the view, and the corresponding camera pose. This is the most critical information for understanding spatial geometry, and it can bring various emergent capabilities that we see in downstream applications of the model, all of which are shown in our blog. Therefore, on the road to spatial intelligence, generating pixels is of course an early step. The large number of models you just mentioned have already demonstrated this capability. But making the generated pixels truly based on spatial context and truly corresponding to space is another major progress. Atlas has crossed this extremely difficult threshold.
Of course, we can continue to move forward. The fourth dimension, namely time, will bring dynamic changes; we can also further improve the fidelity of simulation to depict space in more detail. These are all part of the spatial intelligence roadmap.
From Marble to Atlas: What Challenges Has 3D Reconstruction Overcome
Martin Casado: Very good. I do want to dig deeper into where this will go next, but let's first talk about how you got here. How long has World Labs been established?
Fei-Fei Li: Two and a half years.
Martin Casado: Two and a half years. You actually released models before. So why not go straight to Atlas in one step?
Fei-Fei Li: Good question. It sounds like it can pop out all at once, right? Justin's team needs a lot of chips.
Justin Johnson: Yes, we really need a large number of GPUs to truly scale up this model. Last year, we released the Marble world model, which was our first important world model, and it is also the model behind the current Marble product. Marble is cool, it can take images, videos and text prompts, and generate 3D worlds based on them. But one of the biggest differences between Marble and Atlas is precisely what form the output takes. Marble mainly uses Gaussian Splatting as the 3D representation of the output. No matter what the input is, its final output is a 3D world represented by Gaussian Splatting.
Justin Johnson: Gaussian Splatting is very practical and has many advantages: it is easy to render, can be efficiently presented on mobile devices and VR devices, and can work with game engines and simulation engines. But I think this also became a bottleneck for the previous Marble model. So we readjusted the design in Atlas. We realized that we need to separate the processing paths of different modalities earlier, and at the same time let these modalities work together in a more unified way inside the model. Now, the most fundamental task of Atlas is no longer "generate a Gaussian Splatting world", but the aforementioned "novel view prediction". It can generate RGB images, and it can also generate 3D information; when you need it, we can use these results to generate a beautiful Gaussian Splatting world. But if you don't need it, we don't have to let all outputs go through the restriction of Gaussian Splatting. We really put a lot of effort into figuring out the respective advantages and disadvantages of these different representations.
This is one aspect. On the other hand, you have to move step by step along the path of scaling: first do small-scale experiments, train small models, gradually build confidence, and judge which methods work and which can continue to scale. If you know from the beginning which route can be scaled, of course you should go for it directly. But when we founded the company, the situation was very different from today, and there was no scaling law for spatial intelligence at that time. The technology at that time was at another stage of development. We had great ambitions for the future, but we went through several rounds of iterations before finding the current task setting that made us feel: this is it, this is the direction that can truly be scaled up.
Martin Casado: Ben, you are the creator of NeRF and have done a lot of work in the field of 3D and reconstruction. To me, it is not so self-evident that you can get a 3D scene by taking a few different views. But you have been working on this almost your entire career. Can you tell us how this is achieved in the process?
Ben Mildenhall: That's right, many years of my career, in fact most of my time, has been spent researching how to generate 3D content from images. When the company was first established, we also discussed over and over again: what method should we ultimately use to generate 3D? Should we first synthesize multiple views and then build a 3D scene based on them, or directly generate 3D? There has always been a lot of uncertainty in this field about which route will eventually win, or which route can show advantages earlier. But I do have great confidence in the former idea, because in the previous three years, we have seen the power of scaling on dense reconstruction. I would even call it a kind of progress that was pushed hard by increasing scale. Of course, that was only a very preliminary, very small-scale expansion, which is not the same as real large model scaling.
Martin Casado: Why is it called "dense reconstruction"? What does "dense" mean? Because we are going to talk about "sparse" later, I want everyone to understand the difference between the two first.
Ben Mildenhall: I think this is actually a fundamental problem that 3D reconstruction technology encounters when moving towards product and commercial applications. Ordinary people would think that if I take three photos of this object, or six photos of this room, I can look at them, piece them together in my mind, fill in the parts that were not captured, and understand what the whole space looks like. But in the past, we have never been able to well combine this prior knowledge learned from a large amount of data with reconstruction methods that rely on dense acquisition. The dense reconstruction we did in the past was actually closer to scientific imaging or medical imaging: everything you want to appear in the reconstruction result needs to be captured from at least three or four views. Just think about this room: under the microphone, under the table, every gap, between the leaves of the plants, all need to be captured. To truly cover all these positions, you have to walk back and forth in the room to take very detailed and exhaustive photos. You must have seen me running around all over the place collecting images. For a trained person, it might only take a few minutes; but if you give an ordinary mobile phone or acquisition device to an average user, or even a professional who is doing this for the first time, they might spend an hour. I have seen someone scan a space with multiple rooms for the first time, walking around for two hours before getting enough coverage. This is a very tedious and time-consuming process.
