The world model "Cloud Gaming" has been unveiled, which is a product developed by Li Fei-Fei's World Labs.

Today, people in the AI circle are playing games.

Li Feifei's startup World Labs has launched its first "application" Frameboy, which generates first-person perspectives in real time via AI, turning generative AI into an experience that can be directly explored.

Frameboy is an optimized demo version of the Atlas world model. Using cloud GPU streaming, the frame rate can reach up to 15 fps. Actual tests show that the model's response time is only about 70ms, which is already very close to cloud gaming.

Experience link: https://frameboy.worldlabs.ai/

It currently offers curated scenarios for users to explore, including various real-world landscapes, cyberpunk spaces, and the campus of Stanford University. Some of these scenarios start from a single image, allowing Atlas to fill in other perspectives and gradually build an explorable environment, while others input multiple photos of the same location, and the model then generates new perspectives between and beyond the photos.

World Labs has also directly integrated Frameboy as a streaming overlay rendering mode in its beta web app Viewfinder, which enables users to build scenes frame by frame with their own input content, then explore and expand the virtual world in real time.

Just like using a camera, you can navigate from different perspectives and generate 3D-consistent world views. Every frame is generated by Atlas and contains color and depth information. When ready, users can trigger the Frameboy overlay to play the generated frames in real time as they explore.

Atlas: Deduce a 3D World From a Single Photo

To understand the significance of this Frameboy real-time demo, you first need to figure out what the Atlas behind it actually is.

On September 1 this year, World Labs founded by Li Feifei officially released Atlas, a general world model pre-trained from scratch that can process text, image, video and 3D spatial information. Different from video models that mainly generate continuous frames, Atlas tries to solve a more complex problem: what would the world in front of you look like if the camera moved to another position?

World Labs showed a very intuitive example at the time of release.

The researchers first provided Atlas with a photo of a garden, and asked it to generate a top-down aerial view of this area. Since the original photo only contains part of the garden, the model reproduced the existing scenery but had to infer the surrounding houses and environment on its own.

Then the researchers added a second photo showing a cabin next to the garden. This time, the frame generated by Atlas includes both the garden and the cabin, but the buildings further away are still completed by the model on its own. It was not until the third photo was added, showing the main house nearby, that the model reconstructed the entire scene to be closer to the real environment.

This process demonstrates an important feature of Atlas: it can not only reconstruct the already observed space based on limited images, but also use the world knowledge learned during training to infer what might exist outside the camera's view.

World Labs calls its core mechanism "Spatial Context".

Simply put, Atlas does not only recognize what objects are in the picture, but also associates each image with the position and orientation of the camera in 3D space. When the user specifies a new observation position, the model generates the view that should be seen from that position based on the existing spatial information.

This is also an important difference between Atlas and ordinary video generation models. The latter usually relies on text descriptions such as "pan the camera to the left" and "rotate around the character" to control camera movement, while Atlas can directly receive precise camera positions and angles, allowing users to explicitly specify the perspective to observe the world.

In terms of technical architecture, Atlas adopts a multimodal autoregressive diffusion Transformer. It combines the ideas of language models and diffusion generation models: it organizes text, images, camera poses and depth information into a unified spatial context, and then gradually generates new frames and spatial data.

In the initial demo, users were already able to pre-design camera paths, allowing Atlas to generate videos of up to one minute in length with a maximum 1440p resolution, and maintain scene consistency as much as possible between different shots.

But Atlas's capabilities go far beyond generating videos.

It can also generate images and depth information at the same time, and output 3D representations such as point cloud and 3D Gaussian Splatting. In other words, the model can not only present the frame from a certain angle, but also produce data that can be used in workflows such as 3D rendering and spatial reconstruction.

For example, researchers once used three to five ordinary mobile phones and action cameras to shoot the same scene from different angles. Atlas reconstructs the space based on these frames, allowing users to switch observation angles at the exact moment the action takes place, achieving the "bullet time" effect similar to that in the movie *The Matrix*, without the need to deploy a traditional large-scale multi-camera shooting system.

Another application direction is robot simulation. Researchers can record the real environment through mobile phone videos, then use Atlas to reconstruct the space, and generate the frames and depth data that the robot's on-board camera may observe when the robot moves along different routes. This provides a new method for building robot simulation environments with a small amount of real recordings.

Now, the emergence of Frameboy has changed the way people create based on world models. We can move around while observing, and decide the next step as we go. This experience has great potential in location scouting for films, spatial concept design, and game level ideation.

Reference content: https://www.worldlabs.ai/blog/atlas-realtime

This article is from the WeChat official account "Machine Heart" (ID: almosthuman2014), edited by Neo and Zenan, published with authorization from 36Kr.