Breaking: Li Feifei's latest AI achievement has been released, which embeds the world model into a retro handheld game console and outputs 15 frames in real time from a single image.

Click the generate button, then wait idly for several minutes staring at the spinning progress bar. Anyone who has tried 3D world models or video generation tools in the past must have been worn down by that excruciatingly long rendering process.

Just today, World Labs, founded by Li Feifei, released a research preview demo named Frameboy, which brings its proprietary autoregressive world model Atlas straight into the era of real-time interaction.

Official Blog Link 🔗 https://www.worldlabs.ai/blog/atlas-realtime

You only need to open your browser, and the cloud GPU will stream the footage back in real time at a maximum frame rate of 15 fps, allowing you to freely control the camera just like playing a game, and wander around at will in the 3D world generated from a single image.

Even if you accidentally go the wrong direction or try a bad idea, the cost of trial and error is almost zero: just swipe the mouse to change the angle and keep exploring.

What is more notable is that this update is another technical achievement delivered by Li Feifei's top spatial intelligence team, right after AMD announced at the end of September this year that it would acquire World Labs for $8.2 billion (approximately 550 billion RMB) in an all-stock transaction.

Port the World Model to a Retro Handheld Console, Run 15 Frames Per Second in Real Time From a Single Image

To fully understand this update, we first need to talk about Atlas, which was unveiled just two months ago.

As an autoregressive world model, Atlas's biggest selling point at the time was its ability to generate frame-by-frame footage and support pixel-level accurate camera perspective control. It can do everything from filling in a virtual 3D world from a single image, restoring real landmarks from multiple photos from different perspectives, checking construction site progress, to scouting locations for films.

But the problem was very obvious: it was far too slow.

If you had to wait dozens of seconds or even several minutes to see the rendering result every time you adjusted the camera position, creators would become very constrained when building scenes and planning camera movements, and every press of the generate button felt like opening a blind box.

To ease the long waiting time for world model generation, the team conducted in-depth inference optimization on Atlas, and launched Frameboy, a real-time interactive demo equipped with this optimized version of the model. Its retro handheld console design is highly reminiscent of the classic Game Boy.

Now that the image generation speed has been increased to a maximum of 15 frames per second, users can no longer clearly feel the disjointed action of clicking the generate button.

You only need to pan, tilt, and move the camera in the browser, and the cloud will continuously push the footage from the new perspective to your screen in real time.

Take the official comparison demonstration of the living room as an example: in the past, when using the slow model for manual iteration, users had to carefully click four key frame camera positions in the room, press generate and wait for the video to render, find that the angle was wrong, then drag and retry, and go back and forth for several rounds.

But in Frameboy's real-time mode, you can just move the virtual camera around the room freely, and the footage within your field of view will be updated synchronously in real time.

If you are familiar with RTFM, World Labs' research preview project from last year, you might find the retro handheld form factor somewhat familiar.

However, compared to that lightweight frame generation model from back then, today's Atlas not only has a far larger model scale and is trained on orders of magnitude more data, but also has a completely different capability boundary.

Last year's RTFM could only mostly do interpolation transitions between several existing perspectives, but now Atlas, even if you only feed it a single static image or a text description, can fill in the entire surrounding scene from scratch, and put it into Frameboy's spatial context for you to interact and explore freely.

At the same time, it has fully retained and upgraded its multi-view novel view synthesis capability: input several multi-angle photos of real-world places of interest, and you can directly start a cyber immersive roaming session in your browser.

Upload Images, Build Your Own 3D Playground

Just wandering around in the several pre-set official scenes is obviously not enough fun.

To allow users to build their own 3D worlds by hand, World Labs has also integrated the real-time streaming capability into its web-based application Viewfinder, creating a real-time rendering overlay mode, which is currently in closed beta and has opened the waitlist application.

In Viewfinder, you can use the image you uploaded as the starting point, control the virtual camera to move outward step by step, and build the entire 3D space frame by frame.

Every time Atlas generates a new frame, it will output color and depth information synchronously, helping you visually align the entire 3D spatial structure.

When the main part of the scene is almost built, you can call up the Frameboy overlay with one click, and the static project that was being built frame by frame just now will instantly turn into a real-time streaming 3D playground.

The team has a very clear positioning for the future: they want world models like Atlas to become the next generation of generative renderers.

However, the official team did not avoid the current computing power threshold at the end of the blog.

At this stage, Frameboy must run on high-performance GPU servers in the cloud, and the cost is definitely not cheap compared with traditional local rendering.

But looking back at the history of computer graphics, any disruptive technology is usually slow and expensive when it is first born. With the coordinated evolution of underlying hardware and software algorithms, today's expensive cloud lab toy will sooner or later become a mass-market tool that costs next to nothing.

$82 Billion Deal with Lisa Su, Accelerate Integration With Underlying Hardware

It is precisely on the matter of reducing costs and increasing speed through software-hardware coordination that World Labs has now found the most suitable backer.

Just over a month ago on September 28, chip giant AMD officially announced a major acquisition agreement, planning to acquire World Labs, which was founded only two years ago, through an all-stock transaction valued at $8.2 billion.

This transaction, which is expected to be completed by the end of this year, directly marks the second largest acquisition in AMD's history, second only to its acquisition of Xilinx.

As it stands, this star unicorn headquartered in San Francisco, co-founded by Li Feifei, Justin Johnson, Ben Mildenhall and Christoph Lassner, was valued at $5 billion in its previous round of financing.

According to the acquisition arrangement, after the transaction is completed, Li Feifei will officially serve as AMD's Executive Vice President and Chief Scientist, reporting directly to Lisa Su; the original R&D team will remain fully intact, continuing to focus on spatial models and physical AI under the leadership of Justin Johnson and Ben Mildenhall.

At that time, the outside world was still discussing why a hardware giant that sells chips would spend 550 billion RMB to acquire an algorithm team that researches 3D world models?

Now, combined with Frameboy's pursuit of extreme inference speed, the considerations of both parties are crystal clear.

For Lisa Su, the next must-win battleground after large language models is physical AI, which covers robot simulation, spatial understanding and 3D reconstruction.

By acquiring World Labs, AMD can get first-hand insight into the real computing power load characteristics of next-generation world models, so as to reverse guide the roadmap planning of its own GPU architectures, AI chips and ROCm software stack, and compete head-on with NVIDIA in the robotics and spatial computing track.

For Li Feifei's team, as she stated in the acquisition announcement, next-generation AI requires close collaboration between model research, systems and computing power.

In the past, running a real-time world model in the cloud meant worrying about the expensive GPU bills, but now, working side by side with its own chip team to polish the hardware and underlying operators, the goal of truly bringing real-time generative 3D rendering into end devices in millions of households has clearly taken a huge step closer.

This article is from WeChat Official Account "APPSO", author: APPSO that discovers tomorrow's products, published with authorization from 36Kr.