When world models achieve real-time inference, what entrepreneurial opportunities can they bring to AI applications?

Over the past two years, video models and world models have flourished, spawning a large number of outstanding startups, while the models themselves are evolving at an extremely fast pace. The advancement of models has driven the progress of applications. For example, ByteDance's Seedance has become its core revenue driver in the AI sector.

The progress of applications, in turn, has put forward new stringent requirements for infrastructure. For instance, digital human live streaming and the application of world models in the simulation field do not only demand simple content generation, but also require interaction with users and the "environment".

This has brought about a fundamental transformation of AI: AI is no longer just a prompt-driven tool, but has evolved into an experience where users can immerse themselves and conduct real-time interactions.

Real-time performance has become a new requirement that AI models and applications set for infrastructure, but there is a lack of platforms in the current market that can run multimodal models in real time in production environments.

However, a startup named Reactor is trying to build this infrastructure layer. It hopes to enable developers to build real-time interactive applications with only a few lines of code through SDKs and APIs, and its platform supports real-time model operation with a frame rate above 60FPS and a latency of less than 40 milliseconds.

This may enable entrepreneurs to build a new generation of interactive AI applications in fields including media and entertainment, physical AI and robotics.

Reactor previously obtained $59 million in combined seed round and Series A financing led by Lightspeed Venture Partners, with participation from WndrCo, Amplify Partners, Sky9 Capital, FPV Ventures and other investors. Recently, it secured an additional tranche of Series A financing from NVIDIA and Sapphire Ventures, bringing its total financing amount to $74 million.

World/Video Models Moving to Real-Time Interaction Put Forward New Requirements for Infrastructure

Technological changes first appear on the model side: Google DeepMind's Genie 3 can generate 720p interactive worlds at a frame rate of 24 FPS; StreamDiT achieves 16 FPS text-to-video generation on a single H100. Alibaba, Tencent, and NVIDIA have all released interactive video models or world models.

In the early stage of the development of generative multimodal models, what they generated was one-off, passive content, and everyone has had the experience of using prompts to draw cards in batches.

However, after the emergence of interactive video models, especially interactive world models, the form of models has changed. These models are essentially physics engines that have been trained. By observing billions of samples, they internalize the underlying physical laws, so as to "understand" that fire spreads, water splashes, and metal dents when hit. Users operate in the environment, and the model gives real-time feedback frame by frame.

New forms of models have also spawned new forms of applications, and a large number of new interaction forms have emerged in the fields of real-time video, games and digital humans.

The requirements of models for infrastructure have changed, but the infrastructure that supports these models to run in production environments has not kept pace synchronously. The original infrastructure was built for batch processing requirements, and it is fundamentally different from the infrastructure required for real-time reasoning in the underlying architecture.

The infrastructure required for real-time reasoning needs to maintain state persistence in multiple interactions, have two-way streaming transmission capability with latency lower than 50 milliseconds (the previous average industry latency was generally above 400 milliseconds), support real-time control input processing, and have session orchestration capability that can maintain consistency among thousands of generated frames.

The demand for computing power has also increased exponentially. In the past, when using prompts to "draw cards" for models, each content generation only needed to run once, while interactive world models and video models run continuously every second in each session.

Therefore, the growth of reasoning demand no longer depends on the number of requests, but is directly linked to the user's participation duration. Every minute a user stays in the world model corresponds to one minute of GPU real-time computing. This means that no matter which model or which type of application scenario stands out in the end, the underlying service layer will directly enjoy the dividends brought by the growth of this track.

Building a real-time generative video reasoning platform is Reactor's entry point, and this demand comes from the founding team's keen insight into paradigm shifts.

Reactor was co-founded by Alberto Taiuti (CEO) and Bryce Schmidtchen (CTO), both of whom previously served as technical leaders for Apple Vision Pro. Taiuti was also the co-founder and CTO of Luma AI, and he built the underlying infrastructure for this 3D and video generation unicorn. Bryce Schmidtchen worked at Apple for 7-8 years and participated in the construction of visionOS software.

Reactor was co-founded by Alberto Taiuti and Bryce Schmidtchen, Image source: Reactor

Their team also brings together engineers and researchers from Apple, Netflix, Meta, Google, Adobe, Replicate and Microsoft, who have profound experience in graphics, real-time systems, interactive media and extended AI infrastructure.

The experience at Luma AI made Alberto Taiuti see the gap between model R&D and model production deployment: the research team devoted all their energy to model training, which led to them having no extra resources and time to think about how to run these models and perform reasoning in large-scale environments.

When the two founders built products in the Apple Vision Pro team, they thought about how to optimize models under the extremely low power consumption limit on the device, and how to ensure that they can run at interactive frame rates with latency within 10 milliseconds. They are good at optimizing for extreme engineering conditions such as "stuffing an extremely large system into a very small space".

Past experiences made them realize that when the demand for multimodal models and applications explodes, a platform is needed to bridge the gap between model R&D and model production deployment.

When they actually started to do it, they found that every link of multimodal model reasoning, from the underlying reasoning layer, GPU cloud architecture, streaming media and network layer, to developer experience, API design, and even the architecture mode of the entire system, is fundamentally different from that of large language models from the bottom to the top.

The platform built by Reactor has an on-demand schedulable global GPU cloud at the bottom, a self-developed reasoning engine optimized for low-latency reasoning of video and world models in the middle, and low-latency streaming transmission at the upper layer, which returns the model output to applications in the form of real-time video streams.

This platform enables developers and enterprises to run models at a frame rate above 60 FPS and a latency of less than 40 milliseconds. All of this can be achieved through a unified SDK and API, thus shielding the complexity of large-scale deployment and scaling of these systems.

In order to achieve such performance, Reactor's team must rethink many details of machine learning systems: how to carry out kernel level optimizations, how to design the cloud control plane. It even needs to make improvements to the model at the machine learning level, customize CUDA code and Torch optimization, and convert it into a model that meets the production-level availability standards.

A practical example is that Reactor cooperated with the AWS Neuron Science team to deploy the real-time autoregressive diffusion video model on AWS's self-developed AI chip Trainium.

Different from ordinary video generation, the bottleneck of the system running real-time interactive video/world models is not only the computing power itself, but also links such as video frame position calculation, attention calculation, cache copy, video memory occupation and decoding.

The two parties rewrote and fused the underlying calculation through the Neuron Kernel Interface: for example, the 3D-RoPE position encoding calculation that originally took about 5 seconds was compressed to 1.8 milliseconds, the cache copy of about 23 milliseconds per layer was reduced to 1.9 milliseconds, and the process that originally easily overflowed the video memory was controlled to run within 11GB HBM. In the end, the team realized real-time video generation on a single Trainium2 chip.

Reactor's platform has gathered a number of video models and world models that can run in real time. Among them, there are both real-time world models and world generation models that support motion control; it also provides real-time video models with synchronized audio, as well as models for video editing, digital humans, live streaming generation and infinite video streams. A large part of them are open source models from China.

When the Infrastructure Platform Implements Real-Time Reasoning, It Will Unlock New Application Entrepreneurship Opportunities

Reactor has been put into use in the media and entertainment industry. Major Hollywood studios, advertising platforms and video streaming service providers are all advancing active projects on the Reactor platform.

Image source: Reactor

Its founders shared some examples of how their users use interactive videos to build applications in a blog interview. For example, in terms of education, if you want to learn about something from AI, a real-time video demonstration is obviously a better way than reading a paragraph of text. Many startups are already using Reactor to build educational applications.

AI video editing applications will unlock prosumer-level applications. When users modify videos, they can see the effect immediately, instead of waiting 5 minutes for a modified effect (if you make a modification in PS and have to wait 5 minutes, no one will use it).

Interactive film and game works have movie-level image quality, while users can carry out real-time control and guidance. This opens up a brand new medium - a hybrid of movies and games.

The last part is advertising marketing: when users are watching movies or other content, the system can very skillfully implant a personalized advertisement specially generated for the user.

Physical AI and robotics are another natural application scenario for Reactor. As robot policies and world models become more powerful, the computing power required by many models far exceeds the limit that robots or autonomous systems themselves can carry. Reactor can run these models in the cloud, and transmit observation results and action instructions between the model and the physical system in real time at the same time.

Reactor can also run the world model as a closed-loop simulation environment, enabling robot policies to be trained and evaluated in a realistic and responsive generated world, without relying entirely on physical robots or manually built simulators. Whether it is an open source model or a customer-specific private system, the team can use Reactor to deploy, evaluate and train robot policies and world models, without building low-latency reasoning infrastructure by themselves.

The premise for all these applications to be established is twofold: interactive video generation and world models are gradually maturing, and there is a stable platform that can perform real-time reasoning on these models. The joint transformation of models and infrastructure unlocks new possibilities on the application side.

Building this new type of infrastructure sets high requirements for founders: they must not only understand the needs on the application/model side, but also have experience in building and solving infrastructure problems. Such founders are more likely to discover and solve new problems.

For entrepreneurship in this NeoCloud direction, the progress of models, technological breakthroughs of chips, and new forms of applications are all positive factors. Because it is at the basic layer of the entire ecosystem, the development of the ecosystem will drive its growth.

Its moat may not only be scale, computing power reserves and technology, but also the ultimate experience provided to developers and customers. It is a comprehensive capability including core technologies. Founders need to build it as a product rather than a technology.

This article is from the WeChat Official Account "Alpha Commune" (ID: alphastartups), written by the team that discovers exceptional entrepreneurs, and is republished with authorization from 36Kr.