About this brand-new competition over real-time generative models

In the past two years, the most common use case for AI video has been: input a prompt, wait for dozens of seconds or even longer, and then get a complete video. This approach is more like a production tool — users put forward demands, the model generates results, and users then select and modify from the results. The video itself is a one-off product, and once the generation is finished, the interaction comes to an end.

But this year, another type of video is emerging.

Videos do not necessarily have to be fully generated before users can view them. A digital human can be interrupted in the middle of a conversation and then continue to respond; a video that is being generated can have its characters, costumes, and background changed in real time, or even its style altered. The model is no longer just "generating a single video", but continuously generating new frames based on user input in an ongoing live scenario.

This means that AI video may be facing a different problem than before. What it competes for may not only be content production demands such as short dramas, advertisements, and film and television production, but also live streaming, games, socializing, companionship, and various interactive scenarios that require continuous visual feedback.

Major tech companies have also started to enter this field from different directions.

ByteDance's moves are quite typical. On one hand, Seedance 2.0 and Seedance 2.5 continue to push video generation toward a more powerful creative tool; on the other hand, the Seed team has begun to solve the problem of real-time audio and video interaction. SeedRealtime, released in August, is a native full-duplex audio and video model that can continuously understand audio, frame, and temporal information, and handle interruptions, environmental changes, and context during conversations.

It is not a traditional real-time video generation model, but it points to a similar shift: AI no longer just waits for user input before generating a result, but begins to continuously exist in a live scenario.

Regarding this brand-new competition in real-time video, I had a long conversation with Zhang Jintao, head of streaming video generation and inference at Shengshu Technology.

Shengshu Technology is a model company founded in 2023 by the team of Zhu Jun from Tsinghua University. Jintao, a post-2000s generation member, previously focused on model inference acceleration. In 2024, during his first year pursuing a doctoral degree under Professor Zhu Jun, he and the team developed SageAttention, which accelerates Attention using low-bit computing, achieving more than double end-to-end acceleration in Shengshu's video models. This experience later perfectly addressed the practical challenge of real-time video: the model must not only generate content well enough, but also generate it fast enough, and the cost must be affordable for ordinary users.

In May 2024, when Shengshu was preparing to release its first version of the model and launch it globally, Zhang Jintao began participating in the inference optimization of video models as an intern. By March 2026, he became the head of the real-time video generation team. Four months later, Shengshu released Vidu S1, pushing video generation to real-time interaction: users can upload an image to make a digital character respond and perform actions in real time, and this capability can already run on consumer-grade GPUs. Two months later, Vidu S2 was released, further extending the real-time capability from digital humans to video editing, allowing users to change characters, costumes, backgrounds, and styles in real time.

Behind the product changes lies a bigger question:

If videos can also be generated in real time like language, what kind of medium will it eventually become?

At the same time, major tech companies are entering this track. In this brand-new competition, what will end up being the most valuable — the model itself, or the platforms that control user entry points?

The following is our conversation.

01

What is the difference between this round of real-time video and the metaverse wave?

Gu Lingyu: Why is AI video moving from offline generation to real-time interaction?

Zhang Jintao: It's not that we are deliberately moving from offline generation to real-time interaction, but that real-time interaction is a new demand, and we aim to meet this new demand.

Offline generation is mainly a creative tool, and its market ceiling will not be particularly large, which is determined by demand. Suppose everyone watches videos for an hour a day, and there are 6 billion people around the world, but a piece of content may be played repeatedly by many people, so the final generation demand has to be divided by the average play volume. Even if a large amount of offline video content is generated by AI in the future, its market ceiling will still be limited by this logic.

Real-time interaction is different. In language models, everyone has their own session. Technical practitioners consume a lot of tokens every day when writing code and asking technical questions; ordinary users also chat with Doubao, ChatGPT, Gemini, and Claude. The same goes for real-time video: it meets everyone's independent real-time interactive entertainment needs and interactive visual enjoyment, and it also corresponds to an independent session for each person. Humans have extremely large demand for real-time interactive visual enjoyment, and almost everyone spends a large part of their daily activities on real-time interactions, including going out to play, communicating with friends, playing games, etc.

Therefore, I believe that real-time interactive video, or real-time interactive multimodal entertainment demand, is a new and very large market.

Gu Lingyu: What is the difference between this round of real-time video and the wave of metaverse digital humans?

Zhang Jintao: The first difference is cost. Traditional digital humans use CG plus motion capture, which requires you to design the character image and the set of motion capture standards in advance, meaning the initial cost is relatively high.

But with current models, you only need to upload one or two images, click start, wait four or five seconds, and you can start chatting and using it in real time. There is a clear difference in initial usage cost. Secondly, digital humans in the traditional metaverse cannot interact in real time, they can only perform actions set by templates.

With current real-time interaction models, the content you interact with determines what the model generates, there are no preset templates at all, and whatever you want it to do can happen in real time.

Gu Lingyu: What is the key factor that truly makes this round of real-time video feasible?

Zhang Jintao: The capability of the model itself has reached a new stage.

Many things that were impossible before are not because no one has imagined them, but because the model could not complete them at a sufficiently low cost. Now the quality of video generation has improved, the model's understanding capability is also rising, and with inference optimization, it is possible for the technology to enter real-time scenarios.

Real-time video generation technology has only started to develop in the past six months, and the supporting downstream products and industries have not fully matured yet, but the trend has already begun. A year from now, I believe there will be many mature applications, and the competition will become more intense.

Gu Lingyu: When will this industry truly explode?

Zhang Jintao: In about the next two years, we will reach an explosion point that is relatively close to that of GPT-3.5.

Gu Lingyu: If we look further ahead, what scale will the real-time video demand eventually reach?

Zhang Jintao: 3 to 5 years later, the market for real-time interactive video, or real-time interactive multimodal entertainment demand, will be at least comparable to that of language models, or even larger than the language model market.

Gu Lingyu: Do you think it can be achieved in just 3 to 5 years?

Zhang Jintao: At least there are no such technical bottlenecks.

02

The first metric for real-time video is not image quality, but "keeping up"

Gu Lingyu: What is the core technical indicator of real-time video?

Zhang Jintao: First of all, the speed must at least keep up with the playback speed. For example, if a video is played at 24 or 30 frames per second, you cannot take hundreds of milliseconds or even several seconds to generate one frame, otherwise it cannot truly enter real-time scenarios. Therefore, the first thing we need to solve is end-to-end latency and throughput, so that the model can continuously generate frames at the very least.

Of course, after the speed reaches the real-time standard, image quality is still very important. But if the speed does not cross this threshold, no matter how good the image quality is, it cannot form a real real-time experience. So I will regard it as a relatively hard threshold: achieve real-time performance first, and then discuss issues such as quality, stability, and interactive experience.

Gu Lingyu: If we just keep increasing the generation speed, can we move from offline video to real-time video?

Zhang Jintao: Increasing the generation speed is necessary, but it does not equal the realization of real-time video generation. Acceleration technologies like TurboDiffusion can greatly shorten the video generation time. But if we follow the offline generation approach, essentially we still give a prompt first and then generate the entire video. It only solves the problem of "how long it takes to finish generating this video", and does not mean that if the user changes the prompt halfway, the frame can respond immediately.

Real-time video cares more about another type of speed: how long it takes from the user entering a new instruction to the instruction actually affecting the frame.

For example, when the video is playing forward, the user suddenly says "turn left", the system needs to respond promptly (for example, respond within 0.1s) to follow up this action, while maintaining the coherence of the frames before and after. If you have to regenerate the entire video and then play the result, even if the generation is very fast, it is difficult to form a real-time interactive experience.

Therefore, this is not just a problem of inference acceleration, the design goal of the algorithm and the corresponding engineering construction all need to change. Generation speed answers the question of "how fast the video is made", while real-time performance answers the question of "how fast the system can respond to you", and these two things cannot be equated.

Gu Lingyu: What specific changes has Vidu S2 made in real-time video?

Zhang Jintao: In S2, we focus more on how to make the generated video truly support real-time interaction and real-time editing.

There are two main directions this time. One is S2 Avatar. In addition to simple speaking, facial expressions, and limb movements, it can also respond to relatively complex action instructions such as dancing in real time. New reference images can also be added at any time during the generation process. For example, halfway through the interaction, you show the model an image of an object, and it can make the character pick up the object. That means the Avatar is no longer just a preset digital human, but can continuously receive new information and instructions during the interaction. The other direction is S2 Editing. Its input can be an ongoing video stream: the model receives the video while modifying it in real time, and continuously outputs new frames. For example, during video playback, you can switch the style, character, costume, or background at any time, while retaining the original character's movements and video content.

Gu Lingyu: Can NPCs in games also be made using this type of model?

Zhang Jintao: I think it is a very natural direction. The behaviors of traditional game NPCs are mostly preset: what they say and what actions they take under what circumstances are mostly designed in advance. If the model can understand the player's behavior in real time and then generate corresponding visual feedback, the freedom of NPCs will be greatly improved.

But there is also a very practical problem here, which is cost. If every reaction of an NPC requires a large amount of GPU resources, whether it is commercially viable is another matter. So in the end, it will return to what we said earlier: the model must not only be good enough, but also fast enough and cheap enough.

03

Real-time video does not lack raw data, what it lacks is high-quality data

Gu Lingyu: The embodied intelligence industry is still restricted by data, and some people think there will be no sudden, explosive GPT moment. Do you think the same is true for real-time video generation?

Zhang Jintao: I think it is not completely the same. The obvious problem with embodied intelligence now is that there is very little real-world data that can be directly used for training. When a robot performs an action, it needs to actually execute it and collect the data, which is relatively costly.

But video does not lack raw data. A huge number of videos are generated in China every day, and content such as short videos and live streams itself is very rich data. So our problem is not "there are no videos", but how much of the data in these videos can actually be utilized by the model.

Gu Lingyu: So this is an advantage for Chinese models?

Zhang Jintao: Yes, I think this is a very obvious advantage, which means there are many sources of raw data. But having a large amount of raw data is only the first step. What really determines the effect of the model is whether you can turn these raw data into data suitable for training. This includes cleaning, filtering, labeling, understanding, and finally how to construct training tasks, all of which will affect the model.

Gu Lingyu: What is high-quality data?

Zhang Jintao: We all use Diffusion Models to generate videos now. I think its essence is a renderer, and it does not need to have any intelligence at all. So you only need to have the data of the content you want to generate, and label all the elements in the frames of this data.

First of all, a good piece of data must have very high image quality, and the video data itself is in the state that you want the Diffusion Model to generate.

Secondly, you can make very detailed annotations for this video, and describe what happens in the video in great detail with text. Because if you don't label it, it is very difficult for the renderer to render the unlabeled parts.

04

What are the difficulties for models to shift from passive response to active judgment?

Gu Lingyu: If future real-time video relies more and more on Agents, will the model also become more and more like a real-time decision-making system?

Zhang Jintao: There will be this trend. Because in real-time scenarios, users will not give the model a very complete prompt every time. Many times the user just says a sentence, or even just makes an action, and the model needs to judge what he wants to do based on the context.

Therefore, the work of the model will actually change from "generating according to instructions" to "understanding the context and then deciding what to generate". VLM Agents are very important here, because they can help the model combine the user's natural language, visual information, and context.

But this will also make the system more complex.

In the past, it might just be a single model that maps input to output, but now it may become the collaboration of multiple modules, including visual understanding, Agent, generation model, post-processing, and security control. The real-time requirement also means that the latency between these modules cannot be too high.

Gu Lingyu: After the model becomes more active, will it become more difficult to control instead?

Zhang Jintao: This is indeed a problem that needs to be considered. The more active the model is, the more things it can understand and complete, but at the same time you also want it to be controllable at some boundaries. Especially in real-time scenarios, the model does not leave much time for manual intervention, so things like security, content control, identity and age judgment all need to be built into the system as much as possible.

For example, content involving celebrity faces and personal identities needs to have relatively strict restrictions. Real-time video does not mean that the model can generate whatever it can, it will eventually run in a real product environment, so technical capabilities and security capabilities advance side by side.

05

The computing power for real-time video must first be affordable for ordinary people

Gu Lingyu