Tsinghua University, NVIDIA and other institutions have launched a new ranking, exposing the shortcomings of 38 real-time AI video models.

AI real-time interactive video is one of the most popular technology directions this year.

In the past more than half a year, relevant models have continuously gained widespread attention outside the professional field. Alibaba launched HappyOyster, Kunlun Wanwei released Matrix-Game 4.0, Aishi Technology released PixVerse R2, Shengshu Technology launched Vidu S1 in July and immediately released Vidu S2 in September...

Capital is also rushing into this track, many companies have completed new rounds of financing, world models, digital humans, real-time editing... participants from all directions are pouring in from different paths.

However, behind the boom, this track is currently in a state of chaos.

The chaos first comes from definitions. Even for products that focus on real-time interactive video, some call it "interactive world model", some call it "real-time interactive multimodal model", and some regard it as a third independent route separate from video generation and world models. There is no consensus on what the track is called, let alone unified evaluation standards.

Existing evaluation tools still follow the old framework: input a prompt to the model, generate a video, and check whether the picture is clear, the movement is reasonable, and the style matches the description. Mainstream benchmarks such as VBench and EvalCrafter all follow this logic, but for real-time interactive video systems, this set of methods cannot test the most challenging part at all.

The core challenge of real-time interactive video lies in that user instructions are continuous, and the model needs to respond to new instructions in time without interrupting generation, while keeping the character identity, existing scenes and all previous valid modifications unchanged. Existing evaluation tools have hardly touched on these points.

Recently, Tsinghua University, NVIDIA, CMU, UC Berkeley, Princeton, NUS and other domestic and overseas universities and institutions jointly launched a new evaluation benchmark — StreamEval.

Leaderboard: https://streameval.ai

Paper: https://arxiv.org/pdf/2610.28402

Github Code: https://github.com/StreamEval/StreamEval

Benchmark Data on Huggingface: https://huggingface.co/datasets/StreamEval/StreamEval

Benchmark Data on ModelScope: https://www.modelscope.ai/datasets/StreamEval/StreamEval

This is the world's first streaming interactive video generation evaluation list in the industry, covering five types of tasks: digital human dialogue, virtual human performance, world generation, instruction editing and reference editing, with a total of 300 test cases covering 38 models.

Compared with existing benchmarks, StreamEval not only tests "generation quality", but also focuses more on "interaction capability". It integrates instruction execution, state consistency, visual quality and audio-video coordination into the same quantifiable evaluation system, so that models of different technical routes can complete tests under unified conditions.

At present, all codes, datasets and scoring rules are fully open source, developers can independently retest and verify the scoring process according to the public process. The leaderboard is continuously updated, and any developer can connect their own model to participate in the test.

What does StreamEval specifically test? How does it test?

To evaluate a real-time interactive system, it must be run under near-real conditions, where instructions arrive in batches, scenes evolve continuously, and the model keeps responding as time flows.

StreamEval is designed based on this logic, covering five types of tasks: digital human dialogue, virtual human performance, world generation, instruction editing and reference editing. Each type contains 60 test cases, covering five durations of 15 seconds, 30 seconds, 60 seconds, 120 seconds and 300 seconds respectively, with very comprehensive evaluation dimensions. Each test case is run independently three times by the same model to reduce the influence of accidental factors on the results.

The overall framework of StreamEval. This benchmark covers five types of tasks, involving digital human interaction, world generation and real-time video editing.

The entire dataset contains 1347 online events. Voice, text instructions, reference images and source video frames are released step by step along the timeline. The model can only see the currently released input, cannot know the subsequent requirements in advance, and cannot access the expected answers used for scoring.

Composition of the benchmark dataset. (a) Task and function distribution; (b) Preset input types; (c) Number of input events in each test session; (d) Time interval between adjacent preset input events. The figure shows 200 test cases, including 562 online input events in total, excluding initialization input and source video frames of continuous input. When counting input intervals, event pairs involving trigger events are excluded.

The five types of tasks have their own priorities:

Digital human dialogue examines whether the model can understand the user's voice, give context-compliant responses, and maintain character identity and dialogue state at the same time;

Virtual human performance requires the character to complete actions such as sight, expression, posture and gesture according to real-time instructions;

World generation focuses on scene changes, camera movement and spatial continuity under continuous control;

Instruction editing and reference editing require the model to complete modifications according to text or reference images while the source video is playing continuously, and retain the unspecified content.

Coverage range and evaluation duration of the five types of tasks. Dlg. stands for digital human dialogue, Perf. stands for virtual human performance, World stands for world generation, Instr./Ref. stand for real-time video editing based on text instructions and reference images respectively. ✓ means the task is included, × means the task is not included.

StreamEval divides the performance into four dimensions and ten indicators.

Task execution focuses on dialogue performance, instruction following and complete editing; state consistency examines picture stability, temporal continuity and non-target content retention; visual fidelity covers image quality, character identity and reference image matching; audio-video coordination mainly evaluates audio-video synchronization.

Each indicator is scored from 0 to 100. For each task, only applicable indicators are calculated, and then the comprehensive score within the task is obtained with equal weight. For digital human tasks, an additional 5 points are added to systems that support custom images. The five types of tasks are ranked separately, and will not be merged into a cross-task overall list.

38 models submitted results, continuous instructions are the hardest part to score

The paper evaluated 38 models in total, forming 49 model task configurations. The results show that image quality, character identity maintenance and real-time control are several relatively independent capabilities. The fact that a model can generate videos stably does not mean it can reliably execute changing instructions.

In the digital human dialogue task, Vidu S2-Avatar's basic comprehensive score (Q-Overall) is 88.98, and after adding 5 points for custom image capability, its comprehensive score (Overall) reaches 93.98, achieving the highest result among the participating models that support custom images.

Evaluation results of the digital human dialogue task.

For the virtual human performance task, Vidu S2-Avatar also obtained the highest comprehensive score of 81.71, with its audio-video synchronization, stability and identity consistency reaching 99.60, 97.25 and 86.12 respectively. However, its instruction following (Following) score is only 36.90, which is already the highest value among the seven participating models.

This means that while the character can remain stable and the lip shape can keep up with the sound, the specific actions may still not be completed as required. Continuous interaction puts more complex requirements on the model: it needs to not only understand the instructions, but also make the correct object change at the correct time.

The world generation task also shows obvious capability differentiation.

MemFlow achieved the highest comprehensive score of 33.36, Krea Realtime got the highest instruction following score of 25.93, and Matrix-Game 3.5 obtained the highest continuity (Continuity) score of 34.50. HappyOyster 1.0 Adventure's image quality and stability reached 58.34 and 56.67 respectively, both the highest values in this task, but its instruction following score is only 5.56, and the continuity score is only 2.78.

World generation. Model configurations under text conditions and image conditions are ranked together; when the same model is evaluated under both input conditions, only its evaluation results under text conditions are reported.

These results indicate that visual perception, instruction response and long-time continuity need to be judged separately. The fact that a model looks more stable does not mean it is more obedient to instructions.

In the real-time editing section, Vidu S2-Editing achieved a comprehensive score of 45.43 in the instruction editing task, with instruction following and edit success (Edit Success) being 72.10 and 57.20 respectively; JoyAI-Video-Edit has a comprehensive score of 42.39, but its content retention (Preservation) score reaches 83.33, higher than Vidu S2-Editing's 60.93. The former has higher scores on indicators related to instruction execution, while the latter is better at retaining non-target content in the source video.

The reference editing task is more difficult. Vidu S2-Editing obtained the highest Overall score of 39.02, with Preservation at 64.44 and Edit Success only 15.00. This score is already the highest value among the four reference editing configurations publicly reported in the paper.

Reference image switching, modification range control, existing state retention and undo recovery need to be completed together in the same continuous generation process. The model may successfully change the character's appearance but destroy the background; or it may retain the original video content but fail to adopt the new reference image. Only when all conditions are met at the same time can the full editing score be obtained.

Take the scenario where the system first loads a red velvet cloak, switches to a gold satin cloak at the 10th second, and issues an instruction to remove the edit at the 20th second as an example: the model needs to change the costume in time, keep the character's movements and background, and finally accurately restore to the original state of the source video.

Reference image editing during input change. Reference Image A is preloaded, Reference Image B is input at the 10th second, and the instruction to remove the reference image is issued at the 20th second. Each column pairs and displays the source video frame and the model output frame at the corresponding session time point. The partial enlarged view shows the selected cloak area. The results show that in the two sets of outputs during the effective period of Reference Image B, the cloak still remains gray and does not change to gold satin as required.

Therefore, the focus presented by StreamEval is not about who takes the lead in all tasks comprehensively. Different models have their own advantages, and the common problem faced by the whole industry is how to stably execute continuous instructions in the process of continuous generation and correctly maintain the established state.

Conclusion

The value of StreamEval is far more than just ranking models.

Behind it are top academic and industrial institutions including Tsinghua University, NVIDIA, CMU, UC Berkeley, Princeton, NUS and so on. The joint participation of multiple research forces determines that it is a public evaluation framework for the entire industry from the very beginning, serving the common needs of this field.

More critically, it is establishing a reproducible and continuously evolving evaluation standard for real-time interactive video. The public test data and scoring rules allow developers to accurately locate the shortcomings of their models, and the progress of different technical routes thus has a clear and verifiable basis. In a track where even the definition has not been unified, providing such an open, transparent and verifiable benchmark is scarce in itself.

The gold content of the leaderboard depends on how many models participate