Unreal Engine CEO observes the latest world model: code-driven, and the video model generates visual content.

In recent years, video generation models have made rapid progress in clarity, temporal consistency and controllability. Given text, images, camera trajectories or player actions, models are already capable of synthesizing coherent subsequent frames, which also gradually pushes the "interactive video world" from a concept to a viewable and operable prototype.

Being able to continue frames does not equal being able to run a world. Displacement and collision in the footage are only the most intuitive layer of interaction; in more complex environments, characters also have goals, identities and relationships, events are constrained by rules, and will leave causal consequences that span a relatively long period of time. Many states even occur outside the footage, but they will still change the subsequent plot and actions. It is difficult to explicitly maintain these high-level semantics only by predicting the next observation from local pixels.

A complex world first requires "understanding why". Goals, rules, memories and causal chains usually cannot be directly read from the current single frame, but they determine how the world should change next. The knowledge invocation, reasoning, planning and programming capabilities of large language models are suitable for handling such low-frequency but high-complexity decisions, and further implement the decisions into executable rules.

Another often overlooked fact is that game videos are not the world itself, but the pixel projection of the program execution results after rendering. At this stage, games and simulators still constitute the most important interactive data source for video world models. If only actions and pixels are retained, and then the video model is allowed to directly learn the mapping between the two, it will bypass the originally existing programs, rules and explicit world state, and instead require the model to inversely fit the entire set of program behaviors from the visual results. Such a learning path is not only inefficient, but also couples "how the world evolves" and "how the world is presented" into the same model.

Based on this observation, the research teams from Westlake University AGI Lab and Nanyang Technological University propose the Code World Model, a new world model paradigm that uses large language models as the "brain". Its core idea is to let the coding agent maintain and update the executable world state by continuously writing, calling and modifying code; the state related to the current observation is then converted into a proxy to guide the video model to generate fine visual frames.

Paper Link: https://arxiv.org/abs/2608.25927

Project Homepage: https://buaacyw.github.io/cwm/

Code Repository: https://github.com/buaacyw/code-world-model

The large language model here does not replace the video model. Code World Model re-divides the responsibilities of the two: knowledge, goals, rules and long-term causality are the responsibility of the coding agent and code, while appearance, motion details, lighting and texture are the responsibility of the video model. In other words, the former decides what happens in the world, and the latter decides how all of this looks.

Figure 1  Core division of labor of Code World Model: the coding agent evolves the world state through code, the proxy converts the relevant state into visual conditions, and the video model generates the final frames.

Method Overview

Centering on the above division of labor, Code World Model splits the operation process of the world into three interconnected parts.

Coding Agent: Understand the player's intentions and new events, call world knowledge, infer potential consequences, and decide which mechanisms need to be executed, combined or modified.

Code: Execute deterministic updates such as position, attributes, collision, cooldown and event triggering at a higher frequency, and turn high-level decisions into inspectable, reusable and continuously running rules.

Video Model: Read the evolved state conditions, and use the appearance, motion and interaction priors learned from large-scale visual data to generate visual observations with rich textures, lighting and local dynamics.

This division of labor also corresponds to different calculation frequencies. The coding agent only needs to make sparse decisions when new goals, complex events appear, or the existing mechanisms are insufficient; once the decisions are written into the code, the program can continuously perform dense state updates.

More importantly, what the coding agent rewrites is not only the value at a certain moment, but also the subsequent operation mode that the world follows.

Figure 2  Overall framework of Code World Model. The complete concept includes visual feedback; the current prototype focuses on verifying the forward link of coding agent / code → world state → proxy → video model.

Proxy Connects Executable State and Video Generation

After completing the division of responsibilities, there remains a key interface problem: the coding agent and code maintain the executable state, while the video model receives text, images or video tokens. A common language that is easy to be constructed by the program and can provide frame-by-frame spatial constraints is needed between the two.

Structured text is flexible enough, but it is difficult to stably describe the entity position, relative relationship, motion trajectory and precise camera change of each frame; although fully constructing and rendering a 3D world has stronger control, it will reintroduce the high cost of assets, geometry, materials, animation and rendering pipelines.

To this end, this work introduces the proxy. It extracts the information that the current observation really needs to follow from the world state, and organizes it into a coarse-grained visual representation. A lightweight, deterministic compiler renders the proxy into a proxy video, which is then sent to the video model together with the structured text.

Text is responsible for explaining "who it is, what it is, and what to do", while the proxy stipulates "where it is, how it moves, and how the camera shoots".

The proxy can express the camera and perspective, entity position and orientation, scale and trajectory, scene layout and occlusion, as well as the coarse-grained state required for current interaction; textures, materials, fine lighting and complete local motion are intentionally left to the video model. The paper summarizes this design principle as: only retain the minimum sufficient state required for the current observation, to achieve a balance between control capability and state encoding cost.

The proxy in the current implementation is only a quarter of the target video in both width and height, and the number of pixels is about 1/16 of the target. It is composed of simple, reusable geometric primitives, does not require production-quality assets, but can provide frame-by-frame, editable spatiotemporal skeletons.

Obtain Strictly Aligned Data from Game Runtime Records

To make the video model learn to understand the proxy, the training samples need to contain proxy video, structured text and target RGB video at the same time, and the proxy and RGB must be strictly aligned in time, camera and entity identity.

The runtime record of the game naturally retains this correspondence. The research team synchronously records RGB, camera state, entity identity, position and orientation, approximate scale, scene layout and interaction state from the same gameplay execution, and then compiles the proxy offline through code. Since the two come from the same execution, a one-to-one correspondence can be established for each frame, and the cross-frame identity can also be mapped stably.

Figure 3a  Target RGB (top) and same-frame proxy (bottom) in game data.

Figure 3b  Target RGB in real video (top), and the proxy (bottom) constructed offline using camera calibration, 3D reconstruction and object annotation.

The gameplay data used in the paper includes 157 recordings, with a total of about 5.6 hours of source video. The research team samples at 2-second intervals to obtain 9,420 5-second training clips; each RGB target contains 124 frames, with a resolution of 1344×768 and a frame rate of 24FPS. The corresponding proxy also contains 124 frames, with a resolution of 336×192, combined with fixed-log depth and categorical semantic-ID map.

The same runtime record can also be recompiled into proxies with different coverage ranges and information granularity, without re-recording RGB.

The paper also demonstrates the offline construction process of real video proxy-observation pairs on KITTI-360 to illustrate the possibility of this interface expanding to real data; the adaptation of the current video model only uses paired gameplay data.

How Does the Prototype System Run?

The current prototype adopts the Ref2VA backbone of MiniMax-H3 as the video model, performs rank-128 LoRA adaptation on all 50 transformer blocks, and the trainable parameters are about 596 million. The training uses 8 NVIDIA H800 GPUs, completing a total of 3 epochs and 3,534 optimization steps.

In the inference stage, GPT-5.6 Sol acts as the coding agent. The system provides it with basic player control, collision processing, runtime update loop, existing game-engine scene and gameplay logic templates, as well as the proxy primitives used in the training phase. The coding agent can combine, extend and rewrite these codes to construct a simple, executable, player-controllable world; the coarse-grained 3D geometry in it is only used to express the proxy, not directly as the final frame.

For each 5-second clip, GPT Image 2 generates the appearance anchor according to the first frame proxy and text; MiniMax-H3 then combines the first frame, the complete proxy sequence and text to generate a 124-frame, 1344×768, 24 FPS RGB video.

Longer videos are generated through a sliding window with 34-frame overlap: the latter window inherits the RGB context at the end of the previous window, and reuses the same appearance anchor to maintain local continuity and overall identity appearance.

Experimental Results

Qualitative results show that even with only about 5.6 hours of gameplay source video for LoRA adaptation, the model can still follow the character position, motion trajectory, scene layout and camera motion specified by the proxy when the characters, environment and style deviate significantly from the training appearance, while supplementing rich textures, lighting and local dynamics.

Figure 4  The proxy provides character positions and motion trajectories, and the video model presents the same coarse-grained skeleton as characters of different identities and art styles. Top: proxy; Bottom: generation result.

Figure 5  Simple geometric primitives impose constraints on complex objects and camera motion. Top: proxy; Bottom: generation result.

For example, the same coarse human primitive can be presented as a character with completely different identities and art styles; wireframe or box can constrain the position and general motion of vehicles, ships and other complex objects; the same set of proxy interfaces can also express running, dancing, swimming, falling and camera orbiting.

From this, we can see that the proxy fixes the spatiotemporal skeleton that the current observation must follow, rather than the assets and art style of the training game.

The project homepage also provides video comparisons with action- or camera-conditioned video world models.

This comparison focuses on control granularity and visual results: the proxy directly provides frame-by-frame entity motion and perspective changes, so it can impose finer-grained and more direct spatiotemporal constraints; the paper does not present this set of results as an inference latency comparison.

Some netizens commented: "Will Unreal Engine 6 be the last engine that creates games using traditional 3D methods? Maybe Unreal Engine 7 will adopt a similar technical solution. Only Tim, the CEO of Unreal Engine, knows the answer."

Then Tim also came to watch and commented: "I don't know either!"

Conclusion: Decide What Happens in the World First, Then Generate How It Looks

In the past, video world models mainly learned "how observations continue". Code World Model takes the problem a step further: before generating the next observation, first maintain what the current world is, what rules it follows, and why the consequences of an event will persist.

A truly open generative world requires not only the visual imagination of the video model, but also an execution mechanism that can maintain knowledge, goals, rules, relationships and causal consequences. The core of Code