The world awaits Zhang Yiming

In 2021, Zhang Yiming once wrote in his resignation letter: "The impacts of virtual reality, life sciences, and scientific computing on human life have already shown the first glimmer of dawn, and these require us to break through the inertia of our existing businesses to explore."

In June 2021, Caixin reported that Zhang Yiming personally traveled to Weifang to negotiate the acquisition of Pico, and had shown his acquisition intention as soon as they met. This acquisition later became a landmark event in this round of metaverse investment boom, but Caixin specifically noted in the news that the two parties did not disclose their intention to "enter the metaverse" at that time.

Five years later, in June 2026, ByteDance's world model was exposed. The disclosure said that the world model is one of ByteDance's four key AI priorities in 2026, with the goal of releasing at least one version by the end of the year, benchmarking against Genie 3.

This project is based on Seedance, with users accessing through Pico's head-mounted devices. By saying a word or making a movement, the cloud will generate a dynamically changing world accordingly.

The "virtual reality" Zhang Yiming mentioned does not refer to the metaverse, but a far more grand world model. His resignation letter can actually be interpreted as a preview notice.

In September 2026, another disclosure came out that Zhang Yiming, who has become the richest man in China, still personally supervises a real-time spatial video model of ByteDance, coordinating different business departments to tilt AI resources and computing power to this project, which shows how much he values the world model.

It is not clear how well ByteDance's research on Genie 3 is going, but Google took the lead and released the world model Genie 4 on October 28, which undoubtedly put some pressure on Zhang Yiming.

Compared with the previous generation, the biggest feature of Genie 4 is that users can continuously "intervene" in the world. In the demonstration released by Google, you can step into the world of Genie 4, make it rain, turn the camera around, and make a person appear on the street corner. The wind has sound, and the characters can respond to your words.

People who participated in the test said that what is set for Genie 4 is no longer just a camera view. You can assign it an event that is about to happen, and Genie 4 will automatically generate all corresponding content based on this event.

What is Genie 4?

Rather than saying that Genie 4 is a world model, it is more like a game.

Previous world models are essentially a type of video model that pre-generates possible future frames for you to view, while Genie 4 generates the next frame based on your actions.

If you ask the protagonist in the video to turn left, what traditional video models can do is to regenerate a new video clip of him turning left.

But with Genie 4, when you press WASD, the character can move and turn according to your operation. At this time, the model needs to judge how the relationship between the camera, characters, objects and space will change next, and then present the result to you in real time.

The former generates a segment of content, while the latter simulates the consequences of actions.

Google has been working on this path for several generations.

Genie 1 in 2024 learned potential actions from unlabeled action internet videos, allowing humans to control the generated environment frame by frame. Its importance does not lie in the exquisite picture quality, but in that the model began to learn the relationship between "actions" and "changes" from videos.

Genie 2 in the same year turned a single image into a 3D space that can be operated by humans or AI with mouse and keyboard, and began to demonstrate capabilities including object interaction, character animation, gravity and lighting, as well as re-rendering scenes outside the field of view.

By 2025's Genie 3, the world was upgraded to 720p, 24fps, and could maintain content consistency for several minutes. In addition to navigation, users can also change the weather, add characters and objects through prompt words. Videos first became controllable, then accessible, and finally could be continuously intervened.

The main improvement of Genie 4 is to make this world more worth "playing".

You can use natural language to change the weather, move the camera, introduce a person or an object in real time. Gemini will combine the current scene to understand your instructions, instead of rebuilding the entire world from scratch every time.

Between your two inputs, it will also continue to monitor and guide the scene to keep the world progressing.

Google emphasized that Genie 4 includes physical simulations such as aerodynamics and collision, as well as more natural movements of characters and their responses to the environment.

In addition, Genie 4 is also the first model with sound in the Genie series, which can generate wind sound, wave sound and background music synchronously with the picture.

In the past, people kept telling the world "what will happen next". Now, Genie 4 begins to understand "how this thing will happen step by step".

Gemini 3.8 Flash observes the frames generated by Genie 4 in the loop, splits complex goals into steps, checks whether each step is completed, and adjusts the plan after the world changes or users intervene.

Of course, world models do not only have the route of "letting people go in to play games".

For example, Nvidia's Cosmos uses the world foundation model for simulation, reasoning and data generation, to provide training for robots, autonomous driving and industrial vision.

Autonomous driving needs to be trained with rare traffic scenarios such as extreme weather and complex lighting.

If you want to simulate such scenarios in the real world, it is not only very costly, but also not suitable for active creation. If the simulation world can generate these situations, machines can be trained, tested and verified in it first.

Robots also need to understand the consequences of actions. For example, when grabbing a cup, will it slip? How will the object move after the palm collides with the cup? If the cup is placed in another position, can the original strategy still work?

This route requires not only realistic pictures, but also controllable synthetic data, physical interaction, action consequences and repeated verification.

Google is also exploring the combination of Genie 4 and Gemini Robotics, but the feature of this generation still focuses on entertainment, emphasizing immersion and interactivity.

After all, Genie 4 is only a preview version. Google says it can only maintain consistency in short-term sessions. When running for too long, or after multiple complex modifications, visual details and object permanence may still drift. At the same time, issues such as complex physics, real-time audio, spatial accuracy and long-term consistency of the model have not been solved yet.

ByteDance has made layout long ago

ByteDance's world model has similarities with Google's.

In 2024, Zhou Chang, former technical leader of Tongyi Large Model, joined ByteDance from Alibaba and was responsible for the research related to world models.

However, at that time, ByteDance internally judged that the technical route and commercial scenarios of the world model were not clear enough, and the more important thing was to win the battle of video models first.

By 2025, ByteDance began to form a VLA research group on a small scale. The team led by Li Hang, head of ByteDance AI Lab, mainly trained based on simulation data, and the team led by Wang Wenqian, Seed multimodal researcher, mainly trained based on natural data.

After the 2026 Spring Festival, Seed set up a new 3D simulation world model group, led by Fan Haoqi, former researcher of Meta FAIR, reporting to Zhou Chang; the original two VLA groups were merged and also reported to Zhou Chang uniformly.

Thus two routes were formed: VLA focuses on embodied intelligence, pursuing real actions and environmental responses; 3D simulation focuses on entertainment and games.

ByteDance's training data budget for the world model has reached tens of millions of yuan, covering modalities such as VLA, long video and 3D. A data supplier said that its investment is 3 to 4 times that of other manufacturers.

However, with money spent and people recruited, no satisfactory results have been achieved. According to reports, internal evaluation of ByteDance in early 2026 showed that the comprehensive performance of its world model was still about 10% behind the global SOTA at that time.

Just like Google, ByteDance also has technical support for developing world models.

Seedance 2.5 can generate video content up to 30 seconds in a single run, and can continue the story through multiple rounds of prompt words, so as to output a complete short film.

It can generate videos by referring to multiple materials, and can also modify content by timestamp. Character poses, motion trajectories and camera movements can also be set up with white molds first, and then generate videos based on the white molds.

In the past, video models were essentially random draws: after the prompt word was sent out, the output of the model could be good or bad. Seedance 2.5 is a combination of AI video generation + AI editing, and the output quality is more stable.

The world model not only needs continuous frames, but also requires all objects in it to be three-dimensional, so that you can view them from different angles.

Thus comes Seed3D. It can turn a single image into a 3D model with geometry, texture and PBR materials, and after being imported into Isaac Sim, it can also be used for physical simulation and robot training.

In other words, the assets generated by Seed3D are 3D assets that conform to physical rules and can be used for action testing.

Finally, there is SeedRealtime, a native audio and video full-duplex large model. It is a real-time interactive model with a single architecture that end-to-end models audio + video + text + time series streams.

No matter how beautiful the picture of a world is, if it cannot understand what people are saying and does not know when to respond, it is still just a video clip.

On October 11, Tim from MediaStorm showed such a video: the user first used a sentence to generate a mini-perspective room, then rode a paper plane with mouse and keyboard to fly freely in the scene.

If you want a spoon, the world will generate a spoon, and you can even climb up along the spoon. Then draw a "any door", and the originally unrelated room and outer space will be connected together.

Although ByteDance has not officially announced the specific information of this world model, it has let the outside world see the rudiment of ByteDance's world model for the first time. It emphasizes spatial exploration, as well as modification and interaction at any time.

In the 2026 Hurun Rich List released on October 27, Zhang Yiming became the richest man in China with a fortune of 670 billion yuan.

However, according to reports, on the world model project, Zhang Yiming personally coordinated different business departments of the company to allocate AI resources and computing power for the project.

Zhang Yiming hopes that this model can connect four things: AI model, cloud computing, content platform and Pico hardware. Eventually, he hopes that the world model will become ByteDance's new flywheel.

Seed is responsible for generating the world, Volcano Engine provides cloud computing power, Jianying and Jimeng lower the creation threshold, Douyin, live streaming and short dramas are responsible for distributing experiences, and Pico provides the spatial entrance.

The advantages of Pico lie in light weight and low price.

Traditional XR devices need to cram a large amount of computing into the head-mounted display, which makes the headset heavy and expensive, such as Apple's Vision Pro. Zhang Yiming wants to put complex computing generation in the cloud, so that the headset is only responsible for perception, transmission and display.

The cost has not disappeared, it has just been transferred from the headset to the data center.

Therefore, the more users there are, the higher the cost of concurrent computing. If the network fluctuates slightly, the delay will directly destroy the sense of immersion.

At present, the generation index of ByteDance's world model is about 20fps, which is not smooth, and at best it can only be regarded as "not stuttering".

From "Recommending the Next Piece of Content" to "Generating the Next Second"

It is not only Zhang Yiming who can develop world models. Li Fei-Fei, Yann LeCun, DeepMind, Nvidia, and a large number of robotics and autonomous driving teams are all developing world models from different directions.

All of them are technical experts, but for the world model to move from the laboratory to the public, more than technology is needed. It also requires low-friction interaction, creator ecosystem, content distribution and so on.

What Zhang Yiming is best at is not inventing every underlying technology, but better at technology implementation.

For example, the recommendation algorithm that made ByteDance rise to fame was obviously not invented by Zhang Yiming, but Zhang Yiming combined this technology with ByteDance's existing products, and compressed the whole process of "finding the next piece of content" into a single swipe.

You move your thumb, and the next piece of content you might like comes out.

Users do not need to understand what a recommendation system is, nor do they need to learn a set of operation methods. Swipe once, if you don't like it, swipe again.

The world model also needs such an implementation method.

Can we expect users to learn 3D engine, camera parameters, action space and prompt word format before entering the world model? If so, no matter how advanced ByteDance's world model is, it can only be a decorative vase.

Zhang Yiming shortened the distance between people and content, and the world generated by the world model is just a piece of content.

Similarly, organizing creation, distribution and feedback are also what Zhang Yiming is good at.

A little-known fact: infinitely generating worlds does not mean infinitely generating good experiences.

Randomness does not equal richness, realism does not equal fun, and interactivity does not equal that users are willing to interact. A space that can extend infinitely, without goals, characters, conflicts and rhythm, is ultimately nothing more than a space.

Therefore, the new "world" needs creators, a large number of creators. What they need to create are initial settings, character relationships, rules, and a scene that allows users to explore and interact.

Once generation is no longer scarce, "what is worth generating" will become scarce.

ByteDance's existing creators, tools and distribution system may make "world model designer" a profession like AIGC creators. These people are responsible for making the world work, and continuously modifying it according to user behaviors.

The deeper change is the shift from "recommending the next piece of content" to "generating the next second".

The recommendation system faces an already existing content library.

You have just finished watching a video, the system will predict what you are most likely to watch next based on your preferences and user portraits, and then pick out the next piece of content from the library.

What the world model faces is the next second that has not fully happened yet.

You step forward, turn left, say a word, or pick up an object. The model needs to predict how the environment will change after your action, and generate this moment in real time.

On Douyin, ByteDance distributes already produced content based on user behaviors. In the world model, ByteDance can generate the content itself in real time based on each behavior.

In a broad sense, this is still a kind of "recommendation", because they are all predicting what should appear in front of you. The only difference is that in the past, what was recommended to you was content that others had already made. In the future, what is recommended to you will be a result that did not exist before.

This is also why the assets Zhang Yiming left behind during the metaverse period are still worth re-examining today.