Baidu's autonomous driving technology has created a scenario similar to *The Matrix*, where its world model conducts real-time global deduction, so that one day of operation can achieve the effect that would otherwise take three years of real-world driving.

1 Million Kilometers ——

It is approximately 2.6 times the average distance between the Earth and the Moon, enough to circle the Earth 25 times along the equator.

What does it mean for an L4 autonomous driving system to complete real-world road tests of such a massive scale?

Calculated on the basis that a single vehicle travels 150 kilometers per day, it is equivalent to continuous driving for more than 6600 days. Even if hundreds of test vehicles are deployed on the road at the same time, it will still take more than two months. What's more, complex conditions on real roads often do not emerge as scheduled in the test plan.

For a long period of time, this has been a practical difficulty facing the large-scale development of autonomous driving. Every expansion of the fleet operation scope and update of the driving system is often accompanied by a large number of traffic scenarios that need to be verified.

But now, this time-consuming and labor-intensive verification method in the real world has gained a new solution.

Just now, Baidu Apollo officially unveiled its brand-new world model, ApolloWorld. It can use AI to build a continuously changing virtual traffic environment, allowing autonomous vehicles to experience different weather, roads and traffic participants in it, and complete a full Robotaxi trip.

The acceleration, deceleration or lane change of an autonomous vehicle may change the choices of surrounding traffic participants, and then derive different outcomes.

Well... it looks like an interactive game with extremely high freedom.

And this virtual city already has practical applications now.

According to introductions, ApolloWorld has been applied to the global testing and verification of Apollo Go autonomous vehicles, moving more training and verification work that originally needed to be completed on real roads to the cloud.

The most intuitive change is efficiency.

According to the data disclosed by Baidu, the virtual road test mileage completed by ApolloWorld in one day is equivalent to a real test vehicle equipped with a safety operator driving continuously for about three years.

This means that part of the test tasks that once relied on the accumulation of a large number of vehicles, personnel and time can now be carried out in parallel with the help of computing resources.

The vehicle-side technology supporting this system has also made progress. Baidu's new-generation L4 end-to-end model architecture has been put into actual operation on a certain scale of vehicles in China, and is gradually replacing the previous traditional solutions.

The world model provides a virtual training and verification environment, and the driving model is repeatedly tested in it before being put into operation on real roads. To put it in an intuitive way, it is using AI to teach AI to drive.

For Apollo Go, whose business covers 28 cities around the world, this technical system happens to emerge at the stage when its global expansion is accelerating.

As autonomous vehicles enter more cities, the traffic environments that R&D needs to cope with are becoming more and more complex. However, on real roads, the most unexpected scenarios that are most worthy of research are often the hardest to encounter.

A script-free virtual city, one day of testing equals three years of real driving

People who have watched the movie The Matrix are probably deeply impressed by the world constructed by programs. No matter the streets, buildings, vehicles or pedestrians, they look exactly the same as those in the real world, while a huge computing system is running behind them.

Unexpectedly, only more than 20 years have passed, and the virtual world in this script has really stepped out of the movie and entered reality.

ApolloWorld demonstrated by Baidu generates various virtual cities for autonomous vehicles to drive in. The difference is that traffic participants in the virtual city can also make judgments independently according to the environment.

For example, when an autonomous vehicle drives to the entrance of an elevated road, a car on the adjacent branch road is preparing to merge into the main road.

If the autonomous vehicle chooses to drive onto the elevated road, the vehicle on the branch road will get enough space and naturally merge into the main road.

But if the autonomous vehicle changes its route, continues to drive along the right auxiliary road and occupies the original merging space, the other vehicle will also change its decision, decelerate and stop to wait:

Dr. Yang Ze, Distinguished Architect of Baidu Intelligent Driving Group and Head of the World Model, introduced that in this system, the world model team adopts the technical solution of multi-agent driving, and traffic participants such as vehicles and pedestrians on virtual roads are all modeled as AI agents with decision-making and interaction capabilities.

Each agent has its own action goals and behavior logic, and will adjust its decisions according to the movements of other traffic participants.

Not only the traffic participants, but also the traffic conditions in the virtual city can be adjusted independently, for example, setting different sparse, moderate or dense traffic flows, or changing the driving styles of other vehicles to make them drive more cautiously or more aggressively.

Similar complex interactions, which in the past may have required autonomous vehicles to go on the road repeatedly and wait for specific scenarios to appear, can now be actively organized in the virtual environment.

However, to truly complete the Robotaxi task, it is far from enough to only realize operations such as lane changing and obstacle avoidance according to the environment.

In real scenarios, a trip may last 20 to 30 minutes, passing through dozens of intersections on the way.

Every decision made by the autonomous vehicle may change the traffic conditions encountered later. It needs to complete the entire driving task in a continuously changing environment, interact with a large number of traffic participants, and the previous driving decision may also affect the subsequent trip.

What ApolloWorld demonstrates this time is virtual road testing at the level of complete trips.

In different scenarios such as Binhai Avenue in Shenzhen and the vicinity of Hong Kong Airport, several autonomous vehicles can start from the same location and drive to different destinations. The behaviors of surrounding traffic participants, changes in road environments, and observation data from cameras and LiDAR are all continuously generated by the world model.

Dr. Yang Ze mentioned that the operation of this system is supported by the collaboration of technologies such as agent interaction modeling, multi-view and multi-modal generation, and complete driving closed loop.

To put it simply, the world model predicts the traffic changes at the next moment based on the existing environmental state and vehicle actions; after reading the generated sensor data, the driving model makes decisions such as steering, acceleration or braking, and then feeds back the actions to the world model.

In order to make the generation speed keep up with the driving decision, Baidu also uses methods such as knowledge distillation to accelerate model inference, enabling the virtual environment to respond in real time to vehicle actions.

One AI is responsible for building the road environment and continuously providing new driving challenges, while the other AI practices repeatedly in it. The relationship between the two is like coach and trainee. In this cycle, autonomous vehicles can complete driving tasks in a continuously changing environment.

If the driving system makes a wrong judgment in a certain scenario, R&D personnel can also reproduce the scenario by retaining the same initial conditions, adjust the behaviors of surrounding vehicles, and observe the performance of the system under different conditions.

With no fixed script, interactivity, error correction and traceability, this virtual city thus has the conditions to become a training ground for autonomous vehicles.

However, continuous operation is only one of the prerequisites. The more critical question is what kind of world model can truly undertake the training and verification of autonomous driving?

Setting a verifiable standard for the world model

Simulation technology has always been an important part of autonomous driving R&D.

Long before generative world models became a hot spot in the industry, engineers were accustomed to recording traffic scenarios on real roads and moving them into virtual environments for repeated testing.

But this R&D method has a long-standing problem.

For a long time, a common type of method in autonomous driving simulation still relies on non-responsive playback of real road data.

To put it simply, even if the autonomous vehicle changes its driving strategy, surrounding vehicles still drive according to the originally recorded trajectory, and the traffic environment will not make corresponding adjustments.

This is exactly where the problem lies.

Driving on real roads is a continuous interaction. When an autonomous vehicle changes its action, other vehicles may also change their decisions accordingly. If the test environment cannot respond, it will be difficult to verify the real performance of the driving system in dynamic traffic.

The generative world model solves exactly this limitation, allowing virtual traffic participants to re-make judgments according to the actions of autonomous vehicles, so that the driving system can be tested in a continuously changing environment.

Dr. Wang Liang, Chief Architect of Baidu Intelligent Driving Group and Chairman of the IDG Technical Committee, summarized it as "interactive generation".

This is Baidu's understanding of the world model starting from the actual needs of autonomous driving.

Looking at the entire AI field, the term "world model" is being used more and more broadly.

Models that can generate realistic videos can be called world models; systems that can construct 3D spaces and allow people to roam freely can also be called world models; technologies that can continuously deduce environmental changes according to external actions are also called world models...

These types of technologies seem to be building virtual worlds, but their actual capabilities are not at the same level.

Li Feifei's team once divided world models into three types: renderer, simulator and planner, focusing on visual generation, environmental state deduction and action planning respectively, which also illustrates from another perspective the wide range of technologies covered by this concept.

As the conceptual boundary continues to expand, the problems it brings are gradually emerging. If the world generated by the model is realistic enough, does it mean that it truly understands the laws of physics? If it can continuously generate traffic images, does it mean that it can deduce different driving results according to the actions of autonomous vehicles?

For autonomous driving R&D, these differences are far from irrelevant technical classifications.

If the model cannot respond accurately to vehicle actions, or the generated traffic changes do not conform to real physical laws, no matter how realistic the images are, it can hardly become a reliable basis for training and verification.

Baidu's emphasis on world model standards stems precisely from this practical demand.

In Baidu's technical ideas, the capabilities that a world model needs depend first on the tasks it ultimately undertakes. Since the goal is to train L4 autonomous vehicles without safety operators, it is necessary to clarify what kind of traffic relationships the model needs to understand, which physical laws it must follow, and whether it can support long-term, real-time interactive driving tasks.

Think clearly about the target tasks, capability boundaries and engineering requirements first, and then build the technical system around these requirements. This forward design idea also makes the world model proposed by Baidu this time have a clear target orientation from the very beginning.

This idea also has its realistic background.

Li Feifei once compared autonomous driving cars to a square box performing 2D obstacle avoidance, to illustrate the difference between cars and robots that can complete complex 3D operations.

Dr. Wang Liang believes that the difficulties of autonomous driving go far beyond the driving action itself.

A Robotaxi without a safety operator is on a fully open road, and needs to continuously face a large number of traffic participants with independent intentions. An seemingly ordinary lane change decision may involve judgment and game between multiple vehicles behind it.

More important than the complexity of actions is the change of safety responsibility.

Dr. Wang Liang made an analogy with the documentary Free Solo.

The climber in the film challenged a nearly vertical rock wall about 900 meters high. To complete the challenge, he went through a long and boring practice process.

In Dr. Wang Liang's view, L2 assisted driving is like rock climbing with a safety rope. Climbers can try beautiful and thrilling movements because protective measures are still in place; the main driver of the assisted driving vehicle is also present, and can take over at critical moments.

However, L4 autonomous driving is like free solo climbing — the "safety rope" is removed, and the main driving seat is empty.

The driving action does not seem to have fundamentally changed, but the risks and safety requirements behind it are completely different. Those repeated trainings out of the spotlight and verification of every detail are ultimately responsible for the reliability of the entire system.

Dr. Wang Liang once participated in L2 mass production R&D and has long been engaged in L4 fully unmanned system work, so he has a personal experience of the gap between the two.

Precisely because of this, in his view, to verify whether a world model can be truly used for L4 training, a more rigorous and clear judgment method is needed.

Baidu has proposed a "2+3" world model evaluation framework for physical AI.

Among them, "2" refers to the two basic conditions of causality and interactivity.

Causality emphasizes that the model can deduce subsequent changes according to known states; interactivity requires that external actions can continuously change the environment, so that different decisions produce different outcomes.

And "3" respectively represents physical accuracy, persistence and real-time performance, which are used to measure whether the model can truly undertake tasks in the physical world.

After all, no matter how realistic the world model is, if there is a deviation in its understanding of real physical laws, the test results may lose reference value.

Rainy days are a typical example.

Wet road surfaces will change the braking