After a decade, Ren Shaoqing has published a new paper.

After a decade, Shaoqing Ren, the AI luminary and author of ResNet, has released a new original research paper.

This latest paper enables autonomous driving to "think ten steps ahead while taking one step", making the system behave more like an experienced human driver!

Recently, Shaoqing Ren and his collaborators published the paper 《MM-Future: Multi-Mode Joint World–Action Modeling for Autonomous Driving》, proposing a novel multi-mode joint world-action model.

In public academic records, his representative research was mainly concentrated between 2014 and 2016, covering a series of classic computer vision works including Faster R-CNN and ResNet.

This time, he reappears as the corresponding author on an autonomous driving paper. The primary institution of the paper is NIO, multiple authors are from NIO, and the research direction is directly targeted at world models and planning.

The core problem this paper aims to solve is very clear: in the same driving scenario, the system needs to process not only a single trajectory, nor just one determined future.

The team's idea is to let the model generate multiple sets of scene-action hypotheses at once, so that the vehicle trajectory and the corresponding future scenarios evolve together, before making the final selection from them.

In simple terms, autonomous driving can not only clearly perceive the surrounding environment, but also simulate multiple possible futures, and finally select the safest path to execute.

Expanding from One Single Trajectory to Multiple Futures

The basic logic of end-to-end autonomous driving is to directly map sensor observations to the ego vehicle's trajectory or control commands.

The World-Action Model further adds future scenario modeling, so that planning is not only based on the current environment, but also refers to the possible consequences of actions.

Existing World-Action Models generally fall into two categories of technical routes:

One category is the cascade scheme. For example, it first generates several candidate trajectories, then separately predicts the future scenario corresponding to each trajectory; or predicts the future scenario first, and then completes planning based on the scenario. This method can generate multiple candidate results.

However, the problem of this method is also very obvious: information is transmitted in one-way, and variables generated later cannot go back to correct previous decisions.

If the model has already determined that the ego vehicle will move forward through the intersection, and then predicts that the oncoming vehicle is not decelerating, this future result can hardly re-participate in the previous action generation process.

The other category is the joint scheme. Scenarios and actions are placed in the same generation process, and the two can interact with each other.

When the vehicle's action changes, the predicted environment adjusts accordingly; once the environment changes, it will in turn affect the vehicle's trajectory. Existing methods of this type usually only generate one set of scene-action results.

The paper points out the existing gap: cascade models can cover multiple driving results, but there is a lack of two-way interaction between scenarios and actions; joint models have two-way interaction capabilities, but usually only generate a single set of results.

However, real-world roads often require handling both of these two things at the same time.

The solution proposed by MM-Future is: generate multiple sets of scene-action hypotheses at one time, and the scenarios and actions within each set continue to evolve together.

Each set of results is called a paired scene-action hypothesis.

Assuming the model generates dozens of candidates at the same time, one set may be the ego vehicle accelerating while the oncoming vehicle gives way, another set may be the ego vehicle decelerating and the oncoming vehicle going first, and there are other different degrees of braking, passing methods and scenario changes.

These results no longer only have trajectory differences, and the future environment corresponding to each trajectory is also different. As a result, multiple possible futures are introduced into the planning process.

Deducing Dozens of Futures Simultaneously: Three Thresholds to Overcome

The paper splits the difficulties of multi-mode joint modeling into three items:

* Where does diversity come from, how to control computational costs for multiple futures, and how to screen after generating multiple candidates. The core design of MM-Future is also developed around these three problems.

The first is the diversity problem.

Real driving data has natural limitations: a single recording only records one result that has already happened. If the driver finally decelerates, the real scenario after choosing to accelerate at the same moment will not be left in the data. The model needs to learn multiple reasonable results under the supervision of a single result.

MM-Future establishes Gaussian Mixture Noise at the action end according to the distribution of training trajectories, starting from different action priors; the scenario end uses independent noise.

The initial states of actions and scenarios are paired with each other to form independent starting points for different driving results.

Best-of-Many supervision is added during training. The model generates multiple sets of candidates at once, first finds the set that is closest to the real trajectory, and then uses the same winner index to supervise both the action stream and the scenario stream.

This can avoid all candidates being pulled to similar results by the same ground truth trajectory, and also ensure that one trajectory is always paired with its corresponding future scenario.

The second is computational cost. If multi-view videos are fully expanded for each candidate, the amount of calculation will rise rapidly once the number of modes and prediction time increase.

The authors propose MM-Tokens, which compress the visual features of multiple cameras and multiple time frames into a compact representation oriented to planning.

MM-Tokens do not undertake RGB image reconstruction, nor do they need to restore the complete BEV. The limited token budget mainly retains information related to future evolution and driving planning.

From the attention visualization provided in the paper, the information that MM-Tokens focus on is mainly concentrated in areas such as road structures, intersections, preceding vehicles and surrounding traffic participants.

Thus the model does not need to generate complete high-definition future videos separately for dozens of candidates, and what is retained internally is a set of compact future scenario representations oriented to planning.

The third item is how scenarios and actions evolve together, and how to screen multiple sets of candidates. MM-Future uses a modality-aware Transformer to process both the action stream and the scenario stream at the same time.

Shared attention is responsible for the interaction of the two types of information, and modality-specific branches retain the respective statistical characteristics of the two types of data.

Each set of candidates shares model parameters, but different modes do not exchange tokens.

Therefore, the action trajectory will continue to update as the paired scenario changes, and the scenario prediction will also be readjusted according to the ego vehicle's actions. Two-way coupling between actions and the environment is thus introduced into multi-trajectory planning.

After generation is completed, the Future-Conditioned Proposal Scorer is responsible for the final screening.

When evaluating a candidate trajectory, it reads both historical information and the predicted future exclusive to this trajectory, and then gives a score. Each candidate can only read its own paired future, and cannot borrow scenario results from other modes.

The paper also applies stop-gradient to trajectories and future tokens to prevent the generator from changing the output distribution in order to improve the score.

The final inference process can be compressed into four steps: encode historical observations, generate multiple sets of scene-action results, score candidates using history and paired futures, and then output the trajectory with the highest score.

World Models Begin to Enter the Planning Field at a Deeper Level

How much improvement has MM-Future finally brought?

The paper uses two benchmarks, NAVSIM and HUGSIM, for testing:

On NAVSIM-v1 navtest, MM-Future trained with trainval data achieves 94.0 PDMS. In the same result table, the WAM scheme DriveFuture gets 90.7, and the E2E scheme DrivoR trainval gets 93.7.

On NAVSIM-v2, MM-Future achieves 91.5 EPDMS, which is higher than the 90.1 of UniTeD and 89.9 of DriveFuture listed in the paper.

What can better illustrate the effectiveness of the method is the ablation experiment.

When only actions are generated and only one mode is retained, the PDMS is 84.1; after the number of action candidates is increased to 32, it rises to 92.3.

After adding joint scene-action generation, the single-mode result is 85.1, and the 32-mode result reaches 92.9. Finally, letting the scorer read the predicted future corresponding to each trajectory, the PDMS continues to rise to 93.3.

Several groups of controls verify the three links respectively: increasing the number of modes brings the main gain; joint generation of scenarios and actions can further improve performance; after the paired future participates in candidate scoring, the planning indicator is improved again.

The improvement brought by the scorer is most obvious on the TTC indicator. The paper therefore believes that the candidate-exclusive future mainly helps the model exclude trajectories with high interaction risks.

Multi-mode training also shows faster convergence speed. The 16-mode and 32-mode models take about 3.8k steps to reach the 0.80 validation PDM score, while the single-mode model takes 17.5k steps.

This result shows that multiple hypotheses not only increase inference branches, but also improve training efficiency and final indicators under the setting of the paper.

However, multiple sets of future hypotheses also bring additional computational costs.

The main model of the paper samples 64 sets of scene-action candidates at a time. Under the condition of a single NVIDIA H800, batch size 1 and bf16, the end-to-end forward latency is about 233ms.

The latency of 16 candidates is relatively close to that of the single-mode scheme, and the latency increases significantly after increasing to 32 candidates. A balance still needs to be struck between candidate coverage and computational cost.

Closed-loop testing also has similar limitations.

MM-Future was not further fine-tuned for HUGSIM, and achieved an average HD-Score of 32.3 on 436 scenarios, which is higher than the 28.9 of Latent-WAM and 28.6 of UniAD listed in the paper.

The average Route Completion is 44.5, slightly lower than the 45.9 of Latent-WAM; the HD-Score under Extreme difficulty is 8.6, while Latent-WAM reaches 18.1.

The paper also actively points out the interpretability problem. MM-Tokens belong to implicit scenario representations, and it is still difficult to directly observe what planning information the model has retained.

The authors plan to add an auxiliary perception module to further visualize the scene structure related to planning, so as to improve model transparency and assist fault diagnosis.

Therefore, the value currently demonstrated by MM-Future is more suitable to be understood at the methodological level:

The role of the autonomous driving world model is further moving from future prediction to planning. The model not only needs to estimate how the scenario will change, but also compare different scenario results corresponding to different actions.

The more critical change lies in the definition of multi-mode. In the past, multi-mode planning mainly answered which trajectories are feasible, and MM-Future extends the problem to how the environment may change respectively after different trajectories are taken. The world model begins to undertake part of the function of modeling action consequences.

This technical route is still in the stage of public dataset and simulation verification, and computational overhead, implicit representation, and extreme scenario stability all need to be further verified.

The conclusion that the paper can currently support is clearer: in autonomous driving planning, covering multiple possible results at the same time and maintaining two-way interaction between scenarios and actions can bring stable indicator gains.

There are always multiple reasonable results on real roads. As the world model continues to enter the planning link, the objects that autonomous driving needs to process are also expanding from one candidate trajectory to a set of joint results of actions and futures.

Team Introduction

The research team behind MM-Future mainly comes from NIO.

The first author Shuai Liu is affiliated with both NIO and the School of Computer Science of Sun Yat-sen University. He has accumulated research experience in the direction of end-to-end autonomous driving before, and published GaussianFusion as the first author in 2025, which attempts to complete multi-sensor fusion with a compact Gaussian representation and was accepted by NeurIPS 2025.

Among the other authors of the paper, Hechangle Gong is affiliated with both NIO and Beihang University, Hao Jiang, Runlin He, Junxiang Zhan, and Sheng Yang are all affiliated with NIO, and Kai Huang comes from Sun Yat-sen University.

Shaoqing Ren serves