10x speedup in half a year, NVIDIA DLSS 5 flexes its muscles, is the "neural rendering" sector about to face a total disruptive shakeup?

When NVIDIA released DLSS 5 earlier this year, few people might have expected that this controversial technology could iterate and improve at such a fast pace.

On October 13, NVIDIA announced a new model upgrade for DLSS 5: the number of model parameters is reduced by 24%, video memory usage is lowered by 20%, while the running speed is doubled again compared with the version released in September. Taking previous optimizations into account, the running speed of the DLSS 5 model has increased by over the past six months:

A full 10 times.

Image source: NVIDIA

You don't even need the two RTX 5090s as required in the initial announcement, a single RTX 50 series graphics card is enough to run DLSS 5. Such an upgrade goes far beyond a conventional performance optimization.

It should be noted that over the past decades, the improvement of game graphics quality has largely relied on the continuously enhanced traditional graphics computing capability of GPUs, from rasterization to ray tracing. But from DLSS 4 to DLSS 5, neural rendering technology is undoubtedly becoming a key variable, and this update introduces the latest 3D-guided neural rendering technology.

In short, "neural rendering" means that neural networks directly participate in graphics rendering, and even use AI to generate more realistic visual details such as lighting and materials, instead of relying entirely on traditional graphics algorithm calculations.

It's not just NVIDIA, not just PCs. Apple has provided the ability to integrate machine learning into the real-time rendering pipeline via Metal 4 as early as last year's WWDC, and has added neural network accelerators to its GPUs starting from the A19 series and M5 chips released last year; Qualcomm and MediaTek have also introduced neural rendering technology for the first time on their new flagship chips.

Image source: Apple

Different vendors have different implementation methods and different development stages, but they are all moving in the direction of making neural networks, or in other words AI, truly go deep into the core of the rendering pipeline.

For computer graphics technology that has been developed for decades, this may mean a more profound change than simply improving GPU performance.

10x speedup in half a year, what exactly has neural rendering changed?

It needs to be clarified first that the focus of the latest DLSS 5 upgrade lies in efficiency.

According to NVIDIA, through the joint optimization of the neural network model and the rendering pipeline, the number of parameters of the new model has been reduced to about 118 million, while the GPU running overhead has been cut in half, basically maintaining the original visual effect. According to test results, the new model can increase the frame rate of *NBA 2K27* by up to 40%.

Image source: NVIDIA

DLSS 5 is also being rolled out to more games at an accelerated pace. Apart from *NBA 2K27* which will be launched on October 16, the new model will also be successively applied to *Hogwarts Legacy* and *Where Winds Meet*, and will be connected to RTX Remix on October 23, giving more classic game remaster projects the chance to use this technology.

A 10-fold speedup achieved in more than half a year is enough to demonstrate how much progress NVIDIA has made in model optimization. Of course, the 10x improvement here refers to the running speed of the neural rendering model, not the overall frame rate of the game.

Looking back at the past few years, from DLSS super resolution to frame generation and ray reconstruction, NVIDIA has actually been trying to make neural networks share the graphics computing load of GPUs. For example, super resolution does not require the GPU to directly render a complete high-resolution image, but renders at a lower resolution first, then reconstructs details through the neural network; ray reconstruction uses AI to process the incomplete, noisy sampling results in ray tracing.

These technologies have changed the calculation method of traditional graphics rendering to a certain extent, but they are mainly focused on reconstructing, enhancing or generating supplementary frames based on existing rendered results. DLSS 5 further applies neural networks to generate the visual presentation of images.

For example, to present a sufficiently realistic human face in a game, traditional graphics rendering requires fine geometric models, material textures, as well as calculations of lighting and skin subsurface scattering, to jointly simulate the visual effect of real skin. Theoretically, investing more modeling and computing resources can get more realistic results, but the cost will also get higher and higher.

DLSS 5 provides another solution. It still requires the game engine to provide basic 3D structure, material and lighting information, but does not need the traditional renderer to complete all visual details alone. Instead, it uses the knowledge learned from real-world images, and the generative neural network to supplement complex effects such as skin, lighting and materials.

Image source: NVIDIA

From this perspective, DLSS 5 tries to break through the limitation of traditional real-time rendering on image realism. Of course, this layer of neural rendering itself also requires additional calculation, and this optimization reduces its own operating overhead.

In a nutshell, what neural rendering really changes is that GPUs no longer need to completely rely on pre-written graphics algorithms to gradually calculate all effects in the image. For some complex visual problems, neural networks can also complete the tasks through learning, prediction and even generation.

This does not mean that all future game images will be generated by AI. The geometric structure of the game world, the position of objects, the actions of characters, and many visual information that need to strictly follow game rules are still inseparable from traditional graphics rendering. The more likely future is that traditional graphics algorithms and neural network calculations will gradually integrate:

The former is responsible for the parts that require precise calculation and clear constraints, while the latter undertakes some tasks that can be solved by approximation and have very high computing costs.

This also explains why neural rendering is not only valued by NVIDIA, Intel and AMD, but also has become a new direction for mobile chip manufacturers.

Mobile phones embrace "neural rendering", Qualcomm and MediaTek are accelerating their layout

Compared with PCs, mobile phones actually have greater demand for neural rendering.

The reason is not hard to understand. Over the past decade, although the performance of mobile GPUs has been continuously improved, limited by power consumption, heat dissipation and memory bandwidth, they have never been able to continuously increase computing resources to obtain better game graphics quality like desktop graphics cards.

In particular, starting from *Genshin Impact*, large-scale mobile games such as *Honkai: Star Rail* and *Wuthering Waves*, as well as more and more PC and console games landing on the mobile platform, players have put forward higher requirements for the graphics quality and frame rate of mobile games. The introduction of technologies such as ray tracing has also further increased the computing burden of GPUs.

The problem is that the power consumption budget that mobile GPUs can allocate is limited after all. Manufacturers can improve energy efficiency through more advanced processes and more efficient architectures, but it is difficult to change the basic limitations of mobile phones' slim body and passive heat dissipation.

Neural rendering provides another possibility.

Since some complex graphics calculations can be handed over to neural networks for approximate solutions, it is not necessary for GPUs to continue to rely on traditional methods to complete all calculations. In particular, super resolution, ray denoising, and more complex material and lighting calculations in the future, all have the opportunity to reduce computing costs through neural networks, and achieve better graphics quality under limited power consumption.

To a certain extent, mobile phones have a more urgent need for this capability of "exchanging computing power with algorithms" than PCs. However, it is more difficult to implement neural rendering well on mobile phones.

Neural network calculation itself also consumes computing power, memory bandwidth and power, and real-time games are very sensitive to latency. Taking 60fps as an example, the total time budget for each frame is only about 16.7 milliseconds. If after introducing the neural network, the additional overhead of model inference and data transmission offsets the cost saved by traditional rendering, the final result may be counterproductive.

What's more troublesome is that although mobile SoCs have long integrated NPUs, the data such as color, depth and motion vectors obtained by GPU rendering need to be sent to the neural network for processing in time. GPUs and NPUs can share system memory, but making the two computing units work together may still involve additional memory access, data format conversion and task synchronization. If these overheads are not handled properly, they may not only increase latency, but also bring additional bandwidth and power consumption pressure.

Therefore, what mobile chip manufacturers need to solve now is not only to improve the AI computing power of GPUs, but also how to make neural network calculations efficiently integrate into the rendering pipeline.

The new generation flagship platforms of Qualcomm and MediaTek this year have just provided ideas with different focuses.

Qualcomm launched the Adreno Neural Fusion neural rendering technology on the 6th Snapdragon 8 Elite Edition, and for the first time added dedicated Matrix Cores matrix computing cores inside the Adreno GPU. The GPU can use dedicated matrix computing hardware to process neural network tasks more efficiently.

6th Snapdragon 8 Elite Edition, Image source: Leikeji

For example, the color, depth and motion vector data required for neural super resolution already exist in the graphics rendering process, and the GPU can directly call Matrix Cores for inference, without specially sending the data to another processing unit.

Qualcomm is also equipped with 18MB Adreno HPM on-chip high-speed cache for this purpose, which retains the intermediate data required for model operation inside the GPU as much as possible, reducing access to external memory.

MediaTek has introduced the Dimensity Neural Rendering Architecture for the first time on Dimensity 9600 Pro. The core lies in the instant synchronous interconnection between GPU and NPU, enabling the two originally relatively independent computing units to work together more efficiently.

Dimensity 9600 Pro, Image source: Leikeji

In short, Qualcomm puts more emphasis on putting neural network calculations directly inside the GPU, while MediaTek focuses on opening up the connection between GPU and NPU, so that traditional graphics calculation and neural network calculation can be combined more closely.

With the new dynamic cache architecture, MediaTek hopes to improve the graphics quality and processing speed of super resolution through neural networks while maintaining the rasterization and ray tracing capabilities.

The ideas of the two manufacturers are not completely separated. According to Qualcomm's SDK description, on devices without Matrix Cores, Adreno Neural Fusion can also complete inference through the collaboration of GPU and NPU. After all, matrix calculation inside the GPU also takes up chip area and power consumption budget. If the independent NPU can participate in rendering through efficient interconnection, it can also give play to its own advantages in AI computing energy efficiency, and there is no absolute distinction between good and bad.

But no matter it is Apple, Qualcomm or MediaTek, the current mobile neural rendering is still in a relatively early stage.

Especially compared with DLSS 5, the capabilities that have been clearly announced and opened on the mobile end are still more concentrated in super resolution, denoising, frame generation, and some programmable neural shading technologies, and there is still a considerable distance from using generative AI to massively enhance the material and lighting performance in games.

But this does not mean that mobile phones need to completely follow the development path of DLSS 5.

In fact, the iteration of DLSS 5 over the past six months has proved that there is still considerable optimization space for the computing cost of neural rendering models. Smaller models, more efficient inference, plus GPU architectures redesigned for neural network calculation, may gradually make some technologies that could only be implemented on high-end PCs in the past meet the conditions to run on mobile phones.

More importantly, a series of actions from Apple to Qualcomm and MediaTek mean that mobile chips have begun to prepare for more complex neural rendering from both the hardware and development tool levels.

Closing remarks

If the competition of mobile GPUs in the past was mainly focused on traditional graphics performance and ray tracing capabilities, then in the next step, how low the cost of neural networks participating in rendering can be, and how much expensive traditional calculation can be replaced, will likely become the new focus of competition.

For mobile games and players, this may also be an anticipated change: in the future, the improvement of game graphics quality will no longer have to wait for the rhythm of the improvement of traditional graphics computing power.