GPUs are underutilized while CPUs are stretched thin, can DPUs stage a "coup" to take over their dominant position?
In March 2019, NVIDIA outbid giants including Intel, Microsoft, and Xilinx to acquire server and networking chip vendor Mellanox for $6.9 billion.
A year and a half later, NVIDIA officially named Mellanox's BlueField "smart network card" DPU, ranking it alongside CPU and GPU as the "third core chip" in data centers: the CPU is responsible for general-purpose computing, the GPU for accelerated computing, and the DPU for data movement and processing.
According to the public timeline released by Jensen Huang: within five years, every data center around the world will be deployed with DPUs.
With this grand narrative, chip companies of all kinds quickly followed suit: Intel launched its IPU product line in 2021, AMD acquired DPU vendor Pensando in 2022, and domestic DPU startups including ChinaSys, Nebula Micro, and Leopard Smart have all received large-scale financing.
In terms of results, Jensen Huang's prophecy has only been partially fulfilled.
DPUs have indeed become popular in hyperscale data centers, but the chips that have achieved large-scale deployment are mainly self-developed chips from cloud vendors: AWS launched the Nitro card on a large scale as early as 2017, Google customized the IPU E2000 in partnership with Intel, Microsoft launched the Azure Boost DPU after acquiring Fungible... The commercial DPUs from NVIDIA and AMD have long been widely praised but rarely adopted at scale.
That was until the wave of agents in 2026.
According to AMD's forecast, about 60% of global AI computing power will be used for model inference in 2026. With the popularization of long-context inference and agents, how to quickly feed massive KV Cache and real-time memory to GPUs is directly related to the actual output of a data center.
When data movement and processing become the key link restricting Token production capacity, DPU has finally got the opportunity to seize the dominant position from traditional processors.
01. The "replace computing with storage" anxiety: Why has the CPU become a bottleneck?
The DPU being pushed to the forefront again is inseparable from the "memory wall" and "economic accounting" of large model inference.
In the process of autoregressive inference, large models need to save the calculated KV Cache to avoid recalculating the entire context every time a new Token is generated. The problem is that the longer the context window of the model, the more KV Cache needs to be saved.
From 2023 to 2025, the context window of mainstream large models jumped from 128K to the million-Token level. Taking some large-parameter models as an example, the KV Cache of a single million-Token sequence may occupy 10GB to 100GB of storage space.
Calculated based on the 1.1 million Token throughput per second demonstrated by Microsoft on a 72-GPU rack, under a specific model configuration, the daily KV Cache data generation volume corresponding to a single GPU can reach 92TB. For a simple conversion, 80GB of HBM VRAM is only enough to store data for about 1.2 minutes, and even 1TB of DRAM is only enough for about 16 minutes.
Agents further exacerbate the "storage pressure".
When an agent executes a task, it usually involves cycles such as model thinking, tool calling, waiting for results, rethinking, and re-calling. Each step switches back and forth between GPU generation and CPU/I/O tool execution, and each round needs to carry the system prompt, tool list, and returned results.
According to the calculation of Micron Technology, the demand for memory capacity in agent scenarios may be 10 to 40 times that of ordinary inference.
Since neither VRAM nor memory can hold all the data, the industry has begun to shift: offload temporarily inactive KV Cache to NVMe SSD, and pull it back to VRAM when needed.
According to the medium cost counted by Omdia in 2025, HBM3E is about $14 per GB, and DDR5 is about $8.50 per GB; calculated by the cost per byte after deployment, DRAM is about 44 times that of NVMe SSD.
The vLLM team conducted a test. In an environment with 2×H100 and PD separation, when the number of concurrent sessions exceeded 128 and the CPU memory was filled, offloading to the storage layer could still maintain a high hit rate, doubling the Prefill throughput.
The problem lies precisely in "pulling data from SSD back to GPU".
In the traditional server architecture, the CPU has long acted as the hub of data flow: when the GPU wants to read data from external storage, it usually has to go through the host operating system, file system, storage protocol stack, and memory management mechanism to finally complete data movement.
Different implementation paths vary, but under the conventional I/O mode, metadata query, protocol processing, encryption and decryption, and multiple memory copies may consume a large amount of CPU resources. For traditional enterprise applications, this overhead is still acceptable.
When faced with inference demands that race against time, it becomes a bottleneck: As inference concurrency continues to increase, massive KV Cache read and write requests will continuously impact the CPU and host I/O system. The CPU, which was originally responsible for coordinating data flow, may instead become a bottleneck that slows down the GPU.
As early as during the GTC in October 2020, NVIDIA demonstrated the feasibility of GPU Direct accelerating storage: by reducing the data movement path involving the CPU, it achieved a bandwidth increase of up to 2.8 times in specific tests, and reduced CPU-related overhead by 3 to 5 times.
When "how many valid Tokens can be produced per megawatt of power" becomes the core operating indicator of AI factories, how to make data bypass the redundant processing links involving the CPU has become a problem that architects must face, and it also gives DPUs the opportunity to return to the core stage.
02. Giants have entered the market one after another, and DPUs are accelerating to seize the dominant position
Among the players trying to restructure the power boundary of the CPU, NVIDIA can be said to be the most radical one.
In October 2025, NVIDIA released the BlueField-4 series, further blurring the line between traditional smart network cards and independent computing platforms.
The standard BlueField-4 DPU has about 126 billion transistors, integrates a 64-core Grace CPU, up to 128GB of LPDDR5X memory, PCIe 6.0 x16 bus, and up to 800Gb/s network interface... It is already difficult to define it with the traditional concept of "smart network card".
On the storage node side, NVIDIA has also prepared solutions in different forms.
For example, the STX version demonstrated at OCP 2026 adopts the Vera CPU architecture; the dual BlueField-4 Vera solution demonstrated at Hot Chips 2026 pushes the storage access bandwidth to 3.2Tb/s, claiming to achieve a 10-fold increase in IOPS and a 5-fold increase in energy efficiency.
Behind this lies NVIDIA's greater ambition.
In the past, DPUs were mainly responsible for offloading network, security, virtualization, and part of storage tasks, freeing CPUs from repetitive work; now, NVIDIA hopes that DPUs will further take over the most critical data flow links in the large model inference process.
DPU is evolving from an auxiliary processor of the CPU to a new hub between the GPU and storage.
To achieve its goal, NVIDIA has prepared two sets of mutually coordinated solutions.
The first set: Redefine the storage hierarchy.
The traditional data center storage architecture can be roughly divided into four levels: G1 is the HBM VRAM inside the GPU, which has the fastest speed and the highest cost; G2 is the host DRAM; G3 is the local SSD; G4 is the shared storage with larger capacity but higher access latency.
This system has operated for many years in traditional computing scenarios, but it has gradually exposed its limitations in the era of large-scale agent inference.
Although local SSDs are fast, they are difficult to flexibly share between different computing nodes; although remote shared storage has sufficient capacity, access latency and network overhead may affect the response efficiency of GPUs; especially when multiple agents share the same set of knowledge, tools, and historical context, if each node needs to save and reload KV Cache separately, a large amount of redundant data movement will be generated.
NVIDIA's answer is to insert a new G3.5 layer between the traditional storage hierarchies, that is, the Context Memory Layer (CMX).
According to NVIDIA's design, CMX uses BlueField-4 to organize rack-level flash resources into a shared storage pool, which is dedicated to undertaking KV Cache generated by large-scale inference, so that GPUs no longer need to care about which SSD or which storage node a certain segment of context is stored in.
The second set: DOCA software ecosystem.
People familiar with NVIDIA's history can easily associate it with CUDA. NVIDIA built a software system covering development tools, programming frameworks, and application ecosystems through CUDA, gradually transforming its hardware performance advantages into platform competitiveness.
A similar idea is being repeated in the DPU field.
In the BlueField-4 architecture, software components such as DOCA Memos are responsible for connecting the inference framework and the underlying storage system: the upper-layer inference engine determines when which context data is needed, and the underlying DPU handles tasks such as NVMe-oF, RDMA, encryption and decryption, verification, and cross-node data flow through hardware acceleration and software offloading.
CUDA has helped NVIDIA build a GPU computing ecosystem, and DOCA carries NVIDIA's ambition to extend platform capabilities to the data channel. Once more and more inference frameworks and storage vendors carry out deep adaptation around DOCA, NVIDIA will have the opportunity to extend its competitive advantages from the GPU computing link to the entire AI data channel.
Obviously, NVIDIA is not the only one who has the same idea.
In the Helios rack-level solution launched in 2026, AMD combines EPYC Venice CPU, Pensando Salina DPU, and Vulcano 800G network card, trying to build AI infrastructure based on open Ethernet.
It also uses DPUs and high-speed networks to expand the storage resources accessible to GPUs, so that large-scale KV Cache can get rid of the limitation of local VRAM capacity.
Huawei offers another idea.
At the 2026 Connect Conference, Huawei released the OceanStor M900, further integrating computing, network, and storage control capabilities into the system architecture, and shortening the data access path between Ascend NPU and SSD through technologies such as UnifiedBus, reducing redundant overhead in the traditional host I/O link.
The technical routes of the three giants are not exactly the same, but they have reached a consensus: in the competition of next-generation AI infrastructure, data flow efficiency will become as important a variable as computing performance. A large number of high-frequency tasks in networking, storage, and even context scheduling are gradually getting rid of CPUs and being handed over to DPUs or other dedicated hardware for processing.
Following this logic, the "seizure of dominant position" by DPUs is essentially the transfer of control over the data channel that has long been mastered by CPUs to dedicated processors.
03. Alternative routes and TCO scrutiny: Is it mandatory to buy DPUs?
From the perspective of cloud vendors and large enterprises, the answer is probably not that simple. At least three variables are challenging the commercial value of DPUs.
The first variable is the "slimming down" on the model side.
The storage pressure brought by KV Cache depends on the model architecture, context length, concurrency scale, and the organization of cached data. If the generation of KV Cache can be reduced through model architecture innovation, there is naturally no need to invest in the same scale of external storage resources.
The second variable is CXL memory pooling.
Different from offloading KV Cache to SSD, CXL provides another idea: let multiple computing nodes flexibly access and share memory resources. A study by the team of Peking University shows that in the test of DeepSeek-V3.2, the throughput of CXL memory pooling reaches 2.1 times that of the RDMA baseline.
The third variable is the most realistic TCO ledger.
To build a complete rack-level KV Cache storage system, in addition to SSDs, you may also need to purchase DPUs, switches, high-speed optical modules, and bear additional costs for rack space, power consumption, software licensing, and operation and maintenance. At present, there is still a lack of sufficiently transparent, cross-platform, and same-caliber TCO ledgers.
What is more difficult to answer than technical variables is the commercial problem.
The first wave of DPU commercialization has proved one thing: advanced technology does not mean that cloud vendors are necessarily willing to purchase.
If cloud vendors integrate KV Cache management capabilities into the next generation of self-developed network cards or dedicated processors, DPU as a technical architecture may accelerate its popularization, but the market space left for independent commercial DPUs may not necessarily expand simultaneously.
Different from the last wave of commercialization, NVIDIA has already received support from many storage vendors.
From the end of 2025 to the beginning of 2026, many AI-native storage vendors have successively announced their embrace of BlueField-4: WEKA announced that it will run the next-generation NeuralMesh storage software on BlueField-4; DDN plans to combine Infinia with BlueField-4 to build the next-generation platform; VAST Data also introduced VAST AI OS into the BlueField-4 ecosystem to enter the CMX-related market...
Many vendors are also trying to find breakthroughs from underlying protocols and storage media.
For example, Huari Index Cloud launched the native distributed KV storage WQS, trying to bypass part of the redundant I/O paths and kernel overhead in the traditional file system, and directly manage NVMe bare disks; aiming at the QLC flash write amplification and tail latency problems that may be caused by high-concurrency KV Cache writing, Silicon Motion launched the MonTitan reference design for KV Cache offloading.
When context memory becomes a new infrastructure demand, it is far from settled who will take the lead, whether it is DPU vendors, storage vendors, controller vendors, or cloud vendors themselves.
NVIDIA's biggest opportunity window may still lie in agents.
Just as Jensen Huang's role division for the Vera Rubin platform: Rubin is responsible for the thinking of agents, Vera is responsible for execution, and BlueField undertakes governance tasks.
As enterprises begin to deploy a large number of autonomous agents, the infrastructure is facing new security challenges.
An agent may access the enterprise database at the same time, call external tools, and share context with other agents. The permission boundary, data isolation, and behavior audit between different tenants all require more fine-grained management.
Through BlueField Astra and DOCA-related security components, NVIDIA is trying to sink capabilities such as identity permission, data access control, and behavior audit to the infrastructure layer, and use DPUs to establish an independent security control point. AI cloud vendors that lack self-development capabilities, and data centers that are eager to make up for security shortcomings, may directly purchase the entire set of infrastructure.
04. Final notes
Looking back at the seven-year development history of DPU, it is quite dramatic.
When NVIDIA first acquired Mellanox, DPU carried the imagination of data center network and infrastructure offloading. It was not until AI factories began to calculate how many valid Tokens each watt of power and each dollar of investment could create, that data movement and memory scheduling finally came to the forefront from behind the scenes, giving DPUs the opportunity to prove themselves again.
Unfortunately, Jensen Huang's departure has added new variables to DPUs.
From the acquisition of Mellanox to the promotion of the Vera Rubin full-rack architecture, Jensen Huang has always been the key promoter of NVIDIA's extension to data center infrastructure. Whether the new management can continue this long-term strategy, especially how to balance the commercial return of BlueField with the software ecosystem expansion of DOCA and Dynamo, will affect the future of DPUs.