Half of the workload is no longer handled by the GPU.

At the T-Head Computing Power Summit, I attended two sessions on site: one focused on CPU, the other on network interface cards.

On my way back, I kept wondering why the summit reserved dedicated time for CPUs and network cards. After going through the content again later, I realized the two speakers on stage spent a lot of time circling around one core topic: the competition beyond GPUs.

How to understand this? Let me try to explain it clearly.

Starting with the CPU session, the speaker was Huang Wei, Product Director of T-Head Semiconductor. He set the tone right at the start: in the agent era, the role of CPUs has undergone fundamental changes. Why? He said we have to look at whether the workload itself has shifted.

In the past, large model inference followed a simple question-and-answer pattern, and this workload was almost entirely borne by GPUs.

Agents work differently: to complete a task, they run in continuous loops, running one inference step, calling a tool, checking the result, then moving on to the next inference step until the task is finished.

This step of calling tools, whether running code, querying web pages or reading files, is all CPU workload.

There is also the long context that must be kept in memory all the time, which records the full context of the interaction between the model and the user. It is used as input for every round of inference, and is mostly supported by the memory on the CPU side.

He presented three sets of data.

In agent scenarios, tool execution alone accounts for roughly 60% of the total request response time; in coding agent scenarios, container initialization plus tool execution takes up 55% to 60% of CPU resources. Overall, CPUs are already bearing more than half of the total workload.

These figures were released on site, without detailed experimental conditions, but the conclusion is straightforward: CPUs have returned to the critical path of AI computing.

He divided all workloads into two categories, which is the most impressive framework I took away from the entire speech.

The first category works closely with GPUs, which he calls the head node. The GPU handles the heavy computing work, while the head node takes care of all miscellaneous tasks, like a driver with a co-pilot: incoming requests are preprocessed by the co-pilot first.

It manages the KV cache, which is the draft notes the model keeps in real time to record previously calculated results, so that it does not need to recalculate everything from scratch when generating the next token. Where these notes are stored and how they are transferred are all managed by the head node.

When to wake up the accelerator to start working, and how to assemble the final results, are also scheduled and finalized by the head node. This workload sits on the critical path: if the head node lags one step, the GPU that costs a fortune to run per hour will be left idle waiting.

Therefore, the CPU designed for the head node scenario is fully optimized for single-core speed.

Widening the front-end execution pipeline is equivalent to expanding the single-lane access road, allowing multiple instructions to be processed in one clock cycle.

Enlarging the out-of-order window is like an experienced chef who does not wait for one dish to be fully cooked before preparing ingredients for the next: a larger window allows the CPU to look dozens of steps ahead, and execute instructions that can be processed earlier first.

When the KV cache is constantly transferred between the CPU and the accelerator, address translation often fails to find the corresponding entry, requiring page-by-page lookup like flipping through a dictionary. The solution is page table prefetching, which teaches the CPU to load the next page in advance.

There is also a hidden optimization: two unrelated operations may be misjudged by the system as occupying the same address, so they are queued sequentially even though they could run in parallel. Memory renaming eliminates this misjudgment and clears the execution path.

On the software side, even the overhead of Python interpreter instruction dispatch is optimized down to every single operation.

On the cache side, the metadata of the KV cache is placed hierarchically between L3 cache and memory, with frequently accessed data stored closer to the compute unit, to avoid making the accelerator wait for notes that should be readily available.

The second category is decoupled from inference, called agent rack: a cluster of agent servers in the computer room.

Hundreds of sandboxes are deployed in one cabinet, with one agent running in each sandbox, calling tools, executing code, planning steps, arranging task chains, and managing its own status.

None of these workloads touch GPUs, all run on CPUs. This scenario does not prioritize single-core speed, but density: one cabinet should accommodate as many agents as possible, while ensuring that hundreds of instances do not compete for resources with each other.

A completely different set of optimizations applies here.

The agent loop contains a large number of irregular jumps. Traditional CPU branch prediction often makes wrong guesses for agent workloads, so neural branch prediction is introduced to let the CPU build up execution intuition based on accumulated experience.

Workloads like RAG retrieval and graph data traversal are similar to digging for a series of tubers along a vine: the CPU does not need to wait until the data is needed to look for the next entry in the dependency chain. Graph prefetching digs ahead along the chain in advance.

Hundreds of sandboxes share the same cache, which is like hundreds of families sharing one storage room. Dynamic cache replacement replaces content based on the activity level of each sandbox, evicting unused data to free up space.

MPAM sets resource quota for each sandbox, isolating core cycles and bandwidth, so that no single instance can crash the shared resources.

He used a very vivid description for these two types of workloads: the head node scenario cares about memory performance, how fast data can be moved; the agent rack scenario cares about memory capacity, how much data can be stored.

There are two more figures easily overlooked by the audience, which I believe are the fundamental foundation.

The first is 41%. He stated that in one iteration of head node tasks, 41% of the workload cannot be parallelized, especially the scheduling and control plane part.

This figure corresponds to Amdahl's Law: no amount of extra workers can speed up serial workloads, just like you cannot boil a pot of water faster by adding more firewood once it is already boiling.

The lower limit of the entire system's response time is determined by this 41% serial segment. To make the system faster, there is only one way: make the single core itself faster. The obsession of head node scenarios for single-core performance comes from this root cause.

The second figure is 28GB.

Actual tests in the server room show that a single agent instance at peak load requires memory of this magnitude. To run hundreds of instances in one cabinet, extremely high memory capacity is a non-negotiable requirement.

This is why this CPU cannot rely only on core count: memory capacity itself is a core competitive advantage.

These analyses are not done for no reason. The next day at the main Yunqi Summit forum, T-Head officially released the Yitian CPU roadmap for the first time: Yitian 720 and 730 will be launched in 2027, followed by the 750 series, which features the second-generation self-developed core and self-developed inter-chip interconnect that directly connects to the Zhenwu AI chip.

Huang Wei only briefly mentioned this roadmap at the end of his session, and his entire speech was essentially the detailed explanation for this roadmap.

The second session was about network interface cards. The speaker was Fu Binzhang, Head of High Performance Network R&D at Alibaba Cloud Intelligence Group. He started with a cost calculation.

He mentioned that several previous speakers have noted that in the current inference stage, communication performance often determines more than half of the overall system performance. However, network cards account for less than 20% of the total cluster cost, and sometimes even less than 10%.

In other words, the highest return on investment in the entire server room comes from this small board that most people rarely pay extra attention to.

He listed three major pain points.

The first pain point is mixed workload operation. After the cluster is leased out, the tenants' workloads are not predictable: training, inference, and reinforcement learning traffic all run on the same network. Training traffic is like large trucks driving at constant speed on the highway, while inference traffic is like cars that keep changing lanes. Once a truck convoy is interrupted, the speed of the entire fleet drops significantly.

The second pain point is long-distance transmission.

Current inference systems often use PD separation: one inference task is split into two parts. The first part parses the input prompt and prepares the draft, the second part generates the output token by token. The two parts run on separate GPUs to improve card utilization.

The problem is that if the two parts are not deployed in the same cabinet, the data has to be transmitted outside the server room over the wide area network. A round trip inside the server room takes 100 microseconds, but across cities it takes 5 to 10 milliseconds, two orders of magnitude slower.

Essentially, network data transmission is similar to a fleet on the march: you need to send a scout car to report road conditions before the main fleet can speed up. If the scout's report is 100 times slower, the main fleet has to slow down, and the effective bandwidth plummets.

The third pain point is elasticity.

Agent workloads arrive unpredictably: containers are spun up in seconds and destroyed immediately after finishing the task. But high-speed communication methods like RDMA, which allows machines to directly access each other's memory bypassing the CPU, require pre-allocating all resources and addresses along the entire path before they can work, similar to chartering a dedicated freight line where the schedule has to be arranged before departure.

Containers that only run for dozens of seconds cannot afford this waiting time, so many users end up falling back to general-purpose protocols like TCP that are available on demand, accepting the resulting performance loss.

To solve these three pain points, he laid out a full architecture map, which is information-dense and can be summarized in one sentence: build dedicated networks for different scenarios.

Training workloads use HPN (High Performance Network), which supports more access ports with each new generation. The previous generation used 51.2T switch chips to power version 7.0, the mainstream 8.0 version uses 102.4T chips, and the under-development 9.0 version will adopt 200T chips, where T refers to terabits, the unit of chip throughput.

The highlight of this architecture is that only two layers of network can support hundreds of thousands of 800G access ports, which is like an overpass with only two layers that can accommodate tens of thousands of vehicles at the same time.

Inference workloads run on the new TPN architecture (Token-Performance oriented Network), whose core goal is to connect two previously isolated networks into one.

There used to be two separate networks in the server room: the head node network for service traffic in and out, and the rear network for high-speed data transmission between GPUs, with no connectivity between the two.

Small cards without a dedicated rear network that perform PD separation can only use the head node path, leaving the rear network underutilized. TPN opens up connectivity between the two paths, so that different groups of machines in the same cluster (called pods in industry terminology) can transmit data to each other, and the two ends of the separated inference task can communicate without leaving the server room, eliminating the need for long-distance transmission.

For cross-cluster transmission, traffic is routed through the dedicated intelligent computing line on the backbone network, which Alibaba named ECORE. Its bandwidth is more than one order of magnitude higher than the ordinary backbone network, and supports RDMA across the entire path.

There is a separate interconnection architecture inside the super node called UPN (Ultra-Node Network), which uses optical interconnection, so that machines do not need to be physically bound, and one layer can support thousands of GPUs.

Then he introduced the core product, Panmai 920. Its published specifications include 400G bandwidth, which can transmit 400 billion bits of data per second, targeting both intelligent computing and storage scenarios.

His most enthusiastic introduction was about another unique design.

This network card has a built-in switch, which is equivalent to carrying a dedicated power distribution board. Different CPUs have different compatibility characteristics for the PCIe data channel. Without this built-in switch, changing to a different computing platform may cause compatibility issues.

With this design, the same network card can deliver stable performance that allows GPUs to directly read and write data on the network card without passing through the CPU, no matter which platform it is plugged into.

It supports two sets of protocols: the industry standard RoCEv2, plus Alibaba's self-developed SolarRDMA.

This self-developed protocol was first polished in storage scenarios, and later applied to intelligent computing. As he stated, it has gone through all the pain points in both storage and computing scenarios. Its most valuable feature is automatic per-connection path selection: when two new cards communicate, they automatically use the self-developed fast path; when connecting to older cards, it falls back to the standard protocol.

Old clusters do not need to be torn down and rebuilt: the new cards can be replaced one by one without interrupting running services. He noted that this non-disruptive incremental deployment has extremely high engineering value.

He shared four sets of performance test results, all measured on site during the event.

Before presenting the first result, he corrected a widely spread misconception. He said two years ago, the view that pre-training does not need advanced protocols was correct. Large model pre-training is a single large task that uses ring algorithm for data transmission, each network card only communicates with its neighbors, and bandwidth is the only thing that matters.

This view is outdated now. Modern clusters support integrated training and inference, so inference traffic can enter the network at any time and interfere with pre-training workloads, slowing down the entire training process.

Actual tests on thousand-GPU tasks show that splitting large transmissions into small packets and sending them over multiple paths at the same time, with automatic rerouting when congestion or hardware failure occurs, reduces communication time for data parallelism by 43.2%, and improves end-to-end performance by 12%.

The second test result is for cross-cluster long-distance transmission.

The end side marks the traffic, and the wide area switch selects the optimal path according to the mark. Only the lost packets are retransmitted, instead of retransmitting the entire batch of data. The end-to-end throughput is increased by 55%.

The third test result is about elasticity.

The new virtualization technology moves resource allocation to the software control path, delivering 64 times higher connection density than traditional network card virtualization. Container spin-up time is reduced by at least 35%, up to 93.4% in the best case.

He emphasized that this elasticity does not come at the cost of performance: after applying virtualization, the communication speed is almost the same as hardware passthrough. As he said, currently only this network card can deliver both the speed of a dedicated line and the flexibility of on-demand ride-hailing services.

The fourth test result is for small packet transmission. The data payload in high-performance computing is usually very small, even thinner than the packaging header. This network card uses the "Ethernet+" protocol, which reduces preamble overhead and bypasses link layer overhead, so that the valid payload accounts for 85.4% of total transmission, which he noted is the highest in the industry.

The product deployment update is very brief: Panmai 920 has already been launched with the Lingjun Zhenwu M890 super node, which is equipped with this exact network card.

The two sessions seem to cover separate topics, but when you look at them together, they are very insightful.

Huang Wei said that CPUs bear more than half of the workload in agent scenarios. Fu Binzhang said that communication determines more than half of the performance in the inference stage. The two "more than half" refer to different metrics: one measures time consumption, the other measures performance contribution.

But they point to the same conclusion: after a request is received, the part processed by the GPU accounts for a shrinking proportion of the entire processing pipeline, and the majority of the remaining workload falls on CPUs, memory, and networks.

How