From TOPS to real-world workloads, the evaluation criteria for AI chips are changing.
When Agent moves from demo prototypes to production-grade deployment, the industry is facing a set of sharp practical contradictions: the iteration cycle of upper-layer models and agent algorithms has been shortened to half a year, while an AI chip often takes several years from requirement definition, architecture design to tape-out and mass production. In the past, the industry was accustomed to chasing the nominal peak computing power of a single chip, but feedback from front-line businesses constantly reminds the industry that the characteristics of agents, including long-cycle operation, multi-round interaction and frequent tool calls, will continuously expose various hidden bottlenecks in storage, scheduling and heterogeneous collaboration. The evaluation criteria for computing power are changing imperceptibly, and end-to-end execution efficiency as well as the stability of long-running systems have gradually become hard constraints for commercial implementation.
Agents are reshaping the characteristics of AI workloads, and industrial thinking is shifting accordingly. On September 21, the AICC2026 Artificial Intelligence Computing Conference concluded at Zhongguancun International Innovation Center in Beijing. Being at the venue, people can clearly feel that the industry is re-exploring what kind of hardware and system architecture can meet the new business demands brought by agents.
Workload paradigm is completely shifted, Agent rewrites hardware evaluation logic
The change brought by Agent is not a simple surge in computing power demand, but a fundamental shift of the entire workload paradigm.
Zhou Zhengang, Vice President of IDC, brought the *2026 China Artificial Intelligence Computing Power Development Assessment Report* at the conference. He broke down the explosion of computing power demand into three multipliers: task execution frequency, single-task Token consumption, and multi-agent synergy effect. The superposition of multiple factors jointly pushes up the computing power consumption of the whole society. Different from the one-off question-and-answer reasoning of traditional large models, agents in production environment belong to the long-cycle task form, which involves massive knowledge base reading, multi-round reasoning generation, external tool invocation, session state retention, error rollback and retry, and needs to support concurrent operation of large-scale instances at the same time.
The change of workload form also directly reshapes the division of labor at the hardware and system collaboration level. Liu Jun, Chief AI Strategy Officer of Inspur Information, mentioned in the exchange that after entering the Agent stage, AI no longer only completes one model call, but requires continuous task planning, tool call, state management, result verification and multi-round iteration. As a result, the computing workload becomes longer, more complex and more dynamic, which puts forward new requirements for the collaboration of CPU, accelerator, memory, storage, interconnection and system software. In his judgment, intelligence is no longer a scarce capability unique to human beings. AI computing presents two important development directions: one direction goes upward to continuously approach the upper limit of intelligence; the other direction expands widely to realize sufficient supply. This change at the industrial level will be transmitted to all aspects of system architecture design from top to bottom.
The implementation practices of many enterprises also raise practical problems worthy of vigilance. Jiang Peihang, Senior Vice President of the Software and Systems Research Institute of the Fifth Electronic Research Institute of MIIT, observed that many chips perform well in standardized single-point benchmark tests, but when put into the real production environment of Agent to face complex tasks with multi-steps and high concurrency, they are very prone to task interruption and output deviation. The nominal performance index of hardware is not equivalent to the available capability in real business scenarios. The stability of the full link has become an unavoidable hard threshold for large-scale Agent implementation, which has been repeatedly confirmed in the front-line implementation practices of many enterprises.
In the agent scenario, long context brings huge pressure, and the continuous expansion of KV Cache scale will lead to a series of cascading problems such as memory overflow, first-token latency deterioration, and Token interval jitter. Tasks are no longer fixed computing processes, filled with a large number of unpredictable branches and retry logic. Memory capacity, bandwidth, inter-chip interconnection, resource isolation, and global task scheduling may all become system bottlenecks. This also leads to the core proposition that the industry needs to answer: in the face of dynamic and changeable new workloads, what problems can hardware solve, and what demands must be fulfilled by systems and software.
Facing the storage wall directly, multiple technical routes run in parallel with their own capability boundaries
Storage and memory access efficiency is one of the most frequently discussed topics at the conference.
In the past decade, AI computing power has increased hundreds of times, but the improvement of memory bandwidth is far behind the pace of computing power expansion. Agent services need to continuously generate, update and store a large amount of KV Cache and task states, which further amplifies the memory access pressure. This means that the collaboration between the storage system and the computing system becomes more important. For the domestic market, under the background of increasingly diversified chip architectures, storage technologies and supply chain conditions, the industry is also exploring breakthrough paths in parallel in multiple technical directions.
For the promising technical route of processing-in-memory, Wu Huaqiang, Vice President of Tsinghua University, gave a sober judgment combined with the team's many years of R&D practice: We cannot pin all our hopes on the single-point breakthrough of devices. The full-stack collaboration covering devices, processes, circuits, architectures, compilers and inference frameworks is indispensable. Building crossbar arrays with non-volatile memory devices to bring computation close to memory units, processing-in-memory can deliver outstanding energy efficiency benefits in scenarios with high memory access proportion such as RAG retrieval and vector database. However, analog computing itself comes with inherent practical problems such as device fluctuation and array mapping, which also defines its capability boundary: it is more suitable for specific memory-intensive scenarios, and it is difficult to cover all scenarios for the complete agent business link with a large number of branch jumps and complex general computing.
Restricted by the physical upper limit of area, power consumption and process of a single chip, Chiplet and 3D hybrid stacking have become another important solution. Ma Kaisheng, Founder & Chief Scientist of CoreXi, predicted combined with his own product R&D experience that in the next one to two years, the industry will welcome Agent systems with large parameter models, high Token throughput, thousands of containers collaborating, and thousands of hours of uninterrupted operation. A single chip can hardly cover all business demands. The value of Chiplet and 3D stacking is not only to increase the total computing power by splicing dies, but the core significance lies in the architectural flexibility brought by splitting and combination, so as to alleviate the practical contradiction between fast model iteration and long hardware R&D cycle.
However, advanced packaging is not a universal solution. Lou Jianguang, Chief AI Scientist of Suanmiao Technology, admitted in the sharing that 3D stacking can compress the interconnection distance to the micron level, greatly improve bandwidth and reduce data migration power consumption, but yield control, thermal design and supporting software ecology are all practical obstacles that cannot be bypassed on the way to implementation. We cannot simply copy and reuse the existing GPU/NPU architecture, and need to carry out memory-centric native architecture design for Transformer.
Even with advanced architectures such as 3D stacking and Chiplet, the bottleneck at the memory level still cannot be completely eliminated. Zhang Jie, Director of DDR Product Line of InnoGrit Technology, disassembled the practical risks from the perspective of front-line implementation of the IP industry: Agent tasks often last for more than ten minutes or even longer. Once a critical data error occurs in the memory, the entire completed task will be completely invalidated, resulting in real business losses. It is for this reason that for Agent services, the status of memory operation stability has been elevated to the same level as performance. LPDDR6 adds new protection mechanisms such as Meta space, and will build in PIM processing-in-memory capability in the future, which can complete part of simple calculations inside the memory particles to reduce the pressure on the main chip, but it can only alleviate part of the data migration pressure and cannot completely eradicate the storage wall. HBM, 3D-DRAM and the new generation of DDR have their own adaptation ranges, and trade-offs between bandwidth, capacity, reliability and cost are always required at the practical level.
Therefore, hardware can overcome bottlenecks such as device limits, computing power scale, and physical-level data migration. However, dynamic task scheduling, instance state maintenance, fault retry recovery, and heterogeneous multi-chip collaboration in Agent tasks have gone beyond the scope of pure hardware solutions, and ultimately need to be implemented by the system software stack. No matter how advanced the hardware architecture is, if the adaptation of upper-layer software cannot keep up, even the most eye-catching hardware parameters can hardly be converted into real and usable business capabilities.
New exam questions for RISC-V, hardware capabilities are ultimately realized through a complete software ecology
The rise of Agent mixed workload is expanding the application boundary of RISC-V, making it move from traditional MCU and low-power embedded scenarios to the server and AI computing fields. Whether it is forum sharing or enterprise exclusive interview, many practitioners have mentioned the same reality: Polishing good hardware IP is only the starting point, and the maturity of stability, compatibility and the complete basic software base is often the biggest stumbling block to large-scale implementation.
Zhu Weidong, Product Lead of D-RISC Co., Ltd., expressed based on the front-line experience accumulated in the RISC-V chip R&D process that in the past, the industry's impression of RISC-V mostly stayed in the end-side and MCU fields, but the business demands of long context, large-scale KV Cache and 7×24 hours uninterrupted operation in the Agent era have brought server-level reliability requirements to the forefront. The success of a server chip does not only depend on the score of the CPU core itself. The completeness of the entire base including RDMA network, virtualization, operating system adaptation and AI inference framework is crucial. In many scenarios, the workload of software ecology construction even exceeds the hardware IP R&D itself.
In his view, the pressure brought by long context is not only at the computing power level, but the greater pressure falls on the memory subsystem. The expansion of KV Cache scale will lead to cascading problems such as memory overflow, first-token latency deterioration, and jitter of intervals between Tokens. The industry generally adopts the software-hardware coordinated hierarchical storage solution to cope with this: hot caches with high-frequency access are placed in high-speed memory, and cold historical context is sunk to low-speed storage media, combined with software algorithms such as Page Attention to reduce the pressure on the hardware side. In the product selection stage, the priority of stability and RAS reliability in server scenarios is even higher than the original performance indicators.
For AI mixed workloads, RISC-V needs to introduce AI vector and matrix instruction extensions, which naturally comes with the risk of ecological fragmentation. The guests at the conference sorted out the three AI instruction extension routes of IME, VME and AME in their reports, and different schemes have trade-offs in register organization and computing unit design. If a large number of manufacturers adopt private customized extensions, the upper-layer inference framework will have to undertake heavy adaptation work, raising the cost of the whole industry. Promoting the formation of publicly recognized public instruction standards by the community is a feasible path to reduce the industrial burden.
The instruction set standard is only the starting point. Xie Tao, Distinguished Professor of Peking University, Chief Scientist of Beijing Open Source Chip Research Institute, Director of Beijing Transminghu Information Technology Application Innovation Center, and Dean of the Institute of Advanced Computing Systems of Fudan University, pointed out that The openness of RISC-V should not stop at the instruction set itself. We should raise our perspective from hardware IP to the complete system ecology.
Xie Tao analyzed that in the AIGC era, the focus of computing power is highly biased to accelerator chips. A large number of orchestration and tool execution tasks in Agent scenarios increase the CPU load, and the ratio of CPU and acceleration units tends to be more balanced. In the traditional off-chip heterogeneous architecture, the CPU is responsible for scheduling and the XPU completes AI computing, which will generate a large amount of data copy overhead. RISC-V turns AI computing capability into the native capability of the CPU instruction set through matrix and tensor instruction extensions, moving towards isomorphic computing, which can reduce data migration overhead and simplify the programming model. Of course, isomorphic computing is not a universal solution. Off-chip accelerators are still irreplaceable in super-large computing power cloud-side scenarios, and the two types of routes have their own applicable boundaries.
He also objectively pointed out the practical contradiction of promoting the unified RISC-V AI instruction extension standard: different business scenarios have huge differences in computing power and power consumption demands. If manufacturers make private extensions, the hardware iteration will be more flexible and agile, but the subsequent software adaptation cost will rise exponentially; strictly following the community standards will restrict some micro-architecture innovations, which is a proposition that the industry needs to continuously weigh. AI-assisted development paradigms such as Vibe-Coding can assist operator generation and compiler migration to reduce part of the adaptation workload, but we cannot expect to completely skip the co-construction of the basic software ecology with this. Overseas commercial enterprises are generally on the sidelines, and China's organized promotion of the full-stack construction of RISC-V+AI OS based on practical industrial demands is our unique opportunity of the times.
Apart from hardware IP and instruction standards, the mainline adaptation of the software stack is an unavoidable practical problem. Zhu Weidong mentioned combined with product practice that many RISC-V related codes currently are still in private branches of various manufacturers and have not been merged into upstream mainlines such as Linux kernel and PyTorch. After the upstream version is iterated, the downstream needs to invest manpower to continuously synchronize and maintain, and the maintenance cost remains high. Promoting the code to be merged into the community mainline is the only way for RISC-V servers to achieve large-scale commercial use. The practice of the Flag OS open source software stack of the Beijing Academy of Artificial Intelligence also proves that no matter how advanced the hardware design is, it needs support from compilers, operator libraries, model Day-0 adaptation and automated deployment tools. A large number of heterogeneous domestic computing resources in China urgently need a unified southbound adaptation framework to convert the scattered hardware capabilities into computing power that upper-layer services can call.
Computing power competition is essentially a contest of full-stack ecology
Based on the information obtained from this AICC conference, we can draw a conclusion: There is no single chip architecture that can fully undertake the entire business link of Agent.
Processing-in-memory has energy efficiency advantages in high memory access scenarios, but it is not good at handling a large number of general branch tasks; 3D-Chiplet brings valuable architectural flexibility, but its implementation is restricted by yield, heat dissipation and software support; RISC-V has an open and flexible base, but its value release is highly dependent on the complete software system composed of public instruction standards, operating systems and inference frameworks. At the same time, no matter which hardware route is adopted, for Agent services that run for a long time, the stability of the memory subsystem is a key weight that cannot be ignored in industrial selection.
Therefore, in the Agent era, the criteria for evaluating chips have shifted. Nominal peak computing power is no longer the only measurement standard, and the weights of end-to-end efficiency in real services, long-cycle