Squeezing the Limits of Silicon: How to Keep GPUs From Sitting Idle?
The AI industry has suddenly kicked off a "chip-building race".
On July 20, The Information reported that Google is developing a server chip codenamed "Frozen v2" internally, planning to solidify part of the Gemini model's architecture directly into silicon wafers.
One month earlier, on June 24, OpenAI unveiled its first inference chip Jalapeño co-developed with Broadcom, which took only nine months from design to tape-out.
In addition, Anthropic, Zhipu, and DeepSeek have also successively revealed that they are making relevant layouts in this field.
However, not every company has the capital and capability to redevelop a chip from scratch. As a result, a market is drawing the attention of capital giants and tech behemoths: AI infra. The underlying logic is that the already deployed data centers and GPUs still have room for optimization.
The AI inference platform Baseten has seen its annual revenue surge 20 times, with its valuation skyrocketing from $2.1 billion to $13 billion. Fireworks, a player in the same track, saw its valuation quadruple to $17.5 billion within seven months, with annualized revenue exceeding $1 billion.
Recently, this track has also welcomed a strong new force. The open-source inference engines "Gemini" vLLM and SGLang have successively announced their commercialization, with seed round financing both exceeding $100 million. Almost all the giants and top venture capitals in the AI industry have gathered behind them, kicking off a showdown between strong players.
Therefore, "GPU utilization" has become the latest keyword in Silicon Valley, and has also spawned a new hundred-billion-dollar market.
In this article, we invited the co-founders and core members of RadixArk incubated by SGLang, as well as data center experts, to discuss the four-layer architecture of AI Infra, how the industry can squeeze the potential of "silicon" to the limit, and the key technologies behind it.
The focus of large models has shifted from training to inference, which puts forward completely different requirements for computing power.
Training costs are extremely high, but the task comes to an end once the training is completed. Inference is different: whether it is a Chatbot or an Agent, the more AI applications there are, the more requests the GPU needs to process, and these requests are completely non-stop, leading to an explosive growth in inference demand.
Zhu Banghua
Co-founder of RadixArk
As new applications gradually increase, the total amount of inference will be much larger than before. As people continue to expand applications and push the intelligent boundaries of models, the demand for inference will keep growing. The demand is now so high that GPUs on the market are almost impossible to find, and every company is facing a shortage of GPU resources.
The current situation is that every company on the market is short of GPUs. For wealthy tech giants, the most straightforward solution is of course to continue building data centers and purchasing more GPUs.
Tech giants including Meta, Google, and Microsoft continue to increase their capital expenditures, with the total expected to exceed $1 trillion this year. Jensen Huang also predicts that the annual global investment in AI infrastructure will reach $4 trillion by 2030.
But Wall Street is worried.
Ethan Xu
Former Microsoft Energy Strategy Manager, Director of Research at Breakthrough Energy
Wall Street's concerns are not unfounded, as we have precedents from the past. For example, the burst of the dot-com bubble: a large amount of capital was invested in fiber-optic construction and internet companies at that time. Although these investments realized their value in the long run, they caused huge problems in the short term.
Therefore, if computing power cannot be increased on a large scale in a short period of time, what other methods can meet the rapidly growing demand of AI? The answer is: improve GPU utilization. The industry behind this is called AI Infra (Artificial Intelligence Infrastructure Layer).
First of all, from the perspective of the overall architecture of AI Infra, let's clarify a common misconception about GPUs: You may think GPUs are very busy, but in fact, most of the time, GPUs are very "idle".
GPU is only one part of the entire AI system. From the moment a user's request enters the system, it needs to go through network transmission, resource scheduling, model loading, memory management, and finally complete the inference and return the result. As long as any link in the entire process encounters a bottleneck, the GPU can only stop and wait.
A study released by Carnegie Mellon University (CMU) in April this year conducted fine-grained telemetry on 756 GPUs in a large academic AI cluster for 31 days, covering six models from A6000, L40S, A100, H100 to the latest B200. The results show that most of the time, GPUs are in a state called "execution-idle".
How serious is this state of "apparently busy but actually idling"? In the cluster they observed, nearly 20% of the total execution time and about 11% of energy consumption were wasted on waiting. What's more noteworthy is the inference scenario: Azure Code workloads have up to 65% of their energy consumption wasted on this kind of idling, and OpenAI's Chat requests also reach 52%.
So why can't GPUs run at full capacity? First, we need to understand the four-layer architecture of AI Infra, and each layer will affect GPU utilization.
Layer 1: Power Infrastructure, which refers to power supply. This layer determines whether the GPU can operate continuously and stably.
Layer 2: Hardware Infrastructure, which refers to computing cards. In addition to GPUs, it now also includes High Bandwidth Memory (HBM), high-speed interconnection (NVLink, InfiniBand) and CPUs. This layer determines the theoretical upper limit of computing power that can be provided.
Layer 3: System Software, including CUDA, compilers, communication libraries, memory management and underlying kernel libraries, etc. It determines whether each calculation can reach the physical limit of the hardware.
Layer 4: Service Orchestration. This layer is responsible for coordinating GPU resources, determining which tasks run with priority, and performing dynamic scheduling between different businesses. Inference engines, training frameworks, request scheduling and resource management all belong to this layer. Representative projects include vLLM, SGLang, etc., which is also the area with the most intensive innovations in recent years.
After clarifying the architecture of AI Infra, let's go back to our question: why so many GPUs cannot run at full capacity, and which link is the main bottleneck?
Theoretically, problems in any of the four layers will affect GPU utilization: insufficient power supply and heat dissipation will force the GPU to run at a reduced frequency; lagging hardware interconnection will make GPUs spend a lot of time waiting for data instead of processing data; insufficient optimization of system software will prevent each calculation from reaching the physical limit of the hardware; improper request scheduling will split calculations that could have been merged and recalculate results that could have been reused, resulting in waste.
However, the nature of problems in these four layers is different. The power infrastructure layer and computing hardware layer are "hard" problems: the transformation cycle of power supply, heat dissipation and chips is measured in years, and their optimization space is rapidly peaking. The system software layer and service orchestration layer are "soft" problems, which are essentially engineering and algorithm problems. Optimization in these two layers can bring several times of performance improvement.
Against this background, the attention of the industry is rapidly shifting to these two layers of "soft" problems.
Chen Zhenlin (Richard)
Member of Technical Staff, RadixArk
Although most costs occur at the physical layer, most of the work should be done at the software layer and service orchestration layer. Because investing manpower in this area can bring the greatest possibility of optimization.
Let's take a specific example. The procurement cost of a NVIDIA GB200 NVL72 cabinet is about $4 million. However, if the software stack does not do a good job in scheduling and optimization, the actual utilization of GPUs may only be 50%, which is equivalent to only using $2 million of the value, and the remaining $2 million is wasted in the form of electricity bills, depreciation and opportunity costs. On the contrary, if the utilization rate can be increased from 50% to more than 90%, the effect is equivalent to getting an extra cabinet out of thin air.
Chen Zhenlin (Richard)
Member of Technical Staff, RadixArk
When the software layer is optimized to the best, it will bring better services and lower token consumption. Through rewriting and optimization at this layer, we can help inference engines and training frameworks achieve better interaction, including scheduling for post-training processes such as reinforcement learning, which is very popular now, to improve GPU utilization.
This is why when the hardware cost reaches this level, every optimization at the software layer has become a direct business issue.
Not long ago, netizens found that the largest proportion of Anthropic's current employees are engineers related to AI Infra, which is also closely related to this logic.
Chen Zhenlin (Richard)
Member of Technical Staff, RadixArk
Anthropic's inference team now has more than 200 people. In the past three years, they have gone from extremely unstable infra to extremely stable infra, and achieved stronger cost control and optimization, which has directly enabled their revenue to turn profitable in the second quarter.
OpenAI, Anthropic and Google will keep such AI Infra teams in-house, as it has become one of the competitive barriers for cutting-edge labs. However, for smaller model teams or startups that do not have enough capital to carry out optimization, two open-source frameworks have become their pillars: SGLang and vLLM.
Chen Zhenlin (Richard)
Member of Technical Staff, RadixArk
This is why open-source communities like SGLang emerged two years ago. You need such a new framework to redefine this new generation of hardware system combinations, so as to achieve better adaptation with them.
After understanding the four-layer architecture and the two "soft" problems that need to be optimized, what specific measures should we take?
All current optimization work can be summed up as answers to four questions: Which calculations do not need to be redone? Which waiting time can be eliminated? Which computing power is not fully utilized? Which resources are not coordinated?
The SGLang and vLLM we mentioned earlier are all developed around these questions, but their design ideas are slightly different. vLLM has received widespread attention for proposing PagedAttention, and it focuses more on conventional and general inference deployment scenarios. SGLang starts from the inference execution process, and focuses more on calculation reuse and system-level optimization.
