The latest judgment from former Intel CEO: the vast majority of chip design companies will disappear, and manufacturing is the only moat.

In this wave of AI boom, AI computing power is undoubtedly the biggest winner, and NVIDIA has rapidly become one of the companies with the highest market capitalization in the world.

Relying on GPUs, NVIDIA has not only risen to become one of the world's most valuable companies, but also driven the prosperity of the entire AI chip startup market. In just a few years, a large number of startups have flooded into the track, trying to grab a slice of the market from NVIDIA.

But here comes the question: among so many AI chip companies, how many can survive in the future? Pat Gelsinger, former CEO of Intel, gave a not-so-optimistic judgment:

There are hundreds of AI inference chip companies on the market today, but this prosperity may only be temporary. In the future, most chip architectures may be eliminated, and the industry will re-converge to a small number of platforms.

Speaking of Pat Gelsinger, he is absolutely one of the most senior figures in the chip industry. He joined Intel at the age of 18, participated in the R&D of 386 and 486 processors, and later served as CEO of VMware and Intel successively. It can be said that he has almost fully experienced the development of the modern semiconductor industry.

Recently, he participated in a long-form podcast hosted by Silicon Valley venture capital firm a16z, talking about AI chips, storage, optical interconnection, power supply, and the future of the entire AI infrastructure.

In addition to the above views, Gelsinger also put forward several other viewpoints that I think are very worthy of attention.

First, AI chip design is becoming easier and easier, but manufacturing and packaging have instead become new bottlenecks. A chip may only take three months to complete the design, but it may take another nine months before it can be actually put into use.

Second, HBM, the hottest technology today, is actually a rather poor storage technology. The outbreak of AI demand may push the storage industry to usher in a major innovation rarely seen in 30 years.

Today, the Silicon-based Observer will use this interview to talk about several important changes taking place in the AI computing power industry.

AI Makes Chip Design Faster, While Manufacturing Can Hardly Keep Up

Over the past few decades, the chip industry has been pursuing one thing: to make chip design easier.

Back in the 1980s, when Intel was developing the 486 processor, many design tools that today's chip engineers take for granted were not yet mature.

For example, engineers needed to use hardware description languages to define chip logic, but at that time there was no complete tool system, so they had to develop languages and compilers on their own, and even participate in exploring automatic placement and routing technologies.

Today, AI is further changing this process. The logic design, coding and optimization work that used to take engineers a lot of time can now be increasingly handed over to AI tools.

Especially in the field of digital logic design, AI has been able to significantly improve development efficiency.

The problem is that as design speeds up, manufacturing has not accelerated synchronously.

For example, suppose you want to develop a brand-new AI chip now. The design team has a deep understanding of the computational requirements of the model, with the assistance of AI tools, they only spend three months to complete the chip design.

But then, the chip still needs to be sent to the wafer fab for manufacturing. Even if the process goes smoothly, the wafer fabrication process alone may take three months.

Then comes advanced packaging. Current AI chips usually need to integrate computing chips, HBM and other components together, making packaging increasingly complex, which may take several more months.

Finally, the integration of the whole machine and rack-level system needs to be completed. The result is that the chip design only takes three months, but from the completion of design to actual availability, another nine months may have passed.

If we further count the expansion of production capacity, improvement of software ecosystem and large-scale deployment, the entire cycle may even last as long as one and a half years.

This has created a very realistic contradiction: The iteration speed of AI models may have exceeded the speed of chips from design to deployment.

For example, when a company is designing a chip, it may believe that the core demand for future inference is to improve Token generation efficiency.

But by the time a year and a half later, the market may have entered a new inference paradigm. Longer context, more complex inference processes, and even new model architectures may all change the original performance advantages of the chip.

As a result, before a chip is actually put into large-scale use, the optimal design assumptions made for it at the beginning are no longer valid.

This means that as AI continues to lower the threshold of chip design, the key to determining the competitiveness of an AI chip company will increasingly depend on the manufacturing cycle and large-scale mass production capacity.

100 AI Chip Design Startups, Only a Few May Survive in the End

In the past two years, a very obvious trend has emerged in the AI computing power track: there are more and more chip architectures.

Especially in the inference market, some people specialize in designing low-latency chips, some focus on high throughput, some bet on computing-in-memory, and other companies choose different routes such as optical computing and probabilistic computing.

Even around the inference process of the same large model, chips have begun to further divide labor.

For example, Prefill and Decode. Prefill is responsible for processing input content, which can be simply understood as that before the model formally answers, it reads the user's question and context first.

Decode is responsible for generating output Tokens step by step. Due to the different computational characteristics of these two stages, the industry has begun to discuss whether hardware should be designed separately to obtain higher computing efficiency.

But I think this trend of extreme specialization actually has a big problem.

That is, the computing demand of AI changes too fast.

Today, people may think that separating Prefill and Decode can improve chip utilization.

But tomorrow, with the development of inference models, more CPU-like computing power may be needed.

Later, when AI enters fields such as scientific computing, molecular simulation, and chemical analysis, the high-precision computing demand in traditional high-performance computing may become important again.

Even the boundary between training and inference may gradually blur with the development of technologies such as continuous learning. This means that the dedicated chips designed for a certain computing stage today may not be able to maintain their advantages in the future.

This actually involves a very classic problem: the balance between dedicated and general-purpose.

If a chip is deeply optimized for a specific task, it can often achieve high efficiency. But the more extreme the optimization, the more limited the workload it can adapt to. Once the model architecture changes, the original advantages may disappear.

What's more troublesome is that chips also need to be deployed on a large scale.

Data centers do not only buy a few chips. When a company needs to deploy hundreds of thousands or even millions of accelerators, each additional architecture means new costs for software adaptation, network connection, equipment management and maintenance.

Suppose there are 100 different inference chips in a data center. Some of them are suitable for Prefill, some are suitable for Decode, and some are specially used to process other computing tasks.

Viewed separately, each type of chip may be very efficient. But in a large data center, scheduling, maintaining and upgrading these devices will become very complicated.

Moreover, AI chip startups also face the second problem: capital.

Developing a chip is only the first step. To truly form commercial competitiveness, it is also necessary to complete tape-out, mass production, software adaptation, and find sufficiently large customers.

When the computing power investment of the entire industry reaches the level of tens of billions of dollars, it is difficult for a startup to independently complete all links from design to large-scale deployment.

In the end, it is still large computing power platforms and customers such as NVIDIA, OpenAI and Anthropic that may determine which architectures can survive. Because only these big customers have sufficient capital and workload to support a brand-new chip architecture.

Therefore, my judgment on a large number of AI chip startups today is not that optimistic. There may be more technical routes in the future AI chip market, but there will not necessarily be more independent chip companies.

Leading chip performance is only a ticket to the entry. Whether it can enter the computing power system of big customers truly determines its commercial value.

HBM Is Rather Poor, But AI May Spur the Biggest Storage Innovation in 30 Years

HBM is a rather poor storage technology. But it is already the best choice available today.

This sounds a bit contradictory. After all, in this round of AI computing power boom, HBM has almost become one of the most sought-after hardware.

But the problem is that HBM itself still has many limitations.

For example, storage density, power consumption and heat dissipation pressure, as well as data transmission are constrained by packaging and physical connection conditions. Especially as the computing power of AI chips continues to grow, more and more computing resources are waiting for the storage system to provide data in time.

This is the so-called memory wall. This is also why I think the next major innovation in AI computing power will most likely occur in the storage field.

Interestingly, the storage industry has not seen a new mainstream technology paradigm for too long.

DRAM, SRAM and Flash are still the main technical foundations. Over the past few decades, although many new storage technologies have emerged, few of them can achieve low-cost, large-scale mass production and completely change the industrial landscape.

One of the important reasons is that the business model of the storage industry was too poor in the past.

Storage chips have long been highly cyclical. When the market is good, manufacturers may make profits for a year; after supply increases and prices fall, they may face profit pressure for several consecutive years.

In this case, it is difficult for enterprises to continuously invest a lot of funds to bet on new materials and new technologies with a long commercialization cycle.

But AI has changed this situation.

With the outbreak of HBM demand, the market value and profitability of storage manufacturers have changed significantly, and capital has begun to re-value this once highly cyclical industry.

In the past four years, the overall market value of the storage industry may have increased by about 2.5 trillion US dollars. The storage industry is gaining more capital support, and AI has created a sufficiently strong new demand.

Higher bandwidth, lower power consumption, higher density, easier stacking, these requirements that were difficult to meet at the same time in the past are now becoming the driving force for promoting the commercialization of new technologies.

There are two directions that are particularly worthy of attention at present.

One is new storage materials and device structures, such as ferroelectric materials and new solutions that can get rid of the limitations of traditional capacitive storage.

The other is to make storage and computing physically closer. Because the shorter the data transmission distance, the more opportunities there are to reduce latency and energy consumption.

However, this does not mean that all computing-in-memory technologies are worth looking forward to.

For example, PIM, namely Processing-in-Memory. This type of solution hopes to put part of the computing power directly into the memory to reduce data transmission. But it usually needs to adjust the architecture for specific computing tasks, which may limit the flexibility of the system when facing new types of workloads.

In contrast, placing high-bandwidth and high-density storage as close as possible to the computing unit while maintaining the versatility of the computing architecture may be a solution that is easier to adapt to future needs.

Technical routes that were difficult to commercialize in the past 30 years may get new opportunities in the AI era.

This also means that the competition in the future storage market may not only take place among traditional giants such as Samsung, SK Hynix and Micron.

New materials, new device structures, and new storage architectures may all create new entrepreneurial opportunities.

Copper Wires Were Declared Dead 25 Years Ago, Is This Finally Coming True?

If storage is an important bottleneck for AI computing power, then another increasingly prominent problem is interconnection.

In the past, the connection distance between GPUs was relatively short, and copper interconnection had the advantages of low cost and mature technology. But as a single computing power system expands from several GPUs to dozens, hundreds or even more, the difficulty of interconnection has increased significantly.

Copper wire has a natural problem. The higher the transmission rate and the longer the distance, the more obvious the signal loss. In order to ensure signal quality, the system needs to use more complex cables, signal processing and compensation technologies, which will increase cost and power consumption.

Under certain high-speed interconnection conditions, making copper wires work normally at a distance of 5 meters may be more expensive than making optical fibers work at a distance of 100 meters.

This is also why the AI computing power industry is paying more and more attention to optical interconnection.

However, there is a very important difference here.

What optical interconnection has the best chance to change first is the high-speed I/O connection between computing systems, especially the Scale-up interconnection between GPUs.

As for separating the core computing unit from the memory and then relying entirely on optical connections, it may not be a good idea.

Because photoelectric conversion also has cost and energy consumption. If every access to the memory requires multiple electro-optical and photoelectric conversions, the loss may outweigh the gain.

Therefore, a more reasonable direction is to use optics in places that require high bandwidth and long-distance transmission, and minimize unnecessary data transmission in the core area where computing and storage are highly coupled.

From the perspective of industrial development, I think 2028 to 2029 may be a window period worthy of attention.

At that time, advanced optical interconnection technologies such as NPO and CPO will have the opportunity to be applied on a larger scale.

Why this time node?

One important reason is that AI computing power systems are approaching the engineering limit of traditional electrical interconnection.

The most typical example is NVIDIA's NVL72. This system organizes 72 GPUs into a large computing system through complex high-speed interconnection. From an engineering perspective, this is of course a very remarkable achievement.

But from another perspective, it is also a manufacturing nightmare. In order to realize large-scale deployment of such a complex system, the industrial chain needs to solve a large number of problems in manufacturing, assembly, heat dissipation and reliability.

If the optical interconnection supply chain had been more mature earlier, the industry would have had the opportunity to achieve a more efficient Scale-up system earlier.

But there are also practical reasons why optical interconnection has not fully entered the GPU system for a long time.

It is not just replacing a transmission medium. Optical devices need to enter a more complex packaging environment, and high-performance GPUs are exactly systems with high temperature and high power consumption.

Lasers, optical components, advanced packaging and heat dissipation must all be solved together.

What's more troublesome is that when the system needs to be expanded to millions or even tens of millions of GPUs, optical interconnection must also have extremely high manufacturing consistency and reliability.

Therefore, the implementation of optical interconnection is not only a matter of technological maturity, but also depends on whether the entire supply chain can support large-scale deployment.

But compared with simple material replacement, I think there is another change that deserves more attention.

That is, optical interconnection may promote changes in the network architecture of the entire AI data center.

Traditional data center networks need to process a large number of unpredictable data packets. A data packet may be sent to any server, and the network must have flexible forwarding capabilities.

But part of the traffic of large-scale computing communication in AI training and inference has stronger regularity, which creates opportunities for optical circuit switching, namely OCS.

Compared with traditional packet switching, optical circuit switching can pre-establish optical connections for relatively stable and predictable communication demands.

With the maturity of optical interconnection and new switching technologies, the boundary between Scale-up and Scale-out today may also gradually blur in the future.

From an industrial perspective, this means that optical modules, optical switching, advanced optical packaging, as well as related lasers and materials, will have the opportunity to enter a larger market.

This article is from the WeChat Official Account "Silicon-based Observation Pro", author: A Qi, published with authorization from 36Kr.