DeepSeek and its peers are churning out chips at a frantic pace, has NVIDIA finally lost its shine?

Wait a minute, is DeepSeek developing its own chips???

Here's what happened: A while ago, Reuters reported that DeepSeek, which has long been focused on large model development, has been quietly working on AI chips for a full year, and is currently in contact with chip design, wafer foundry and storage manufacturers.

But just a few days after the news came out, OpenAI unveiled the Jalapeño chip cluster it developed in partnership with Broadcom. Around the same period, Anthropic also announced that it would start building its own chip team...

Wait a second, all of you are frantically snapping up Nvidia's production capacity while publicly announcing that you are going to develop your own chips. Don't the GPUs sold by Jensen Huang taste good anymore?

After some research, I found out that they really don't taste as good anymore...

And all of this is because of: inference chips.

This so-called inference chip is actually not a new term at all. Many years ago, Google's first-generation TPU was already used for inference research, and there were quite a few startups working on inference chips a few years ago, such as Groq, Cerebras, SambaNova, each with its own unique strengths.

But the reason why most people find it unfamiliar now is obvious: This industry has been lukewarm in the past few years, a field that big tech companies didn't even bother to pay attention to.

So the question arises: Big tech companies didn't care much about this a few years ago, why are they suddenly placing such high value on it all at once this year?

Because it wasn't cost-effective back then... There was no clear market prospect for this matter at that time.

As we all know, the large model industry is developing extremely fast. A model or technical direction that is all the rage today may be completely forgotten in two months. We don't even know where the OpenCLaw project that was hyped this year is now, let alone the large model vendors from two years ago.

After all, there were many people who openly opposed LLM back then (we are looking at you, Yann LeCun)... No one could tell how far large models would go, so model vendors were even less likely to specially develop chips for inference.

After all, Nvidia's chips worked perfectly fine, and performed extremely well in training. Spending time researching inference chips would be totally pointless. Google's TPU, the only dedicated inference chip back then, was not originally designed for large models, because large models didn't even exist at that time.

But the turning point happened around this time two years ago.

In August 2024, the Google DeepMind team published a landmark paper, which found that as long as the thinking time of the model is extended, the accuracy rate of the model will surge directly.

Starting from September, OpenAI's o-series models caused a huge wave in the industry; after the new year, deep thinking models such as DeepSeek-R1 brought another major impact to the global tech circle.

Then from last year to this year, the industry gradually realized that AI Agent, multi-Agent collaboration clusters, and super-long context analysis of hundreds of thousands of words have become the standard configuration for large models to land in real work scenarios. The technical route of deep reasoning and Agent has been basically recognized by the entire industry.

With the confirmation and development of this technical route, apart from the front-end service teams, GPUs are the ones facing the greatest pressure...

Open any AI chat dialog box, you input text here, and the AI returns text to you, which looks very natural and smooth.

But from the AI's perspective, when you ask the AI to "act as a catgirl", it first needs to understand "what exactly this sentence means". This step is called Prefill. After the AI understands the instruction, the process of outputting responses in the tone of a catgirl to you is called Decode.

In the Prefill stage, the AI splits the whole sentence into individual tokens, and understands the meaning of these tokens respectively. This process is very similar to the training stage of the AI.

How does the model judge the meaning of these tokens? This involves matrix operations.

Among all hardware in the world, the best device for matrix operation is the GPU.

However, for GPUs, the calculation process of the Prefill stage (sending the question) and the Decode stage (outputting the answer) is completely different...

In the Prefill stage, we provide all the data at once, the GPU can split the data on its own, and all cores perform parallel calculations together, which is very efficient.

You can get a sense from the simplified formula in the picture below: Assuming the input tokens require a matrix with 4096 parameters, then for every 1 byte of data moved from the memory, the GPU can perform thousands of operations on this data.

But once we enter the Decode stage, the situation goes wrong.

As we all know, current large models are autoregressive, which means they can only calculate the next word after getting the previous word. So the input matrix with 4096 rows in the Prefill stage will shrink sharply to only 1 row in the Decode stage.

After the same calculation, for every 1 byte of data moved, the GPU can only perform one operation...

Now we run into a big problem...

Inside the chip, the energy consumption of multiplying two pieces of data is very low. But moving a piece of data from the video memory to the GPU core often consumes dozens or even hundreds of times more energy and time than the multiplication operation itself.

So to make full use of the computing power of a chip, the ideal situation is to move the data into the core once, and then perform tens of thousands of operations locally on it. The most inefficient situation is to move the data once, perform one operation, and then move it again when you need it next time.

Therefore, in the Decode stage, for a large model with 70B parameters and 140GB weights, every time the chip outputs one Token to the user, it has to read the entire 140GB of data from the HBM video memory completely. It will repeat this full reading process for every subsequent word it outputs.

Even with top-tier video memory with a bandwidth of 3 TB/s, the output speed is only 21 tokens per second...

In other words, you spend a huge amount of money buying high-end graphics cards, but 80% of the chip's time is spent idling and waiting for data to be transferred. Who can stand this?

Worse still, in addition to longer inference time, in order to remember the thinking process of hundreds of thousands of words before, the model must cache all the features of historical Tokens in the video memory. For every additional step of thinking, the amount of data (KV Cache) that needs to be moved in the video memory explodes accordingly. (This is why the prices of video memory and RAM have risen sharply recently)

In the past when video memory was sufficient, major vendors could easily handle this problem with pure software solutions.

For example, they use continuous batching to accumulate dozens of user requests to spread the data movement cost; there is also the grouped query attention mechanism to compress KV Cache and reduce the data movement volume of the GPU.

But these are only temporary solutions that do not address the root cause... GPU is essentially a general-purpose chip, not an inference chip, and it is not designed for the Decode scenario at all!

In fact, Google discovered this problem many years ago. The solution they came up with was very straightforward: Since general-purpose GPUs perform so poorly in this scenario, why don't we develop a dedicated chip only for inference?

Thus, the first-generation TPU was born. Its core technology to solve the data movement problem is called Systolic Array.

Compare the inside of the chip to a factory, HBM video memory is the central warehouse outside the factory, SRAM is the small basket next to the assembly line, and the computing cores are the screw tightening stations. When an ordinary GPU calculates matrix operations, it is like a worker who has to go back to the video memory warehouse to get one part every time he tightens one screw, and then sends the part back to the warehouse after finishing the work.

But Google's Systolic Array turns the screw tightening process into a pipeline, and it is a two-dimensional pipeline.

They welded tens of thousands of multiplication computing units (PE) into a square matrix. Once the weight data of the matrix enters the pipeline, it is passed horizontally between the stations like a relay baton for calculation, and the calculated results are passed to the next layer of computation in turn.

Since data is transferred directly between adjacent cores, every time data enters a core, it will be continuously transferred and reused dozens or hundreds of times inside, there is no need to return to the external video memory at all, and no extra data movement is required halfway.

Therefore, this unique design of TPU has become one of the key factors that allowed Google to get rid of Nvidia's supply constraints and achieve excellent performance in the Gemini 2.5 era. (Of course, Gemini's performance has declined again recently)

In the past when the technical route was not yet finalized, dedicated solutions like TPU did not show great advantages for large models, so everyone continued to buy GPUs.

But now that deep reasoning has caused the inference load to surge, the TPU design idea has become really attractive, even Google has started to split TPU into training versions and inference versions.

Thus, major model vendors have woken up one after another during this period: it is time to invest heavily in developing dedicated inference chips.

Of course, after being choked by Nvidia's supply constraints for so long, no large model vendor is willing to be a passive buyer this time...

Compared to paying tens to hundreds of billions of dollars to Jensen Huang every year while having to look at his face, the cost of running your own chip development team is simply trivial with that amount of money.

But when they actually start developing their own chips, no one just copies Google's TPU design. After all, everyone has their own proprietary models, they need to do custom tuning and development to find the most suitable hardware for their own scenarios!

Take Anthropic as an example. Although the systolic array is very fast, it cannot handle a large number of miscellaneous non-matrix operations efficiently. Therefore, Anthropic not only develops systolic pipelines, but also carries out in-depth cooperation with AWS on Train