Google's AI chips, a huge transformation

According to sources familiar with the matter at The Information, Google is developing a new type of server chip that will directly integrate the blueprint of its Gemini artificial intelligence model, enabling the company to deliver AI model services to users far more efficiently.

As is widely known, in the early years of the AI boom, general-purpose GPUs dominated artificial intelligence computing. After recognizing the opportunities in the AI market, chips such as TPUs, NPUs, LPUs, and solutions from companies like Cerebras emerged one after another. However, for a very long time, the dominant position of GPUs remained unshaken.

Yet, driven by companies like Google and Amazon that build their own models and infrastructure, the landscape of AI chips has begun to shift. Especially as AI deployment accelerates, AI chips have found new development opportunities. Even Google, when developing TPUs, designed separate products for inference and training workloads. Still, these products did not break away from the characteristics of traditional general-purpose chips.

Now, with Google embedding its models directly into silicon, profound changes are taking place in AI chips, and the traditional rules of the industry are being rewritten.

What Kind of Chip Has Google Developed?

According to reports, Google plans to use this new chip, informally named "Frozen v2," to solve the severe shortage of AI computing power. This shortage has created tensions within the company and forced Google Cloud to turn down collaboration requests from some external clients.

Against this backdrop, Google is believed to have begun exploring a new path for AI chips: instead of simply pursuing stronger general computing performance, the company is deeply integrating chips with specific models to improve AI computing efficiency through hardware-software co-design.

Sources say the name "Frozen" comes from the concept of permanently etching parts of the model onto silicon wafers. Engineers are still finalizing the core functions of the new chip and how its various components work together. Insiders indicate that Google plans to deploy this chip as early as 2028.

In fact, this is not the first time Google has tried to improve AI efficiency through dedicated hardware. As early as 2016, Google launched its Tensor Processing Unit (TPU) to reduce its reliance on general-purpose GPUs. However, like NVIDIA GPUs, TPUs are essentially still general-purpose AI accelerators that need to support a wide range of different models and algorithms.

This means that both GPUs and TPUs need to dynamically schedule tasks during runtime based on model structures, computing tasks, and data flows. For example, during large model inference, chips need to handle massive matrix operations, memory access, and model scheduling tasks. While these general capabilities offer greater flexibility, they also introduce additional computing overhead.

The Frozen project takes a completely different approach.

It attempts to solidify parts of the Gemini model's computing logic, inference workflow, and even decision-making mechanisms directly into the chip, allowing the hardware to "understand" how a specific model operates, thereby reducing runtime scheduling overhead. In other words, traditional AI chips are more like general-purpose computers, while Frozen is closer to a "dedicated machine" built for a specific AI model.

This model is similar to the ASIC (Application-Specific Integrated Circuit) path in the past semiconductor industry. Compared with general architectures like GPUs, ASICs can often achieve higher performance and lower power consumption for specific tasks, at the cost of reduced flexibility. If the algorithm direction undergoes major changes, dedicated chips may face the need for a full redesign.

This is also the biggest challenge facing the Frozen project.

Since chip design cycles typically take several years, and AI models iterate much faster than the traditional semiconductor industry, the Frozen chip may not be able to fully deliver its performance if the Gemini model architecture changes significantly in the future. Therefore, Google needs to strike a balance between "dedicated efficiency" and "sustained model evolution capabilities."

According to insiders, Google will be able to modify the chip, including updating model weights — the parameters that determine how the model generates outputs. This means Frozen is not completely fixed, but rather attempts to create a new compromise between hardware solidification and model updates.

If this path succeeds, Frozen could represent an important direction in AI chip development: as large model architectures gradually stabilize, chip design will shift from "supporting all models" to "deeply optimizing around mainstream models."

Similar trends have already begun to emerge. The core of current competition in the AI industry has shifted from simply competing for peak chip computing power to the overall synergy of models, software, chips, and systems. NVIDIA has built its GPU advantage through the CUDA ecosystem, while Google hopes to form a tighter hardware-software closed loop with its Gemini models, TPUs, and future Frozen chips.

According to predictions from R&D participants, after the Frozen chip is released, its efficiency, calculated by the number of tokens processed per unit of power consumption, could reach 6 to 10 times that of Google's latest existing AI chips. If this goal is achieved, it will not only ease Google's own pressure on AI computing resources, but may also change the way AI infrastructure competes in the future.

Entering the Track That Taalas Has Long Adhered To

In fact, before Google's news broke, Taalas had already made the decision to embed models into chips.

As stated on its official website, artificial intelligence models are changing the world. As software, they place extremely high demands on computing power. The company aims to increase efficiency by 1000 times, a goal that is far beyond the reach of today's most advanced general-purpose computers.

To achieve this goal, Taalas states that the industry needs to recognize the following points:

The model should not be simulated on a traditional computer;

It is the computer itself;

Embodied in native hardware;

Optimal, Hard Wired, Silicon;

Human language is its software;

Taalas makes hardcore models;

In their view, any AI model can be upgraded to a Hardcore model through the Taalas Foundry. Hardcore models support fine-tuning, and their supporting applications are written in human language.

Driven by this philosophy, the company launched its first product, the HC1 — a chip optimized for the open-source Llama 3.1 8B language model. In terms of specifications, the chip is manufactured using TSMC's N6 process. It has an area of 815 square millimeters, close to the current limit of semiconductor photomask sizes. Each HC1 chip package contains 53 billion transistors, most of which are likely used for ROM and SRAM memory. According to Bajic, the HC1 card consumes about 200 watts of power, while a dual-socket X86 server equipped with ten HC1 cards consumes up to 2500 watts.

According to Taalas, the chip can generate 17,000 output tokens per second, which is 73 times faster than NVIDIA's H200 GPU. In addition, the processor delivers such high performance while consuming only one-tenth of the power of the latter.

Incidentally, because the HC1 card is so fast, low-latency inference can be achieved without batch processing queries, meaning the bandwidth pressure on Taalas devices is very low. So low that even when multiple cards are run in parallel to handle larger models, the PCI-Express bus is more than sufficient. Taalas will allow customers to distribute workloads across multiple HC cards using pipeline parallelism later this year. In fact, by this summer, Taalas will launch a Llama 3.1 model hard-coded into the HC chip, which contains 20 billion parameters.

Customizing chips for specific AI models can improve efficiency, in part because it allows engineers to remove redundant components. For example, off-the-shelf graphics cards may contain more DRAM than required by AI models. Custom chips can replace unused RAM with additional transistors, allowing the model to run faster.

For most AI projects, developing a fully custom processor is too costly. According to Taalas, its engineers reduced costs by customizing only two of the more than 100 layers that make up the chip. In most processors, only a few layers contain transistors. The remaining layers house auxiliary components, such as wires that transmit data between different regions of the chip. Typically, the layers containing wires are located near the transistors.

The custom layers in Taalas chips use a technology called mask ROM call architecture. ROM is a type of memory where users can write data only once. After that, users can read the data but cannot edit it. Reportedly, each mask ROM call structure module can store 4 bits of data. Taalas's architecture uses only a single transistor to process this data. This transistor performs matrix multiplication operations, the mathematical calculations that AI models use to make decisions.

A major advantage of Taalas's design is that it eliminates the need for high-bandwidth memory modules (HBM). HBM is memory chips integrated on graphics cards to store data for AI models. Transferring information between HBM and the chip takes a long time, which slows down processing. Taalas says its solution not only avoids this latency but also eliminates the various auxiliary components required to support the chip's HBM modules.

It is worth mentioning that by the end of this year, Taalas will launch a cutting-edge large language model — possibly Llama, DeepSeek, or both — that will run inference across multiple HC cards. This architecture will be named HC2.

Is Focusing on Selected Models an Inevitable Choice?

Etched, a startup, is also a pioneer in the field of in-depth optimization around selected models.

Founded in 2024 by Harvard dropouts Gavin Uberti (CEO) and Robert Wachen (President), the company's founders said that as early as 2023, they submitted a 30-page memorandum arguing that artificial intelligence would ultimately require dedicated chips, not just general-purpose GPUs. But every major investor they visited rejected their investment requests.

Later, as AI advanced, they found their opportunity, and the market heat grew. The company completed its latest round of financing last December, with a valuation of $5 billion. Participants in this financing included VentureTech Alliance, a TSMC-related venture capital fund, and more than a dozen other investors, including Geoffrey Hinton, Li Fei-Fei, and Andrej Karpathy.

Insiders revealed that the company's valuation will quadruple to approximately $20 billion in a new round of financing led by existing investor Jane Street.

Returning to their products, it can be seen from related reports that the Etched team firmly believes that Transformer chips will dominate the AI field, which gives them a clear vision: to build a chip that outperforms general-purpose GPUs in running Transformer models, thereby revolutionizing the artificial intelligence industry.

In a recent press release, Etched stated that the company plans to use TSMC's N4P process to produce its inference chips. The N4P process is an upgraded version of TSMC's 5-nanometer process, with 11% higher performance than the original process. According to Etched, its first batch of prototype chips came off the production line earlier this year.

Etched has also added several other performance optimizations to its chips.

The more calculations a GPU performs per second, the more power it consumes, which in turn increases its operating temperature. Beyond a certain threshold, excessive heat can cause malfunctions. Graphics cards mitigate this problem by reducing their clock speeds when approaching peak frequencies. This mechanism, known as "thermal throttling," reduces computing speed.

Etched has developed a technology called LVI that reduces the need for thermal throttling. The company says its chips can run trillion-parameter AI models at "over 80% of peak floating-point performance" without reducing clock speeds. This significantly improves inference speed. Etched claims that its chips have several times higher floating-point operation density (a measure of performance) than existing AI processors on the market.

The company plans to deliver its chips as part of rack-mounted inference devices. The system uses multiple chips mounted on a custom circuit board. Etched has also developed custom cold plates, components that play a key role in liquid cooling systems. A cold plate is a flat metal sheet that transfers heat generated by the chips to the coolant in the rack.

Etched's devices use a mix of SRAM and HBM memory. AI chips use SRAM — currently the fastest RAM on the market — to store the most critical data of workloads. Other information is sent to HBM memory, which is slightly slower but has significantly larger capacity.

AI chips in a rack usually need to access data in each other's memory. Etched's devices use custom interconnection technology to achieve this data transmission. According to the company, the device has a system-level shared memory pool with lower request processing latency compared to earlier technologies.

Etched is currently ramping up chip production and plans to deliver its first batch of chip racks this summer. The company revealed at the end of June that it has received over $1 billion in customer orders.

Final Thoughts

From GPUs to TPUs, to Google's Frozen and Etched, we can see that AI chips are entering a new stage of development.

In the past, the core of chip competition was to improve general computing capabilities, and GPUs, with their strong flexibility, became the infrastructure of the AI era. However, as large models enter the stage of large-scale application, inference costs, energy consumption, and efficiency have gradually become key challenges. Hardware-software co-design optimization around specific models is becoming a new direction of exploration.

Of course, this does not mean that the GPU era is coming to an end. AI is still in a stage of rapid evolution, and general-purpose chips still have irreplaceable value. But as model architectures gradually mature, dedicated AI chips are expected to become an important part of future infrastructure.

In the future, AI chip competition may no longer be a simple replacement between GPUs and ASICs, but form a new division of labor: GPUs are responsible for supporting model innovation, and dedicated chips are responsible for improving the deployment efficiency of mature models.

More importantly, what Frozen represents is not just a change in a single chip, but a change in the competitive logic of the AI industry — in the future, models may in turn define chips, and the boundaries between software, algorithms, and hardware will further merge. This may also become the core direction of the next round of AI chip competition.

This article is from the WeChat official account "Semiconductor Industry Watch" (ID: icbank), authored by the editorial department, and published with authorization from 36Kr.