Google releases new AI chip with doubled HBM bandwidth

According to a report from TechXuexi on November 9, Google has just released two new TPU models: the TPU 8i Max, which is designed for ultra-large context windows and features the highest computing power density per single rack, and the TPU 8tx that supports secure and confidential inference.

Google claims that compared with TPU 8t, TPU 8tx can reduce latency by up to 50% with almost negligible performance loss. The computing power, SRAM and HBM bandwidth of TPU 8i Max are all 2 times that of TPU 8i.

This marks the first time that Google has launched 4 TPU models within a single year. Back in April this year, Google rolled out two new dedicated TPUs, namely the 8t for large-scale training and the 8i for real-time inference.

So far, the 4 TPU models of Google's 8th-generation TPU product line can cover the full lifecycle of agents.

In addition to launching the latest TPUs, Google has also upgraded a unified open-source software stack for developers.

01. Google Unveils 2 New TPUs Targeting Confidential Computing and the Memory Wall

Google has long been collaborating with Google DeepMind on TPU design. However, as the computing paradigm shifts from human interaction-oriented to agent-oriented tasks, enterprises demand accelerators that can complete inference within ultra-large context windows. Clients in some highly regulated industries also require dedicated encryption capabilities for exclusive business workloads.

This is the reason why Google launched these two new TPUs this time, with the goal of enabling every developer and cloud customer to use TPUs conveniently.

Google's 8th-generation TPU product portfolio targets different segments of the AI lifecycle:

The TPU 8t released in April this year is designed as the training core for large foundational models, aiming to achieve the highest "effective throughput", which refers to the effective intelligent capability output per watt of electricity. TPU 8i is an inference engine built to break the memory wall, reducing inter-chip communication latency by up to 50%.

The newly launched TPU 8tx and TPU 8i Max are targeted at confidential computing and the memory wall.

TPU 8tx, the third model in Google's 8th-generation TPU lineup, is optimized for secure and confidential inference, and is also suitable for small-scale training. It is built for regulated enterprises and government agencies, which can run cutting-edge models and agents while processing sensitive data.

Normally, data is encrypted when stored or transmitted over the network, but it must be decrypted in memory during computer processing. TPU 8tx adopts the same chip as TPU 8t, paired with the 5th-generation AMD EPYC CPU host, and directly embeds security capabilities into the hardware layer, so that sensitive data and model weights can be continuously protected and isolated even when they are in active operation.

At the same time, TPU 8tx inherits Google's low-latency Boardfly network built for real-time inference, which ensures the smooth operation of multi-step agent tasks. Google claims that compared with TPU 8t, TPU 8tx can reduce latency by up to 50% with almost no performance loss.

Combining training-level computing power with a high-speed inference network also means that enterprises can run and customize models on the same set of systems. With TPU 8tx, R&D teams can fine-tune models using private data, distill large models into small ones, and train agents through reinforcement learning, without the need to migrate sensitive data between multiple independent clusters.

TPU 8i Max solves the memory wall problem.

When dozens of agents work simultaneously on the entire software codebase or hours of video content, the bottleneck of memory bandwidth will become prominent. Therefore, Google encapsulated two low-latency TPU 8i chips into the same packaging substrate.

Google says TPU 8i Max can achieve synchronous expansion of computing power and memory, with computing power, SRAM and HBM bandwidth all twice that of TPU 8i.

In addition, TPU 8i Max integrates 2 times the chip area within the original rack space, so data centers can deploy more computing power with the same footprint and power consumption.

02. Google Opens TPUs to All Developers with Unified Software Stack Upgrade

Google also announced that in order to open TPUs to all developers, it will invest in the following aspects:

First, Google will open TPU development capabilities to all PyTorch developers without requiring them to rewrite code.

Last month, Google open-sourced TorchTPU, its native PyTorch framework for TPUs. Now PyTorch code and models on Hugging Face can run natively on TPUs, and mainstream open-source inference engines such as vLLM and SGLang also provide out-of-the-box TPU backends.

In addition, to lower the threshold of hardware optimization, Google's open-source TPU Agent Toolkit can directly connect to programming tools such as Claude Code, Cursor and Google Antigravity, complete TPU environment access within minutes, automatically locate performance bottlenecks, and support automatic tuning of custom operators using natural language. The toolkit is currently in preview.

For production deployment scenarios, Google has open-sourced the XLA Serving Engine workflow, which supports model training in Python and direct deployment in a low-latency C++ runtime, providing services across heterogeneous hardware including TPUs, GPUs and CPUs.

Second, Google states that its TPUs have achieved industry-leading production-grade performance on the InferenceX and AgentX benchmarks.

The industry standard evaluation sets InferenceX and AgentX target real-world agent code generation and long-context tasks. According to the test results released by Google, relying on the native vLLM-TorchTPU software stack on TPUs, it has achieved the top cost-performance ratio at present. This advantage covers a wide range of cutting-edge open-source models and various open-source service stacks, while ensuring that model accuracy is not affected.

Third, Google will further eliminate the idle computing power waste of autonomous agents.

AI agents are stateful programs that often pause for seconds or even hours while waiting for database query results, information returned by external tools, or human feedback. To make accelerators more power-efficient during these waiting periods, Google's Agent Compute Engine software can automatically suspend idle agent sessions and migrate working memory out of the chip within milliseconds.

Fourth, Google will break through the memory wall with TPU Sync.

Google will open-source TPU Sync, a model state layer for TPU clusters. TPU Sync integrates on-chip HBM, host DRAM, shared storage and data center network into a co-designed unified memory hierarchy; it can automatically offload KV cache outside the chip within milliseconds, stream context between decoupled prefill nodes and decoding nodes, and synchronize reinforcement learning weights, ultimately reducing KV transmission latency and first token generation time by over 90%.

Fifth, Google will horizontally scale distributed tasks across multiple TPUs through a single controller.

Cloud Pathways, Google's in-house large-scale distributed machine learning runtime system, is now officially available on Google Kubernetes Engine (GKE), opening the Google DeepMind runtime to all Google Cloud customers.

Leveraging the single-controller architecture and Pathways tools, developers can orchestrate large-scale TPU clusters spanning multiple TPU slices. Google says whether it is pre-training, reinforcement learning or multi-host inference, developers can operate it just like a local machine.

In addition, with co-located Python sidecars, sub-minute TPU slice hot switching and elastic shard rearrangement capabilities, Cloud Pathways allows long-running tasks to continue running even when hardware failures occur.

Finally, Google will splice custom TPU topologies on demand relying on GKE dynamic slicing capabilities.

The GKE dynamic slicing feature is now officially available. It uses Optical Circuit Switching (OCS) technology to decouple physical hardware resource provisioning from logical slice composition. This feature can dynamically combine TPU cubes into larger topologies (super slices) in real time, or split them into smaller, isolated workloads (sub slices).

Google claims that this topology flexibility can reduce task startup time to as low as 1/5 of the original, and increase fault recovery speed by up to 4.5 times, maximizing the resource utilization of the entire TPU pool and the effective throughput of tasks.

03. Conclusion: In the Agent Era, Are Accelerators Moving Towards Scenario-Specific Customization?

Google says that more and more enterprises are using AI to conceive and build the next generation of AI infrastructure, and Google is no exception. Whether using Google DeepMind's AlphaChip reinforcement learning system, or relying on Gemini to complete chip floorplan design in a few hours, Google is using TPU-based AI services to accelerate the R&D of next-generation TPU hardware.

Now, with the explosive growth of computing power demand for agent scenarios, computing power vendors are shifting from general-purpose accelerators to scenario-specific optimizations for training, inference, confidentiality, long context and other use cases. Google has formed a closed loop of mutual promotion among chips, models and systems, and built this accelerator solution customized around the agent lifecycle, which is a new exploration for the above trend.

This article is from the WeChat Official Account "TechXuexi", written by Cheng Xi, edited by Panken, and published by 36Kr with authorization.