The chip is overheated.
AI chips are getting larger and hotter.
Over the past few years, the chip industry has come up with many solutions to continuously improve AI computing power. As transistor miniaturization becomes increasingly challenging, the industry turns to Chiplet; when a single die can no longer accommodate more components, CoWoS is used to integrate multiple computing chips together; when on-chip memory bandwidth is insufficient, HBM is continuously scaled up; when data transmission between chips becomes a bottleneck, technologies including NVLink, UALink and even CPO are under active development.
But all these technologies eventually push the problem to an increasingly unavoidable physical constraint: heat.
At SEMICON Taiwan 2026, James Chen, head of advanced packaging R&D at TSMC, presented a set of staggering data. According to the disclosed technology evolution roadmap, from 2024 to 2029, the CoWoS packaging scale will expand from approximately 3.3x reticle size to over 14x, the number of computing transistors in a single package will increase by roughly 48 times, and the total HBM bandwidth will rise by more than 34 times. Along with this trend, the power consumption of AI systems and advanced packages is growing rapidly. Relevant information shows that TSMC expects the power consumption of AI systems to increase by around 6 times in the next five years, and the power consumption of advanced AI packages will gradually rise from hundreds of watts to the level of approximately 4.1kW.
The AI chip industry is facing an increasingly intractable problem: chips can continue to be stacked, HBM can be further expanded, and packages can keep getting larger, but how to dissipate the heat generated by these transistors? Heat dissipation is evolving from a supporting issue for AI servers into a fundamental technical problem that determines whether next-generation AI chips can continue to scale.
AI Chips Enter the Thermal Scaling Era
Looking back at the semiconductor industry over the past few decades, performance improvement has been largely built on the concept of Scaling.
The first stage is Transistor Scaling. Transistors have been shrinking in size, evolving from 28nm, 16nm all the way to 7nm, 5nm, 3nm, 2nm and even angstrom-level nodes, allowing more transistors to be integrated per unit area.
However, as advanced process nodes become increasingly costly and the size of a single chip is constrained by limits such as the reticle size, the industry has begun to explore a second scaling path: Packaging Scaling. Chiplet, 2.5D, 3D IC, CoWoS, HBM and Hybrid Bonding all essentially solve the same problem: since a single chip cannot be scaled up infinitely, more chips can be packaged together.
The advanced packaging roadmap released by TSMC this year shows that by 2028, the 14x reticle size CoWoS plan will be able to integrate approximately 10 large computing chips and 20 stacks of HBM; by 2029, it will further break through the 14x reticle size limit, and simultaneously promote the SoW-X system-level wafer technology at a scale of around 40x reticle size.
However, after packaging continues to scale, a new physical constraint is becoming increasingly prominent. A single package can hold 10 computing chips and 20 HBM stacks, but this does not mean that all these chips can run at full load indefinitely. The core problem that ultimately determines the upper limit of performance becomes: can the heat generated by these transistors be removed in a timely manner?
Therefore, AI chips are very likely entering the third scaling stage: Thermal Scaling. It does not focus on "how many more transistors can be crammed in", but on how many watts of computing power can operate safely in the same area and volume.
This will bring a notable change to the performance competition of AI chips: we will not only pay attention to the number of transistors per unit area, nor only to TB/s bandwidth, bump pitch and Die-to-Die bandwidth, but also increasingly focus on an indicator that used to be more common in the thermal engineering field: W/cm², that is, how many watts of heat can be dissipated per unit area.
At a liquid cooling technology session during NVIDIA GTC 2026, Supermicro disclosed that the power density of server racks has increased rapidly from around 45kW in the past few years, and the Vera Rubin reference architecture can reach approximately 227kW per rack. According to its presented roadmap, the next stage may even reach the 400-500kW level, and gradually move towards megawatt-level racks in the long run. Supermicro also stated that the Vera Rubin architecture requires 100% liquid cooling.
A 600W GPU can use a large heat sink or cold plate to solve the heat dissipation problem, but if a single package eventually integrates ten large computing chips and 20 HBM stacks with power consumption rising to several kilowatts, the pressure on traditional heat dissipation paths will increase rapidly.
A 2.5D packaging thermal management study published in Applied Thermal Engineering in July 2026 constructed a realistic test structure containing epoxy molding compound and RDL, and studied the thermal coupling problem between Chiplets under a total power of about 948W and a heat flux density of 400W/cm². The paper listed "suppressing thermal crosstalk between different Chiplets" as an important goal for microchannel cooling. 【1】
As a result, heat dissipation issues are now continuously moving upstream to the manufacturing side. The current trend we are witnessing is: logic chip, memory, advanced packaging and equipment manufacturers are all bringing thermal management deeper into the semiconductor manufacturing process.
TSMC Bets on Microchannel,
Heat Dissipation is Getting Closer to Transistors
At SEMICON Taiwan 2026, James Chen identified Microchannel Cooling as one of the key technical directions for future high-power advanced AI packages. According to TrendForce reports, its research structure has demonstrated heat dissipation capacity of over 5kW, which is designed to cope with advanced AI packages with 4kW or even higher power consumption in the future.
Before introducing the microchannel cooling technology, let's first take a look at the pain points of current heat dissipation solutions.
Looking back over the past few decades, the basic idea of chip heat dissipation has not undergone fundamental changes: as much heat as possible is transferred from the silicon die to the outside, and then taken away by air or liquid.
Early data centers mainly relied on air cooling. After the chip generates heat, the heat is transferred to the air through thermal interface materials, heat spreaders and large heat sinks, and then discharged by fans and data center air conditioners.
As GPUs entered the era of hundreds of watts, direct liquid cooling began to gain rapid popularity. The heat dissipation path of a typical cold plate is roughly: silicon die → thermal interface material → heat spreader → thermal interface material → cold plate → coolant. Compared with air, liquid can withstand higher heat flux density, but there is still an unavoidable problem here: multiple material interfaces exist between the chip and the coolant, and each interface introduces additional thermal resistance.
IBM's research on advanced chip thermal management points out that traditional flip-chip packages usually dissipate heat from the back of the chip to the liquid cold plate, where a copper lid and thermal interface materials are required to provide mechanical protection and thermal conduction. Removing one thermal interface, or even bringing the cold plate closer to the bare die, can further reduce thermal resistance. IBM believes that interlayer cooling is the only volumetric cooling solution that can scale with the number of chips in a 3D chip stack (the figure a below), thus enabling extremely high 3D integration density. Since electrical interconnections exist near the water cooling fluid flowing in the chip stack cavity, specific sealing structures such as annular structures are required. Two-phase medium cooling can reduce this complexity. However, even under high pressure, the package needs to provide structural integrity support (Figure 4b, c). 【2】
a) Schematic diagram of interlayer cooling topology. b) Interlayer cooling module. c) Cross-sectional SEM image of interlayer cooling chip stack with TSVs embedded in the microchannel walls. (Source: IBM)
Therefore, from the perspective of engineering logic, since the external cold plate is still too far away from the transistors, we can continue to push the cooling structure closer to the silicon die.
Microchannel Cooling, as the name suggests, is receiving increasing attention under this exact background. Its basic principle is to process a large number of micron-scale flow channels near the chip, so that the coolant can directly pass through the high heat flux area, thereby shortening the distance for heat to transfer from the silicon die to the coolant.
In fact, microchannel is not a suddenly emerging new technology. As early as 1981, academia had begun to study the use of silicon microchannels to dissipate heat for chips with high heat flux density; in 2012, DARPA launched the ICECool program, which explicitly proposed to bring microfluidic cooling into substrates, chips and packages.
TSMC has also been conducting research for many years. In 2021, TSMC publicly disclosed a direct silicon water cooling solution for 3D IC: a silicon cover plate with grooves and grid-like cooling structures is bonded to the logic chip through low-temperature bonding, so that the cooling water is as close to the back of the chip as possible. Its test structure achieved a total heat dissipation power of over 2600W for a single SoC, corresponding to a power density of about 4.8W/mm²; after further adopting direct silicon backside water cooling, the demonstrated power density exceeded 7W/mm².
Therefore, the real innovation of Microchannel is not just "making the water channels smaller", but gradually moving the cooling structure that used to be located outside the chip into the package and even inside the silicon die. It is worth noting that Microchannel Cooling itself is not a single technical route. A key point of divergence lies in: how close to the silicon die should the microchannels be fabricated?
A study published in Communications Engineering, a Nature journal, in 2026 chose a relatively compromise route. The researchers did not fabricate the microchannels directly into the semiconductor substrate, but embedded them in the package substrate to form Direct-to-Package cooling. In the experiment, this solution handled a heat flux density of approximately 625W/cm² with only 2-4mL of coolant; compared with ambient air cooling, its junction temperature and thermal resistance were reduced by about 6-7 times, and compared with traditional liquid cooling heat sinks with TIM, it was also reduced by about 2-3 times. 【3】
Design and prototype of the direct-to-package cooling concept
The upper diagram shows the schematic of the proposed concept, and the lower diagram shows the assembly schematic of the microfluidic substrate and package prototype. (Source: Nature)
The other route is more radical — bringing the coolant even closer to the inside of the silicon die. In 2026, researchers from KAIST and Georgia Tech demonstrated a manifold microchannel cooler that embeds the microfluidic structure directly into the silicon thermal test chip, and is fabricated using a CMOS-compatible process at a temperature below 350℃. Under room-temperature single-phase water cooling conditions, this structure experimentally demonstrated a chip-level heat flux density of over 2000W/cm², while keeping the junction temperature below 100℃, with a total pressure drop of only 7.8kPa. 【4】
Nowadays, Microchannel is no longer just a research topic in the laboratory.
Microsoft has previously tested a set of on-chip microfluidic cooling systems, which etch tiny flow channels directly on the back of the silicon die to allow the coolant to flow as close as possible to the high-temperature silicon die. Laboratory tests show that according to different workloads and system configurations, the heat dissipation performance of this solution can reach up to 3 times that of traditional cold plates, and reduce the maximum temperature rise of the silicon die inside the GPU by 65%. Microsoft also uses AI to identify the distribution of hot spots in different areas of the chip, and designs the microchannels in a targeted manner to make the coolant flow more precisely to the high-heat areas. Microsoft has begun to study how to integrate this technology into actual chip manufacturing and data center deployment. In the next step, Microsoft plans to continue to cooperate with wafer manufacturing and chip partners to promote the application of Microchannel in its future self-developed chips.
Putting all these advances together, we can find that Microchannel is undergoing a very important change: it is gradually evolving from a "heat dissipation technology" to a "chip technology". Its industrialization also faces a very practical contradiction: the closer the coolant is to the transistors, the greater the heat dissipation potential; but every step the cooling system moves closer to the silicon die, the manufacturing difficulty, sealing requirements and reliability challenges will also increase accordingly.
If we continue to move forward along this route, the heat dissipation technology will roughly follow this evolution path: Air cooling → Cold plate liquid cooling → In-package microchannel → Direct silicon cooling.
HBM May Be More Sensitive to Heat Than GPUs
This trend is not limited to computing chips. After AI chips entered the 2.5D advanced packaging era, the heat generated by the GPU or AI ASIC itself is no longer the only heat dissipation problem that needs to be solved. The adjacent HBM may also become the heat dissipation bottleneck of the entire package.
HBM has a different structure. It is not an ordinary planar DRAM, but stacks multiple layers of DRAM dies vertically through TSV. As HBM gradually develops from the early 4-layer and 8-layer designs to 12-layer and even 16-layer designs, more and more active silicon dies are compressed in a very small three-dimensional space.
This means that the thermal problem faced by HBM has two dimensions: one is the heat generated by itself, and the other is the heat transferred from the adjacent computing chip. This is exactly a new problem brought by 2.5D advanced packaging — thermal coupling.
A 2026 study on 2.5D AI accelerator packaging released by Vinci points out that the power density of modern AI accelerators and HPC devices has often exceeded 1W/mm², and advanced 2.5D packaging is integrating high-power AI ASICs with 8-16 HBMs together. The resulting problem is thermal coupling: local hot spots formed inside the ASIC will transfer heat laterally to the adjacent HBM, raising the HBM junction temperature, which may further lead to reduced operating frequency, performance throttling and accelerated memory aging. A representative package demonstrated in the study consists of 2 AI accelerator ASICs and 8 HBMs, and under maximum power consumption conditions, the HBM junction temperature is close to 100°C. 【5】
After the HBM temperature continues to rise, it will also affect the operating frequency and system performance, accelerate memory aging, and ultimately affect the reliability of the entire AI accelerator. What's more troublesome is that as advanced packages continue to grow larger, this thermal coupling may become increasingly complex.
In the past, chip layout was more focused on signal distance, bandwidth, routing and package area. For example, placing HBM as close to the GPU as possible can shorten the signal path and improve interconnection efficiency. But from the perspective of heat dissipation, the closer two high-power chips are, the more serious the mutual thermal influence will be.
As a result, a new contradiction has emerged: electrical design expects chips to be placed closer, while thermal design may expect them to be farther apart.
Therefore, we can see that in the past, HBM competition was more focused on stack layers, capacity and bandwidth, and now memory manufacturers have begun to directly modify the internal structure of HBM to solve heat dissipation problems.
In May 2026, SK Hynix released the iHBM solution, which integrates an integrated cooling element ICE inside the HBM package (as shown in the figure below). This structure made of electrically insulating, high thermal conductivity silicon-based materials will form an additional thermal conduction path inside the HBM. SK Hynix stated that this solution can reduce thermal resistance by more than 30%, and plans to apply it to next-generation high-performance HBM products.
