HBM, New Great War

In the era when the number of parameters in AI models is growing exponentially, data centers are undergoing a profound transformation from "computing power supremacy" to "bandwidth-driven." In the wave of this computing architecture innovation, HBM (High Bandwidth Memory) is quietly emerging as the core infrastructure supporting large model computing.

Entering the "post-AI" era, HBM has not only become a standard component of high-performance AI chips (such as GPUs and TPUs) but has also evolved into a strategic high - ground for fierce competition among semiconductor giants.

Whether it's Samsung, SK Hynix, or Micron, these leading enterprises in the storage field have all identified HBM as the key engine for future revenue growth. They seem to have reached a consensus: to dominate the storage market, one must first master the core technology of HBM.

So, in this smokeless competition, which technologies are worth paying attention to? Let's conduct an in - depth analysis together.

Is customization the only way out?

Customization may be one of the ultimate destinations for HBM.

In fact, more than two years ago, when HBM was just emerging, SK Hynix and Samsung discussed the trend of customization. As cloud giants began to customize their own AI chips, the demand for HBM has only increased, making customization an inevitable requirement.

In August last year, Yoo Sung - soo, vice - president of SK Hynix, said, "All M7 (Magnificent 7, referring to the seven major technology stocks in the S&P 500 index: Apple, Microsoft, Google Alphabet, Amazon, Nvidia, Meta, and Tesla) companies came to us and asked us to make customized HBM (High Bandwidth Memory)."

In June this year, South Korean media reported that SK Hynix has locked in companies such as Nvidia, Microsoft (MS), and Broadcom, which are expected to become "heavy - weight customers" in the customized HBM market. Recently, it has reached agreements with Nvidia, Microsoft, and Broadcom to supply customized HBM and has started design work according to the needs of each company.

It is reported that SK Hynix determines the list of other customers based on the supply plan of its largest customer, Nvidia. Industry insiders said, "Considering SK Hynix's production capacity and the launch schedule of major technology companies' AI services, it cannot meet the needs of all M7 customers at once," but also pointed out that "considering the changes in the HBM market, several new customers may be added in the future."

SK Hynix also announced in April this year that it will shift to customization starting from the seventh - generation HBM (HBM4E) and has cooperated with TSMC. It plans to use TSMC's advanced logic process on the HBM4 base die. The first batch of customized HBM products is expected to be launched in the second half of next year.

Notably, since SK Hynix has successfully won several heavy - weight customers, its possibility of maintaining a leading position in the next - generation customized HBM market has greatly increased. According to TrendForce data, SK Hynix currently has a market share of about 50% in the HBM market, far exceeding Samsung Electronics (30%) and Micron (20%). In the latest HBM3E products, SK Hynix's market share is as high as 70%.

On the other hand, Samsung Electronics is reported to be discussing the supply of customized HBM with several customers. Since it has successfully supplied HBM3E to AMD, the world's second - largest AI chip manufacturer, the industry expects it to win customers for HBM4 and customized HBM soon. It is reported that Samsung is currently having specific consultations with customers such as Broadcom and AMD regarding HBM4 products.

Compared with the two South Korean manufacturers, Micron in the United States seems to be lagging behind. In June this year, Raj Narasimhan, senior vice - president and general manager of Micron's Cloud Memory Business Unit, said that the production plan of HBM4 will be closely integrated with the preparation of customers' next - generation AI platforms to ensure seamless integration and timely increase in production to meet market demand.

He said that in addition to providing the latest HBM4 to mainstream customers, customers are also seeking customized versions, and the development of the next - generation HBM4E is also in progress. Collaborating with specific customers to develop customized HBM solutions will further enhance the value of memory products.

At this time, many people may wonder what the benefits of customized HBM are and why DRAM manufacturers and cloud giants are so eager for it?

It should be clear that the key to customized HBM (cHBM) is to integrate the functions of the base die into the logic die designed by the SoC team. This includes controlling the I/O interface, managing the DRAM stack, and hosting the direct access (DA) port for diagnosis and maintenance.

This integration process requires close cooperation with DRAM manufacturers, but it gives SoC designers greater flexibility and stronger control over accessing the HBM core chip stack. Designers can more closely integrate memory and processor chips and optimize between power consumption, performance, and area (PPA) according to specific applications.

SoC designers can freely configure and instantiate their own HBM memory controllers and directly interact with the HBM DRAM stack through the DFI2TSV bridge. The logic die can also integrate enhanced functions, such as a programmable high - quality built - in self - test (BIST) controller, a die - to - die adapter (D2D adapter), and high - speed interfaces (such as the Universal Chiplet Interconnect Express UCIe), enabling communication with the processor chip in a complete 3D stack. Since the chip is manufactured using a logic process rather than a DRAM process, existing designs can be reused.

An important advantage of customized HBM is that it significantly reduces the delay introduced by the interposer in the data path and reduces the associated power consumption and performance loss. By reusing existing high - speed die interconnects (such as UCIe), it effectively reduces the distance between the memory and the processor chip. This flexibility can be applied in various scenarios, such as edge AI applications by cloud service providers where cost and power consumption requirements are extremely high, and complex AI/machine learning computing scenarios where systems pursue maximum capacity and throughput.

However, customized HBM also faces some challenges at present. The entire concept is still new, and the technology is in the early stage of development. Like all innovations, there are bound to be challenges ahead. Integrating the functions of the base die into the logic die means that end - users need to consider the entire lifecycle from the perspective of chip lifecycle management (SLM) - from design, trial production, mass production, to on - site application. For example, after wafer - level HBM chip stacking, the responsibility for screening DRAM cell defects will fall on the end - users. This brings some questions, such as how users should handle the specific DRAM algorithms recommended by suppliers and whether users can conduct comprehensive on - site HBM testing and diagnosis during planned outages.

At present, to successfully deploy customized HBM, a complete ecosystem is needed, bringing together IP providers, DRAM manufacturers, SoC designers, and ATE (Automatic Test Equipment) companies. For example, due to the large number and high density of interconnects, traditional ATE cannot be used for customized HBM testing.

In general, customized HBM has become a major trend. Whether manufacturers like it or not, it will occupy a very important position in the HBM4 standard.

Is hybrid bonding an unavoidable technical challenge?

In addition to customization, hybrid bonding is also one of the important development directions for future HBM.

Currently, as the number of stacking layers continues to increase, traditional soldering technologies are facing significant challenges. The flux currently used can remove metal surface oxides and promote solder flow, but its residues can cause problems such as increased stacking gaps and concentrated thermal stress, especially in the field of precision packaging such as high - bandwidth memory (HBM), where this contradiction is more prominent.

Companies including Samsung, SK Hynix, and even Micron are considering using hybrid bonding technology in the next - generation HBM.

Let's first understand the current bonding technology for HBM chips. In traditional flip - chip bonding, the chip is "flipped" so that its solder bumps (also known as C4 bumps) are aligned with the bonding pads on the semiconductor substrate. The entire component is placed in a reflow oven and uniformly heated to about 200ºC - 250ºC according to the solder material. The solder bumps melt, forming an electrical interconnection between the bond and the substrate.

As the interconnection density increases and the pitch shrinks to less than 50µm, the flip - chip process faces some challenges. Since the entire chip package is placed in the oven, the chip and the substrate will expand at different rates due to heat (i.e., different coefficients of thermal expansion, CTE), resulting in deformation and causing interconnection failures. Then, the molten solder will spread beyond its designated area.

This phenomenon is called solder bridging, which causes unnecessary electrical connections between adjacent pads and may cause short - circuits, leading to chip defects. This is where the TCB (Thermal Compression Bonding) process comes into play because it can solve the problems that occur in the flip - chip process when the pitch shrinks below a certain point.

The advantage of TCB is that heat is locally applied to the interconnection points through a heating tool head instead of being uniformly applied in a reflow oven (flip - chip). This can reduce heat transfer to the substrate, thereby reducing thermal stress and CTE challenges and achieving a more robust interconnection. Pressure is applied to the chip to improve the bonding quality and achieve better interconnection. The typical process temperature ranges from 150ºC - 300ºC, and the pressure level ranges from 10 - 200MPa.

TCB allows a higher contact density than flip - chip, reaching 10,000 contact points per square millimeter in some cases, but the main drawback of higher precision is lower throughput. While a flip - chip machine can achieve a throughput of more than 10,000 chips per hour, the throughput of TCB is in the range of 1,000 - 3,000 chips.

The standard TCB process also requires the use of flux. During the heating process, copper may oxidize and cause interconnection failures. Flux is a coating used to remove copper oxides. But when the interconnection pitch shrinks to more than 10µm, the flux becomes more difficult to remove and leaves a sticky residue, which can cause slight deformation of the interconnection, resulting in corrosion and short - circuits.

Fluxless bonding technology has emerged as a result, but it can only further reduce the pitch size to 20μm, up to a maximum of 10μm, and can only be used as a transitional technology. When the I/O pitch is less than 10μm, hybrid bonding technology is needed.

Hybrid bonding technology achieves DRAM chip stacking through direct copper - to - copper bonding, eliminating the need for traditional bump structures. This method can not only significantly reduce the chip size but also double the energy efficiency and overall performance.

According to industry insiders, as of May 7th, Samsung Electronics and SK Hynix are promoting the mass production of hybrid bonding technology for their next - generation HBM products. It is expected that Samsung will adopt this technology in HBM4 (the sixth - generation HBM) as early as next year, while SK Hynix may introduce it first in the seventh - generation product, HBM4E.

The current fifth - generation HBM, HBM3E, still uses thermal compression bonding technology, and the chips are fixed and stacked through heating, pressurization, and bump connection. Samsung mainly purchases TC equipment from its subsidiary SEMES and Japan's Shinkawa, while SK Hynix relies on Hanmi Semiconductor and Hanwha Semiconductor. Micron in the United States, which supplies HBM to Nvidia, also purchases equipment from Hanmi and Shinkawa.

With the initial opening of the hybrid bonding market, this technology is expected to trigger a major reshuffle in the semiconductor equipment field. Once successfully introduced, hybrid bonding may become the mainstream process for future HBM stacking.

To gain an early advantage, Applied Materials in the United States has acquired a 9% stake in Besi, a Dutch company that is the only one in the world with the mass - production capacity of advanced hybrid bonding equipment, and has taken the lead in introducing its hybrid bonding equipment into the system - level semiconductor market to seize the application opportunity.

Meanwhile, Hanmi Semiconductor and Hanwha Semiconductor are also accelerating the research and development of next - generation chip stacking equipment. These two South Korean manufacturers are not only rapidly advancing the research and development of hybrid bonding equipment but also actively developing fluxless bonding equipment to enhance their market competitiveness.

If customized HBM is a competition between DRAM manufacturers and cloud giants, then hybrid bonding is a game between DRAM manufacturers and bonding equipment manufacturers. As HBM officially enters the HBM4 era in the second half of this year, the attention paid to hybrid bonding may further increase.

What other new technologies are there?

Notably, in June this year, the Korea Advanced Institute of Science and Technology (KAIST), a South Korean national - level research institution, released a 371 - page research paper systematically depicting the evolution path of HBM technology from HBM4 to HBM8. The content covers improvements in bandwidth, capacity, I/O interface width, thermal design, etc., as well as packaging methods, 3D stacking structures, memory - centric architectures with embedded NAND storage, and even power - consumption control methods based on machine learning.

It should be emphasized that this document is not a product roadmap released by a commercial company but an academic prediction of the potential evolution of future HBM technology based on current industry trends and scientific research progress. However, it is enough for us to glimpse the possible development direction of future HBM.

Let's first look at the technical features of each generation of products from HBM4 to HBM8:

HBM4: The pioneer of customized design

As the beginning of the new - generation HBM technology, the biggest innovation of HBM4 lies in the customized base die design. By integrating an NMC (Near - Memory Computing) processor and an LPDDR controller, HBM4 enables direct access to HBM and LPDDR without CPU intervention. This design significantly reduces data transmission delay and improves overall system efficiency.

HBM4 supports a variety of flexible data transmission modes, including direct read and write between the GPU and HBM, data migration between HBM and LPDDR, and the GPU indirectly accessing LPDDR through HBM. The introduction of dual - command execution capability further improves multi - tasking efficiency and provides strong support for complex AI workloads.

HBM5: A breakthrough in 3D near - memory computing

HBM5 takes 3D near - memory computing technology to a new level. By integrating an NMC processor die and a cache die and using a dedicated TSV interconnection and power network, HBM5 achieves a high - energy - efficiency computing architecture. The introduction of a distributed power/ground and thermal TSV array effectively reduces the IR drop and improves heat dissipation efficiency.

Notably, HBM5 begins to introduce AI - designed agent optimization technology. Through intelligent algorithms, it optimizes the TSV layout and decoupling capacitor placement, significantly reducing power - supply noise - induced jitter (PSIJ). This innovation not only improves system stability but also lays the foundation for the intelligent design of subsequent products.

HBM6: Innovation in multi - tower architecture

The biggest highlight of HBM6 is the introduction of a quad - tower architecture. Four DRAM stacks share a base die, achieving an amazing bandwidth of 8 TB/s through 8,