Is NVIDIA also stepping into the 3D DRAM market?

According to South Korean media reports, Groq, an artificial intelligence semiconductor company that has been effectively acquired by NVIDIA, is planning to apply a 3D stacked DRAM packaging technology to its next-generation chips. This move is interpreted as Groq's attempt to significantly increase chip capacity by moving away from the current SRAM-centric chip design, where SRAM delivers fast speed but comes with small capacity.

The method adopted by Groq is called 3DS (3D Stacked Memory). 3DS is a high-capacity packaging solution that uses Through-Silicon Vias (TSV) to vertically connect DRAM chips. While it is similar to High Bandwidth Memory (HBM) in that both use TSV, unlike HBM which focuses on data transmission speed, 3DS is designed to expand capacity while retaining the existing DRAM interface.

An insider in the semiconductor industry stated that "Groq plans to introduce 3D packaging technology in its next-generation chips", adding that "this technology involves stacking DRAM instead of the existing SRAM". Industry insiders regard Groq's move as an effort to substantially increase chip capacity. SRAM has fast speed but limited capacity, while DRAM has relatively lower speed yet is more suitable for implementing high-capacity chips.

It can be seen from this news that 3D DRAM seems to be an inevitable development path.

The capacity growth of SRAM can no longer keep up with the expansion of models

As a competitor of NVIDIA in another track, the evolution of d-Matrix's technical route, from SRAM computing to 3D stacked DRAM, well explains the reason behind Groq's transformation. Because previously, Groq's solutions also used SRAM.

In a previous interview with Tae Kim, Sid Sheth, CEO of d-Matrix, said that Corsair, the company's first product, used SRAM as part of its computing architecture. According to Sheth, when d-Matrix selected the memory technology, it mainly considered two factors: first, the memory must support flexible programming and adapt to the ever-changing workloads in data centers; second, the calculation results must meet the predictability and accuracy requirements of digital circuits. Based on these considerations, the company chose SRAM.

The advantage of SRAM lies in its support for high-speed and low-latency data access, so it is suitable for AI inference tasks that require high interaction speed. However, as the scale of models and the length of context continue to grow, the capacity expansion capability of SRAM has begun to become a limiting factor.

Sheth pointed out that SRAM and logic chips use the same logic process, and its capacity growth is also affected by the slowdown of transistor scaling. In the interview, he presented a comparison that the capacity of SRAM can only increase by about 10% to 20% every two years, while the scale of models may double every six to nine months.

This means that relying solely on SRAM is increasingly difficult to meet the simultaneous demands of models for memory capacity and bandwidth.

The core of the problem is not just the high cost of SRAM, but that its capacity growth speed can no longer match the expansion speed of AI models. If the original architecture is continued to be used, even if the computing power is continuously improved, the performance may not be fully utilized due to insufficient memory resources.

Therefore, d-Matrix needs to find a new memory expansion method that retains the advantages of SRAM in high-speed access while introducing larger-capacity memory resources. This leads them to turn to 3D DRAM.

The so-called 3D DRAM refers to an architecture that stores bits along the vertical direction, which is similar to 3D NAND.

From the perspective of design, 3D DRAM stacking technology places DRAM on top of the processor, realizing a more compact design, thus improving data transmission efficiency. This in turn enables AI processors to handle larger models and generate results faster, paving the way for the continuous expansion of AI hardware in the future.

Some notable advantages of 3D DRAM stacking technology include improved bandwidth, reduced latency and higher energy efficiency, which are critical to the future development of artificial intelligence. In addition, space utilization is equally important, because stacked chips allow designers to integrate more functions in a smaller space, which is crucial for applications such as mobile devices and wearable devices. Artificial intelligence systems need to process an increasing amount of data, so reducing the data transmission distance is essential to improve performance and maintain high-quality results.

The key of 3D DRAM is not just to increase memory capacity

As for Groq's 3D DRAM solution, we cannot elaborate on it here because there is not much publicly available information. Considering that there were previous rumors that NVIDIA was considering acquiring or investing in d-Matrix, we will analyze their solutions in the following article to understand the multi-layered reasons behind this transformation.

It is introduced that in the field of 3D DRAM, d-Matrix's solution is to carry out 3D stacked packaging for customized DRAM dies and SRAM computing dies.

Sheth specially emphasized that this is not a traditional HBM solution. According to his introduction, Raptor uses two different dies: one is a logic computing die, and the other is a DRAM die. The former is manufactured by TSMC, and the latter is manufactured by Powerchip Semiconductor Manufacturing Corporation (PSMC) in Taiwan, and then the two are integrated by the packaging factory using Chip-on-Wafer (CoW) technology.

The significance of this architecture is first reflected in the expansion of memory capacity. By adding a layer of DRAM, d-Matrix can provide more storage space on the basis of retaining the original SRAM computing architecture.

But in Sheth's view, what is more noteworthy is the change in data access methods brought by 3D stacking.

Although the traditional combination of GPU and HBM has achieved high memory bandwidth through advanced packaging, the data connection between the computing chip and HBM is still limited by the physical layout and connection interface. Sheth vividly described it as only being able to use the "beachfront" area of the memory or GPU to access data, that is, the interface actually used for connection and data transmission is limited.

In d-Matrix's solution, the computing die and the DRAM die are stacked vertically, and the DRAM can use a larger surface area to establish data connection with the computing part, allowing data to be directly transmitted downward to the computing units.

In other words, 3D DRAM is not just about placing more memory near the chip, but trying to change the connection relationship between memory and computing, creating conditions for improving data transmission bandwidth.

This is also why d-Matrix regards it as an important breakthrough for the next-generation inference architecture: it hopes to solve the problems of insufficient memory capacity and limited data access bandwidth at the same time, rather than only improving one of them.

It should be noted that the above is Sheth's explanation of the architectural design idea, which does not mean that 3D DRAM has surpassed HBM in all performance indicators. The actual performance still depends on the specific design, packaging implementation and product verification.

If we only understand 3D DRAM from the perspective of memory capacity and bandwidth, it is still not enough to explain why d-Matrix adopts this technical route. In the interview, Sheth put forward an important distinction: in the competition of AI inference chips, there is a trade-off between throughput and interactivity.

Throughput focuses on how much work can be completed per unit of time; interactivity focuses on how fast the system can give a response after the user sends a request. The two are related, but they are not the same indicator.

The HBM solution represented by GPU has advantages in high-throughput tasks and is suitable for processing a large number of parallel workloads. d-Matrix does not think it needs to replace GPUs in all scenarios, but hopes to serve another type of demand: AI applications that are particularly sensitive to response latency.

Sheth said that d-Matrix adopts the combination of SRAM and 3D stacked DRAM, prioritizes the optimization of memory bandwidth, and then the memory capacity, hoping to occupy a different position on the trade-off curve between throughput and interactivity.

This positioning is closely related to the changes of AI applications.

In batch processing tasks, the system can improve overall efficiency by processing a large number of requests at one time. But in scenarios such as programming agents, cybersecurity agents and voice interaction assistants, users or other agents need to continuously interact with the model, and the waiting time will directly affect the experience.

Especially for agent applications, their working process is often not to generate results at one time, but to repeatedly perform inference, call tools, read information and make next-step decisions. The lower the response latency, the smoother the entire interaction process will be.

Therefore, what d-Matrix bets on is not all AI computing tasks, but inference loads that require fast response and continuous interaction.

Sheth predicted in the interview that 70% to 80% of AI applications in the future will be latency-sensitive. This is his judgment on market development, not a verified industry statistical data.

Final Notes

Although this is a highly promising technology, as Applied Materials stated in its blog, the speed of DRAM is almost 1000 times that of NAND, and the realization of its high-speed performance benefits from the use of high-mobility silicon substrates as the initial material of the channel. The high-speed performance of DRAM also comes from the fact that charges can move quickly in capacitors. With the continuous expansion of DRAM in the vertical direction, new material innovations are needed to achieve high-mobility and ultra-low-defect channels.

Experts from Rambus also pointed out in a blog that when memory and processors are stacked together, yield and reliability are also issues that need to be considered. As he said, in a stacked structure, a defect in any layer may affect the entire stacked structure, thus reducing manufacturing yield and increasing costs. This highlights the importance of repair and redundant resources, which can improve manufacturing reliability, thereby improving yield and reducing production costs.

Designing 3D stacked memory is also more challenging, because engineers must consider how memory interacts with processors and accelerators to optimize performance and energy efficiency. These challenges, coupled with higher packaging costs, mean that although stacked memory is a powerful solution for artificial intelligence, its effective implementation requires careful planning and substantial investment.

The semiconductor industry has been working hard to address these challenges and has made a series of progress that will help realize the vision of future super chips. Built-in Self-Test (BIST) measures, advanced EDA tools, optical interconnection and AI-aided design are all examples of continuous efforts to address these challenges and improve chip efficiency. As the industry accumulates more experience in stacking technology, challenges related to the debugging and verification of vertically integrated chips will be solved through new methods, thus reducing costs and improving yield.

Finally, it is worth mentioning that many domestic manufacturers are also competing in the 3D DRAM track. Do you know who they are? Everyone is welcome to have a more extensive discussion on the current status and future of this technology in the comment area!

This article is from the WeChat Official Account "Semiconductor Industry Watch" (ID: icbank), written by the editorial department, and authorized for release by 36Kr.