The memory anxiety of large models has found a solution in USB drives
Wait, isn't Flash supposed to be really slow? How did SanDisk and SK Hynix end up using it to power large models???
IEEE has recently published a counterintuitive new approach: The memory anxiety of large models may be alleviated by the exact same technology used in USB drives.
That's right, we're talking about NAND Flash.
In many people's minds, the biggest features of Flash are low cost, large capacity, and non-volatile data retention when power is cut off. But when it comes to speed, especially for driving large model inference, it never seems like a legitimate solution.
But now, SanDisk and SK Hynix are pushing a new innovation: High Bandwidth Flash, abbreviated as HBF.
Their idea is straightforward: apply the advanced packaging and chip stacking methods used for HBM (High Bandwidth Memory) to NAND Flash.
In other words, this means enabling storage-focused flash memory to participate in high-speed data delivery for AI inference workloads.
According to public information from SanDisk, HBF is projected to support stacking of up to 16 NAND Flash chips, with a target maximum single-stack capacity of 512GB and a target peak read bandwidth of 1.6TB/s.
In subsequent second and third-generation roadmaps, the read bandwidth will be further increased to 2TB/s and 3.2TB/s respectively.
While this figure still cannot match top-tier HBM performance, it has completely broken away from the "sluggish storage" stereotype of traditional flash memory.
Behind this technological breakthrough, lies an urgent supply shortage crisis that the global storage industry needs to resolve.
Just a few days ago (Beijing time on July 11), SK Hynix CEO Goh Ro-jung publicly stated during the company's Nasdaq debut that the global storage industry is expected to face the worst supply shortage in history in 2027, a statement that immediately sparked widespread attention in the capital market and industry circles.
Against this backdrop, SK Hynix partnering with SanDisk to launch HBF is likely their technical response to ease the storage supply crunch.
Let's take a closer look.
Wait, Flash is supposed to be slow — how can it be used for AI?
Here's the obvious question: Isn't Flash famously slow for writes? How can it suddenly be suitable for AI workloads?
Jim Handy, Director of semiconductor market research firm Objective Analysis, noted in an IEEE Spectrum interview that most people's first reaction is exactly this: How does this make sense? Flash is ridiculously slow.
But the key point is that Flash's slowness is primarily limited to write operations.
NAND Flash stores data using charges in floating-gate transistors, organized by blocks and pages. This mechanism allows it to retain data even after power loss without requiring constant refreshing like DRAM, making it naturally ideal for long-term storage, combining high density with low cost advantages.
The tradeoff, however, is that writing data requires moving charges in and out of the insulated gate, a process far slower than charging and discharging a capacitor.
Therefore, in traditional computing paradigms, Flash has never been considered in the same category as high-speed memory like DRAM or HBM.
But when we only look at read performance, the picture changes entirely.
Under the latest ONFI 6.0 flash memory interface standard, the theoretical maximum bandwidth per single die can reach 4.8GB/s.
On its own, this number doesn't seem particularly impressive.
But a single DDR5 DIMM can peak at 70.4GB/s, and a single HBM4E stack can reach up to 3.6TB/s — roughly a 750x performance gap compared to a single NAND flash chip.
What HBF aims to do is use 3D packaging and vertical stacking to combine multiple NAND Flash dies together, drastically increasing the total read bandwidth.
Hoshik Kim, Senior Vice President of Memory System Research at SK Hynix, pointed out: "By applying advanced 3D packaging and vertical stacking technologies to NAND flash memory, HBF can achieve bandwidth far exceeding standard NVMe storage."
Just as HBM stacks DRAM chips, HBF stacks NAND flash dies to create chips with extremely high storage density.
Their goal was never to magically make a single Flash chip faster, but to multiply read channels through stacking and packaging.
HBF is not meant to replace HBM — it has found its own unique niche
The truly interesting aspect of HBF is not that it intends to replace HBM.
Instead, it has identified a scenario perfectly suited for Flash: AI inference.
Training and inference have fundamentally different memory requirements.
When training large models, the system must continuously read weights, calculate errors, and update weights through backpropagation;
This process involves heavy read operations paired with equally heavy write operations, which is very unfriendly to Flash;
Because Flash has slow write speeds, excessive writes also introduce issues with lifespan and wear management.
But the inference stage is completely different.
At this stage, the model has completed training, and its weights are largely frozen. Most of the time, the system only needs to quickly read these massive model weights, rather than rewriting them frequently.
This falls exactly within the capabilities where Flash excels.
SK Hynix Senior Vice President Kim stated: "In inference environments, massive read-intensive data such as static multi-billion parameter model weights or precomputed KV caches can be reliably stored at the HBF tier."
This way, expensive and capacity-limited HBM can be freed up to serve exclusively as a high-speed scratchpad.
This architecture is somewhat like adding a "super-large capacity read-only memory pool" to AI systems.
HBM continues to handle the most urgent, high-frequency high-speed computing data exchange.
HBF takes on the task of storing the massive, rarely modified model parameters.
For current large model inference workloads, this approach has significant practical value.
Because the challenges facing large model inference today extend far beyond insufficient computing power — multiple pressures including insufficient memory capacity, inadequate bandwidth, excessive energy consumption, and prohibitive deployment costs are all compounding together.
Especially as model sizes continue to expand, limited per-GPU VRAM capacity combined with the high cost of HBM has become a major bottleneck restricting development.
In many cases, it's not that the GPU can't compute fast enough, but that the model can't fit in memory, or the data can't be fed through quickly enough.
If HBF can strike a balance between capacity, bandwidth, and cost, it could enable much larger models to run at significantly lower costs.
512GB per stack: AI servers might need fewer GPUs
The first-generation HBF target parameters announced by SanDisk reach a maximum 512GB capacity with 1.6TB/s read bandwidth.
This combination is critical — HBM's strength is extreme speed, but it comes with high cost and limited capacity; HBF, while slightly slower, offers far greater capacity at a much lower per-unit cost.
This means that in the future, when deploying large model inference, AI servers may not need to fit all weights into HBM.
Some colder, more static, read-heavy data can be stored in HBF.
△
The potential benefits of this approach include:
First, alleviating the HBM capacity bottleneck.
Second, reducing the need to deploy more accelerator cards just to fit the model.
Third, lowering the overall cost and energy consumption of inference systems.
Kim's assessment is that: HBF and HBM are not competitors, but complementary tools.
He noted: "HBF can effectively alleviate the severe capacity bottleneck facing HBM without sacrificing data delivery speed, thereby potentially reducing the number of accelerators required to run large-scale models."
Building on this, he expects the solution to also improve energy efficiency and reduce costs, further helping data centers scale out their AI inference hardware infrastructure.
This is the most compelling industry logic behind HBF: It doesn't aim to turn Flash into HBM, but to place different types of data in AI systems in the most appropriate storage tier for their needs.
Hot data remains in HBM; large-capacity, static, read-intensive data moves to HBF; lower-speed, cheaper data continues to reside in SSDs or other storage.
If this tiered architecture is widely adopted, a new layer will be added to the memory hierarchy of AI inference hardware.
Large-scale deployment is still distant, but a new signal for large model memory architecture has emerged
Of course, HBF is not yet a mature product today.
According to IEEE Spectrum reports, HBF will take at least another year before shipments begin, and large-scale deployment could still be several years away.
On February 25 this year, SanDisk and SK Hynix jointly held a launch event, announcing that they would promote HBF standardization under the Open Compute Project (OCP).
As an important open industry organization in the data center hardware space, OCP has led the development of numerous specifications for servers, racks, power supplies, and accelerators, including the DC-MHS server specification and OAI-OAM accelerator specification.
But the HBF standard is still in progress, with no official release date confirmed yet — which means HBF won't immediately transform the AI server market in the short term.
It's more like an early emerging signal: AI inference is forcing the storage and memory architecture to redefine their division of labor.
In the past, the focus of large model hardware was centered on GPUs, HBM, and interconnection technologies.
But as models grow larger and inference requests multiply, the bottleneck has expanded from "how fast can it compute" to "can the model fit in memory," "can data be read fast enough," and "can we afford the power bills."
HBF is stepping directly into this gap.
The AI memory war won't be fought by HBM alone
Interestingly, SK Hynix itself is one of the biggest winners in the HBM market.
HBM is currently generating record revenues for the company, so logically, launching a cheaper alternative technology would seem counterintuitive to business sense.
But from a system perspective, HBF won't necessarily cannibalize HBM's market share.
What AI servers need in the future is not a single type of memory, but a tiered memory hierarchy.
Let HBM handle ultimate performance, HBF handle high-capacity high-bandwidth reads, and SSD handle low-cost mass storage.
If this combination works, it will make AI inference systems far more scalable.
To put it simply: Not every piece of model data deserves to live in HBM's "luxury mansion." Some data only needs a location with sufficient capacity, fast enough read speeds, and lower cost.
That's exactly the position HBF aims to occupy.
From NAND Flash in USB drives and SD cards to high-bandwidth model weight pools in AI servers, this seems like a huge leap, but the problems of large models are crystal clear: parameters keep growing, inference costs keep rising, and HBM supplies keep getting tighter.
And the solution doesn't necessarily have to come from more expensive memory — it might just come from repackaging a familiar, long-established technology.
References:
[1]https://spectrum.ieee.org/high-bandwidth-flash?itm_source=homepage&itm_medium=hero&itm_campaign=hero-2026-07-15&itm_content=hero6