Tang Jie's team from Tsinghua University unveils the full landscape of large model memory.

In recent years, the breakthroughs of Large Language Models (LLMs) have been largely attributed to the Scaling Laws (parameters, data, and computing power).

However, in the process of model evolution, an equally important dimension is becoming increasingly critical — Memory.

From the KV Cache generated along with computation in Transformer, to the implicit state compression of linear RNN/SSM, and then to the recently popular explicit and persistent memory modules such as Titans, TTT (Test-Time Training) and Engram, the underlying memory mechanism of large models is undergoing a profound paradigm shift.

Based on this direction, the team of Professor Tang Jie from Tsinghua University, the National University of Singapore (NUS) and Bosch AI jointly released a comprehensive survey on the cutting-edge of large model memory architecture. The paper not only proposes for the first time a three-dimensional memory taxonomy (Representation, Update Dynamics, Persistence), but also integrates scattered frontier works into a coherent theoretical framework, pointing out the direction for the design of the next-generation LLM that is more efficient and capable of lifelong learning.

It needs to be specially clarified here that the "memory" discussed in this paper specifically refers to the architectural-level internal memory of the model — that is, the representation, compression and state update of historical information by the model at the underlying network level; rather than the equally popular agent-level memory realized through external Agent frameworks (such as external dialogue logs, local file retrieval engines, etc.).

Background: Paradigm Shift in Large Model Memory

Traditionally, the internal "memory" of LLMs is often implicit and exists as a by-product of computation. The self-attention mechanism of Transformer provides a content-based instantaneous working memory (KV Cache), but its computation and storage overhead increases quadratically with the sequence length, and is severely limited by the context window.

Recent research trends show that simply "extending" the context window is no longer enough. To endow models with persistent learning capabilities across sessions, researchers have begun to introduce explicit, independently addressable and persistent memory mechanisms (such as parameter modules that can be updated during the inference phase).

As the authors pointed out in the paper: Memory is rising from being a mere by-product of computation to a first-principles dimension in large model architecture design.

△

Core Framework: Three-Dimensional "Taxonomy" of LLM Memory

Facing the massive and fragmented architectural innovations (such as attention cache, recurrent state, test-time adaptation, retrieval module, conditional parameter routing, etc.), this survey proposes a unified three-dimensional architectural-level classification framework, to clarify the boundaries and commonalities of different memory mechanisms:

△

Representation: Implicit vs. Explicit

Implicit Memory: Deeply bound to the model's forward computation, with no independent control interfaces (such as KV Cache of the attention mechanism, hidden states of RNN/SSM).

Explicit Memory: Equipped with independent storage components and clear read-write semantics, not limited to the standard forward computation process (such as external Datastore, test-time updatable parameters, memory slots, etc.).

Update Dynamics: Offline vs. Online

Offline: Updated only through gradient descent during the training phase (such as pre-trained parameters, traditional MoE).

Online: Supports dynamic updates during the inference phase, endowing the model with the ability to absorb new knowledge without re-fine-tuning (such as surprise-driven update in Titans, TTT).

Persistence: Short-Term vs. Long-Term

Short-Term: Dissipates with the end of the local context window or inference session (such as Attention).

Long-Term: The information or its influence can persist across local contexts and even across independent sessions.

△

In-depth Analysis: Design Philosophy of Implicit and Explicit Architectures

Under the unified taxonomy, the authors take the representation form as the entry point, and conduct in-depth technical decomposition of the two mainstream camps at present:

Implicit Memory: A "Passing Traveler" in the Computational Flow

Although implicit memory is extremely efficient, its essence is a "computation-bound transient state".

Attention Mechanism: As a point-like transient working memory with content addressing, although sparse attention or sliding window (such as StreamingLLM) alleviates the computational overhead, its capacity is still limited by short-term storage.

Recurrent Sequence Memory: Including SSM architectures such as Mamba, and linear attention models such as RWKV and Gated DeltaNet. They compress historical information into structured hidden states. Recent evolution directions (such as Kimi Delta Attention) even upgrade simple scalar decay to channel-level dynamic state editing.

Bottlenecks and Limitations: The capacity of implicit memory is strictly limited by the hidden layer dimension and window, and usually lacks independently controllable read-write semantics.

△

Explicit Memory: An Independently Addressable "External Brain"

This is the current most cutting-edge research hotspot, aiming to endow LLMs with longer-lasting vitality and adaptive capabilities:

Parametric Memory Modules: Such as Titans and TTT-E2E, which solidify memory in specific parameter blocks and perform high-frequency updates during Test-Time. This design that decouples "context storage" from the "general knowledge backbone" greatly mitigates long-term catastrophic forgetting.

Lookup-based Memory: Such as Engram and kNN-LM. Information is stored in non-parametric slots or databases, and retrieved through sparse routing during inference. This approach greatly expands the upper limit of the model capacity and gets rid of intensive computational overhead.

Conditional Parameter Memory (MoE): MoE architectures such as Mixtral are essentially a variant of explicit memory — the Router performs context-dependent retrieval and addressing in the parameter space.

△

Towards Integration: Hybrid Architectures and System-level Challenges

As can be seen from the above, no single memory paradigm can solve all problems: Attention has high recall rate but extremely high video memory consumption; SSM is computationally efficient but may lose fine-grained information; parametric memory has strong adaptability but faces optimization drift.

Therefore, Hybrid Memory Architectures are becoming the mainstream.

Some well-known large models have adopted this approach: for example, Kimi Linear interleaves Kimi Delta Attention (KDA) and Multi-head Latent Attention (MLA) at a ratio of about 3:1, Qwen3-Next also mixes Gated DeltaNet and full attention layers at a similar ratio; models such as Jamba and Samba represent the earlier inter-layer combination of Attention-SSM. These designs prove the complementarity of different memory paths, but their allocation ratio is usually still pre-fixed by the architecture.

A more cutting-edge direction is adaptive memory routing: for example, AMOR decides when to enable high-cost Attention refinement according to the model's uncertainty; HAM uses prediction error to determine which Tokens are worthy of entering the high-fidelity sparse KV Cache, and the rest of the information is processed by the high-compression-rate recurrent state.

However, integrating multiple types of memory does not mean simply stacking modules. This field still faces several key challenges:

What to write and when to write: Surprise degree, entropy or prediction error cannot necessarily accurately measure the long-term value of information; tokens that seem irrelevant at present may become critical much later.

Balance between Stability and Plasticity: Parameter updates during test time can quickly absorb new information, but may also bring memory drift, old knowledge interference and error accumulation. There is still no mature solution to achieve reliable writing, forgetting and rollback when necessary.

Capacity, Precision and System Cost: High-fidelity KV storage continuously consumes video memory and bandwidth, while fixed-size recurrent states may over-compress history. PagedAttention, KV Cache quantization and memory integration need to be co-designed with the model structure.

How to truly measure "good memory performance": Long context length is not equivalent to effective memory. In addition to long-range tests such as RULER and LongBench, future evaluations also need to distinguish between recall and reasoning, and measure persistence, anti-interference, update consistency and computational efficiency.

Conclusion and Future Directions

Research on memory architecture is receiving more and more attention. At the end of the paper, the research team proposes several enlightening directions for future work:

Unified Memory Theory: Establish a theoretical framework that can quantify Attention, SSM and retrieval mechanisms from a unified perspective of information compression and state transition.

Lifelong Parametric Memory: Break through the short-term adaptability of TTT, and enable the model to achieve long-term stable incremental learning without full fine-tuning.

Adaptive Memory Allocation: Abandon static architecture design, introduce a learnable controller, and dynamically determine the storage medium (Cache or State) and persistence level of information.

Algorithm-Hardware Co-Design: As the context extends to millions of Tokens, memory reading and writing must be deeply coordinated with the GPU memory hierarchy.

From "Passive Forgetting" to "Active Control", the memory mechanism of large models is reshaping the computational boundary of AI. Whether you are a system engineer focusing on underlying Kernel optimization, or a researcher dedicated to exploring next-generation model architectures, this survey will provide you with a theoretical map.

Paper link: https://arxiv.org/abs/2607.25380

This article is from the WeChat public account "QbitAI", written by the LLM Memory Team, and republished by 36Kr with authorization.