Three key pieces of the RSI puzzle, MemT 1.0 fills in the "memory" gap at the model layer

Recently, the AI community has been collectively fixated on three letters: RSI.

In Hinton's RSI paper, among its 22 named authors are Bengio, Pachocki (Chief Scientist at OpenAI), Jack Clark (co-founder of Anthropic)... nearly half of the top talents in the global AI field are on the list.

As everyone turns their attention to AI self-improvement, a more fundamental question worthy of in-depth exploration emerges:

How can one improvement become the starting point for the next?

Valid methods need to be preserved, detours that have been taken need to be identified, and the very entity that undertakes this task is essentially memory.

The conventional line of thinking is straightforward: directly connect an experience base externally to the AI, so that all operating methods and historical lessons can be saved and retrieved.

The exploration of memory tensor in this regard presents a completely different solution: memory-native architecture.

What does memory-native mean? It means making the retention, update and invocation of experience a capability that the model can learn.

When memory takes root and sprouts inside the model, it forms a memory-native model.

Its latest open-source released MemT 1.0 series offers a very specific path:

It encodes experience into internal dynamic states, then learns through training how to write, update and read these states, and finally makes these states participate in subsequent calculations via the memory branch.

Compared with external memory, it not only retains full capability performance, but also has a much higher ceiling for future expansion.

This also marks the first time that a memory-native model has achieved engineering verification at a scale of nearly 300 billion parameters.

(Paper link: https://arxiv.org/abs/2610.26825)

Impressively, memory is intangible, but it has truly become the fundamental foundation for the model's growth.

From the early requirement of letting models memorize information to making good use of experience, the model's demands for memory keep escalating, which also pushes memory from behind the scenes to the front stage step by step.

Embed Memory into the Model

MemT 1.0 directly rolls out two models this time:

MemT-1.0-Nano has a total of 36B parameters with 3B activation, and its base model is Qwen3.6-35B-A3B; MemT-1.0-Flash breaks through to a higher level, reaching 288B parameters with 17B activation, and its base model is DeepSeek-V4-Flash after further post-training.

Their common change is that a parallel memory calculation path alongside the original attention mechanism is added to the selected Transformer modules.

We use a simple example for illustration.

For example, when facing the same new task, the approach of external memory is to put the original memory text, retrieved evidence and even abstract methods back into the context, so that the model can reuse experience.

A learning mechanism can also be introduced into the retrieval and sorting process.

While memory-native models like MemT 1.0 choose to go deeper into the interior of the model:

After being encoded, experience becomes an updatable matrix state; when the model generates answers or actions, it reads the state through a dedicated memory branch, then fuses the read result with the original attention output to affect subsequent generation.

The difference lies in how experience is represented and what calculation process it uses to play a role.

The benefits of this approach are also obvious: letting read and write operations participate in training can bring smoother method reuse. But there is also a tradeoff: if you want to perform precise deletion, interpretation and recovery, additional mechanisms need to be added.

External text is definitely more convenient for inspection and traceability, and at present, both approaches have their own advantages and disadvantages.

To observe the real difference, we need to see how the model runs after using experience.

Let's first look at a scenario familiar to programmers: code modification.

Receive bug report → locate code → start modification, this is the basic workflow. But if the problem is not reproduced at the beginning and the boundary conditions are not confirmed properly, the modification will be full of loopholes instead of being fixed, and the error will be passed on from one generation to another.

Take SWE-bench Verified as an example, one of its tasks involves the parse_http_date function of Django, which needs to fix the parsing logic for two-digit years.

The research team compared the execution trajectories under three settings: no experience, text-based experience, and memory-native.

Without experience, the Agent will quickly start modifying the parsing logic after finding the bug, and targeted reproduction and verification will only appear in subsequent steps. The two groups with experience will first build reproduction tests, then modify the code.

Among them, MemT-1.0-Nano also re-checked a boundary condition after the first modification, corrected the implementation, and performed additional verification.

It can be clearly seen that experience changes the order of actions, and also participates in inspection and correction within the task.

Comparing on a larger sample, across 500 SWE-bench Verified tasks with each task running three times independently, MemT-1.0-Nano has an average success rate of 69.73% at key breakpoints, the no-experience baseline is 66.07%, and the text experience group is 69.13%.

The proportion of tasks that succeeded in all three runs increased from 54.00% in the no-experience group to 62.80%, and the text experience group reached 61.80%.

In the SWE-bench Multilingual task, under the same trigger setting, the average success rate of the memory-native group is 63.56%, the no-experience group is 60.00%, and the text experience group is 63.00%.

In other words, these procedural experiences can be preserved through memory-native architecture, with performance close to explicit text-based experience. Its advantages are more reflected in the average success rate and the reliability of repeated execution.

To be more intuitive, let's look at a pixel-style maze exploration demo.

In the same maze with the same movement speed, the left side is the Agent without memory-native capability, and the right side is MemT 1.0 with memory-native capability.

In the first round, both sides need to explore unknown routes, and both will walk into dead ends.

The difference appears after the end of this round: the successful routes and the forks that need to be avoided are sorted and archived, and loaded into MemT 1.0's memory to provide reference for the next round of exploration.

In the second round, starting from the same starting point, the left side will still repeatedly test at the forks, while the right side uses the experience from the previous round to avoid the dead ends that have been passed and reach the end along the effective route.

The path formed in the previous round directly affects how the next round proceeds, which is the specific manifestation of the value of memory-native architecture.

Let's look at another information update scenario that is closer to our daily office work.

The exhibition was originally scheduled for 3 p.m. on Saturday, and was later officially changed to 4 p.m. on Sunday. After that, someone suggested changing it to Monday, but the decision was not finalized.

This kind of repeatedly changing information increment is actually very tricky for AI.

It is not enough to remember the latest date, it also needs to distinguish between formal decisions and casual suggestions, and retain the content that has not changed.

After the experiment went through multiple rounds of conversations, MemT-1.0-Nano still successfully captured all the information, answered the original time, the officially revised time, and confirmed that other arrangements had not changed.

In the comprehensive evaluation of 11 memory benchmarks, the performance of Flash and Nano is equally outstanding.

Their average scores are 50.6 and 47.0 respectively, ranking top two among all parameterized memory methods participating in the evaluation, leading far ahead of other models.

But to be honest, text memory systems still have overall advantages, especially in detail recall of long-term history.

What MemT 1.0 delivers today is only the initial computational implementation of the new memory-native idea, but its actual performance is already comparable to text memory, and it has unlimited potential for further development.

What makes memory-native architecture "native"?

Then the question arises: experience is not stored in the task context, so where does the model use it from?

This is exactly the core of MemT 1.0's architecture.

The native path provided by MemT 1.0 is to first encode relevant experience into internal dynamic states. When generating subsequent actions, the current query reads the state through a dedicated memory branch, then fuses the result with the original attention output.

In the specific parameterized implementation, MemT 1.0 retains the original calculation process of the base model, and adds a native memory branch in the selected Transformer module, which consists of three layers when disassembled:

• Backbone weights: carry the existing general capabilities of the model, and remain frozen when training memory;
• Hypermemory: trainable parameters such as read/write projections and gating determine the specific way to encode, update and read memory;
• Local memory: dynamic matrix state that stores newly added information in interactions.

After this differentiation, the model can not only carry new experiences, but also does not need to fine-tune the entire backbone every time, only the dynamic state corresponding to the task trajectory changes.

The read and write operations of this mechanism are also carefully designed.

When writing, the hidden state passes through projection to generate the key and value of memory; erasure, writing and forgetting gating jointly adjust the associations in the existing matrix; when reading, the current input is mapped to a memory query to read relevant information, which then participates in subsequent generation after normalization and output gating.

The update mechanism adopts token-by-token generation based on Gated DeltaNet-2, with stabilization processing.

(Unconfirmed news: the author of Gated DeltaNet-2 also participated in relevant discussions)

It is triggered after the end of each round, adjusting existing memory associations through gating, and adjusting the state according to the token order within the round.

Ablation experiments prove this point: after replacing token-by-token update with full-round aggregation, the average memory benchmark score of Nano drops from about 47.0 to 41.0.

The 6-point gap fully shows that continuous token-level update can better handle memory decay and correct intra-round associations in time.

But what if the model capacity is insufficient when the memory content keeps expanding?

At this time, MemT 1.0 proposes the MoS (Mixture-of-Storage) solution based on MoE.

MoS extends one memory-native module to multiple memory experts, and separately learns storage and usage routing. It adopts dense fusion: all memory experts participate in the calculation, and the routing weight determines their respective contribution.

This is obviously different from the sparse expert routing of the base MoE model.

However, structure is only the starting point.

In order to make the model truly learn to use this path, the team constructed a 34-billion-token memory training corpus, including 20 billion tokens for continued pre-training and 14 billion tokens for post-training.

The format of continued pre-training is unified: first split long text into consecutive fragments and write them into