Karpathy said it will take another ten years, but this path is already crowded with people.

Recently we have reported a number of startups and research achievements focused on the "Continual Learning" direction, such as "Mind Lab Releases Latest LoRA Progress Consecutively, A New Paradigm for 'Continual Learning' of Large Models Emerges", "Breaking the Shackles of KV Cache, Compressing Long Context into Weights, Is There Hope for Continual Learning Large Models?", "ICML 2026 | Breaking Limits! The University of Hong Kong Proposes the First Continual Learning Architecture Adapted to 300+ Tasks to Solve the Catastrophic Forgetting Problem", "Harness That Enables DeepSeek to Self-Evolve! The Author of LlamaFactory Open-Sources a New Tool: Automatically Build Agents for Only 0.2 Yuan", "When 'Scaling Up' Is No Longer the Only Path, Another Domestic Model Goes Open-Source"...

Yes, the coverage is quite intensive, and "Continual Learning" is also becoming one of the most frequently encountered keywords for us. Behind this lies a judgment that has been mentioned repeatedly.

In October 2025, Andrej Karpathy said on Dwarkesh Patel's podcast: The current large models "have no continual learning. You can't tell it one thing and expect it to remember." He believes that it will probably take another ten years to fix these cognitive defects. Two months later, he included this statement in a broader picture in his annual review: The leap in model capabilities in 2025 mainly came from Reinforcement Learning with Verifiable Rewards (RLVR), but in the entire LLM technology stack, memory, multimodal perception, continual learning, and the ability to operate computers are still obvious shortcomings. "We have prototypes, but we don't yet have agents that can be used as colleagues."

Continual Learning (also called Lifelong Learning) has thus become one of the hottest concepts in the past year.

It refers to the ability of a deployed model to continuously absorb new things from new tasks, new knowledge and new experiences like a human, without forgetting what it has learned before.

This sounds natural, but it is extremely difficult to implement, so difficult that it has become the hardest bone on the road to "AI colleagues". The series of reports above also show that this path is no longer a single direction, but several diverging routes that are advancing forward at the same time.

Around how to enable models to "learn while using", the academia and industry have developed several distinct technical routes in the past year. Some attach memory outside the model, some keep rewriting weights, some simply re-pre-train the model, and a number of newer ideas try to redefine the very act of "learning". This article tries to expand these directions one by one, and clarify what each route is betting on and where it gets stuck.

To make it clear: The hard part is not "learning", but "not forgetting"

The core obstacle of continual learning has a special name: catastrophic forgetting. The knowledge of neural networks is stored in billions of weights. When you fine-tune the model with new data and update these weights, while the model learns new tasks, it often overwrites the part of parameters that carry old capabilities, leading to a sharp drop in performance on tasks it used to be good at.

Schematic diagram of catastrophic forgetting. When an artificial deep neural network is trained on two tasks sequentially, it will quickly and completely forget the first task while training the second one.

This phenomenon has been extensively studied since the era of small models, but new troubles have emerged for LLMs. TRACE, a benchmark specially designed to evaluate the continual learning of LLMs, found that continuous fine-tuning of an already aligned model will not only make it forget old tasks, but also damage its general capabilities and instruction following capabilities. In other words, the price of teaching the model a new thing may be that it becomes dumber overall and less obedient.

Precisely because directly modifying weights has such high risks, "continual learning" has split into several schools. Their fundamental divergence actually lies in different trade-offs on the two issues: "whether to modify weights" and "where to store new knowledge".

Attaching Memory Outside the Model

The most straightforward and fastest-to-implement idea is: do not modify the model weights at all, store new knowledge in a database outside the model, and retrieve it back to the context when needed. This route has evolved all the way from RAG (Retrieval-Augmented Generation), and has now grown into a specialized field: Agent Memory.

The representative work is MemGPT (the company behind it has now been renamed Letta). It analogizes the context management of LLM to the memory management of operating systems, distinguishing between a limited "working context" and a larger "external storage", allowing the model to decide what to load into the context and what to write back to the archive by itself, just like an operating system scheduling memory.

Following this idea, a number of systems have their own focuses: Mem0 focuses on production-grade, scalable long-term memory access; Zep adds a temporal knowledge graph on top of retrieval to handle cross-session temporal reasoning; A-MEM draws on the Zettelkasten note-taking method, labels each memory with structured tags and automatically associates them with related old entries, making retrieval more context-aware.

Karpathy himself also bets on this direction. He has repeatedly emphasized that what is needed in the future is not "larger hard disk" style memory, but a streamlined "cognitive core" (he estimates that 1 to 2 billion parameters are enough) plus a set of structured external memory that can compound growth on its own.

Based on this idea, he released a pattern called "LLM Wiki" on GitHub: instead of using RAG to fetch original text chunks every time you query, let the agent actively compile the materials into a continuously updated, interlinked knowledge base, and then query it.

Karpathy's LLM Wiki document has gained nearly 45,000 stars and has been forked more than 9,000 times

Letta itself has elevated this set of practices to the proposition of "Continual Learning in the Token Space". They pointed out that the current default practice is "append first, then summarize", that is, keep piling original experiences until the context overflows, then compress them into summaries. This has two drawbacks: the append operation pushes all representation work to inference time, and every forward propagation needs to reprocess the original logs; while summarization is lossy and abrupt, important details will disappear without warning. Letta's bet is that the memory learned in the token space in the future will be more valuable than the model weights themselves.

The advantages of this direction are safety, controllability and interpretability, and wrong content can be deleted at any time; the disadvantage is that it does not essentially "internalize" knowledge into the model, and every access has to go through the narrow gate of retrieval and context. Once the memory library expands, retrieval accuracy and cost will both become bottlenecks.

Let the Context Evolve into a "Playbook" on Its Own

Taking one step further from external memory, there is a more sophisticated idea: do not modify weights, but let the input context of the model evolve continuously. This is called Context Engineering.

In October 2025, ACE (Agentic Context Engineering) proposed by Stanford, SambaNova and UC Berkeley is a representative of this idea. It treats the context as a continuously growing "playbook", and maintains it through three roles with clear divisions of labor: the Generator is responsible for generating reasoning trajectories, the Reflector extracts specific experiences from successes and failures, and the Curator organizes these experiences into structured incremental updates and merges them into the playbook.

ACE aims to solve two common flaws of similar methods. The first is "brevity bias": when the LLM is asked to rewrite the context repeatedly, it tends to compress and discard domain-specific details; the second is "context collapse": repeated rewriting will lead to the gradual loss of details.

ACE uses incremental small changes (delta updates) instead of rewriting the entire paragraph to avoid these two problems. According to the data in the paper, ACE improves performance by 10.6% on agent tasks and 8.6% on financial reasoning compared with the baseline, while reducing the adaptation delay by about 86.9%; on the AppWorld leaderboard, using the small open-source model DeepSeek-V3.1 paired with ACE can achieve an average score comparable to the production-grade agent IBM CUGA based on GPT-4.1.

It is worth noting that ACE emphasizes that it can work without labeled supervision; it relies on the feedback signals naturally generated during execution (such as whether the code runs successfully or reports an error) to guide reflection and organization. This connects it to the larger narrative of "agents learning from their own experiences".

Continual Post-training: Modify Weights, But Do It Smartly

External memory and context engineering avoid the risk of modifying weights, but also avoid real knowledge internalization. Another group of researchers believe that in the long run, some knowledge and skills must eventually be written into the parameters to be practical. This is Continual Post-training, which is mainly divided into stages such as continuous instruction fine-tuning and continuous preference alignment.

John Schulman from Thinking Machines put forward a hierarchical view: he analogizes learning to several types of motor learning, episodic memory and procedural memory in psychology, and believes that in-context learning will continue to handle short-term learning tasks well, while parameter fine-tuning (including methods like LoRA) will be stacked on top of it, which is especially suitable for tasks that require larger capacity and real knowledge absorption — when the time span is too long for in-context learning to work, parameter fine-tuning will prevail.

The biggest enemy of this route is still catastrophic forgetting, and an interesting recent solution comes from Tinker of Thinking Machines.

Tinker is their first product released in October 2025, a fine-tuning API based on LoRA that abstracts away the complexity of distributed training, only exposing low-level primitives such as forward_backward and optim_step, allowing researchers to focus on data and algorithms.

On the track of continual learning, they mainly promote a formula called Self-Distillation Fine-Tuning (SDFT): the core insight is that ordinary supervised fine-tuning is "off-policy", the model is forced to imitate some tokens that it would never generate itself, so every time it learns a new skill, it will erode its old capabilities; SDFT allows the model to act as its own teacher, learning new skills from demonstrations without forgetting the old ones. A startup called Trajectory has used Tinker as the core infrastructure of its continual learning platform.

By the way, Tinker uses LoRA instead of full fine-tuning, which also has a practical benefit: multiple fine-tuning tasks can share the same computing power pool, and the cost is diluted. This also explains why "Continual Learning as a Service" is becoming a business: turning frequent model updates into an on-demand callable API is exactly the direction that the industry is currently trying to explore.

Re-pre-training and Continual Pre-training

If post-training is to make adjustments on the "surface layer" of the model, then Continual Pre-training (CPT) is to go back to the bottom layer, continue to pre-train the model with new corpora, and make it adapt to new domains, new languages or knowledge distributions that drift over time. Compared with re-training from scratch by mixing old and new data, CPT is built on the basis of existing models, which is much more cost-effective in terms of computing power.

There are three typical application scenarios for it: knowledge drifts over time, cross-language expansion, and cross-domain adaptation. But the cost is also clear: a number of empirical studies have repeatedly shown that continual pre-training has high computational costs and is prone to triggering catastrophic forgetting of learned knowledge. Therefore, the academic community has been looking for "replay-free and task-label-free" continual pre-training methods, trying to achieve non-forgetting at the LLM scale.

For the vast majority of companies, the cost of re-pre-training a cutting-edge model from scratch is unrealistic, which is exactly the origin of Karpathy's statement that "large models are not suitable for frequent re-training". So the "re-pre-training" in the strict sense is more of an option for cutting-edge laboratories, while continual pre-training is a more feasible compromise.

Newer Ideas: Let the Model Learn to Modify Itself

The first four directions are more or less trapped in the dual framework of "external attachment vs. weight modification". But a number of new works that emerged in the past year tried to jump out of this framework and redefine learning itself.

One idea is to let the model generate training data by itself and decide how to update itself.

MIT's SEAL (Self-Adapting Language Models) is a typical example. Given a new input, the model will generate a "self-edit"; this is a natural language instruction that explains how to reconstruct information, what hyperparameters to use to update weights, and even what tools to call for data augmentation; then the model fine-tunes itself accordingly to form persistent weight updates. And the outer reinforcement learning loop is used to train "what kind of self-editing is effective", the reward signal is the performance of the updated model on downstream tasks.

The NeurIPS 2025 version further proves that this adaptive ability will increase with the expansion of model scale, and alleviates forgetting with the help of reinforcement learning. Its authors look forward to a model that can judge by itself "whether to learn now" in the middle of inference, distilling the one-time chain of thought into permanent capabilities.

Another more radical idea comes from Google's Nested Learning, published at NeurIPS 2025. Its core is to re-interpret a model as a set of nested, multi-level optimization problems with different "context streams" and different update frequencies — from this perspective, the architecture and optimization algorithm are unified into different levels of the same thing.