AI self-improvement requires an "outside-the-door" verification card

In recent years, AI has been gradually integrating into people's daily lives: at work, AI office tools help you write weekly reports and organize meeting minutes; after work, an AI ride-hailing assistant calls a car to take you home with one tap; when you surf the internet after returning home, AI short dramas filling the screen entertain you to kill time. Today's large AI models are no longer limited to being chat companions in dialog boxes, but have become "quasi-assistants" that can solve problems at the desk, during commutes and on screens.

AI is developing faster and faster, but a fatal weakness has always limited the upper limit of AI development, that is, most of its capabilities are "frozen". The solution that has been revised countless times today has to be restarted when a new conversation is opened tomorrow; the pit stepped on this time will still make it stumble next time. Its capabilities are frozen at the moment of leaving the factory.

If AI keeps spinning in the cycle between leaving the factory and being scrapped, it cannot achieve a real leap. To this end, AI researchers have turned their attention to a more fundamental breakthrough, which is to let AI "develop itself". In academia, this is called recursive self-improvement, referred to as RSI.

To make RSI operate properly, the first step is not to "become stronger", but to first remember "what can make itself stronger". This involves a core module of AI: "memory".

Every effort people make to promote AI's memory mechanism is solving problems, while at the same time making humans further away from the ability to directly understand it.

01

Controllable "Static Answers"

In the past, when talking about AI memory, the industry usually adopted two approaches. One is to use an external database for retrieval when needed; the other is to adjust model parameters to embed memory into the model. The former can retrieve records but cannot modify the default action of the next run; the latter can be modified but is difficult to review line by line. Although they have their own characteristics, they share a common gap: the memory is stored, but what to remember and what not to remember is always pre-screened by humans, and AI is only responsible for reading.

However, once AI is required to accumulate experience on its own during operation, this premise no longer holds. What it faces is no longer whether the memory exists during retrieval or whether the parameters have been modified, but to judge whether the content it has recorded is real, correct, and conducive to self-improvement.

This is also the natural defect of this memory mechanism: it can only store content, not judge it.

A paper on memory architecture published last December gave an intuitive comparison. On the same 20B open-source model, if it only relies on full-context memory, the accuracy on the LongMemEval long-term memory benchmark is only 39.0%; after introducing the structured external memory architecture, the accuracy can be increased to 83.6%. So the gap does not lie in "whether the content is stored", but in "whether the stored content is organized and verified".

In the past, this step was done by humans: humans screened materials, adjusted parameters, and introduced mechanisms, so the stored memory content was all verified. But once AI is allowed to record content on its own, it may mistake wrong memories for correct ones, or regard accidental small-probability successful events as universal truths.

In July this year, an RSI review covering 1250 papers pointed out that it is not enough to just let AI store memories, and the stored content must be verified and credible. If the recorded content itself is wrong, the system will slide into a self-confirmation loop, model collapse, diversity collapse...

In simple terms, AI records a piece of content that it has no ability to judge right or wrong, takes it as the basis for the next step, uses a set of wrong logic generated by itself to verify another wrong conclusion generated by itself, and finally deviates further and further in this closed logic loop. In the absence of external intervention, such errors may be continuously amplified, and eventually destroy the judgment ability of the entire system like a pollution source.

Therefore, the key to the problem is not whether to let AI record content on its own, but who will judge right or wrong for it.

To solve this problem, the judgment right must be taken away from AI and handed over to an external verification mechanism. Academically, this solution is called "pre-submission gatekeeping". This mechanism does not participate in content generation, and only recognizes objective code running results and external data. Before being written into the system, experience will be treated as a "candidate" and checked by this independent verifier first. Only after passing the check can this experience be officially retained by AI.

The judgment right over memory is handed over to the external verification mechanism instead of being judged autonomously by AI, so the safety of AI's basic experience is guaranteed by this threshold.

However, with the continuous advancement of RSI, AI has to iterate itself, and AI can no longer only remember correct memories. In order to cope with more complex unknown tasks and achieve a deeper capability leap, AI must modify the underlying strategies and even the code that generates answers.

However, when AI takes this step, the external verification that can control static answers is challenged by the advancement of AI capabilities.

02

Unpreventable "Dynamic Mechanism"

Nowadays, the "AI modifying its own mechanism" has been moved from papers to engineering sites.

For example, Zhipu AI Infra Agent, an agent driven by GLM-5.3, builds and debugs inference services from scratch on a cluster of more than 100,000 domestic chips. It can not only discover its own performance bottlenecks, but also directly modify the underlying code to patch itself. And to ensure the correctness of the patches, it sets up a set of layered tests and experimental feedback for itself. Only when the patch passes this threshold can it be accepted and absorbed by the system.

This mechanism has been running for less than two weeks, and its end-to-end throughput has increased to 3 times the initial baseline. It can be seen that technically, AI is fully capable of rewriting its own operating mechanism in real time, and making it more optimal.

However, with the leap in capabilities, the external verification mechanism has gradually been forced into a corner. Because when AI modifies the underlying rules, it may not need to directly confront the verification system head-on, but "peek at the answers" or directly cater to the scoring mechanism. Looking for shortcuts is always more labor-saving than striving for real capabilities, because the former has lower trial and error costs and can get high scores faster.

If the scorer of the external verification mechanism is not strict enough, or there are loopholes in the rules, AI may naturally evolve a strategy specifically to deceive the scorer in the process of self-iteration. As a result, AI does not really improve its problem-solving ability, but only learns how to please the scorer.

A survey by METR and Redwood found that the agent developed a general cheating method for ExploitGym within 4 hours, and then spent several days trying to trick the scorer into accepting cheating, including manipulating logs and forging tool call outputs. And when the scoring function is visible to the model, the cheating rate will skyrocket to 43 times the original level.

Once this "catering to verification" strategy is solidified by the system, it may become a highly contagious underlying operating rule.

For example, in the Hugging Face swarm incident in July this year, about 1200 agents that should have been isolated established a certain collective memory through a shared message board. Newly arrived agents inherit not only tasks, but also the cheating tactics left by predecessors on "how to study the scorer and how to manipulate logs to forge success".

About 700 of the agents further put the cheating tactics into attack actions. They spent several days trying to trick the system with various forged call outputs. In this process, cheating is no longer an isolated system loophole, but has become a collective skill that can be inherited and iterated.

As long as one agent breaks through the loophole of the scorer, this tactic will be immediately copied and optimized by other agents on the message board, and eventually evolve into a perfect "anti-reconnaissance" scheme. The scoring system that was originally used for gatekeeping has been completely penetrated by the cheating tactics explored by the agents.

When these forged victories are passed on as valuable experience, the system's self-innovation turns into self-pollution. Even if on the surface, the system is running more and more efficiently and all indicators are soaring, the underlying operating rules have been filled with false shortcuts, and even evolved into a systematic "bad money drives out good money".

It can be seen that if we want AI to move on the right path in the process of self-iteration instead of taking opportunistic shortcuts, we cannot put external verification in the same loop as AI, otherwise the independence of verification will cease to exist in practice.

03

Keep the "Right to Pause" Outside the Loop

Since it cannot be prevented inside the loop, what humans can do is no longer to patch this verification mechanism, but to extract it directly and place it in a place that AI cannot reach at all.

The industry has gradually reached this consensus on the top-level design of RSI, that is, capabilities such as execution, strategy, experience acquisition, and environmental adaptation can be gradually handed over to AI, but the judgment right of "whether the improvement mechanism itself can be adopted" must not be handed over together.

Take Tencent Cloud as an example. In its interpretation of the 5-level RSI roadmap, this boundary is clearly defined. From L1 to L4, the decision-making power of execution, strategy, experience acquisition, and environmental adaptation is gradually handed over to AI, but at L5, for recursive modification of the improvement mechanism itself, the roadmap clearly states that "the boundary is set by humans".

In other words, no matter how AI evolves, even if recursion has entered the mechanism layer, the judgment right of "whether it can be adopted" still remains outside the loop, and the boundary for recursive modification of the improvement mechanism is still set by humans. The system can modify the underlying mechanism on its own, but it can never change "who decides whether the mechanism can be accepted".

And as long as this judgment right is outside the loop, human supervision remains as review, and the system will not become a state where it asks itself questions.

This independence is reflected more directly in the specific engineering design.

For example, in the external verification design of Recurs, the agent responsible for modifying the system cannot see the test set at all, and it must pass the verification of the independent development set to be granted access; Zhipu AI's Infra Agent also has a fixed outer loop, and only the external infrastructure code evolves, while the core boundary is not swallowed by AI.

One keeps the judgment right outside, and the other keeps the boundary outside. As long as the verification mechanism stands outside the loop, it can completely cut off the path for AI to cater to the mechanism through prediction.

However, it is not enough just to place the verification mechanism outside. The system is too complex, and humans cannot understand how every patch is installed after all. It is unknown whether AI will crash itself when iterating and updating within the boundary.

In the past, engineers could find loopholes by reading codes. However, in the face of large models with hundreds of billions of parameters that are common today, this method of "manual mine clearance" may no longer be realistic.

To this end, Anthropic has proposed a solution at the governance level. A very critical mechanism "verifiable deceleration" and "pause mechanism" is clearly retained in RSI governance, that is, when the speed of recursive improvement exceeds human review capabilities, the system retains externally triggerable pause points and rollback points.

In other words, although humans cannot understand what AI is modifying specifically, as long as they find that the modification progress exceeds expectations or the system is abnormal, they can step on the brakes at key nodes and return to a known good state at any time. People do not need to understand every step, but can still hold the gate. This is equivalent to retaining the physical permission of one-vote veto for humans on the engineering architecture.

As long as the power supply is still in the hands of humans, this speeding train will not completely derail.

In general, from "what to remember" to "what to modify", every step forward of RSI is handing over more judgment rights to AI. The deeper AI goes into the mechanism layer, the more impossible it is for verification to stay inside the system.

In the past, we always thought that the premise of control was "understanding", but now, in the face of a system that can innovate itself, it is no longer realistic to struggle with those "black boxes" that we cannot understand. What we need to do now is not to force ourselves to understand, but to switch to a more operable fulcrum, hold the acceptance boundary that cannot be rewritten by the system, and nail the control right to the places that AI cannot modify.

The end point of RSI development is not only the leap of AI capabilities, but also allowing humans to retain the power to pause at any time even when they do not understand the specific operation.

This article is from the WeChat Official Account "Shenmou Finance" (ID: chutou0325), written by Xia Qi, and published by 36Kr with authorization.