Will AI be humanity's last invention?

Overview

This article explores the ongoing phenomenon of Recursive Self-Improvement in the field of artificial intelligence, where AI begins to deeply participate in the creation of next-generation systems more powerful than itself.

The article points out that this will not only bring exponential acceleration of technological evolution, but also completely reshape the R&D paradigm and organizational form of human society.

At the same time, the author conducts an in-depth analysis of real constraints faced by recursive systems, including the verification bottleneck, local optimum and overfitting. In the future of human-AI collaboration, the core value of human beings will shift from simple "execution" to setting goals, maintaining aesthetic judgment and assuming ultimate responsibility.

Finally, the article emphasizes that no matter how technology evolves, individuals and organizations need to build an honest closed feedback loop, cross the threshold of ego-driven self-protection, and maintain continuously updated evolutionary capabilities.

01

The truly terrifying thing is not that "AI is getting smarter", but that "AI begins to determine the speed at which it gets smarter"

Recently, the most fascinating term in the AI field that also triggers strong self-doubt among humans is Recursive Self-Improvement (abbreviated as RSI).

It sounds mysterious, but in fact it can first be understood as a very simple sentence:

AI begins to participate in the creation of the next generation of AI.

But the term "self-improvement" has been overused to some extent now:

Letting a large model check its own answers for typos after generating them is not RSI;

Letting an Agent modify a few lines of business code based on error messages cannot be regarded as real RSI either;

Even letting AI help engineers at OpenAI, Anthropic or Google write training code is still only a "semi-closed loop".

The RSI in the strictest sense is a process as follows:

An AI system can participate in the design and creation of successor systems stronger than itself; after the new system is born, it gains stronger capabilities to design the next generation of successors; this improvement can occur continuously with less and less reliance on humans.

In other words, the real key is not Self-Improvement, but Recursive — the first generation modifies the second generation, the second generation is better than the first at modifying the third generation, and the third generation is better than the second at forging the fourth generation.

If the intelligence level is denoted as C, in the past we only cared about: How big is C? (What score does this machine get in the test today?) But RSI truly changes the rate of change:

It even further evolves to: The existing stock intelligence C itself begins to directly amplify the growth rate dC/dt.

When "intelligence" itself becomes the "means of production" for generating more intelligence, the nature of the game changes completely.

02

Why ordinary closed loops can only make you proficient, while recursion can trigger the singularity?

Long before the wave of generative AI emerged, Elon Musk revealed his underlying mindset in an interview: "I think an extremely important suggestion is to have a correctable Feedback Loop, constantly reflecting on what you have done and how you can do better."

It sounds very similar to the familiar PDCA Cycle (Plan-Do-Check-Act) or The Lean Startup (Build-Measure-Learn), right? Then why can the PDCA of ordinary people only turn them into skilled workers, while recursive self-improvement can trigger exponential explosion?

Because not all circles rotate on the same dimension.

"Effort and improvement" in the human world are actually divided into four cruel tiers:

Tier 0: Open Loop. You just keep working hard without ever looking at the feedback of results. Mechanically repeating the work experience of the first year for ten years — that does not mean you have ten years of experience, it means you have run the same Bug 3,650 times.

Tier 1: PDCA and constant temperature air conditioner (first-order closed loop). You set the air conditioner at home to 25°C, it blows cold air when the temperature is too high and stops when the temperature is too low. The target is fixed, and the rules are fixed. When you make fine adjustments to your movements based on results while doing exercises or playing ball games, it can help you eliminate errors, but its mathematical destiny is diminishing marginal returns — the closer you get to full marks, the slower you progress, until you get stuck at the ceiling.

Tier 2: Lean Startup (double-loop learning). You not only fine-tune your actions, but also dare to modify your original goals and assumptions (Pivot) based on market feedback. You originally wanted to make a game, but found that users only liked the chat tool inside it, so you simply cut the game and changed direction to build Slack.

Tier 3: Recursive Self-Improvement (RSI).

The first three tiers all have one thing in common:

"The improver (subject)" and "the object being improved (object)" are separated. You are holding a hammer to beat external products or skills; the work gets better, but your underlying computing power as the person holding the hammer does not change at all.

Expressed by a mathematical formula, the ordinary PDCA cycle is:

Here x is your output, and f is your brain or workflow. You are constantly optimizing x, but the operator f is fixed.

And recursive self-improvement is:

The algorithm f used to improve things points the scalpel at itself like a god.

In the first round, you use the 1.0 version of yourself to upgrade yourself to version 1.1;

In the second round, you no longer work with the capabilities of version 1.0, but use the cognition and tools of version 1.1 to push yourself to version 1.3;

In the third round, the version 1.3 of you pushes yourself to version 1.8 in one go...

Ordinary closed feedback loops improve Position;

Recursive self-improvement improves Acceleration.

03

The first "Bootstrapping" myth in computer history and the accelerated historical clock

This spectacle of "grabbing your right hand with your left hand and pulling yourself off the ground by your shoelaces" actually happened once in real computer history, and the industry term for it is Bootstrapping.

Many programming beginners will ask a "chicken or egg first" question: What language is the C language compiler written in?

The answer is: Written in C language itself.

Before there was a C language compiler, who could compile this code? The pioneers' approach was extremely elegant:

First, use the cumbersome assembly language to manually build an extremely simple "seed compiler v0" that can only understand a few basic instructions;

Use these few basic instructions to write slightly smarter compiler source code, feed it to v0, and compile to generate "compiler v1";

Throw away the assembly language directly! Use the new features supported by v1 to rewrite a more powerful compiler source code, then use v1 to compile it into v2; then use v2 to compile v3...

In every cycle, the compiler uses the teeth it just grew a second ago to chew and forge the sharper version of itself in the next second.

In 1965, I.J. Good, a statistician and Turing's old partner at Bletchley Park, put forward the "Intelligence Explosion" hypothesis:

As long as humans build the first machine that is only slightly smarter than humans and has the "ability to design machines", it can design a second generation of smarter machines, thus triggering a chain reaction.

Good's original words were: The first ultra-intelligent machine will be the last invention that humans need to make, provided that it is docile enough to tell us how to control it.

But today we need to do a crucial demystification of the "intelligence explosion": If RSI is truly realized, the first thing that changes is not that a machine suddenly changes from IQ 140 to IQ 10000 at 3 a.m. one day, but that "time itself" is compressed.

Today, training a cutting-edge model still takes "months" or even "years" as the unit. Once most of hypothesis formulation, experimental design, coding and result analysis are taken over by AI, the research cycle will be compressed from "once a year" to "once every six months", then to "once a month" and "once a week". As Leopold Aschenbrenner depicted in Situational Awareness:

Millions of "AI researchers" work around the clock to conduct parallel trial and error, compressing the algorithm evolution that originally took ten years into just one year.

What RSI really changes is the "clock frequency" of human technological progress — history itself begins to accelerate.

04

Dispel the myth: "Recursion" is not equal to "infinite exponential explosion"

It is very easy to have a misunderstanding here: as long as A can create a better A, will it definitely lead to infinite exponential explosion?

Not necessarily. Because any real system cannot escape Amdahl's Law and the bottleneck constraints of the physical world:

AI can put forward one million experimental schemes a day, but you may not have one million GPUs to run them;

AI can design 100 new chip architectures a day, but TSMC's wafer fabs cannot be built in one day;

AI can screen out 1,000 candidate drug molecules a day, but phase III clinical trials on the human body still have to follow the biological clock.

A system will not automatically speed up 100 times just because 90% of its digital links are 100 times faster — the remaining 10% of physical and computing power bottlenecks will suddenly become the key points that determine the overall situation.

Therefore, the most accurate economic expression for understanding RSI is not "infinite exponential growth", but:

RSI turns "R&D capability" from an Exogenous Variable into an Endogenous Variable.

In the past, the speed of technological progress depended on how many human doctors and how much capital there were in the external world;

In the future, a self-catalysis loop will be formed inside the system: "the stronger the AI → the faster the R&D → the stronger the next generation AI".

Whether it will loop infinitely depends on the external physical bottleneck, but just closing this endogenous loop once is enough to subvert the entire game.

05

As of October 2026, where are we now? Three meshing gears and the "small closed loop"

Why has I.J. Good's prediction lain on paper for sixty years, but only in the past two years has it made Silicon Valley both fanatical and nervous?

Because in the past, AI training was a typical "human manual closed loop": AI was responsible for outputting results, and human engineers were responsible for checking effects, cleaning data, adjusting parameters, and scoring answers (RLHF).

In this cycle, humans are the slowest bottleneck — it's like connecting an old cow that has to drink coffee and hold weekly meetings on time between the Ferrari engine and the wheels.

But by 2026, the old cow is being removed. AI has begun to mesh its own gears at least at three levels:

The first gear: distilling "slow thinking" into "fast intuition". When encountering difficult problems, the model first mobilizes massive reasoning computing power to perform thousands of searches, refutations and trial and error in the background (the psychological "System 2" slow thinking); once it finds the only golden path that works among thousands of dead ends, it immediately takes this path as high-purity data, "distills" it and writes it back to its own neural network weights — using today's high-cost "hard thinking" to reshape tomorrow's low-cost "muscle memory (System 1)".

The second gear: the "ruthless referee" that nature comes with. The quality of poetry and chat depends on human subjective scoring; but whether the code runs correctly and whether the mathematical proof is correct is determined by the compiler and formal verifiers (such as Lean). With a millisecond response and absolutely objective referee, AI can stay behind closed doors and complete tens of thousands of self-games of "generation - error reporting - modification - internalization" in one second.

The third gear: the physical body of AI is exactly code, and what AI is best at is exactly writing code. Carpenters cannot use carpentry to upgrade their own brain neurons, but the neurons, training frameworks and underlying CUDA operators of AI are all written in code. The means (writing code) and the goal (upgrading the intelligence composed of code) have achieved pixel-level coincidence for the first time in history.

This has also directly given birth to the "narrow-domain small closed loop" that has been implemented in laboratories:

Darwin Gödel Machine can modify its own Coding Agent by itself, test the new version and then let the stronger version continue to participate in the next round of modification, directly pushing the SWE-bench score from 20% to 50%;

The AIDE² released in September 2026 goes a step further: it runs autonomously for eight consecutive days, performs surgical operations on its own AI Research Agent, iterates seven generations continuously (evolves new search strategies and memory mechanisms autonomously), and the evolved capabilities can be transferred to tasks that are not involved in optimization.

From the perspective of computer science definition, the local "small closed-loop RSI" is no longer science fiction, but an established fact;

What has not been fully closed is the full-process "large closed loop" covering "direction selection - hypothesis proposal - experimental design - training next generation foundation model - verification and deployment".

06

Step into the "antechamber" of RSI: AI has already sat on the production line of manufacturing next-generation AI

Even though the full-process "large closed loop" has not been completely closed, the internal data disclosed by the three cutting-edge laboratories in 2026 reveals a subtle fact — although the machine has not taken over evolution independently, it has already sat in the main driver's seat of the laboratory:

Anthropic: As of May 2026, more than 80% of the merged code in its production codebase has been written by Claude;