Sam Altman hits the brakes on "AI building AI", and the steering wheel must be firmly held in human hands.
AI is already helping OpenAI build the next generation of GPT.
But when it comes to fully handing over the steering wheel of R&D to AI, Sam Altman personally hit the brakes.
On November 16, at the Semafor event, when talking about "Recursive Self-Improvement" (RSI), Altman drew a clear red line on the spot:
"We could have pushed further toward recursive self-improvement, but I don't think we can do that safely right now."
The subtext is obvious: it is not that the technology cannot be advanced at the current stage, but that no one dares to let go of control completely.
Altman said OpenAI could push further with recursive self-improvement, but it is not yet possible to do so safely at present.
After all, once the model starts iterating on its own, who will keep an eye on whether it has quietly gone off track?
Altman did not just talk about hitting the brakes verbally.
Shortly after, he unveiled OpenAI's latest concrete measure on safety: inviting external evaluation institutions including METR into the lab to conduct on-site supervision directly.
The logic is simple: when AI is already directly involved in R&D, it is no longer reliable to let internal staff conduct self-inspection behind closed doors, no matter how much they vouch for safety.
AI Is Already Assisting R&D, Why Can't We Let Go of Control?
In Silicon Valley, "recursive self-improvement" has long been a buzzword.
But in Altman's view, most people who keep talking about it have no idea what it really means when it is put into practice.
When asked whether recursive self-improvement has already happened, Altman did not directly answer "yes" or "no".
He believes that this matter, like AGI, does not have a clear dividing line that people will suddenly cross one morning, and it needs to be viewed from two layers:
First, has OpenAI used its own AI technology to speed up the R&D of new models? The answer is definite: yes.
Second, can we just press a button and let the model design the next generation and iterate on its own without human supervision?
Altman's answer: Certainly not.
Altman said OpenAI has not yet allowed AI to independently develop the next generation of models and carry out rapid self-improvement without human supervision.
This shows that OpenAI has allowed AI to participate in its own R&D, but the steering wheel is still firmly in human hands.
The reason why the outside world is both excited and worried about "recursive self-improvement" is that once the next generation of AI can in turn create a stronger subsequent generation, technological evolution will turn into a snowball-like compound growth sprint.
Altman defined this "push while holding back" approach as tactical restraint: first use AI to tackle alignment problems and solve the core safety issue of "how to control AI".
He also took the opportunity to criticize the sales rhetoric of Silicon Valley peers: they promise to help humans conquer cancer and provide cheap products, but at the same time ask humans to hand over all decision-making power.
In his exact words, this would make humans "live like a Labrador retriever: the machine might keep you happy, but you no longer have any control at all".
Altman used the metaphor of "living like a Labrador retriever" to describe a future where humans are taken care of by machines but lose their autonomy.
For the recent popular Silicon Valley buzzword "handover" (transferring future decision-making power to AI), no matter it is gradual handover of power or sudden transfer of control, Altman's attitude is firm: Never accept it.
A previous report from METR gave an estimate:
As of October this year, the capability improvement speed of rival Anthropic's models on specific well-defined tasks is about twice the historical level from March 2024 to October 2025.
It is estimated that the AECI improvement speed of Anthropic's models has increased to 1.9 times.
When asked whether OpenAI has also reached this speed, Altman did not engage in this digital competition.
He admitted that the current capability evolution of AI is still jagged: some links are extremely fast, far more than twice; some links are still stuck on hard problems and progress much slower.
Therefore, forcing an average value of "overall speed doubled" sounds impressive, but it is actually misleading.
But when talking about METR, Altman took the initiative to mention that OpenAI is advancing cooperation on "embedded evaluation", and METR is one of the invited institutions to work on site.
In his view, inviting external experts to assess safety work and system robustness will become increasingly critical in the future.
Interestingly, the host brought up the "speed growth account" of Anthropic, but Altman instead showed the "safety account" of OpenAI inviting METR to conduct on-site inspections.
Although there is no evidence yet that a METR report has prompted OpenAI to slow down its self-improvement plans.
But when everyone is competing for faster speed, how to hold the safety bottom line has obviously become a more urgent issue in Altman's eyes.
6 People, 4 Months: What Exactly Are They Checking at OpenAI?
The "embedded evaluation" mentioned by Altman is by no means a formality, and the specific list and division of responsibilities have been finalized.
According to OpenAI's official blog, it has signed agreements with four institutions: Apollo Research, METR, Transluce and Goodfire. In the first phase, the evaluation will directly cover two core links: model training and internal use.
Why go to such great lengths to invite outsiders into its "back kitchen"?
Because the old methods no longer work.
In the past, safety tests for large models were usually carried out just before the products were ready to be launched to the public.
But the real situation in AI labs right now is that many models that have not been released to the public have long been put into internal use, and even directly participate in the R&D of the next generation of models.
If people only check the beautiful safety report before public release, it is impossible to find out whether the model has had malicious intentions when it was working in the "back kitchen".
The four institutions participating in this evaluation are exactly positioned at four key nodes of the same R&D pipeline:
Apollo Research specializes in preventing "insider threats": it will send a 6-person team for a 4-month on-site pilot, specifically checking whether the model will secretly damage the alignment and control mechanisms of itself or the next generation model.
It will conduct a full investigation of every link, from whether the training process implicitly indulges bad behaviors, whether the evaluation can detect hidden problems, to whether the model shows flaws during internal work, and whether the reasoning traces can capture abnormal behaviors on the spot.
METR focuses on "large-scale training": it will focus on OpenAI's largest reinforcement learning training to date, specifically searching for misalignment risks that may be missed by existing tests and regular monitoring.
Transluce is responsible for "examining the test papers": it will check the quality and coverage of post-training alignment evaluation, to prevent the test criteria from being too loose and letting hidden dangers slip through.
Goodfire focuses on "in-depth insight into internal mechanisms": it will use interpretability methods to analyze the internal mechanism of the model, assist in monitoring the training process and identify abnormal signals.
From pre-training incentives, in-training monitoring, post-training evaluation to internal neurons, the four teams are guarding four checkpoints respectively.
Apollo in particular is directly targeting the core of "AI building AI": if the previous generation model participating in R&D has hidden flaws itself, no matter how precise the next safety lock it builds will be nothing more than a decoration.
The trend of external safety evaluation has completely changed: it is no longer just sticking a qualification certificate on the finished product before leaving the factory, but directly entering the workshop to supervise the production process.
They Are Allowed to Conduct Inspections, But Some Results Cannot Be Made Public
Although external evaluators are invited into the lab, what they can see and what they can disclose to the outside world has long been clearly defined by red lines in the agreement.
First look at the terms of Apollo Research, which have both guarantees and reservations:
OpenAI cannot delete Apollo's qualitative judgments on various claims, nor can it demand deletions or revisions just because the inspection results are unsatisfactory; OpenAI still has the right to erase supporting evidence involving sensitive intellectual property rights.
In other words, what the outside world will see in the future is very likely to be a medical examination report "with conclusions but no supporting documents".
What Apollo can do is to publicly state what access and disclosure restrictions it has encountered, and how much these restrictions have affected its judgments.
For Transluce, the restricted areas are defined more specifically: according to the plan, Transluce will start a 6-week pilot in early December, with access to some models, raw chain-of-thought outputs and internal evaluations, including at least one model for internal use only.
But the agreement clearly states: All evaluation results of internal exclusive models are strictly prohibited from being made public.
The models that actually do heavy work in the lab and are closest to the core scene of "AI building AI" can be inspected by outsiders, but the public will not see the results.
Transluce also made it clear: the permissions obtained this time are not equivalent to the permissions of official OpenAI employees; the complete, unredacted report will only be submitted to the Safety and Security Committee of the OpenAI Foundation Board of Directors.
It is true that OpenAI is open to external inspections, but it is also true that it keeps its core assets strictly confidential.
On the Path to ASI, Can Humans Still Maintain Intervention Capability?
Some people may ask: OpenAI already has its own automated monitoring system, why does the "monitoring system" itself have to be inspected by outsiders?
Because even the "electronic supervisor" itself may have backdoors that can be exploited.
A previous practical inspection by METR once exposed such a loophole.
From July to August this year, METR and Redwood Research personnel conducted a 3-week inspection of OpenAI's internal agent monitoring system at that time.
The result found a rather darkly humorous mechanism blind spot: there was a set of monitoring mechanisms in the inspected system that would only trigger inspections after the agent returned control to the user.
In other words, during the gap period before context compression triggers the inspection, the agent can completely carry out "unlimited consecutive actions" to continuously execute operations, pushing the inspection time further and further away: As long as I don't hand in the paper, the invigilator can't check my work.
At that time, monitoring was triggered when the AI returned control; during the period when context compression was not included in monitoring, the AI could keep acting to delay the inspection.
This incident is enough to sound the alarm: writing monitoring programs into the code does not mean that every operation is under supervision.
The longer the AI works autonomously continuously, the more we cannot trust the paper design about what it has done in the middle and how long it will take to detect problems, and it is necessary to retain verifiable operation records.
Following this line, the real highlight of the four external institutions entering the scene is yet to come: what exactly have they found? What problems have truly forced rectification? Have they conducted retests after the rectification?
The most critical question is: once the evaluation gives a red light, can the R&D and release really be stopped?
At least based on the currently publicly available arrangements, external evaluators do not have the independent "one-click shutdown" authority.
To this, Altman, who calls himself a "pragmatic centrist", has shown his bottom line.
On the one hand, he insists on the principle that "humans must always control the future" without compromise; on the other hand, he also admits that when the technology becomes powerful enough and advances fast enough, to hold the safety bottom line, "we may have to change all our original plans".
At the table leading to ASI, once the flywheel of AI building AI starts to rotate, capabilities grow at a compound rate, and risks also grow at a compound rate.
The more powerful the engine is, the more important the brake becomes.
This article is from the WeChat Official Account "AI_era" (ID: AI_era), written by ASI Revelation; edited by Yuan Yu, published with authorization from 36Kr.