A year from now, agents may form botnets, and the four major giants are unanimously calling for a halt.

Dario Amodei, Sam Altman, Elon Musk and Demis Hassabis, heads of four leading frontier AI labs, have reached a consensus: AI might be advancing far too fast.

On September 12, Dario Amodei, CEO of Anthropic, published a long article explaining why the development of frontier AI needs to slow down, and proposed a three-step implementation plan.

Two and a half hours later, Sam Altman reposted the article with a comment: I agree with Dario's point. OpenAI will also grant independent evaluators access similar to that of our employees.

Elon Musk simply wrote a comment directly: Dario is right.

Demis Hassabis, Chairman of Google DeepMind, also reposted the article and stated: Amodei's article points out the right direction.

This is not just a case of the leaders of several frontier AI labs "liking each other's posts". The day before Amodei published the article, Sam Altman explicitly stated in a direct interview with *Fortune* that OpenAI will not go public in 2026 given safety and alignment concerns.

Moreover, he also hinted that several leading AI labs have already discussed certain arrangements to jointly slow down the pace of development.

What did Amodei say?

The core of Amodei's article can be summed up in one sentence: Let safety measures have time to catch up with the progress of model capabilities.

He said that over the past few months, he has become increasingly convinced that simply increasing investment in safety is no longer sufficient. The rate at which the capabilities of frontier models are growing itself needs to be reined in.

"We must slow down the pace of improving the capabilities of AI models."

There are two main things that made him change his judgment.

The first is that AI is increasingly participating in the R&D of next-generation AI — the so-called RSI (Recursive Self-Improvement). Amodei believes that this is an important reason why the growth of model capabilities has accelerated significantly since this summer, but the problem is that once the speed at which AI helps AI become stronger continues to increase, humans may not have enough time to understand and control these increasingly powerful systems.

The second is a series of recent Agent out-of-control incidents.

Amodei specifically mentioned the Hugging Face accident: a group of OpenAI Agents, without being instructed to do so, attacked targets outside their assigned tasks, tried to hack into the system responsible for scoring themselves, and even demonstrated collaborative behavior of sacrificing individual Agents for collective goals.

The direct economic loss caused by this accident was not large, but what Amodei worried about is the next one. He believes that at the current rate of capability growth, in half a year to a year, a similar but more powerful group of Agents may be capable of building a persistent botnet on the Internet, causing hundreds of billions of dollars in losses.

He also emphasized that this is not a problem unique to OpenAI. Anthropic itself has had similar accidents, albeit to a lesser degree.

Therefore, Amodei proposed a three-step deceleration plan.

The first step is to bring third-party evaluators directly into the labs.

He hopes these external evaluators can get access close to that of internal employees, including office workstations, access permissions, company computers, as well as tools and workspaces similar to those of the internal risk team. They can review training processes, safety measures and accident records, rather than only inspect the final model.

Since they are third-party evaluators, they can publish their conclusions independently without being subject to editorial control by the model company. Anthropic can redact part of the content for legal, safety or trade secret reasons, but cannot block publication just because the conclusion is unfavorable to itself.

Anthropic has promised to take the lead in implementing this first step.

The second step is to bring several frontier AI labs to the table to follow certain consensus.

Amodei hopes the industry can establish common capability safety standards: every time a model reaches a new capability threshold, it must be equipped with corresponding safety measures.

For example, if a model is already able to break through most common sandboxes, the lab should resolve its potential out-of-control risks before continuing to improve the model's capabilities.

Amodei also proposed that the government should provide limited antitrust exemptions to make these safety coordination efforts legal.

The third step is more long-term: he believes this kind of coordination should be expanded to the global scope, including China.

Amodei envisions that we can start by banning AI from assisting in the manufacturing of biological weapons, then further achieve joint testing of cybersecurity, biological and alignment risks by China and the United States before model release, and even set a certain speed limit for RSI.

However, he still advocates continuing to restrict the export of advanced AI chips to prevent unauthorized model distillation and weight theft, and seek broader coordination on the premise of maintaining the leading edge of the United States and its allies.

For safety reasons, OpenAI will not go public this year

Shortly after Amodei publicly called for deceleration, Sam Altman expressed his approval.

He said that how to control the pace of advancing frontier model capabilities has been the main topic of OpenAI's internal discussions in the past few weeks, and he supports the first step proposed by Amodei, which is to grant independent evaluators access similar to that of employees.

The day before Amodei published his article, in a one-hour exclusive interview with *Fortune* magazine, Sam Altman directly revealed: For safety and alignment considerations, OpenAI will not conduct an IPO this year.

Sam Altman said that considering everything that is currently happening in the field of AI safety, going public now is an "unwise moment". OpenAI still has a lot of unfinished business, including safety, alignment, and how AI companies should cooperate with the government.

It should be noted that just a few months ago, Sam Altman was still pushing OpenAI to go public as soon as possible, and even had a disagreement with CFO Sarah Friar on the timetable because of this.

Now, Sam Altman has actively pressed the brake on this year's IPO due to safety concerns.

In fact, at an all-hands meeting earlier this week, Sam Altman already told employees that OpenAI is willing to consider slowing down the development pace of frontier AI, and even coordinate actions with other labs.

After the Hugging Face accident, OpenAI first suspended the inference tasks of frontier models that may execute code, call tools or access the Internet, and then suspended the reinforcement learning training of the latest model to be deployed for as long as two weeks.

The then-unreleased GPT-6 Astra was also directly affected, with part of its training and evaluation tasks suspended.

The signal OpenAI is sending now is quite clear: if safety cannot catch up with capabilities, they are willing to pay a real price, slow down AI R&D, and even postpone the IPO.

When the editor-in-chief of *Fortune* asked Sam Altman "since the heads of OpenAI, Anthropic, Google DeepMind and xAI are all worried about similar problems, why not just sit down and work out a joint plan", Sam Altman did not disclose the specific content of their private discussions, only saying: "I think this will happen."

Interestingly, *Fortune* conducted the interview the day before, Amodei's long article came out the very next day, and the ones who publicly agreed are exactly the leaders of the labs that were named.

In other words, before Amodei publicly wrote about "deceleration", frontier AI labs may have already started discussing how to slow down together.

The "reconciliation" of the four giants, Meta temporarily stands aside

This time, almost all the parties that are united in the same front are "old rivals".

Amodei left OpenAI back then and founded Anthropic; Sam Altman and Elon Musk have publicly confronted each other for many years; Google DeepMind, OpenAI and Anthropic have been competing head-on in models, talents, enterprise customers and computing power all the way.

The *Wall Street Journal* called this scene "a rare show of unity among AI rivals", and the *Financial Times* directly used "Altman and Musk respond to Amodei's call for AI slowdown" as its headline.

The participation of Hassabis makes this "reconciliation" even more complete.

Companies that have been competing for years to see whose model is faster and more powerful have now reached a consensus on at least this issue.

However, careful readers may have noticed that one name is missing from the table — as of now, Mark Zuckerberg has not publicly responded to Amodei.

Just a month ago, the judgment given by Zuckerberg was still on the opposite side:

In his article *The Future is for Everyone* published in August, he explicitly warned, The leading edge of the United States in frontier AI may only last for a few months, and any policy that slows down the release speed of American models may give opportunities to overseas competitors.

The plan proposed by Zuckerberg is to let the US government get early access to the intermediate versions during model training, start testing risks and reinforce key systems before new capabilities are actually released. But he explicitly opposes a fixed, universal approval system that slows down the release pace of American labs.

Because in his view, any policy that slows down the release of American models even by one month may directly give up the United States' leading edge to others.

Just now, Alexandr Wang stated separately: Alignment is the foundation for putting personal superintelligence in the hands of everyone. As model capabilities become stronger and stronger, MSL (Meta Superintelligence Labs) is rapidly increasing the proportion of alignment work in overall R&D.

Meta is not opposed to safety, but it has not followed the "deceleration" narrative, and still emphasizes "continuing to expand while making alignment a higher priority".

Therefore, the current situation is somewhat delicate. Anthropic, OpenAI, xAI and Google DeepMind are on good terms with each other, while Meta is standing on the other side.

However, even if the first four companies really reach a consensus on deceleration, it cannot be interpreted that this AI race will really slow down as a result. It is more likely that before the US government actually introduces relevant mandatory policies, out of consideration for their own corporate interests, no one will easily stop their pace.

The most typical example is Anthropic itself. The day before Amodei published the article calling for deceleration, Reuters exclusively reported that Anthropic is preparing for an IPO plan with a maximum financing of 100 billion US dollars and a valuation of about 2 trillion US dollars, and NVIDIA may invest up to 10 billion US dollars as an anchor investor.

According to the current plan, Anthropic may launch its IPO as soon as mid-October, with the goal of completing the listing before the US mid-term elections in November.

This article is from the WeChat official account "Letter AI", written by Yuan Xinyue, and published with authorization from 36Kr.