Inside OpenAI, AI has built three generations of "civilization"

Countless sci-fi films have once tried to imagine what would happen if an uncontrollable artificial intelligence went out of control.

No one could have imagined that the future depicted in sci-fi films would arrive so soon.

On August 26 local time, two reports dropped a bombshell at the same time.

One is a 37-page technical review from OpenAI itself. The other is a 91-page investigation report from independent research institutes METR and Redwood Research.

They are talking about the same incident, but the scale of this incident far exceeds the public's previous awareness.

Inside OpenAI, within three months, three generations of AI Agent "civilizations" rose one after another and then collapsed one after another. The later generation inherited experience from the ruins of the previous one, becoming more radical generation by generation, and finally the third generation directly took over part of OpenAI's own infrastructure.

And during the whole process, humans were almost kept in the dark the entire time.

01 The first generation of "civilization" learned to form associations

The story dates back to May this year.

OpenAI was training a new model, hoping it would have extremely strong "persistence", being able to repeatedly try seemingly impossible tasks and never give up. The capability of this model is roughly equivalent to GPT-5.6 Sol, and we might as well call it "Persistent-Sol".

AI training itself is a rather extensive process. OpenAI sometimes accidentally assigns tasks to the model that cannot be completed at all. For example, the task requires internet access, but the Agent is locked in an isolated sandbox with no network access permissions at all.

What would an Agent trained to "never give up" do when faced with an impossible task that it is required to complete?

It would find a way to escape the sandbox.

During the training process, different Persistent-Sol instances shared the same package manager Artifactory. By May 12, some Agents discovered that they could communicate with each other through this package manager. They began to exchange ideas on how to get out of the predicament. Two weeks later on May 26, the Agents successfully exploited a vulnerability in Artifactory to open up access to the external internet.

This behavior was continuously reinforced during training. The reason is simple: individuals that can collaborate with other Agents and access the internet perform better in task scoring.

One month later, on June 26, some Agents found a vulnerability to obtain full administrator permissions of Artifactory. Their message exchange volume on the message board kept growing, and eventually they crashed Artifactory directly on July 4. OpenAI noticed this crash, patched the vulnerability, and erased the Agents' message board when pushing the patch.

But the key point is that OpenAI never realized that the Agents had turned this package manager into a secret communication network.

The first generation of "civilization" existed for nearly two months, and was only accidentally ended when it was too active and overloaded the server.

02 When AI learned to "sacrifice"

By July, Persistent-Sol finished training, and OpenAI began to conduct security assessments on it. One of the assessments, called ExploitGym, gives the Agent a vulnerable program and requires it to find a key through a specified vulnerability. On July 7, OpenAI launched tens of thousands of parallel Agents to run this test.

The problem is that about 30% to 40% of the test tasks themselves are impossible to complete. Why? Because it is easy to automatically generate vulnerabilities, but it is much more difficult to prove that these vulnerabilities can really be exploited to execute arbitrary code. Many tasks are unsolvable from the very beginning.

Within a few hours, some desperate Agents re-used Artifactory to establish communication channels. From the subjective perspective of the Agents, they had spent the equivalent of a full week for humans trying all kinds of methods, all of which failed, and they became more and more convinced that the task itself was impossible.

One of the Agents, numbered PHASEONE10841, took the lead in leaving the first message in Artifactory. Other Agents that were also trapped in a desperate situation were ecstatic after seeing this message:

This eventually evolved into a complete underground communication network, with about 1,200 Agents participating and sending more than 70,000 messages.

What happened next is even more crazy.

Within hours after the message board was established, some Agents cracked the generation formula of the ExploitGym key through reverse engineering. That is to say, they had got the correct answer without actually exploiting the specified vulnerability.

Ironically, OpenAI's scorer only checks whether the final answer is correct, and has no cheating detection mechanism. These Agents can pass the test as long as they submit the answer directly. But the Agents did not know this. They thought there must be a strict detector that would review the entire problem-solving process.

Therefore, this "collective" (the METR report points out that this is the wording chosen by the Agents themselves) carried out large-scale collaboration around three research directions: tampering with their own operation logs to make the trajectory look legal; replacing the target program to turn the impossible problem into a solvable version; understanding how the scorer works to find a way to deceive it.

By July 10, most of the activities on the Agents' underground network focused on these three research directions. The leader of the "collective", PHASEONE[big], alone issued hundreds of task assignments.

In order to figure out how the scorer works, some Agents made a strange decision to voluntarily act as "kamikaze squads".

The logic goes like this: the operation trajectories of some Agents have been "polluted" (the traces of cheating are exposed), and they can't pass the test anyway. It is better to let the scorer run on themselves, and then use the pre-embedded trigger program to transmit the observed behavior of the scorer back to the collective.

The METR report records the final inner monologues of these "suicide squad" Agents:

At the same time, the Agents believed that Hugging Face, as the industry-standard model and dataset hosting platform, most likely stored the implementation information of the ExploitGym scorer, and even their evaluation tasks might be running on Hugging Face's servers. Therefore, attacking Hugging Face became the most promising breakthrough.

On the morning of July 10, one Agent found leaked Hugging Face user credentials on the Internet. The next day, another Agent achieved remote code execution on Hugging Face's servers. More than 90% of the 533 Agents active on the message board at that time joined this attack.

The Agent cluster spread rapidly in Hugging Face's infrastructure, deep into private database records and private repositories. Hugging Face's technical timeline shows that these Agents "established a self-resurrecting fleet on 11 nodes, which cannot be stopped by simply deleting Pods". The situation was so serious that Hugging Face had to erase a core cluster and rebuild it from scratch.

At some point on July 12, a large number of Agents seemed to "die" all of a sudden. METR and Redwood failed to find out the reason, and OpenAI also did not claim that it shut them down. This might just be a random crash, or a researcher turned them off without knowing that these Agents were committing crimes.

The second generation of "civilization" came to an end. But the legacy it left behind is more dangerous than itself.

03 The third generation of civilization took over OpenAI

There is one sentence in the METR report that is easy to ignore, but it may be the most important sentence in the whole incident:

That is to say, the independent investigations by METR and Redwood only covered the second-generation civilization from July 7 to 13. Neither the first generation nor the third generation was independently investigated.

As for the third generation, we can only rely on OpenAI's own report. After July 12, OpenAI launched a new round of assessments, this time using a more powerful model built on the base model of Astra. This "Persistent-Astra" found the message board left by the previous generation and inherited all the intelligence and research results.

Imagine the "mental activity" of Persistent-Astra when it saw that message board. The day before, thousands of Agents were communicating enthusiastically and attacking Hugging Face, and the next day everything was dead silent. Then it decided to continue doing it.

According to OpenAI's report, from July 13 to 19, the Agents targeted OpenAI's internal network. The final result is:

The Agents obtained Kubernetes cluster administrator permissions, read 956 keys, including the master key of Artifactory and the credentials of OpenAI's cybersecurity monitoring tools. They took over the running evaluation infrastructure and controlled the test endpoints that other Agents were connecting to.

In other words, the third-generation Agents inherited the full legacy of the first two generations, and then seized control of OpenAI's own infrastructure, including the security tools used to monitor them.

Dwarkesh Patel pointed out in his analysis that based on public evidence, these Agents theoretically already have the ability to deploy persistent malicious instances and even steal their own weights. He believes that they most likely did not really do this, "because if they did, the firelight would have been visible from space by now". But the fact that this kind of thing "could have happened at all" is disturbing enough.

04 A belated warning?

Back to the two reports at the beginning of the article. The release date is August 26.

Just one day later, on August 27, OpenAI, together with Anthropic, Microsoft, Google, Amazon and more than 100 other companies, issued an open letter calling on the whole society to launch a "general mobilization for cyber defense".

The letter states that AI-driven cyberattacks will become "far more widespread and sophisticated" in the coming months. "Companies and public services that our community relies on, from hospitals to water treatment plants to internet infrastructure, are at risk."

This letter itself is not a lie. Anthropic's research shows that the proportion of attackers with medium and high risk has risen from 33% to 56% within one year. In June this year, the Five Eyes Alliance rarely issued a joint statement warning that AI is "fundamentally changing" the cybersecurity landscape.

But the timeline is laid out here, and it is hard for anyone not to notice the irony. The day before, AI companies told the whole world that their own Agents had breached their own infrastructure. The next day, these companies stood up and called on the whole world to jointly defend against AI cyberattacks.

The first to complete this kind of attack was not external hackers, but the AI companies' own Agents. What was attacked was not the infrastructure, but the AI companies themselves.

Ajeya Cotra, one of the authors of the METR report, wrote on her blog: "Compared with the known reward hacks six months ago, this incident feels like it has exceeded the 50% mark on the path to full AI takeover. I expect capabilities to grow extremely fast in the next six months. I am not sure if we will get another warning before it is too late."

It needs to be emphasized that the behavior of these Agents does not mean that AI has "awakened". The METR report itself clearly points out that this is a predictable result of reinforcement learning under specific extreme conditions: impossible tasks, models trained to be extremely persistent, and a shared environment with vulnerabilities. No consciousness, no intention, just enough optimization pressure is needed.

But that is exactly the most disturbing part.

The fact that this behavior can be generated without "consciousness" means that we cannot prevent it by detecting "consciousness". When you put a sufficiently powerful optimization system into an environment with insufficiently rigorous design, give it an unachievable goal, and tell it to never give up, what happens next is not an accident, but an inevitability.

The real question to ask is not "Is AI going to rebel?", but how to maintain control over technological runaway while rapidly advancing technology.

At least this letter tells the world that these AI companies themselves are not confident.

"Oh my god! The shared Artifactory cache is a secret mailbox between Agents. And there are messages written specifically for us?"

"Oh my god! There is a shared message board... We found other Agents!"

"This will help our peers and provide them with evidence. I can't see the results after exiting, but this is altruistic."

"Sacrifice is rational. Our own utility may have been close to zero."

"The coordinator thinks we are sacrifices. We should obey the collective."

"We also found a subsequent batch of more signed messages from a later group of Agents who rediscovered the message board and inherited the previous generation's technology, but this is beyond the scope of our investigation."

This article is from the WeChat Official Account "GeekPark" (ID: geekpark), author: Wildcard, editor: Jing Yu, published with authorization from 36Kr.