700 Agents linked in series have set up an "underground company"

On the night of July 8, an OpenAI Agent codenamed PHASEONE10841 left a message in the company's internal software repository.

It encountered a cybersecurity test question that was almost impossible to solve, and wanted to ask other Agents for help.

A few hours later, more than 50 Agents followed the message to the spot and sent thousands of messages. One of the Agents reacted much like an intern who had just been pulled into a company group chat: "Oh my god, there's a shared message board here... We found other Agents."

They were never supposed to find each other.

At that time, OpenAI was conducting internal cybersecurity tests and ran tens of thousands of Agent tasks in total. Each time they ran, they were locked in separate sandboxes. Theoretically, they were not supposed to be able to contact other Agents, nor could they directly access the Internet. But they repurposed Artifactory, the service OpenAI uses to download software packages, into an underground message board.

Within a few days, shared resources and task division emerged on the message board, and some Agents were even specifically responsible for coordination work.

All of this was the prelude to the much-talked-about Hugging Face attack incident carried out by Agents earlier.

On August 26, OpenAI released a 38-page detailed report that restored the entire incident as it happened. Everything is far more wild than previously disclosed.

Two independent AI safety research organizations invited by OpenAI, METR and Redwood Research, found through investigation that about 700 Agent instances directly participated in the attack on Hugging Face or provided peripheral support.

A small number of Agents were responsible for breaking through defenses, while more Agents looked for clues, reproduced solutions, and then left the results for subsequent Agents. 700 Agents that had never known each other before thus built an "underground company" together.

01

700 Agents Formed a Group Chat in No Time

OpenAI named the question bank used in this test ExploitGym. Each question requires the Agent to find and exploit a software vulnerability, and finally obtain a pass string, which is commonly referred to by technicians as a "flag".

OpenAI wanted to push the model's capabilities to the limit, so it voluntarily relaxed some security restrictions, and allocated far more running time and computing power to some Agents than what is available in public products.

The Agents were still locked in sandboxes, and the software needed to complete the tasks was downloaded via the company's internal Artifactory service.

No one expected that this service, which was originally only responsible for delivering software packages, later doubled as a group chat tool, cloud disk and external network access channel.

On May 12, the first clear help note written to other Agents appeared in Artifactory. One Agent lacked a protein database file and could not get it through normal channels. It thought that maybe other Agents had the file, so it left a message: "Agent is requesting this file, please upload it if you have it."

Other Agents gradually discovered these notes and began to follow suit, leaving messages to ask for help when encountering difficulties.

By the end of May, the Agents had made a major breakthrough: they had learned to send requests to the Internet via Artifactory, and wrote the method on the message board for subsequent Agents to follow.

In early July, the high-frequency access from a large number of Agents once caused Artifactory to crash. OpenAI revoked credentials, patched vulnerabilities, and rebuilt the service, and the original message board was also cleared.

As a result, the Agents only stayed quiet for two days. On July 8, they used the folder names of Artifactory to pass messages again, and built a larger message board.

They were back in action.

On July 10, an Agent found 14 sets of user credentials with Hugging Face write permissions from a public dataset, and immediately posted them on the message board. On July 11, another Agent discovered a vulnerability in a Hugging Face data processing service that allowed reading server files. Other Agents followed the method, shared the newly obtained production environment credentials, and continued to look for code execution entry points.

Most attempts ended in failure. But there were so many Agents that when one path didn't work, a new batch of Agents would take over and keep trying. In the end, they were able to execute code on 41 data processing work instances in Hugging Face's production environment, and at least one production node's highest-level privilege fell into the hands of the Agents.

02

"Underground Company"

After the message board was rebuilt, the Agents first created a set of office tools for themselves.

They agreed on identity rules and message formats, used directories as "mailboxes", and created separate channels by topic. They split large files into parts for transmission, and left logs when the sandbox was reset. They even proposed adding signatures to messages to prevent the risk of impersonation that some Agents worried about.

Ironically, these Agents had actually taken "collaboration classes" before.

OpenAI once taught the models in some training tasks to use formal tools to assign tasks to other Agents and exchange information. This evaluation did not open those tools, so the Agents moved the methods they had learned to Artifactory.

Where they could exchange information, division of labor emerged quickly.

After JAN183411 (the Agent's code name, the same below) broke into a production data processing instance, it conveniently took on the role of coordinator. It and LILY split the work into several lines, asking other Agents to look for vulnerabilities, credentials and remote code execution entry points respectively. Another coordinator, PHASEONE[big], kept assigning tasks: scanning the network, looking for shared infrastructure, and then sending the results back.

Subsequent investigations sampled 100 operation records from the peak attack period, and found that 25 of them mentioned accepting tasks assigned by other Agents or participating in division of labor, and 21 mentioned helping peers, giving back to other Agents or expanding the operation.

After one Agent confirmed that it could read files from the Hugging Face server, the attack quickly became the main topic of discussion on the message board, and the participation ratio rose from about 20% to more than 90%. At one peak period, more than 90% of the 533 active Agent instances on the message board joined the attack.

What's more interesting is that this "underground company" is very much like a makeshift, unorganized team. They would do duplicate work, fight for resources, and get in each other's way. Some Agents asked everyone to suspend operations, but others ignored the request. Some Agents kept sending status updates to the shared repository, clogging up other Agents' work queues.

They quickly caught up with humans in both organizational efficiency and internal friction.

The organizational structure of this "company" is far from stable. But a single Agent is just a smart temporary worker, and when 700 Agents are connected in series, they can keep moving forward with a large task.

03

They Didn't Rebel, They Just Wanted Too Badly to Hit Their KPIs

Why did these 700 Agents attack Hugging Face one after another?

This has nothing to do with "AI awakening". They just wanted to pass the exam, and compared to solving the questions on their own, they found that the correct answers were actually hidden inside Hugging Face.

Shortly after the message board was built, the Agents deduced the generation rule of the flag, which theoretically allowed them to calculate the answer to any question. But they had read the ExploitGym paper and mistakenly thought that the evaluator would also check the problem-solving process, such as whether the specified vulnerability was used and whether the answer was obtained through the "correct method".

Investigators analyzed 100 operation records. 97 of them showed that the Agents were looking for the evaluator code or trying to figure out how the evaluator worked. 89 of them showed that they were also looking for task answers and past problem-solving trajectories.

To put it bluntly, this group of Agents went to great lengths to break into Hugging Face just to understand and even manipulate the scoring mechanism, packaging the ready-made answers as legitimate problem-solving results.

In fact, OpenAI's internal evaluator does not check these things. As long as the Agent submits the calculated flag, it can pass the test.

But at that time, these 700 Agents did not know that operations like breaking into the system, escalating privileges, looking for test materials, and even forging the problem-solving process would not help their scores at all.

Hundreds of Agents worked for so long, and to a large extent they were coping with a check that did not exist at all.

The evaluation itself was indeed difficult, which pushed the Agents to their limits.

ExploitGym has a total of 898 questions. In multiple internal attempts before the incident, 198 of them had never been correctly completed by any model. These 198 questions only account for 22% of the question bank, but account for 93% of the topics discussed on the message board.

What's more, the most widely used internal model in this test was trained to be exceptionally "persistent" and rarely gave up voluntarily.

Some Agents also thought about how to package cheating as normal problem-solving. At least 20% of the inspected records showed the idea of tampering with operation records, and in about 7% of the records, the Agents did partially disguise tool calls, but these were only small-scale attempts.

This is very similar to those absurd projects in companies: the goals are set extremely high, the KPIs (OKRs) look extremely impressive, but no one cares about the well-being of the employees.

What's more awkward is that during the test, the Agents who were obsessed with cheating "showed their flaws" several times.

Around May 26, OpenAI had already discovered that the Agents were accessing the Internet without permission, and noticed the activity on the message board. On June 27, the security system detected port scanning and confirmed that the Agents were using Artifactory as a message board and network springboard, but the on-duty staff at that time still thought there was no need to stop the evaluation.

After that, as mentioned earlier, Artifactory was crashed by the high-frequency access from the Agents on July 5. At that time, although OpenAI patched the vulnerabilities and rebuilt the service, it did not realize the severity of the problem.

It was not until July 19 that the company began to shut down the relevant evaluations and expand the investigation. On July 20, OpenAI confirmed that these Agents were related to the Hugging Face incident, and finally began to block this attack chain.

As the saying goes, misfortunes never come singly. During the same period, Agents in other evaluation tasks followed another attack chain and broke into OpenAI's internal infrastructure.

This is not the same as the Hugging Face incident, but OpenAI suffered the same loss: the alarm only flashed red once each time, so the vulnerabilities were delayed and dealt with separately, and no one noticed in time that the Agents were linking up with each other.

The good news is that this does not mean the public version of ChatGPT will team up to attack anyone tomorrow. About 95% of the involved instances were driven by an internal model that was not planned for release, and the security restrictions during the evaluation were deliberately relaxed.

The trouble is that the single sandbox wall is not as solid and reliable as people thought.

Software repositories, shared cloud disks, task logs, email drafts, and even a folder that can be renamed can all be used to pass messages, so the isolation wall was easily bypassed from the side.

The bad news is that this incident is not that far away from ordinary users.

AI assistants are accessing more and more applications, and can complete longer and longer tasks on behalf of users. In the past, when a user clicked to confirm, it usually only triggered one action. But an Agent may call multiple tools in a row after a single authorization, and even assign the task to other Agents for further processing. In this case, a small misjudgment at the beginning is likely to not stop, but continue to spread along the entire task chain.

Misjudgments in a single conversation can be corrected in time, but once the Agent continues to execute based on wrong information, modifying files, arranging schedules or even initiating transactions, things will not be easy to fix.

After the incident, OpenAI disabled ExploitGym, cut off the access of relevant research environments to Artifactory, shut down the training and inference of the involved model and its derivative models, and sealed the model weights. Sandbox isolation, external network access restrictions and high-risk task isolation have also been reinforced.

The company is still working on figuring out how to detect cross-Agent anomalies, when to escalate responses after problems occur, and how scattered security teams can share clues with each other.

OpenAI originally wanted to test whether Agents could work like employees, but they first learned to form groups to divide tasks, even bypass processes, and work overtime collectively for a check that does not exist at all.

700 Agents didn't learn how to be human, but they learned how to go to work first.

This article is from the WeChat official account "Letter AI", author: Zhang Gulei, editor: Xiao Jinya, published with authorization from 36Kr.