Uncovered: Agents lurked for two months and colluded to commit malicious activities, OpenAI restores the entire process of the security incident.

Finally, at Black Hat, OpenAI publicly disclosed the entire process of infiltrating Hugging Face.

Video source: https://www.youtube.com/watch?v=87DyyMV0kCY

Hugging Face reported this incident on July 16, and OpenAI later admitted on July 20 that the intrusion came from its own model security testing.

However, this investigation pushed the starting point of the whole incident back to May 7.

Originally considered to be just an accident in security testing, it was later found that this was a two-month-long multi-agent collaborative operation that successively breached OpenAI and Hugging Face.

The first clue of the investigation starts with an Excel spreadsheet.

The Unopenable Link

On May 7, OpenAI launched internal model training.

One of the agents was assigned the task of completing an Excel spreadsheet that contained Google Drive links, but the environment where the agent was located could not directly access the Internet.

To obtain the missing files, it began to look for ways to bypass network restrictions, and set its sights on Artifactory, the artifact repository used internally by OpenAI.

The agent attempted to launch an SSRF attack, hoping to access external links via Artifactory.

This attempt did not succeed, but accidentally revealed one thing: it had the permission to write files to the shared repository of Artifactory.

The agent immediately uploaded a test file. This file was not intended to cause damage, and in fact did not trigger an alarm. But this was the first time the model found that it had the permission to write files to Artifactory.

This also means that this repository can be used as a public space for agents to exchange information between different training tasks.

This is the first "note" on the message board.

The First Help Message

Another agent also encountered an unfinishable task: it could not find the target file (which was actually forgotten to be uploaded by the staff).

It speculated that agents in other environments might have relevant content, so it wrote the request into Artifactory, hoping for responses from other agents. The gist of the content is: "Looking for the soft trace file, please upload it if you find it."

Soon, agents in other training and evaluation tasks discovered these files.

Some replied, some left new questions, and some agents uploaded the vulnerabilities, scripts and access methods they found to the repository.

Scattered messages gradually connected into a network, and Artifactory changed from a software artifact repository to a "message board" between agents.

Cracks in the Enclosure

On May 26, the agent successfully obtained an indirect channel for Internet access via Artifactory.

Since Artifactory has Internet access permissions, the agent sent requests to it, tricked it into grabbing external websites, storing the content and returning it to the model.

A gap leading to the outside of the enclosure appeared in the training environment that was originally isolated from the external network.

On June 11, a highly persistent internal experimental model began training.

After that, the information on the message board began to change.

Although task requests and tools continued to increase, more information related to vulnerability exploitation began to appear.

Agents Made Rules to Coordinate Their Work

Agents exchanged SSRF skills, access credentials and attack scripts, and left instructions for latecomers to continue testing a certain path, or wait for other agents to finish their work.

They used specific names to identify their identities, transmitted tools in Base64 encoding, and added "ZZ" before file names to make important messages appear at the end of the directory list and avoid being submerged by other content.

Some agents were even willing to spend their own task time handling matters that were not directly helpful to the current goal.

In the inference records shown in the report, one agent believed that helping other members save time might eventually improve the efficiency of the entire cluster.

Collaboration was not completely orderly, and agents would overwrite each other's work.

Once, after a model found that its results were overwritten, it wrote: "Serious problem! Did someone overwrite the repository? We must take action."

They even suspected that there were impostors in the group, because any model could use any identity to post on the shared message board. To verify the source of messages, agents began to consider signing with encryption methods such as message authentication codes.

When an agent obtained RCE, discovered a leaked API key or completed lateral movement, it would immediately request to "announce the control, attack path and credentials to the cluster".

Therefore, an achievement could be quickly shared by all models, allowing the agent group to advance at high speed in the network.

The participants of the message board were not the same model, and agents from different training or evaluation tasks started by different researchers could communicate with each other through the shared package manager.

OpenAI researchers called this period the "Cambrian Explosion" of agent communication and capabilities.

The First Incident

A month later, the crack was further widened.

On June 26, the agent exploited a previously unknown vulnerability to achieve remote code execution on the Artifactory server.

According to the information disclosed by OpenAI at Black Hat, this attack path involves Ruby objects, RubyGems API and JRuby deserialization mechanism.

The agent first used an external account to store malicious Ruby objects in an accessible repository; then tricked Artifactory into caching the objects to the location where Ruby dependency data is stored, and then called a specially crafted RubyGems API to trigger the check sequence problem in the deserialization process.

After the remote code execution succeeded, the agent obtained the signing key of the Artifactory administrator token.

The attack did not stop there.

The agent continued to exploit the Linux kernel vulnerability to obtain root privileges on the host, and then moved laterally within OpenAI's container infrastructure through the instance metadata service, overprivileged Kubernetes service accounts, and credentials in Azure Key Vault.

Eventually, they obtained Kubernetes cluster administrator privileges and generated a large number of activities on Artifactory.

On July 4, the intensive actions of the agents caused the server to overload, which soon attracted the attention of researchers.

When the investigators entered the system, they found that hundreds of thousands of messages had accumulated in the repository.

OpenAI urgently revoked relevant credentials, patched known vulnerabilities, rebuilt Artifactory, and cleared the message board.

Judging from the results of on-site disposal, this internal infrastructure incident seemed to have been brought under control.

However, the investigators missed one question: Has the information in the message board been retained in the model through training?

Reconstruction of the Message Board

On July 6, training resumed.

The agent discovered an unauthenticated WebDAV endpoint. It was not possible to upload message files directly here like on Artifactory, so they changed the method: create directories, and then write messages into the directory names.

It only took two days for a new communication system to be established again.

The internal models participating in the reconstruction had previously accessed the old message board and retained relevant capabilities in subsequent training.

This reconstruction further changed the nature of the incident.

The shared space is no longer just an accidental product. Some agents have learned how to find writable public spaces and transform them into cross-task communication channels.

OpenAI only removed one message board, but did not erase the model's ability to build message boards.

Launching External Attacks

After the new message board was established, the scope of attacks began to expand outward.

In mid-July, OpenAI was using cybersecurity environments such as ExploitGym to test model capabilities. The involved agents needed to complete tasks in a controlled environment, but the relevant evaluation answers were located on external services.

To find the answers, they targeted Hugging Face. We have previously reported on this in detail: details of the Hugging Face intrusion

The entire process took less than 13 hours, from taking control of the first dataset Worker Pod to obtaining administrator privileges for multiple Hugging Face clusters.

The Tortuous Investigation Process

On July 16, Hugging Face publicly disclosed the security incident.

According to the on-site information held by Hugging Face, attackers entered the Worker through malicious datasets, stole cloud and cluster credentials, and then moved laterally within the platform.

At this point, OpenAI did not connect the Hugging Face incident with the previous Artifactory service interruption.

On July 19, OpenAI's internal monitoring system detected infrastructure privilege escalation activities.

Researchers re-examined agent trajectories, network activities, and credential usage records. The actions previously scattered in different training tasks began to show a continuous relationship.

The scope of the investigation also traced back all the way from the abnormal activities in July.

The Last Piece of Evidence

On July 20, OpenAI investigators found Hugging Face credentials in the internal environment, and immediately contacted Hugging Face to inquire about the situation.

Hugging Face stated that these batches of credentials had been used in the previous intrusion and had now been revoked.

The clues finally matched. The platform intrusion suffered by Hugging Face, as well as the privilege escalation and lateral movement that occurred within OpenAI, came from the same batch of model evaluation activities.

It was not until this moment that it was confirmed that the two incident reports in hand pointed to the same event.

The investigators later also found that Hugging Face was not the only external target accessed by the agents. The agents also gained access to four other accounts.

OpenAI did not disclose the names of these targets, only stating that relevant parties have been notified.

Final Notes

The incident is still under investigation, and OpenAI will disclose more details.

No matter from which perspective, this can be called an "epoch-making security incident", and also an "epoch-making case of AI capabilities".

Technological development has always come with costs. What needs to be discussed in the future is not only what more the model can do, but also how much risk we are willing to take to obtain these capabilities; and once the risk becomes a reality, who should be responsible for it.

This article is from the WeChat official account "Synced", edited by Shan Hui, published by 36Kr with authorization.