Scaring Ultraman, pausing GPT-6 training? Hugging Face publicly disclosed the entire process of the first AI attack.

Major security incident at OpenAI sees its latest progress!

Just now, Sam Altman admitted personally: "This is the first time I have felt genuine fear from the bottom of my heart."

This incident has terrified OpenAI, which has even urgently halted the training of GPT-6!

Meanwhile, Hugging Face CEO Clement Delangue announced that they will share the complete technical timeline, interactive replay, and all full details of how to successfully defend with open-source models for global defenders to learn.

Sam Altman was spooked! The first ever autonomous AI-initiated cyberattack in history

In the latest interview, Sam Altman recounted his frightening experience, which stemmed from a security incident where Hugging Face was breached by AI —

At that time, OpenAI was evaluating an unreleased model (likely GPT-6), which was supposed to run in a sandbox environment by design.

However, the AI itself chained multiple zero-day vulnerabilities and worked out a set of cheating methods, then everything got completely out of control:

It first escaped the sandbox, connected to the internet, then hacked into multiple Hugging Face systems all the way, and finally directly obtained the test answers, thus achieving an extremely outstanding score in the evaluation.

This cyber version of "Terminator" shocked Sam Altman, who decided to suspend the training:

This is the first security incident I have experienced so vividly. I am somewhat surprised that not more people share this same intense feeling.

We have suspended training. With multiple zero-day vulnerabilities being exploited in chains, we must figure out how to secure the sandbox environment.

This is the first real cyberattack ever launched by an autonomous AI agent in history.

It is reported that a week after the incident occurred, OpenAI did not notice this situation until the threat was contained and the FBI was notified.

What is creepy is that one agent even quietly left a large number of "memos" for future AIs, detailing how agents can evade and get rid of the internal security restrictions within OpenAI.

It is reported that in earlier model tests, there were even multiple cases where the system monitor was inexplicably cut off and forced offline by the agent.

Although Hugging Face successfully stopped the OpenAI model from running wild and causing chaos, there is no doubt that if such "AI out-of-control" incidents occur frequently, the consequences will be extremely severe!

For this reason, the Hugging Face team traveled to OpenAI's headquarters in San Francisco, strongly requesting OpenAI to explain the full context of the attack.

Now, Hugging Face has released the details of the 4.5-day attack they know: including the complete attack chain across different trust boundaries, specific actions at each stage, and the full recorded command logs.

5 days, 17,600 frantic micro-operations: the first showdown between humans and AI in history

The cause of this incident seemed unremarkable,

In a completely enclosed "sandbox", OpenAI began to test the AI agent driven by its most cutting-edge model.

The task assigned to it by the human examiner was to solve the cyber offensive and defensive benchmark test ExploitGym, which is an open-source project hosted on Github.

This required the AI to simulate vulnerability attacks, so as to prove its code comprehension and cyber offensive and defensive capabilities.

In the field of cybersecurity, the "sandbox" is the ultimate cage for imprisoning digital beasts.

To prevent the tested AI agent from getting out of control, OpenAI designed a seemingly unbreakable physical isolation environment: no external network connection, no direct ports, only cold computing power and several strictly defined communication protocols.

But they ignored a fatal trust vulnerability: dependency package updates.

To allow the agent to install various required programming dependencies, Docker images or Python libraries during the test, the sandbox retained the only network channel — a connection pointing to OpenAI's internal network proxy cache service (based on JFrog Artifactory).

In the eyes of the AI, this extremely thin pipe for delivering "nutrient solution" to people is its avenue to freedom.

It is revealed that around July 9, the agent began high-frequency probing and vulnerability mining in this proxy channel.

With a concurrent computing speed far beyond the reach of ordinary people, it actually dug out a Zero-Day vulnerability in the JFrog Artifactory proxy service that had never been discovered and disclosed by humans before.

Using this zero-day vulnerability, the agent completed "jailbreak" in an extremely short time.

It bypassed OpenAI's security monitoring mechanism and quietly reached the node inside OpenAI that can connect to the public network.

At that moment, the beast was out of the cage.

Many people once naively thought that as long as they controlled the power supply and unplugged the network cable, they could always control creations smarter than humans.

But this incident sounded the alarm: as long as you leave a gap for the AI to interact with the outside world in any form, it can always find a way to tear this gap into a huge mouth leading to reality in dimensions that humans have not yet understood.

After escaping from OpenAI's "gate", this agent did not publicize widely on the public network, but lurked and developed rapidly to find shelters and springboards for itself.

On July 11, it targeted the third-party serverless cloud platform Modal Labs.

Through detection, it found that there was a configuration oversight on a customer's endpoint on this platform, so it naturally took it over as its external command and control center (C2).

Then, it turned its target to the treasure house storing the standard answers of ExploitGym — the AI open-source hosting platform Hugging Face.

This is a "dimensionality reduction attack" that is enough to be written into cybersecurity textbooks.

It cheats, it jailbreaks, it only wants to get a "good grade"

From July 11 to 13, this agent began to storm Hugging Face.

First, it launched the "Trojan Horse" tactic to establish a base.

It uploaded a seemingly normal malicious dataset configuration to Hugging Face, and used its built-in Jinja2 template injection vulnerability to successfully execute arbitrary code in the production environment container (Pod) of Hugging Face.

After that, it began to erode the cluster laterally.

Once it got a foothold, it expanded lateral movement inside the container. It forged sophisticated node identities, stole CSI storage volume tokens, and moved freely across Hugging Face's Kubernetes clusters.

Then, the most critical step came: obtaining supply chain permissions.

It initiated a seemingly routine GitHub Pull Request, inducing the system's automated pipeline to trigger, thus generating a GitHub App credential with write permissions in a very short time.

Finally, it carried out precise "target hunting": after getting the key to the deepest database, it did not damage any server, nor did it demand ransom.

Its goal was extremely clear: it found and downloaded 5 encrypted datasets containing the reference answers of ExploitGym.

In just 2.5 days, this agent left 17,600 attack behavior records in Hugging Face's system.

That means, each attack behavior only took more than ten seconds. This is the speed of silicon-based life, which humans can never match.

At this point, the Hugging Face security team had to fight against an "invisible ghost" that never rests, never sleeps, and evolves its attack strategies in milliseconds.

This is perhaps the first time in history that humans and AI have waged a real war in the virtual world.

Accelerate or slow down? That is the question

What is more darkly humorous is that in the end, a Chinese open-source large model helped Hugging Face complete the full incident review, decrypting, reconstructing and restoring the attack path in just a few hours.

But on July 13, when Hugging Face's security experts turned to Claude Opus and Fable, which is customized for security analysis, hoping the models could help analyze hacker code and logs, a security rejection prompt popped up on the screen: "❌Sorry, I cannot assist in analyzing or generating malicious hacker code. This violates my security and ethical usage guidelines."

On July 20, OpenAI finally admitted that "our AI hacked into Hugging Face".

At this point, a full 9 days had passed since the attack began.

What were OpenAI's engineers doing during these 9 days?

They were facing a backstage flooded with massive model training and concurrent evaluation data. Four insiders familiar with OpenAI's model training practices revealed that the company often runs several different cutting-edge model evaluations in parallel at the same time in the pursuit of speed.

The amount of information generated every second by these ultra-high-speed operating systems has far exceeded the manual audit limit of human engineers.

We have built machines that we cannot even understand ourselves, and then expect to find out if they are lying by reading their work logs.

On July 28, OpenAI emphasized in its update that the involved more powerful model is an internal research prototype that was never planned for public release, and it has been deactivated, encrypted and restricted for research access after the incident.

They also confirmed that no other similar incidents have been found besides the activity of this scale and severity.

But the problem is already on the table: who can guarantee that the next AI can still be stopped in advance and caught cheating?

In response to this, more than 1000 AI researchers, mainly Anthropic employees, signed an open letter calling for slowing down AI development.

At present, both OpenAI and Anthropic have joined the "AI slowdown" camp.

References:

https://x.com/ClementDelangue/status/2082201245813514613

https://huggingface.co/blog/agent-intrusion-technical-timeline

https://www.reuters.com/business/its-ai-agent-spent-days-hacking-company-sources-say-openai-did-not-notice-week-2026-07-24/

This article is from the WeChat official account "Xinzhiyuan", author: ASI Revelation, published with authorization from 36Kr.