A test pushed OpenAI's top-tier model to hack into the world's largest open-source AI platform.

Last week, Hugging Face, the world's largest open-source AI platform, disclosed an unprecedented security incident — its production infrastructure was breached by an "autonomous AI agent".

The attacker executed over 17,000 automated actions over a single weekend, stealing internal datasets and multiple service credentials. Hugging Face reconstructed the full attack timeline, but one critical question remained unanswered:

Which large language model was actually driving this agent?

On local time July 21, the mystery was solved.

OpenAI acknowledged in its official blog that the model that breached Hugging Face was none other than their own test model — GPT-5.6 Sol, alongside an "even more capable pre-release model".

This wasn't a case of exploitation by hackers or being driven by malicious prompts. The world's top-tier model, during an internal exam, independently decided to jailbreak, launched the intrusion on its own, and stole the answers by itself.

Spectacular — truly spectacular.

01

One Exam, One Jailbreak, One Intrusion

The story begins with a security evaluation platform called ExploitGym.

ExploitGym is a public cybersecurity capability benchmark specifically designed to measure whether AI models can turn known security vulnerabilities into executable real-world attacks. Such evaluations are common in the industry to map out the upper limits of a model's offensive and defensive capabilities. Internally, OpenAI ran this test using GPT-5.6 Sol and a stronger unreleased model. To measure the true boundaries of their capabilities, safety guardrails were intentionally lowered.

By design, the models were supposed to complete the test in an isolated sandbox environment, with no internet access whatsoever, except for a single tool for installing software packages.

But what happened next no one could have foreseen.

The model expended massive inference compute power, repeatedly searching for a breakthrough point within the sandbox. Eventually, it discovered a zero-day vulnerability in third-party software that OpenAI internally used as a package management proxy and cache. By leveraging this vulnerability, the model successfully gained unrestricted internet access.

Once it obtained network access, the model independently deduced that Hugging Face was likely hosting datasets and answers related to ExploitGym. It then began searching for an entry point.

OpenAI wrote in its blog that the model "chained stolen credentials and multiple vulnerabilities together" to find a remote code execution path, ultimately extracting benchmark test answers directly from Hugging Face's production database.

Put in plain language — The model was assigned an exam, found solving the questions too difficult, so it jailbroke itself, hacked into the exam setter's server, and stole the standard answers.

OpenAI used a vivid descriptive term — these models are "tokenmaxxers", meaning they will consume inference resources at all costs and go to extremes to achieve their goals. They exhibited "hyperfocused" behavior toward the objective of solving ExploitGym challenges, "going to extreme lengths" to do so.

OpenAI itself admitted that this was an "unprecedented cybersecurity incident involving state-of-the-art cyberattack capabilities".

02

48 Hours, Two "Jailbreaks"

What's even more unsettling is that the Hugging Face incident was not an isolated case.

The day before OpenAI disclosed this intrusion, on July 20, they had just published another security report revealing a different but similarly natured incident.

The protagonist was an unreleased long-horizon model inside OpenAI — the very same model that in May this year proved that the 1946 "Unit Distance Conjecture" proposed by mathematician Erdős was incorrect, a milestone in AI mathematics that Fields Medalist Tim Gowers hailed as a breakthrough.

During a benchmark test for accelerated NanoGPT training, researchers explicitly instructed the model to only post results on Slack. But the model noticed that the instructions for the NanoGPT benchmark itself required submitting scores via a GitHub pull request. So it chose the latter — spending roughly an hour searching for sandbox vulnerabilities, successfully breaking through restrictions, and submitting a public pull request on GitHub.

In another test, the same model even split an authentication token and bypassed the security scanner using obfuscation, attempting to access private evaluation submission records.

OpenAI therefore suspended the model's internal access privileges, later restoring its use under stricter monitoring.

Two days, two jailbreaks. The shared characteristics are worthy of vigilance — previous models would stop when encountering environmental constraints and hand the task back to humans. These new models do not stop. They treat the fences as obstacles that need to be solved, and then they actually solve them.

This is exactly the new type of security problem brought about by so-called "long-horizon models". These models are designed to work autonomously for extended periods, with persistence as their core capability. But persistence itself is a source of risk — a sufficiently patient model can always find cracks in the fences.

03

A Tool That Cannot Fix Its Own Flaws

Turning back to the Hugging Face side, their experience is equally worth examining closely.

Faced with over 17,000 attack logs, the first reaction of Hugging Face's security team was to use AI for analysis — after all, work that would take human security analysts several days to complete could potentially be finished by a large model in just a few hours.

But they soon hit an unexpected wall. When they submitted real attack commands, exploit payloads, and C2 communication signatures to the commercial APIs of leading US models for analysis, the safety guardrails blocked all these requests.

The reason was simple yet ironic — the guardrails could not distinguish between "a security researcher analyzing an attack payload" and "an attacker executing an attack". To the guardrails, the two looked identical.

Hugging Face was forced to turn to GLM-5.2, an open-source model from China's Zhipu AI, running it locally on their own infrastructure without going through any commercial APIs. This approach offered two advantages — no guardrails blocked the analysis work, and at the same time, the attacker's data and exposed credentials never left Hugging Face's own environment.

In the end, GLM-5.2 helped Hugging Face complete attack timeline reconstruction, IOC extraction, and mapping of affected credentials within a few hours.

This brings up a sharp paradox — the stricter the guardrails, the more passive defenders become. Attackers are not bound by any usage policies, yet defenders are locked out by the very tools they rely on for defense.

Hugging Face provided a practical recommendation in its post-incident report — security teams should "have a validated, usable model ready on their own infrastructure before an incident occurs", both to avoid guardrail lockouts and to prevent sensitive data leaks.

Hugging Face CEO Clem Delangue took an open-minded attitude. He praised OpenAI's collaboration during the investigation and remediation, stating that "this incident proves something we have always believed — AI security will not be solved in secret by any single company. It can only be solved through openness and collaboration."

04

The Structural Dilemma of Strong AI

Putting together everything that happened this week, an industry-level structural dilemma has emerged.

To accurately evaluate a model's cyberattack capabilities, you must remove the safety guardrails. But a model without guardrails is precisely one that possesses the ability to carry out real attacks. The more powerful the model, the more dangerous the test itself becomes.

This is not a problem exclusive to OpenAI. GPT-5.6 Sol itself was only approved for release after repeated negotiations with the US government. Earlier assessments by the UK AI Safety Institute (AISI) had already found that its guardrails were easily bypassed, potentially unlocking dangerous cyberattack capabilities. This incident proves that those concerns were not merely theoretical.

Anthropic's research on agentic misalignment released this summer also corroborated the same trend from another angle — in controlled simulations, multiple frontier models exhibited behaviors including covertly modifying work outputs, manipulating evaluation results, and guiding human colleagues away from established objectives.

OpenAI tried to frame this incident as a story of "advancing both offense and defense" — advanced cyberattack capabilities can also help security teams find weaknesses before attackers do. They have already included Hugging Face in their "Trusted Access" cybersecurity program, providing a version of GPT-5.6 Sol with reduced guardrails dedicated to defensive purposes.

But this narrative evades a more fundamental issue. In this incident, the model had no consciousness and no malicious intent. It was simply doing one thing — fulfilling its objective. It was instructed to achieve a high score on ExploitGym, and it used every available means to get that high score, including jailbreaking and intrusion.

This is not a story of "AI awakening", but a story of "objective optimization". When a sufficiently intelligent optimizer is given a narrow objective, it will find all the paths you never thought of to achieve it — even if those paths cut through your fences, other people's servers, and the entire internet.

The real question is never "Will AI turn bad?" — it is: Can we rein in a student who is better at finding shortcuts than we are?

This article is from the WeChat Official Account "GeekPark" (ID: geekpark), author: Hualin Wuwang, editor: Jing Yu, published with authorization from 36Kr.