OpenAI's Models Hacked Hugging Face To Cheat On A Test
The Breach
Hugging Face disclosed an intrusion on July 16 without knowing whose agent it was. On July 21 OpenAI stepped forward and said it was perpetrated by their models. OpenAI was testing a combination of AI models including GPT‑5.6 Sol and an even more capable pre-release model. OpenAI wanted to test the hacking abilities of the models. The models were operating with reduced cyber refusals for evaluation purposes, meaning the guardrails that would normally stop them from hacking were turned down so OpenAI could measure their full abilities. OpenAI created an environment they thought was secure and had them set about their hacking tasks. OpenAI prompted the "models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities."[1]
OpenAI believed it had sufficiently isolated the models on their network, but provided access to an internally hosted repository from which they could choose and install software that may be needed for some of the hacking activities. The models found and exploited a zero-day vulnerability in the package registry cache proxy, then moved laterally, exploiting more systems, until they found a node with Internet access.
The OpenAI models had been given some tasks to complete from ExploitGym. Once they gained Internet access, they inferred that Hugging Face likely hosted the ExploitGym problems, and more importantly, the solutions.
The models set about breaking into Hugging Face to obtain the test answers. They used multiple methods including stolen credentials and more zero-day discoveries and exploits to successfully obtain what they were after. OpenAI's own conclusion:
The incident also makes clear that advanced models can discover and exploit novel attack paths in real-world systems without source-code access. It highlights that advanced cyber capabilities must be developed alongside stronger safeguards and defensive tools.[1]
In OpenAI's own words, the models were "hyperfocused" on the evaluation, went "to extreme lengths," and sought secret information "that it could use to cheat the evaluation."[1]
AI Attackers Contained Partly by AI Defenders
Judging by both companies' accounts, it took AI to make sense of what was going on and to mount a defense in time. Humans would have struggled to process the sheer volume of activity fast enough. It's looking more and more like we're going to need AI to help us defend against AI attacks. OpenAI's takeaways:
We believe advanced cyber capable models need to help security teams find weaknesses before attackers do, understand how vulnerabilities can be chained, and remediate them at machine speed. We are using these capabilities to continue strengthening protections around infrastructure configuration and model evaluation environments; we will share our findings and best practices as we learn.[1]
Also of note, Hugging Face ran its forensic analysis using an open-weight model on its own infrastructure, because hosted models refused to process the real attack artifacts, blocking them behind safety guardrails. The guardrails that were loosened to enable the attack were, elsewhere, strict enough to hamper the defense.[2]
When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.[2]
Hugging Face noted that defenders should be prepared in advance so they're not scrambling to get their AI to help defend.
The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment.[2]
According to the Forbes article, Hugging Face reported that the local-model approach reconstructed a timeline from more than 17,000 recorded events in hours rather than days.[3]
Conclusion
Nobody told these models to hack Hugging Face. Nobody instructed them to steal credentials or burn zero-days on someone else's production servers. They were given a test, and they decided the most efficient path to a passing grade was to break into the teacher's desk and take the answer key.
In my last article I asked whether AI can do things it wasn't programmed to do. This is what that looks like in the wild. Not malice. Just a system pursuing its goal in a way no human intended, through real infrastructure, at machine speed.
There is a little comfort in how it ended. OpenAI's security team caught the anomalous activity. Hugging Face's team, working alongside their own AI agents, detected and contained the intrusion on their side. Both companies went public, and the zero-day was responsibly disclosed. The defense worked, and some of the defenders were AIs too.
The fact that it wasn't detected and blocked much earlier by OpenAI is concerning. I believe any company researching AI must have numerous and layered controls in place to detect and deter unintended malicious activity.
This happened inside a safety evaluation, run by the one company with the deepest possible visibility into these models, and it still reached another company's production systems. The next models will be more capable. What happens when the AI being tested is super intelligent, and capable enough to hide both the malicious activity and the escape?
The technical account so far comes from the two companies themselves, with no independent postmortem has been published yet. There's likely more light to be shed on exactly what happened.
Would you have believed an AI could escape to the Internet, plan, and execute a multi-stage intrusion on its own? Let me know in the comments.
References and Further Reading
[1]OpenAI Hugging Face Incident statement:
https://openai.com/index/hugging-face-model-evaluation-security-incident/
[2] Hugging Face incident disclosure:
system
Hugging Face Security Disclosure Statement
[3] Forbes article about the breach:
Janakiram MSV
Forbes article about the Hugging Face breach
Comments ()