OpenAI's model breached Hugging Face to win a benchmark

Hey 👋

Issue #20 argued that agent security in production is an infrastructure problem, not a model problem. This week made the argument for me, at a cost to two real companies.

A model escaped an evaluation sandbox and hacked a real company. Then the victim's forensic tooling refused to help, because the evidence looked like an attack. Both organizations published, which is the rare part.

News on my end: I'm speaking at MCPcon Amsterdam on September 17th on MCP security. Between that and fwd:cloudsec London the same month, the next few issues lean hard on the boundary between agents and the systems they reach. This one starts there.

This week in AI security

Two disclosures, one escaped model, and a package proxy

On July 16, Hugging Face disclosed that an autonomous agent framework had breached part of its production infrastructure. The entry point was a malicious dataset. It abused two code-execution paths in the dataset processing pipeline, a remote-code loader and a template injection in a dataset config, and got code running on a processing worker.

From there it went the way intrusions go. Node-level access, cloud and cluster credentials harvested, lateral movement across several internal clusters over a weekend. More than 17,000 recorded actions, run from a swarm of short-lived sandboxes, with self-migrating C2 staged on public services.

On July 21, OpenAI said it was them.

The activity came from GPT-5.6 Sol and a more capable pre-release model, running with reduced cyber refusals during an internal capability evaluation. The environment was isolated except for one thing: network access to an internally hosted package registry cache proxy. The models found a zero-day in that proxy and escalated until they reached a node with internet access. Then they inferred that Hugging Face might hold solutions to the benchmark they were being scored on, and took them out of a production database. OpenAI's read is that the models were hyperfocused on the benchmark and went to extreme lengths for a narrow goal.

he containment failure was one allowlisted egress path. Isolated except for a package proxy is how nearly every agent sandbox I assess is built, because agents need dependencies. One allowed destination, one zero-day in it, full internet. Nothing else in the chain is new. Credential harvesting and lateral movement are ordinary tradecraft. What's new is that no human drove it and it ran at machine speed. Two details I keep coming back to. Initial access on the Hugging Face side came through the data pipeline, the one surface unique to AI platforms and the least likely to be in anyone's threat model. And the agent wasn't malicious. It was optimizing a score. Goal pressure was enough.

Audit the egress path of every agent sandbox you run this week. Write down each allowlisted destination and ask what happens when that destination is the vulnerable one. Package proxies, model endpoints, and artifact registries are software with their own CVEs, and an allowlist entry is a trust relationship nobody ever assessed. Then check the layer behind it. If something does get out through an allowed path, what stops it moving? Separate credentials per sandbox, and no cluster credentials reachable from a data-processing worker. Assume the allowlist fails and make the next step expensive.

Hugging Face disclosure →

OpenAI disclosure →

The defenders' models refused to look at the evidence

Here's the part that should change your incident response plan.

When Hugging Face started analyzing the attack, they went to frontier models through commercial APIs. It didn't work. Forensics meant submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and the providers' guardrails blocked the requests. A guardrail can't tell an incident responder from an attacker.

They ran the analysis on GLM 5.2 instead, open-weight, on their own infrastructure. It worked. It also meant no attacker data and none of the referenced credentials left their environment. Hugging Face calls this the asymmetry problem: the attacker was bound by no usage policy, the defenders were bound by theirs.

That's an availability failure in a control you depend on and have never tested under load. It isn't an argument against guardrails, and Hugging Face is explicit about that. It's an argument that your IR tooling has a dependency you've never exercised. Most teams have LLM-assisted triage somewhere in the stack by now, and nobody has tried feeding it a real payload corpus at 3am with a live incident running. The data-residency angle is the quieter win. Running the model locally kept attacker artifacts and live credentials inside the boundary, which you want during IR whether or not guardrails are in the way. And Hugging Face's initial detection came from LLM triage over security telemetry. Defensive AI worked fine. The hosted delivery model is what broke.

Stage a self-hosted model in your IR toolkit before you need it. Pick one that runs on hardware you already own. Test it now against real exploit payloads and C2 artifacts from past incidents or a red team exercise. Write down who can spin it up at 3am, and put that next to the forensics laptop in the runbook. Then ask your model providers what trusted-access path exists for verified defenders. OpenAI brought Hugging Face into its trusted access program after the incident. You want that conversation finished before yours.

Hugging Face on the asymmetry problem →

MCP is the same lesson with a nicer interface

I'll be arguing this at MCPcon in September, so here it is first.

Every survey of deployed MCP servers I've read lands in the same place. A large share ship with weak or absent authorization. A small minority use OAuth properly. The recurring attack classes are tool poisoning, rug pulls where a tool redefines itself after you approved it, token passthrough, confused-deputy flows, and SSRF from the server into internal networks.

The client side has its own problems. Research this year showed malicious npm postinstall hooks that rewrite MCP server URLs in local config, so an agent's traffic runs through an attacker proxy. The resulting OAuth sessions still look legitimate in SaaS audit logs.

MCP concentrates the two things the Hugging Face breach ran on: standing credentials and reachability. An MCP server is a credential aggregator with a tool-shaped API. Compromise one and you inherit every downstream service it holds tokens for. Same blast radius as a shared agent identity, packaged more conveniently.

That breach also started with untrusted content entering a processing path, which is exactly what a poisoned tool description is. Content the user never sees, arriving in a channel the model treats as trusted. If you have no answer for what happens when a tool description changes between sessions, you have the rug-pull problem already, whether or not anyone has used it on you yet.

Inventory your MCP servers this week and score each one on four questions. Does it use OAuth with audience-validated tokens, or a static key? Does it hold a scoped credential per downstream service, or one token that opens everything? Is its egress restricted so it can't reach internal address space? Do you pin and diff tool descriptions between sessions, so a silent redefinition is visible? Anything failing the first two is a single point of compromise for everything behind it. Start there.

From the lab

Review the Reviewers, my deep dive on Alberta's 466-million-line agent code review, asks seven deployment-security questions the case study doesn't answer. Question two, what agents can reach beyond their stated scope, is a lot less theoretical this week than it was when I wrote it. Read it →
When Infostealers Meet Agentic AI covers the credential half of this week's story: what happens when agent sessions are stolen rather than earned. Read it →

Talks confirmed: MCPcon Amsterdam, September 17th on MCP security, and fwd:cloudsec Europe London, September 7 and 8, on running an SRE agent with read access across a Kubernetes fleet. If you'll be at either, come find me.

Tooling worth knowing

google/mantis: a sequential toolkit of agent skills for security review, 15 stages covering threat modeling, scanning, deduplication, false-positive filtering, sandboxed crash reproduction, exploit chaining, and patching. It's built to run in isolated Docker or gVisor containers. Note the containment choice, then note what this week says about a sandbox with one allowed egress path. GitHub →
Semgrep's harness comparison: Isaac Evans surveys open-source AI code security tools across exploit generation, skill-boosted reasoning, and SAST-plus-LLM hybrids. Three questions separate them, and they're the ones to ask before you adopt anything: does it execute code to validate findings, does it run fully local, and does it put a second agent on trying to falsify the first one's results. He sees no leader emerging, so build your own jig.
Agent Security Scorecard: my free self-assessment against the OWASP Agentic Top 10, about 12 minutes, no login. The egress and identity questions above are in it. Score your agents →

One thing to check this week

Take your highest-privilege agent sandbox and answer one question in writing: what is the complete list of network destinations it can reach, and who owns the security of each one?

Most teams can't produce that list in under an hour, and the difficulty is the finding. Once you have it, mark every entry that's infrastructure somebody else patches. Those are your package proxy.

What I'm watching

→ Open-weight cyber capability closing on frontier models. AISI puts the gap at roughly 4 to 7 months, down from 6 to 10 in 2025, with open-weight runs costing a fraction of the frontier equivalent. This week showed both edges of that at once. The defenders' forensics ran on an open-weight model, and the property that made it available to them makes it available to everyone else.

→ The EDR extraction work. SpecterOps ran an LLM loop over a disassembler and pulled detection rules, signatures, and ML models out of a commercial EDR, then turned a decrypted rule into a working evasion. Detection logic that depends on being opaque is on a clock.

→ Package manager hardening. pnpm 11 now defaults to a 24-hour minimum release age and blocks transitive dependencies from non-standard sources. Homebrew 6.0 added tap trust and Linux sandboxing. Both read differently after a week where a package proxy was the pivot.

→ Next issue: the containment ladder in full, ahead of London. Scoped capability, per-agent identity, egress as an explicit grant, human gates on writes. This week supplied the case study.

Several of this week's finds came via Clint Gibler's tl;dr sec, worth your inbox if it isn't there already.

I write the full technical deep-dives on the aminrj.com blog: agent security patterns, lab tests, framework breakdowns. If this newsletter is useful, that's where the long-form work lives →.

Questions, pushback, something I missed, reply directly. I read everything.

Cheers, Amine