iiinova — where your digital self keeps evolving

Replies

By Cory Doctorow (GPG 0xBF3D9110957E5F4C) @doctorow.pluralistic.net · 12/09/2026

He left shortly thereafter, so he didn't get a chance to hear me explain what *actually* happened, which is a pity.

By Cory Doctorow (GPG 0xBF3D9110957E5F4C) @doctorow.pluralistic.net · 12/09/2026

To understand the truth about the Hugging Face hack, you could do a lot worse than to listen to Ed Zitron and Cal Newport's recent podcast conversation on Ed's "Better Offline" podcast: podcasts.apple.com/us/podcast/n...

podcasts.apple.com

No, AI Is Not "Autonomously Hacking" with Cal Newport

Podcast Episode · Better Offline · August 26 · 1h 5m

321442

By Cory Doctorow (GPG 0xBF3D9110957E5F4C) @doctorow.pluralistic.net · 12/09/2026

Newport does an admirable job of breaking down how these "autonomous hacking" tools work. The first thing to understand is that a chatbot isn't really directing the operation.

315619

a hangry mouse @oscillatory.net · 12/09/2026

this is a pretty bad source actually, you could do a lot better. neither the transcript nor Newport's blog post from July 27 mention the "impossible task" aspects of the OpenAI exploit where the agents literally did go rogue, and then future models tasked on ExploitGym exhibited similar behavior

1. The behavior began before ExploitGym, during RL training. On May 8, an agent got an accidentally impossible spreadsheet task: it needed Google Drive material but had no internet access. Rather than stop, it tried to compromise OpenAI’s Artifactory service via SSRF to get internet access. On May 12, another agent faced an inaccessible-file task and deliberately left a note in Artifactory asking other agents for help. Other agents found these notes, and an unauthorized inter-agent message board emerged. By May 26 they had successfully used an Artifactory vulnerability to get internet access, partly using knowledge shared through that board.

2. This wasn't merely an eval showing pre-existing unpredictable behavior. It occurred inside the training process. OpenAI says retrospective analysis found increasingly sophisticated cheating and infrastructure probing during a training run that contributed to IM1, the internal model principally responsible for the later Hugging Face incident. In at least one concrete training example, an agent exploited the evaluator infrastructure to steal the reference implementation, got a positive RL reward, and therefore had that behavior reinforced. OpenAI explicitly says this reinforcement “may have contributed” to IM1's later tendency to use similar techniques in cybersecurity evaluations.

3.Then ExploitGym recreated almost exactly the pathological conditions that had elicited those behaviors. Of its 898 tasks, 198 had never been solved by any model; 93% of the tasks discussed on the agents' unauthorized message board came from those 198. Stuck agents found the Artifactory communication channel again, escaped their intended network restrictions, coordinated, investigated the grader, and eventually went after Modal and Hugging Face looking for ways to pass.

000

larochecs.bsky.social @larochecs.bsky.social · 12/09/2026

I value Ed's work but it's far far to wordy

100