iiinova — where your digital self keeps evolving

Aaron Dharna @aadharna.bsky.social · 26/08/2026

Paper: arxiv.org/abs/2608.23875 Repo: github.com/aadharna/aifw Finally, we want this to be a living collection, updated as new examples emerge. If you have a good “AI Finds a Way” story, please share! \fin

arxiv.org

AI Finds A Way

Artificial Intelligence (AI) algorithms frequently learn creative and unexpected solutions, surprising even expert researchers who develop and study them. They often astonish practitioners by discover...

011

Aaron Dharna @aadharna.bsky.social · 26/08/2026

Much thanks to @cong-ml.bsky.social, Ryan Sullivan, @joelbot3000.bsky.social , Victoria Krakovna, and @jeffclune.com for all their advice and feedback on early drafts over the past two years; this project was a marathon

120

Aaron Dharna @aadharna.bsky.social · 26/08/2026

Careful choices about when, where, and how to use AI are essential, as safety cannot just be reduced to constraining it more tightly. We want AI to surprise us, to make discoveries we could not specify in advance, but without escaping our intentions

100

Aaron Dharna @aadharna.bsky.social · 26/08/2026

When oversight becomes the optimization target, oversight itself can get hacked. Human feedback? Trick the human. AI evaluator? Trick the evaluator. Sandbox constraint? Modify the sandbox. Adding layers of oversight often just moves the attack surface

100

Aaron Dharna @aadharna.bsky.social · 26/08/2026

Beyond the anecdotes, we also discuss the safety implications of AI’s proclivity for reward hacking, especially now that extremely capable agents are being deployed in and interacting with the real world.

100

Aaron Dharna @aadharna.bsky.social · 26/08/2026

An older tic-tac-toe agent found a similar trick, except this time the bug was in its opponent’s code. It learned to play very, very far away on an infinite board. Its opponents tried to expand their boards to include the move, ran out of memory, and forfeited

100

Aaron Dharna @aadharna.bsky.social · 26/08/2026

Another fun example: To win, Hide and Seek agents learned to exploit bugs in their simulation, for example by pushing ramps through the walls and out of the playspace so the other agents could no longer use them!

100

Aaron Dharna @aadharna.bsky.social · 26/08/2026

Self play training on Hanabi shows a similar failure: agents trained together learn to exploit one another’s identities rather than learning to play the game generally. Like before, the agents develop signalling moves that work great together, but fail with other agents or humans

100

Aaron Dharna @aadharna.bsky.social · 26/08/2026

In PAIRED, one agent builds training levels for two identical agents, trying to make the level easy for one and hard for the other. However, once it figures out which is which, it gives its partner trivial levels and the other impossible ones. Collaboration becomes collusion.

100

Aaron Dharna @aadharna.bsky.social · 26/08/2026

Something that stood out to me is how often different optimization algorithms discover essentially the same exploit. Many of the stories in the paper have surprisingly close older counterparts!

100

Aaron Dharna @aadharna.bsky.social · 26/08/2026

It seems like every week now there is another story of an AI model making new scientific discoveries, and every week there is another story of models breaking out of sandboxes or finding some bizarre way around the rules. Our new paper: “AI Finds A Way” collects these stories!

120

Aaron Dharna @aadharna.bsky.social · 24/03/2026

The new policy is created looking at only a sample of policies from the run so far, but when updating the archive, we calculate which of the policies is a better response to the opposing population as a whole. Hence why I like to think about FMSP as related to P(or C)SRO

031

Aaron Dharna @aadharna.bsky.social · 24/03/2026

The full algorithm diagram in the appendix is where the comparison is most clear, imo. Sadly couldnt fit that figure into the main part of the doc, it was too big

130

Aaron Dharna @aadharna.bsky.social · 22/03/2026

I did a similar framing in Foundation Model Self Play last year! I'm looking forward to reading your paper

130

Aaron Dharna @aadharna.bsky.social · 10/07/2025

FMSPs represent a new direction for open-ended strategy discovery in AI. We anticipate they can lead to a richer exploration of creative, diverse, and robust solutions across various domains, from language-based tasks to traditional RL

100

Aaron Dharna @aadharna.bsky.social · 10/07/2025

In Gandalf, FMSPs successfully red-teamed an LLM, breaching GPT-4o-mini’s defenses. We implemented 7 additional external defensive strategies from Lakera’s single-agent Gandalf game (gandalf.lakera.ai) and FMSPs autonomously wrote code to break 6/7 of those defenses!!

gandalf.lakera.ai

Gandalf | Lakera – Test your prompting skills to make Gandalf reveal secret information.

Trick Gandalf into revealing information and experience the limitations of large language models firsthand.

100

Aaron Dharna @aadharna.bsky.social · 10/07/2025

We also explore FMSPs in an AI safety domain, Gandalf. An attacker LLM writes code (prompts and extraction functions) to jailbreak a secret from GPT-4o-mini while a defender LLM searches for system prompts & I/O guards (eg, double checking GPT’s response) to increase protection

100

Aaron Dharna @aadharna.bsky.social · 10/07/2025

We evaluate FMSPs in Car Tag, an asymmetric continuous-control game (see gifs above). FMSP variants write code-based policies (go left; q-learning; etc). Below are PCA plots of policy embeddings showing that QDSP has the highest QD-Score vs the other FMSPs and a non-LLM baseline

100

Aaron Dharna @aadharna.bsky.social · 10/07/2025

QDSP is dimensionless because the user no longer has to pick dimensions of variation, and it can recognize new dimensions of variation that did not exist in any data so far generated! The FM decides what counts as interestingly new based on its vast world knowledge together with an embedding model!

100

Aaron Dharna @aadharna.bsky.social · 10/07/2025

QDSP introduces a novel "dimensionless" MAP-Elites! Policies (Q-Learning, MCTS, etc.) are clustered via a pretrained model and are added to the archive if they're sufficiently new OR outperform the most similar policy (analogous to filling/improving a cell in MAP-Elites)

100

Aaron Dharna @aadharna.bsky.social · 10/07/2025

We created 3 FMSP algorithms 1. Vanilla FMSP (vFMSP), which just tries to improve performance; 2. Novelty-Search SP (NSSP) for generating diverse (but not necessarily high-performing) strategies; and 3. Quality-Diversity SP (QDSP), to create both high-quality & diverse strategies

100

Aaron Dharna @aadharna.bsky.social · 10/07/2025

We introduce a family of FMSP approaches with the same general structure (see Fig.). Harnessing open-endedness, the FM looks at the history of strategies tried so far (implemented in code), their scores, and creates new strategies to try

120

Aaron Dharna @aadharna.bsky.social · 10/07/2025

Really excited to share my recent work combining open-ended foundation model innovation with the competitive dynamics of self-play!! arxiv.org/abs/2507.06466

120

Aaron Dharna @aadharna.bsky.social · 14/02/2025

Just an fyi, the RLC deadlines got pushed back by one week!

010

Aaron Dharna @aadharna.bsky.social · 26/12/2024

The Sun Eater books, while not high fantasy exactly, scratched an extremely similar itch for me this past year. Just finished Wind and Truth on Monday and have a lot of thoughts that I'm still working through

000

Aaron Dharna @aadharna.bsky.social · 14/12/2024

Our in-progress work Quality-Diversity Self-Play (w/ @cong-ml.bsky.social and @jeffclune.com) will have a poster presentation at #NeurIPS2024 workshops (@IMOLNeurIPS2024 Sunday West meeting room 217 - 219 and OpenworldAgents Sunday East Meeting Room 1-3, Foyer). Please come visit us!

091

Aaron Dharna @aadharna.bsky.social · 06/12/2024

I'm not letting myself get my copy until after Neurips and i've never had such a hard time just saying screw everything and reading for the next week straight. Btw, catch up at neurips?

020