iiinova — where your digital self keeps evolving

Nick Tomlin

@nickatomlin.bsky.social

1.7K followers 117 following 18 posts

Assistant professor at TTIC. Previously: faculty fellow at NYU, PhD student at Berkeley, undergrad at Brown University. Natural language processing. He/him. 🌐 nickatomlin.github.io

Posts Replies Media

Nick Tomlin @nickatomlin.bsky.social · 27/05/2026

Finally, we show preliminary evidence that user simulators with more human-like memory are more useful. In particular, we find that our most human-like model is more capable of predicting which LLM outputs humans will best understand and remember:

120

Nick Tomlin @nickatomlin.bsky.social · 27/05/2026

Since prompting alone isn’t enough to simulate human memory, we also introduce an approach called COMPACTOR, where an LLM agent writes to a key-value memory store. We find that this leads to more human-like memory behavior:

130

Nick Tomlin @nickatomlin.bsky.social · 27/05/2026

We found that across tasks, language models perform at ceiling (for example, remembering lists of 20 digits perfectly, without any errors), even when prompted to behave like humans with limited working memory. This trend holds for a variety of models and prompting strategies:

120

Nick Tomlin @nickatomlin.bsky.social · 27/05/2026

To compare humans and language models, we built a suite of 10 memory tasks, ranging from classic working memory tests (“remember this list of numbers”) to more open-ended tasks (“study this map and answer questions about it”).

120

Nick Tomlin @nickatomlin.bsky.social · 13/05/2025

This is a difficult benchmark: the best non-reasoning LLMs score around 9%, while the best reasoning models score around 36%. In the future, as models get stronger, we anticipate that they'll also be able to generate harder games

Results table. The best model (o1) wins about 36% of games against the RL baselines.

110

Nick Tomlin @nickatomlin.bsky.social · 13/05/2025

We use o1 to generate natural language rulebooks for 1000 two-player games and then implement these games as Gym environments. For each game, we train baseline agents in self-play with RL and then evaluate whether LLMs can beat the RL baselines

Main paper figure showing a three-step pipeline of game description generation, implementation generation, and self-play training of RL agents

240

Nick Tomlin @nickatomlin.bsky.social · 13/05/2025

I'm particularly fond of this new benchmark paper we wrote, which aims to scalably evaluate whether language models can generalize to arbitrary new tasks. The core idea is to use LLMs to generate new games, and then evaluate whether LLMs can play those games 📄: arxiv.org/abs/2505.07215

Title and abstract of the paper, "Measuring General Intelligence with Generated Games"

3339