iiinova — where your digital self keeps evolving

ICML Workshop on Assessing World Models

This is one way to evaluate world models. But there are many other interesting approaches! Plug: If you're interested in more, check out the Workshop on Assessing World Models I'm co-organizing Friday at ICML www.worldmodelworkshop.org

worldmodelworkshop.org

Date: Friday, July 18 2025 Location: Ballroom B at ICML 2025 in Vancouver, Canada

100

Evaluating the World Model Implicit in a Generative Model

Last year we proposed different tests that studied single tasks. We now think that studying behavior on new tasks better captures what we want from foundation models: tools for new problems. It's what separates Newton's laws from Kepler's predictions. arxiv.org/abs/2406.03689

arxiv.org

Recent work suggests that large language models may implicitly learn world models. How should we assess this possibility? We formalize this question for the case where the underlying reality is govern...

120

Summary: 1. We propose inductive bias probes: a model's inductive bias reveals its world model 2. Foundation models can have great predictions with poor world models 3. One reason world models are poor: models group together distinct states that have similar allowed next-tokens

100

Inductive bias probes can test this hypothesis more generally. Models are much likelier to conflate two separate states when they share the same legal next-tokens.

We fine-tune an Othello next-token prediction model to reconstruct boards. Even when the model reconstructs boards incorrectly, the reconstructed boards often get the legal next moves right. Models seem to construct "enough of" the board to calculate single next moves.

100

If a foundation model's inductive bias isn't toward a given world model, what is it toward? One hypothesis: models confuse sequences that belong to different states but have the same legal *next* tokens. Example: Two different Othello boards can have the same legal next moves.

120

We also apply these probes to lattice problems (think gridworld). Inductive biases are great when the number of states is small. But they deteriorate quickly. Recurrent and state-space models like Mamba consistently have better inductive biases than transformers.

100

Would more general models like LLMs do better? We tried providing o3, Claude Sonnet 4, and Gemini 2.5 Pro with a small number of force magnitudes in-context w/o saying what they are. These LLMs are explicitly trained on Newton's laws. But they can't get the rest of the forces.

We then fine-tuned the model on a larger scale, to predict forces across 10K solar systems. We used a symbolic regression to compare the recovered force law to Newton's law. It not only recovered a nonsensical law—it recovered different laws for different galaxies.

131

To demonstrate, we fine-tuned the model to predict force vectors on a small dataset of planets in our solar system. A model that understands Newtonian mechanics should get these. But the transformer struggles.

141

But has the model discovered Newton's laws? When we fine-tune it to new tasks, its inductive bias isn't toward Newtonian states. When it extrapolates, it makes similar predictions for orbits with very different states, and different predictions for orbits with similar states.

We apply these probes to orbital, lattice, and Othello problems. Starting with orbits: we encode solar systems as sequences and train a transformer on 10M solar systems (20B tokens) The model makes accurate predictions many timesteps ahead. Predictions for our solar system:

We propose a method to measure these inductive biases. We call it an inductive bias probe. Two steps: 1. Fit a foundation model to many new, very small synthetic datasets 2. Analyze patterns in the functions it learns to find the model's inductive bias

140

Newton's laws are a kind of foundation model. They provide a place to start when working on new problems. A good foundation model should do the same. The No Free Lunch Theorem motivates a test: Every foundation model has an inductive bias. This bias reveals its world model.

If you only care about orbits, Newton didn't add much. His laws give the same predictions. But Newton's laws went beyond orbits: the same laws explain pendulums, cannonballs, and rockets. This motivates our framework: Predictions apply to one task. World models generalize to many

140

Perhaps the most influential world model had its start as a predictive model. Before we had Newton's laws of gravity, we had Kepler's predictions of planetary orbits. Kepler's predictions led to Newton's laws. So what did Newton add?

Our paper aims to answer two questions: 1. What's the difference between prediction and world models? 2. Are there straightforward metrics that can test this distinction? Our paper is about AI. But it's helpful to go back 400 years to answer these questions.

150

Golub Capital Social Impact Lab @gsbsilab.bsky.social · 30/06/2025

Can an AI model predict perfectly and still have a terrible world model? What would that even mean? Our new ICML paper (poster tomorrow!) formalizes these questions. One result tells the story: A transformer trained on 10M solar systems nails planetary orbits. But it botches gravitational laws 🧵

24114

Reposted by Keyon Vafa

If we know someone’s career history, how well can we predict which jobs they’ll have next? Read our profile of @keyonv.bsky.social to learn how ML models can be used to predict workers’ career trajectories & better understand labor markets. medium.com/@gsb_silab/k...

medium.com

Keyon Vafa: Predicting Workers’ Career Trajectories to Better Understand Labor Markets

If we know someone’s career history, how well can we predict which job they’ll have next?

192

Reposted by Keyon Vafa

Golub Capital Social Impact Lab @gsbsilab.bsky.social · 30/06/2025

Foundation models make great predictions. How should we use them for estimation problems in social science? New PNAS paper @susanathey.bsky.social & @keyonv.bsky.social & @Blei Lab: Bad news: Good predictions ≠ good estimates. Good news: Good estimates possible by fine-tuning models differently 🧵

183

Reposted by Keyon Vafa

Nikhil Garg @nkgarg.bsky.social · 10/03/2025

*Please repost* @sjgreenwood.bsky.social and I just launched a new personalized feed (*please pin*) that we hope will become a "must use" for #academicsky. The feed shows posts about papers filtered by *your* follower network. It's become my default Bluesky experience bsky.app/profile/pape...

23535299

Reposted by Keyon Vafa

Germans Savcisens (Savčišens) @savcisens.com · 02/01/2025

Happy to write this News & Views piece on the recent audit showing LLMs picking up "us versus them" biases: www.nature.com/articles/s43... (Read-only version: rdcu.be/d5ovo) Check out the amazing (original) paper here: www.nature.com/articles/s43...

nature.com

Large language models act as if they are part of a group - Nature Computational Science

An extensive audit of large language models reveals that numerous models mirror the ‘us versus them’ thinking seen in human behavior. These social prejudices are likely captured from the biased conten...

0137

Reposted by Keyon Vafa

Allison Koenecke @allisonkoe.bsky.social · 16/12/2024

📢Announcing 1-day CHI 2025 workshop: Speech AI for All! We’ll discuss challenges & impacts of inclusive speech tech for people with speech diversities, connecting researchers, practitioners, policymakers, & community members. 🎉Apply to join us: speechai4all.org

Banner for CHI 2025 workshop with text: "Speech AI for All: Promoting Accessibility, Fairness, Inclusivity, and Equity"

2214

Date: December 14, 2024 (at NeurIPS in Vancouver, Canada) Location: MTG 19&20

Our Saturday workshop is focused on incorporating insights from the behavioral sciences into AI models/systems. Speakers and schedule: behavioralml.org Location: MTG 19&20 at 8:45am

behavioralml.org

Behavioral ML

010

Our paper proposes new metrics for world model recovery based on the Myhill-Nerode theorem from language theory: Co-authors: Justin Chen, Ashesh Rambachan, Jon Kleinberg, Sendhil Mullainathan (@sendhil.bsky.social)

282

Our paper asks: how can we tell if a transformer has the right world model? We trained a transformer to predict directions for NYC taxi rides. The model was good. It could find shortest paths between new points But had it built a map of NYC? We reconstructed its map and found incoherence:

Evaluating the World Model Implicit in a Generative Model

At NeurIPS today through Sunday! Today I'll be presenting our spotlight paper on evaluating LLM world models at the 4:30pm poster session (#2301). On Saturday I'll be co-organizing the Behavioral ML workshop. Hope to see you there! Paper: arxiv.org/abs/2406.03689 Workshop: behavioralml.org

arxiv.org

172

Reposted by Keyon Vafa

Sophie Greenwood @sjgreenwood.bsky.social · 11/12/2024

I'm excited to use my first post here to introduce the first paper of my PhD, "User-item fairness tradeoffs in recommendations" (NeurIPS 2024)! This is joint work with Sudalakshmee Chiniah and my advisor @nkgarg.bsky.social Description/links below: 1/

1142

Keyon Vafa @keyonv.bsky.social · 06/12/2024

Can you bring some to Vancouver?

Nicolas Beltran-Velez @velezbeltran.bsky.social · 02/12/2024

Reposted by Keyon Vafa

I am very excited to share our new Neurips 2024 paper + package, Treeffuser! 🌳 We combine gradient-boosted trees with diffusion models for fast, flexible probabilistic predictions and well-calibrated uncertainty. paper: arxiv.org/abs/2406.07658 repo: github.com/blei-lab/tre... 🧵(1/8)

Samples y | x from Treeffuser vs. true densities, for multiple values of x under three different scenarios. Treeffuser captures arbitrarily complex conditional distributions that vary with x.

415223

Keyon Vafa @keyonv.bsky.social · 04/12/2024

want to co-work this week? i'm in the city