iiinova — where your digital self keeps evolving

Keyon Vafa @keyonv.bsky.social · 14/07/2025

Inductive bias probes can test this hypothesis more generally. Models are much likelier to conflate two separate states when they share the same legal next-tokens.

110

Keyon Vafa @keyonv.bsky.social · 14/07/2025

We fine-tune an Othello next-token prediction model to reconstruct boards. Even when the model reconstructs boards incorrectly, the reconstructed boards often get the legal next moves right. Models seem to construct "enough of" the board to calculate single next moves.

100

Keyon Vafa @keyonv.bsky.social · 14/07/2025

We also apply these probes to lattice problems (think gridworld). Inductive biases are great when the number of states is small. But they deteriorate quickly. Recurrent and state-space models like Mamba consistently have better inductive biases than transformers.

100

Keyon Vafa @keyonv.bsky.social · 14/07/2025

Would more general models like LLMs do better? We tried providing o3, Claude Sonnet 4, and Gemini 2.5 Pro with a small number of force magnitudes in-context w/o saying what they are. These LLMs are explicitly trained on Newton's laws. But they can't get the rest of the forces.

130

Keyon Vafa @keyonv.bsky.social · 14/07/2025

We then fine-tuned the model on a larger scale, to predict forces across 10K solar systems. We used a symbolic regression to compare the recovered force law to Newton's law. It not only recovered a nonsensical law—it recovered different laws for different galaxies.

131

Keyon Vafa @keyonv.bsky.social · 14/07/2025

To demonstrate, we fine-tuned the model to predict force vectors on a small dataset of planets in our solar system. A model that understands Newtonian mechanics should get these. But the transformer struggles.

141

Keyon Vafa @keyonv.bsky.social · 14/07/2025

But has the model discovered Newton's laws? When we fine-tune it to new tasks, its inductive bias isn't toward Newtonian states. When it extrapolates, it makes similar predictions for orbits with very different states, and different predictions for orbits with similar states.

110

Keyon Vafa @keyonv.bsky.social · 14/07/2025

We apply these probes to orbital, lattice, and Othello problems. Starting with orbits: we encode solar systems as sequences and train a transformer on 10M solar systems (20B tokens) The model makes accurate predictions many timesteps ahead. Predictions for our solar system:

130

Keyon Vafa @keyonv.bsky.social · 14/07/2025

We propose a method to measure these inductive biases. We call it an inductive bias probe. Two steps: 1. Fit a foundation model to many new, very small synthetic datasets 2. Analyze patterns in the functions it learns to find the model's inductive bias

140

Keyon Vafa @keyonv.bsky.social · 14/07/2025

Perhaps the most influential world model had its start as a predictive model. Before we had Newton's laws of gravity, we had Kepler's predictions of planetary orbits. Kepler's predictions led to Newton's laws. So what did Newton add?

130

Keyon Vafa @keyonv.bsky.social · 14/07/2025

Our paper aims to answer two questions: 1. What's the difference between prediction and world models? 2. Are there straightforward metrics that can test this distinction? Our paper is about AI. But it's helpful to go back 400 years to answer these questions.

150

Keyon Vafa @keyonv.bsky.social · 14/07/2025

Can an AI model predict perfectly and still have a terrible world model? What would that even mean? Our new ICML paper (poster tomorrow!) formalizes these questions. One result tells the story: A transformer trained on 10M solar systems nails planetary orbits. But it botches gravitational laws 🧵

24114

Keyon Vafa @keyonv.bsky.social · 12/12/2024

Our paper proposes new metrics for world model recovery based on the Myhill-Nerode theorem from language theory: Co-authors: Justin Chen, Ashesh Rambachan, Jon Kleinberg, Sendhil Mullainathan (@sendhil.bsky.social)

282

Keyon Vafa @keyonv.bsky.social · 12/12/2024

Our paper asks: how can we tell if a transformer has the right world model? We trained a transformer to predict directions for NYC taxi rides. The model was good. It could find shortest paths between new points But had it built a map of NYC? We reconstructed its map and found incoherence:

110