iiinova — where your digital self keeps evolving

Andrew Lee @ajyl.bsky.social · 30/03/2026

We are thrilled to host the next Mech Interp Workshop @ ICML 2026! 🎉 July 2026, Seoul 🇰🇷 The workshop aims to understand the inner workings of neural nets. Topics: Feature geometry Circuit analyses Interp for {practical applications, safety, scientific discovery}, and many more.

Andrew Lee @ajyl.bsky.social · 04/07/2025

Question @neuripsconf.bsky.social - a coauthor had his reviews re-assigned many weeks ago. The ACs of those papers told him "i've been told to tell u: leave a short note. You won't be penalized". Now I'm being warned of desk-reject due to his short/poor reviews. What's the right protocol here?

000

Reposted by Andrew Lee

nikhil07prakash.bsky.social @nikhil07prakash.bsky.social · 24/06/2025

How do language models track mental states of each character in a story, often referred to as Theory of Mind? We reverse-engineered how LLaMA-3-70B-Instruct handles a belief-tracking task and found something surprising: it uses mechanisms strikingly similar to pointer variables in C programming!

25918

Reposted by Andrew Lee

Lihao Sun @1e0sun.bsky.social · 10/06/2025

🚨New #ACL2025 paper! Today’s “safe” language models can look unbiased—but alignment can actually make them more biased implicitly by reducing their sensitivity to race-related associations. 🧵Find out more below!

1122

This project was done via Arbor! arborproject.github.io Check us out to see on-going work to interp reasoning models. Thank you collaborators! Lihao Sun, @wendlerc.bsky.social , @viegas.bsky.social , @wattenberg.bsky.social Paper link: arxiv.org/abs/2504.14379 9/n

arborproject.github.io

ARBOR

Our interpretation: ✅we find subspace critical for self-verif. ✅in our setup, prev-token heads take resid-stream into this subspace. In a different task, a diff. mechanism may be used. ✅ this subspace activates verif-related MLP weights, promoting tokens like “success” 8/n

We find similar verif. subspaces in our base model and general reasoning model (DeepSeek R1-14B). Here we provide CountDown as a ICL task. Interestingly, in R1-14B, our interventions lead to partial success - the LM fails self-verification but then self-corrects itself. 7/n

100

Our analyses meet in the middle: We use “interlayer communication channels” to rank how much each head (OV circuit) aligns with the “receptive fields” of verification-related MLP weights. Disable *three* heads → disables self-verif. and deactivates verif.-MLP weights. 6/n

100

Bottom-up, we find previous-token heads (i.e., parts of induction heads) are responsible for self-verification in our setup. Disabling previous-token heads disables self-verification. 5/n

100

More importantly, we can use the probe to find MLP weights related to verification. Simply check for MLP weights with high cosine similarity to our probe. Interestingly, we often see Eng. tokens for "valid direction" and Chinese tokens for "invalid direction". 4/n

210

We do a “top-down” and “bottom-up” analysis. Top-down, we train a probe. We can use our probe to steer the model and trick it to have found a solution. 3/n

CoT is unfaithful. Can we monitor inner computations in latent space instead? Case study: Let’s study self-verification! Setup: We train Qwen-3B on CountDown until mode collapse, resulting in nicely structured CoT that’s easy to parse+analyze 2/n

100

🚨New preprint! How do reasoning models verify their own CoT? We reverse-engineer LMs and find critical components and subspaces needed for self-verification! 1/n

1173

Interesting, I didn't know that! BTW, we find similar trends in GPT2 and Gemma2

Shared Global and Local Geometry of Language Model Embeddings

Next time a reviewer asks, “Why didn’t you include [insert newest LM]?”, depending on your claims you could argue that your findings will generalize to other models, based on our work! Paper link: arxiv.org/abs/2503.21073

arxiv.org

Researchers have recently suggested that models share common representations. In our work, we find that token embeddings of language models exhibit common geometric structure. First, we find ``global'...

1215

We call this simple approach Emb2Emb: Here we steer Llama8B using steering vectors from Llama1B and 3B:

Now, steering vectors can be transferred across LMs. Given LM1, LM2 & their embeddings E1, E2, fit a linear transform T from E1 to E2. Given steering vector V for LM1, apply T onto V, and now TV can steer LM2. Unembedding V or TV shows similar nearest neighbors encoding the steer vector’s concept:

Local2: We measure intrinsic dimension (ID) of token embeddings. Interestingly, ID reveals that tokens with low ID form very coherent semantic clusters, while tokens with higher ID do not!

131

Local: we characterize two ways: first using Locally Linear Embeddings (LLE), in which we express each token embedding as the weighted sum of its k-nearest neighbors. It turns out, the LLE weights for most tokens look very similar across language models, indicating similar local geometry:

140

We characterize “global” and “local” geometry in simple terms. Global: how similar are the distance matrices of embeddings across LMs? We can check with Pearson correlation between distance matrices: high correlation indicates similar relative orientations of token embeddings, which is what we find

141

Andrew Lee @ajyl.bsky.social · 31/03/2025

🚨New Preprint! Did you know that steering vectors from one LM can be transferred and re-used in another LM? We argue this is because token embeddings across LMs share many “global” and “local” geometric similarities!

36113

Cool! QQ: say I have a "mech-interpy finding": for instance, say I found a "circuit" - is such a finding appropriate to submit, or is the workshop exclusively looking for actionable insights?

020

Andrew Lee @ajyl.bsky.social · 05/03/2025

My website has a personal readme file with step by step instructions on how to make an update. I would need to hire someone if something were to ever happen to that readme file.

David Bau @davidbau.bsky.social · 20/02/2025

Reposted by Andrew Lee

Today we launch a new open research community It is called ARBOR: arborproject.github.io/ please join us. bsky.app/profile/ajy...

1155

Auditing AI Bias: The DeepSeek Case

@canrager.bsky.social finds that R1’s reasoning tokens reveal a lot about restricted topics. Can we find all restricted topics of a reasoning model, by understanding its inner mechanisms? dsthoughts.baulab.info 6/N

dsthoughts.baulab.info

Cracking open the inner monologue of reasoning models.

100

Similarly, I find linear vectors that seem to encode whether R1 has found a solution or not. We can steer the model to think that it’s found a solution during its CoT with simple linear interventions: ajyl.github.io/2025/02/16/s... 5/N

100

We have some preliminary findings. @wendlerc.bsky.social find simple steering vectors to either make the model continue its CoT (ie, double check its answers) or finish its thought on a GSM8K github.com/ARBORproject...

100

ARBORproject arborproject.github.io · Discussions

Contributions can include: experiments for an on-going project; new resources (model/SAE checkpoints) datasets, or even software (general infra, tools for data collection/annotation, etc.). Of course feel free to launch your own projects! See on-going projects: github.com/ARBORproject... 3/N

github.com

Explore the GitHub Discussions forum for ARBORproject arborproject.github.io. Discuss code, ask questions & collaborate with the developer community.

100

Equational Theories Project

We draw inspiration from previous open-research (eg teorth.github.io/equational_t...) to accelerate progress. Our goal is to facilitate connections between researchers, minimize duplicate efforts, and host a centralized repository of resources. 2/N

teorth.github.io

Mapping out the relations between different equational theories of Magmas

100

Excited about recent reasoning models? What is happening under the hood? Join ARBOR: Analysis of Reasoning Behaviors thru *Open Research* - a radically open collaboration to reverse-engineer reasoning models! Learn more: arborproject.github.io 1/N

arborproject.github.io

ARBOR

1133

Hope you enjoy our work! Thank you team members for an amazing collaboration! @corefpark.bsky.social @ekdeepl.bsky.social YongyiYang MayaOkawa KentoNishi @wattenberg.bsky.social @hidenori8tanaka.bsky.social Special thanks: @ndif-team.bsky.social 9/N

Interestingly, similar observations have been made in humans: human brains also construct graphical (spatial+relational) representations given enough observations of random images from a graph! doi.org/10.7554/eLif... 8/N

Why does this happen? We hypothesize LMs internally perform a graph spectral energy minimization process: Thus we quantify + measure energy of model activations: when energy is reduced, in-context task accuracy *jumps* to near 100%. Phase-transition of in-context reps? 7/N

In-context representations form across models and various graphs. We can intervene on the principal components to alter the model’s belief-state of the graph, and change the model’s in-context task predictions accordingly (ie, causality). 6/N

Instead of random words, we also try words w/ a semantic prior (ex: days of week). We make an in-context task with a new ordering of days (Mon,Thurs,Sun,Wed…): The orig. semantic prior shows up in first principal components & in-context reps in latter ones! The two co-exist. 5/N

@JoshAEngels find that words with a semantic prior (ex: days of the week) can form a ring (x.com/JoshAEngels/...) These rings can also be formed in-context! 4/N

What are in-context representations? We build a graph with random words as nodes, randomly walk the graph, and input the resulting seq to LMs. With a long enough context, LMs start adhering to the graph When this happens, PCA of LM activations reveals the graph (task) structure 3/N

Given an in-context task, and as context is scaled, LMs can form ‘in-context representations’ that reflect the task. Team: @corefpark.bsky.social @ekdeepl.bsky.social YongyiYang MayaOkawa KentoNishi @wattenberg.bsky.social @hidenori8tanaka.bsky.social arxiv.org/pdf/2501.00070 2/N

131