iiinova — where your digital self keeps evolving

David Atkinson

@diatkinson.bsky.social

613 followers 379 following 12 posts

PhD student at Northeastern, previously at EpochAI. Doing AI interpretability. diatkinson.github.io

David Atkinson @diatkinson.bsky.social · 06/10/2026

We find that the cosine similarity between a model's decision and report attributions is 0.34 for faithful models compared to 0.08 for unfaithful ones (95% CI for the difference: 0.16 to 0.36).

110

David Atkinson @diatkinson.bsky.social · 06/10/2026

To find out, we trained 32 new characters into each checkpoint, then used attribution patching to score every new adapter weight on each task. Each adapter in a pair was trained identically, differing only in the underlying base checkpoint.

110

David Atkinson @diatkinson.bsky.social · 06/10/2026

We can test this further: trained on all 40 layers, Qwen3-14B is a terrible self-reporter. But if we train only its first 20 layers, faithfulness reaches 0.74. Once training reaches layer 25 or beyond, faithfulness plummets, although the decisions are ~just as good.

120

David Atkinson @diatkinson.bsky.social · 06/10/2026

What changed? Ablating adapter layers from the front or back shows that the faithful checkpoint stores its preferences 5-6 layers earlier. Our hypothesis: self-report only works once preferences sit early enough for the model's existing verbalization machinery to read them.

120

David Atkinson @diatkinson.bsky.social · 06/10/2026

Although we train solely on decisions, faithful self-report emerges late in training, long after decisions have become accurate! Qwen3-32B at step 1000: decisions 0.82, faithfulness 0.25. At step 3000: decisions 0.92, faithfulness 0.83. This gives us a contrast pair.

120

David Atkinson @diatkinson.bsky.social · 06/10/2026

Then, in a fresh context, we ask the model how it would weigh each attribute. This gives us two metrics: decision performance (how well the choices follow the character's hidden preferences) and faithfulness (how well the stated preferences match those revealed by its choices).

100

David Atkinson @diatkinson.bsky.social · 06/10/2026

We build on Plunkett et al.'s "Self-Interpretability" setup (arxiv.org/abs/2505.17120): train Qwen3-32B to make decisions as 100 different characters (Gregor Samsa buying a washing machine...), each with random hidden preferences.

110

David Atkinson @diatkinson.bsky.social · 06/10/2026

New COLM paper: Identifying Introspection From the Inside When an LLM tells us about its decisions, does it 𝘬𝘯𝘰𝘸 what drives its choices—or is it guessing? In our setting, we find that faithful models decide and report with the same layers. Unfaithful ones don't. 🧵

1173