iiinova — where your digital self keeps evolving

12/ We just think the evidence so far doesn't quite support a strong interpretation of the introspection findings in previous work. And, as always, extraordinary claims require extraordinary evidence. Paper: arxiv.org/abs/2605.26242

Can LLMs Introspect? A Reality Check

Can large language models detect and report their own internal states? A number of studies have argued that the answer to this question is yes. We argue, based on lessons from human metacognition rese...

010

11/ None of this is a takedown. These paradigms are well-motivated and thought-providing, have surfaced fascinating and nontrivial results, and are worth building on. Additionally, we do not rule out "weaker" (but potentially very useful!) notions of introspection.

100

10/ In other words, what makes us think these tasks diagnose introspection is mostly that we can't see the hidden state.

9/ Why is this a subtle distinction? Every computation in an LLM already operates over hidden states. Just because the labels for a task depend on the model’s internal states doesn’t mean the model needs introspective access to solve the task.

100

8/ Turns out they can. A linear probe on the layer-0 embeddings does as well as the model's own ICL predictions. And if we randomly shuffle the labels, so they no longer correlate with the input, accuracy drops to near random: the model was just solving a normal ICL task.

100

7/ We next analyzed "biofeedback", another major paradigm deemed to demonstrate introspection. In this paradigm, models predict in-context labels computed from their own activations. Here, we asked a simple question: can the same labels be predicted from just the input?

100

6/ So a model that passes the introspection test no longer passes it the moment you ask it to tell apart tampering with its “thoughts” from tampering with its prompt. That looks a lot more like general anomaly detection than any special access to the model’s own internals.

100

5/ In Lindsey’s original two-way setup, open models like Llama-3.1-70B reproduce the “steering awareness” effect, with few false positives and decent detection. But with the three-way choice, the models can't distinguish a “thought injection” from an input intervention.

100

5/ To differentiate the two, we add a third condition: we distinguish control, *input* intervention (“gaslight”) and *hidden* intervention. We expect a model with strong introspection capabilities to be able to tell apart the input intervention from the internal one.

100

4/ Jack Lindsey from Anthropic recently showed Claude can sometimes tell when a steering vector is added to its activations (“steering awareness”). We argue that this surprising result may reflect an assessment that "something is off" as opposed to strong introspection.

100

3/ We lay out two conditions for strong introspection: 1. Privileged access, the ability to detect internal features that aren't recoverable from the input; 2. Second-order computation, that is, the introspective behavior can't be explained by a simple first-order forward pass.

100

2/ There are different notions of introspection in cognitive science and philosophy. We target a strong notion (consistent with the framing in some recent work), under which LLMs can access their own activations in a second order computation distinct from a simple forward pass.

1/ Can LLMs introspect, i.e., reason about their internal states? Recent work claims LLMs notice when their "thoughts" get tampered with, and can report the content. We took a closer look and think it's too early to say that. Work led by Shashwat Singh, with @tallinzen.bsky.social and me. A thread 🧵

160

The model’s ability to mention the concept conditioned on a positive answer indeed shows that “steering actually worked”, and that’s exactly why it’s *not* evidence for introspection.

This is, at most (when interpreted favorably), what some people call extraspection. This is not our target nor is it the focus of the previous papers.

Models indeed condition on their context. I don’t see how it follows that introspect.

100

The interesting effect in the Lindsey experiment was the model’s ability to report the intervention in the *first* generated token. When the model mentions the identity of the concept later this can be a first order result of the steering (i.e, no introspection).

210

1. We did verify, in a separate experiment, that the steering works. 2. As I mentioned above, the result is not about the model’s inability to report a textual intervention vs. control, but about its inability to tell if it was a textual intervention or a hidden state intervention (vs. control).

Shauli Ravfogel @shauli.bsky.social · 07/06/2026

In what sense are LLMs “introspection machines”?

300

Shauli Ravfogel @shauli.bsky.social · 07/06/2026

That wasn’t the claim. The claim was that it cannot distinguish *between* the two.

Shauli Ravfogel @shauli.bsky.social · 07/06/2026

Yes, the gaslight text was added as a prefix to the prompt. The key result is that the model cannot distinguish between the gaslight condition and the hidden intervention, which is at least consistent with the interpretation that the original design measured non-introspective anomaly detection.

hakwan lau @hakwan.bsky.social · 07/06/2026

Reposted by Shauli Ravfogel

LLM introspection revisited. if we do the controls properly we may not have strong enough evidence just yet arxiv.org/html/2605.26...

Can LLMs Introspect? A Reality Check

2197

Shauli Ravfogel @shauli.bsky.social · 01/12/2025

I’ll be at NeurIPS in San Diego presenting this paper during the Wednesday, Dec 3 poster session (11 am – 2 pm PST) & at the mechanistic interpretability workshop on Sunday (spotlight). Come say hi, and feel free to DM if you’d like to talk research or just catch up!

081

Shauli Ravfogel @shauli.bsky.social · 02/11/2025

Are there existing theories that explain the subjective aspect of consciousness (“qualia”)? To my understanding most “theories of consciousness” don’t even make an attempt to do so (either aiming to explain other aspects of consciousness, or claiming that qualia doesn’t “really” exist, etc)

Shauli Ravfogel @shauli.bsky.social · 01/11/2025

These findings show, at best, that training data can make an LM act as if it has conscious experience (unsurprising). They don’t support the claim that the model actually has subjective experience, because no mechanistic intervention can establish that (in think you’re already hinting toward this).

130

Beyond truth-encoding, toy models can be adopted to study other fascinating phenomena, like representation of other concepts in transformers, and the way models utilize the structure of their representations to develop nontrivial capabilities. Arxiv: arxiv.org/pdf/2510.15804

040

We show that a similar phenomenon is reproduced when using natural language data, instead of our synthetic data, although in “real”, pretrained LMs, the layer normalization doesn’t seem to play such an important role in inducing linear separability.

100

*Assuming* this structure, we prove that (1) the model represents truth linearly; (2) confidence on the false sequence is decreased. This mechanism relies on a difference in norm between true and false sequence, that the layer normalization translates to linear separability.

100

An analysis of the first gradient steps also predicts the gradual emergence of this structure, with the block corresponding to memorization appearing first, followed by the block that gives rise to linear separability and confidence modulation.

100

To understand the emergence, we study the structure of the attention matrix. It turns out that it is highly structured, with blocks corresponding to mapping subjects to attributes (lower left), attributes to subjects (upper middle), and additional ones.

100

False sequences contain a uniform attribute (instead of the memorized one), so the ideal behavior on them is to output a uniform guess. We see that the probability the model allocates to the memorized attribute starts dropping *exactly* when the linear truth signal emerges.

100

We train a simplified transformer model on this task, with a single layer and a single attention head. We see an abrupt emergence of linear separability of the hidden representations by factuality.

We start with a “truth co-occurrence" hypothesis: false assertions tend to co-occur. Thus, LMs are incentivized to infer the truth latent variable to reduce future loss. We define a factual recall task, where each instance contains two sequences, whose truthfulness correlates.

100

New NeurIPS paper! Why do LMs represent concepts linearly? We focus on LMs's tendency to linearly separate true and false assertions, and provide an analysis of the truth circuit in a toy model. A joint work with Gilad Yehudai, @tallinzen.bsky.social, Joan Bruna and @albertobietti.bsky.social.

1265

8/8 We advocate moving from benchmark- to skill-centric evaluation. Modeling latent structure clarifies strengths and gaps in evaluation, guiding dataset design and LLM development. Shout-out to the first author Aviya Maimon for her principled, and at times painstaking, work!

010

7/8 This factor space lets us (i) flag benchmarks that add little new information, (ii) predict a new model’s full profile from a small task subset, and (iii) choose the best model for an unseen task with minimal trials.

100

6/8 From this analysis, eight main latent skills emerge—e.g., General NLU, Long-document comprehension, Precision-sensitive answers—allowing each model to receive a more fine-grained skill profile instead of one aggregated number.

100

5/8 Using Principal-Axis Factoring, we separate shared variance (true shared skills) from task-specific noise, yielding an interpretable low-dimensional latent space.

100

4/8 We built a 60-model × 44-task performance matrix, harmonizing diverse metrics onto a 0–10 scale. This sets our empirical foundation for the analysis. Our goal is to identify a small set of underlying latent model capabilities that explain these scores in a data-drive way.

100

3/8 Drawing on psychometrics, we treat every benchmark item as a test question and apply Exploratory Factor Analysis to uncover the latent abilities those tasks only approximate.

100

2/8 Today’s practice relies on partially redundant benchmarks that aim to measure latent, underlying capabilities. Evaluation usually collapses them into a single average, obscuring a model’s real strengths and weaknesses.

100

Shauli Ravfogel @shauli.bsky.social · 26/07/2025

1/8 Happy to share our new paper—“IQ Test for LLMs”—co-authored with Aviya Maimon, Amir DN Cohen, @neurogal.bsky.social and Reut Tsarfaty. We propose to rethink how language models are evaluated by focusing on the latent capabilities that explain benchmark results. Arxiv: arxiv.org/pdf/2507.20208

140

I’ll be at #ACL2025! If you’re around and want to catch up or chat, please ping me!

070

Reposted by Shauli Ravfogel

Jackson Petty @jacksonpetty.org · 09/06/2025

How well can LLMs understand tasks with complex sets of instructions? We investigate through the lens of RELIC: REcognizing (formal) Languages In-Context, finding a significant overhang between what LLMs are able to do theoretically and how well they put this into practice.

152

Reposted by Shauli Ravfogel

Gal Vishne @neurogal.bsky.social · 28/03/2025

Introduction \ updates post (better late than never): - I recently graduated from ELSC @hebrewuniversity.bsky.social 🎉 - Moved to NYC 🗽 (view from our balcony below 👇) - And started a postdoc in @columbiauniversity.bsky.social Pls PM if you are in the NYC area and want to talk (or have a beer 🍻)

392

Shauli Ravfogel @shauli.bsky.social · 12/02/2025

There's still more to explore—especially in making the reconstruction process more faithful and generalizable across data distributions—but we see this as a promising step toward improving the interpretability of interventions in natural language. Arxiv: arxiv.org/pdf/2402.11355 (6/6)

Shauli Ravfogel @shauli.bsky.social · 12/02/2025

040

We also demonstrate that the generated counterfactuals can be used for data augmentation, helping the model become more invariant to sensitive features. (5/6)