iiinova — where your digital self keeps evolving

6. Position papers: Critical discussions on the feasibility, limitations, and future directions of actionable interpretability research. We also invite perspectives that question whether actionability should be a goal of interpretability research.

100

5. Developing realistic benchmarking and assessment methods to measure the real-world impact of interpretability insights, particularly in production environments and large-scale models.

100

4. Incorporating interpretability–often focusing on micro-level decision analysis–into more complex scenarios, like reasoning processes or multi-turn interactions.

100

3. New model architectures, training paradigms or design choices informed by interpretability findings.

100

2. Comparative analyses of interpretability-based approaches versus alternative techniques like fine-tuning, prompting, and more.

100

1.Practical applications of interpretability insights to address key challenges in AI such as hallucinations, biases, and adversarial robustness, as well as applications in high-stakes, less-explored domains like healthcare, finance, and cybersecurity.

Tal Haklay @talhaklay.bsky.social · 31/03/2025

🚨 Call for Papers is Out! The First Workshop on 𝐀𝐜𝐭𝐢𝐨𝐧𝐚𝐛𝐥𝐞 𝐈𝐧𝐭𝐞𝐫𝐩𝐫𝐞𝐭𝐚𝐛𝐢𝐥𝐢𝐭𝐲 will be held at ICML 2025 in Vancouver! 📅 Submission Deadline: May 9 Follow us >> @ActInterp 🧠Topics of interest include: 👇

153

Amazing news: our workshop was accepted to ICML 2025! Interpretability research sheds light on how models work—but too often, those insights don’t translate into actions that improve them. Our workshop aims to challenge the interpretability community to go further.

020

12/13 We evaluate our automatic pipeline across three datasets and two models, demonstrating that: 1️⃣ Our pipeline discovers circuits with a better tradeoff between size and faithfulness compared to EAP. 2️⃣ Our pipeline produces results comparable to those obtained when human experts define a schema.

11/13 But where does this schema come from? And how do we determine the boundaries of each span within each example? Sounds like we just added more work for researchers! 😅 Actually, we show that an LLM (Claude) can do a pretty decent job at defining a schema and tagging all examples accordingly.

120

10/13 After defining a schema, we construct an abstract computation graph where each span type corresponds to a single token position. We then map attribution scores from example-specific computation graphs to the abstract graph and identify circuits within it.

9/13 To address this problem, we introduce the concept of a 𝙙𝙖𝙩𝙖𝙨𝙚𝙩 𝙨𝙘𝙝𝙚𝙢𝙖, which defines token spans with similar semantics across examples in the dataset.

8/13 But you may notice an issue... What if the examples in a dataset vary in length and structure? Discovering a circuit in such cases is not straightforward, leading many researchers to focus only on datasets with uniform length and structure.

7/13 First improvement : We introduce 𝗣𝗼𝘀𝗶𝘁𝗶𝗼𝗻𝗮𝗹 𝗘𝗱𝗴𝗲 𝗔𝘁𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻 𝗣𝗮𝘁𝗰𝗵𝗶𝗻𝗴 (𝗣𝗘𝗔𝗣) —an extension of EAP that allows us to discover circuits that differentiate between token positions. The key advancement? Our approach uncovers "attention edges", revealing dependencies missed by previous methods.

6/13 The Problem: Automatic circuit discovery methods like Edge Attribution Patching (EAP) and EAP-IP implicitly assume that circuits are position-invariant—they do not differentiate between components at different token positions. As a result, the circuit may include irrelevant components.

5/13 Since the IOI circuit was first discovered, many new techniques for discovering circuits have emerged, with a clear trend of being automated and efficient. Automated methods offer the advantage of scaling more easily and being less susceptible to human biases.

4/13 Early circuit discovery techniques relied on manual causal analysis to identify circuits. Here’s an example of a well-studied circuit in the IOI task by Wang et al. Notice how different components play crucial roles at different token positions—this is expected!

3/13 What is a circuit? A circuit is a minimal subgraph of a model’s computation graph that executes a specific task. Circuit analysis helps us understand how the model operates and which components (e.g., MLPs, attention heads) are involved.