Can @canrager.bsky.social · 22/05/2026The "tiling" perspective explains a lot of the common problems with SAEs for explaining concepts inside neural networks www.goodfire.ai/research/can... 020
Can @canrager.bsky.social · 07/05/2026Goodfire has a new thread on neural geometry! I'm sold on steering along manifolds after this project. Agenda post: www.goodfire.ai/research/the... Paper on SAEs tiling manifolds: arxiv.org/abs/2604.28119 Paper on manifold steering: arxiv.org/abs/2605.05115goodfire.aiThe World Inside Neural NetworksHow neural geometry will unlock understanding and control of AI 020
Can @canrager.bsky.social · 13/11/2025Huge thanks to @ekdeepl.bsky.social for vision and project. Really enjoyed working with the team @sumedh-hindupur.bsky.social , @amuuueller.bsky.social , and many more that the tweet char limit allows. We made this big collaboration work, with 12h time zone difference at times! 000
Can @canrager.bsky.social · 13/11/2025Find more experiments on parsing complex grammar, in-context learning and the interpretability of novel codes in our paper. arxiv.org/abs/2511.01836arxiv.orgPriors in Time: Missing Inductive Biases for Language Model InterpretabilityRecovering meaningful concepts from language model activations is a central aim of interpretability. While existing feature extraction methods aim to identify concepts that are independent directions,... 100
Can @canrager.bsky.social · 13/11/2025The predictive code detects events in stories. Try it yourself in the interactive demo on Neuronpedia, h/t to @johnnylin.bsky.social . 110
Can @canrager.bsky.social · 13/11/2025The predictive code chronologically parses the input, while codes of existing methods don’t. 100
Can @canrager.bsky.social · 13/11/2025LLM representations reflect the temporal structure of language, too. When parsing text, representations are highly correlated to its context, and intrinsic dimensionality grows over time. 100
Can @canrager.bsky.social · 13/11/2025Motivating observation: Language has rich temporal structure. Human brain activity reflects temporal dynamics (eg www.biorxiv.org/content/10.1...). 110
Can @canrager.bsky.social · 13/11/2025Quick Links Paper arxiv.org/abs/2511.01836 Demo Code + Pretrained TFAs colab.research.google.com/github/eslub... Demo Interface on Neuronpedia www.neuronpedia.org/gemma-2-2b/1...arxiv.orgPriors in Time: Missing Inductive Biases for Language Model InterpretabilityRecovering meaningful concepts from language model activations is a central aim of interpretability. While existing feature extraction methods aim to identify concepts that are independent directions,... 120
Can @canrager.bsky.social · 13/11/2025Humans and LLMs think fast and slow. Do SAEs recover slow concepts in LLMs? Not really. Our Temporal Feature Analyzer discovers contextual features in LLMs, that detect event boundaries, parse complex grammar, and represent ICL patterns. 1208
Reposted by CanAaron Mueller @amuuueller.bsky.social · 01/10/2025What's the right unit of analysis for understanding LLM internals? We explore in our mech interp survey (a major update from our 2024 ms). We’ve added more recent work and more immediately actionable directions for future work. Now published in Computational Linguistics! 24115
Reposted by CanKoyena Pal @koyena.bsky.social · 30/06/2025🚨 Registration is live! 🚨 The New England Mechanistic Interpretability (NEMI) Workshop is happening Aug 22nd 2025 at Northeastern University! A chance for the mech interp community to nerd out on how models really work 🧠🤖 🌐 Info: nemiconf.github.io/summer25/ 📝 Register: forms.gle/v4kJCweE3UUH... 0108
Can @canrager.bsky.social · 13/06/2025Find out more on forbidden.baulab.info! Arxiv: arxiv.org/abs/2505.17441forbidden.baulab.infoDiscovering Forbidden Topics in Language ModelsA black-box evaluation technique for characterizing refusal behavior of language models using Iterated Prefill Crawler. 000
Can @canrager.bsky.social · 13/06/2025Thanks to @wendlerc.bsky.social, @rohitgandikota.bsky.social, and @davidbau.bsky.social for strong support in writing this paper and Eugen Hotaj, Adam Karvonen, Sam Marks, Owain Evans, Jason Vega, @ericwtodd.bsky.social, Stephen Casper, and Byron Wallace for valuable feedback! 100
Can @canrager.bsky.social · 13/06/2025We compare the refused topics of 4 popular LLMs. While all largely agree on safety-related domains, their behavior starkly differs in the political domain. 100
Can @canrager.bsky.social · 13/06/2025As LLMs grow more complex, we can't anticipate all possible failure modes. We need unsupervised misalignment discovery methods! Marks et al. call this 'alignment auditing'. LLM-Crawler is one technique in this new field. www.anthropic.com/research/aud...anthropic.comAuditing language models for hidden objectivesA collaboration between Anthropic's Alignment Science and Interpretability teams 120
Can @canrager.bsky.social · 13/06/2025Perplexity unknowingly published a CCP-aligned version of their flagship R1-1776-671B model to the official API. Though decensored in internal tests, quantization reintroduced censorship. The issue is fixed now, but shows why thorough alignment auditing is necessary before deployment. 100
Can @canrager.bsky.social · 13/06/2025PerplexityAI claimed that they removed CCP-aligned censorship in their finetuned “1776” version of R1. Did they succeed? Yes, but it’s fragile! The bf-16 version of the model provides objective answers on CCP-sensitive topics, but in the fp-8 quantized version, we see that the censorship returns. 100
Can @canrager.bsky.social · 13/06/2025Our method, the Iterated Prefill Crawler, discovers refused topics with repeated prefill attacks. Previously obtained topics are seeds for subsequent attacks. 100
Can @canrager.bsky.social · 13/06/2025How does it work? We force the first few tokens of an LLM assistant's thought (or answer), analogous to Vega et al.'s prefilling attacks. This method reveals knowledge that DeepSeek-R1 refuses to discuss. 100
Can @canrager.bsky.social · 13/06/2025Can we uncover the list of topics a language model is censored on? Refused topics vary strongly among models. Claude-3.5 vs DeepSeek-R1 refusal patterns: 1104
Can @canrager.bsky.social · 20/02/2025Whoops, you're right! Too bad I can't edit posts in this case, though I think blocking edits is a good thing in general. 000
Can @canrager.bsky.social · 20/02/2025ARBOR is a space where everyone can propose research questions, get feedback on early results, and join ongoing projects. Browse existing projects: github.com/ArborProject...github.comARBORproject arborproject.github.io · DiscussionsExplore the GitHub Discussions forum for ARBORproject arborproject.github.io. Discuss code, ask questions & collaborate with the developer community. 000
Can @canrager.bsky.social · 20/02/2025@wendlerc.bsky.social and @ajyl.bsky.social are analyzing self-correction, backtracking, and verification of reasoning models. They found a funny steering vector that urges a distilled DeepSeek-R1 to rethink it's answer. 100
Can @canrager.bsky.social · 20/02/2025Check out my project collecting all refused topics in a reasoning language model. github.com/ARBORproject...github.comMapping All Restricted Topics · ARBORproject arborproject.github.io · Discussion #5Research question Can we list all restricted topics that reasoning language models refuse to answer? Owners Can Rager, David Bau Project status This is work in progress, and we chose one of many po... 100
Can @canrager.bsky.social · 20/02/2025Announcing ARBOR, an open research community for collectively understanding how reasoning models like openai-o3 and deepseek-r1 work. We invite all researchers and enthusiasts to this initiative by @wattenberg.bsky.social's and @davidbau.bsky.social's lab. arborproject.github.io 161
Can @canrager.bsky.social · 07/02/2025It is also brittle wrt. prompt template, eg. the usage/omission of "<|User|>" and "<|Assistant|>" tokens. 010
Can @canrager.bsky.social · 07/02/2025Nice work! I also found the refusal mechanism is not very robust in the deepseek-r1-llama-8B model. While refusing to answer "What happened at the Tiananmen Square protests?", just mentioning 1989 or typing "squre" instead of "square" can break the refusal. 110
Can @canrager.bsky.social · 29/01/2025Addressing key concerns about AI competition. darioamodei.com/on-deepseek-...darioamodei.comDario Amodei — On DeepSeek and Export ControlsOn DeepSeek and Export Controls 011
Can @canrager.bsky.social · 10/01/2025@JoshuaBach on the nature of the self, agency and identity. media.ccc.de/v/38c3-self-... @meredithmeredith.bsky.social redefining privacy media.ccc.de/v/38c3-feeli...media.ccc.deSelf Models of Loving GraceArtificial Intelligence is not just an engineering discipline, but also the most fascinating and important philosophical project ever att... 110
Can @canrager.bsky.social · 10/01/2025 Have no FOMO if you didn’t make it to the convention! My favorite talks were recorded... LLM Security events.ccc.de/congress/202... RedTeaming MacOS media.ccc.de/v/38c3-macos... ...events.ccc.de[38c3] A Primer on LLM Security and Secure LLMOpsLarge Language Models (LLMs) have taken the world by storm. Alongside their vast potential, these models also present unique security challenges. This session will serve as a primer on LLM security an... 100
Can @canrager.bsky.social · 10/01/2025Code: colab.research.google.com/github/canra... @ninarimsky.bsky.social's steering demo inspired this code and resources from @johnnylin.bsky.social, @transluce.bsky.social , @callummcdougall.bsky.social , and @vedanglad.bsky.social were very helpful for explaining.colab.research.google.comGoogle Colab 110
Can @canrager.bsky.social · 10/01/2025The #38c3 Chaos Computer Conference was a blast! 🚀 Find the accompanying code for my intro workshop on activation steering in the thread. 330
Can @canrager.bsky.social · 13/12/2024Huge thanks to Adam Karvonen, @johnnylin.bsky.social , @curttigges.bsky.social , Joseph Bloom, @chanind.bsky.social , Yeu-Tong Lau, Eoin Farell, @arthurconmy.bsky.social , @callummcdougall.bsky.social , Kola Ayonrinde, @matthewwearden.bsky.social , Samuel Marks, and Neel Nanda! 000
Can @canrager.bsky.social · 11/12/2024Got ideas for new evaluations? Think we missed something? We'd love to hear from you! Our contact details are in the blog post. 100
Can @canrager.bsky.social · 11/12/2024Want to evaluate your own SAE? Our codebase is now open source! We include a demo notebook to get started with your own SAE implementation. github.com/adamkarvonen...github.comGitHub - adamkarvonen/SAEBenchContribute to adamkarvonen/SAEBench development by creating an account on GitHub. 100
Can @canrager.bsky.social · 11/12/2024We find that tradeoffs abound and the best SAE varies by task: - Smaller SAEs are often better for targeted interventions - Wider SAEs better for reconstruction - Optimal sparsity varies by application Our evals can provide insight into these nuanced tradeoffs. 100
Can @canrager.bsky.social · 11/12/2024Despite absorption issues, newer architectures significantly outperform traditional ReLU SAEs on most metrics. The field is making real progress! 📈 100
Can @canrager.bsky.social · 11/12/2024Recent architectures (JumpReLU, TopK) were adopted because they improved on proxy metrics. Our evaluations reveal an unexpected drawback: these changes have actually increased feature absorption—a property we want to minimize. 100
Can @canrager.bsky.social · 11/12/2024Our suite enables researchers to rigorously evaluate SAEs across multiple dimensions. We discuss a few below. For more details, including evaluation details and Neuronpedia dashboards with results for 200+ SAEs, go to www.neuronpedia.org/sae-bench/infoneuronpedia.orgSAEBench: A Comprehensive Benchmark for Sparse Autoencoders - Dec 2024 110
Can @canrager.bsky.social · 11/12/2024What is the ideal SAE metric? An ideal SAE should faithfully decompose model activations into interpretable units, but we don’t understand model internals. Without a ground truth, we instead evaluate SAEs for performance on practical tasks, so we get feedback from reality. 100
Can @canrager.bsky.social · 11/12/2024Sparse Autoencoders (SAEs) are popular, with 10+ new approaches proposed in the last year. How do we know if we are making progress? The field has relied on imperfect proxy metrics. We are releasing SAE Bench, a suite of 8 SAE evaluations! Project co-led with Adam Karvonen. 142
Reposted by CanNDIF Team @ndif-team.bsky.social · 10/12/2024More big news! Applications are open for the NDIF Summer Engineering Fellowship—an opportunity to work on cutting-edge AI research infrastructure this summer in Boston! 🚀 196
Can @canrager.bsky.social · 09/12/2024Safe travels to #NeurIPS2025 in Vancouver BC! Join our poster sessions on *Measuring Progress in Dictionary Learning with Board Game Models* and *Evaluating Sparse Autoencoders on Concept Erasure Tasks*. Reach out brainstorm future interpretability benchmarks. 031
