iiinova — where your digital self keeps evolving

Lee Sharkey

@leesharkey.bsky.social

677 followers 108 following 12 posts

Scruting matrices @ Apollo Research

Lee Sharkey @leesharkey.bsky.social · 27/01/2025

And the method lets us identify computations that are spread across multiple layers. This has been conceptually challenging for the SAE paradigm to overcome. (Crosscoder features aren't the computations themselves, but are more akin to the results of the computations).

100

Lee Sharkey @leesharkey.bsky.social · 27/01/2025

Our method lets us identify fundamental computations (or 'circuits') in a toy model of 'Compressed computation', which is a phenomenon similar to 'Computation in superposition'. Each parameter component learns to implement a different basic computation.

100

Lee Sharkey @leesharkey.bsky.social · 27/01/2025

The key idea: Neural networks only need certain parts of their parameters on each forward pass. The rest can be thrown away (on that forward pass). How to identify which parts are needed? Using attribution methods. Hence the name Attribution-based Parameter Decomposition!

100

Lee Sharkey @leesharkey.bsky.social · 27/01/2025

For example, with anthropic's Toy Model of Superposition, we can decompose the parameters directly into mechanisms that are used by individual features.

110

Lee Sharkey @leesharkey.bsky.social · 27/01/2025

New interpretability paper from Apollo Research! 🟢Attribution-based Parameter Decomposition 🟢 It's a new way to decompose neural network parameters directly into mechanistic components. It overcomes many of the issues with SAEs! 🧵

1173