iiinova — where your digital self keeps evolving

Bonnie Berger Lab

@bergerlab.bsky.social

The Berger lab at @csail.mit.edu works on a diverse set of problems in computational biology and biomedicine. Account run by lab members. people.csail.mit.edu/bab

Posts Replies Media

Bonnie Berger Lab @bergerlab.bsky.social · 03/09/2025

8/ Paper: pnas.org/doi/10.1073/... MIT News article: news.mit.edu/2025/researc... Github:

pnas.org

Sparse autoencoders uncover biologically interpretable features in protein language model representations | PNAS

Foundation models in biology—particularly protein language models (PLMs)—have enabled ground-breaking predictions in protein structure, function, a...

000

Bonnie Berger Lab @bergerlab.bsky.social · 03/09/2025

7/ We show that SAE & transcoder features are much more interpretable than ESM neurons, for both protein-level & amino acid-level representations. This has the potential to improve safety, trust & explainability of PLMs. As PLMs improve, SAEs could help us learn new biology.

100

Bonnie Berger Lab @bergerlab.bsky.social · 03/09/2025

6/ We also use Claude to autointerpret SAE features based on protein names, families, gene names & GO terms. Many features correspond to families (like NAD Kinase, IUNH, PTH) & functions (like methyltransferase activity, olfactory/gustatory perception).

100

Bonnie Berger Lab @bergerlab.bsky.social · 03/09/2025

5/ We interpret these SAE features using Gene Ontology (GO) enrichment. Many protein-level SAE features align tightly with GO terms across all levels of the GO hierarchy.

110

Bonnie Berger Lab @bergerlab.bsky.social · 03/09/2025

4/ SAEs have a very wide latent dimension with a sparsity constraint. This forces PLM representations to disentangle into biologically interpretable, sparsely activating features without any supervision.

100

Bonnie Berger Lab @bergerlab.bsky.social · 03/09/2025

3/ We train sparse autoencoders (SAEs) on protein-level and amino acid-level representations from layers 6-10 of ESM2_t12_35M_UR50D. We also train transcoders (an SAE variant) on protein-level representations.

100

Bonnie Berger Lab @bergerlab.bsky.social · 03/09/2025

2/ Protein-level representations from PLMs are used in many downstream tasks. Disentangling their features can enhance interpretability, helping us trust and explain downstream applications.

100

Bonnie Berger Lab @bergerlab.bsky.social · 03/09/2025

1/ PLMs like ESM have made big strides in predicting protein structure & function. But they feel like a “black-box.” What biological information do PLM representations contain? Can we disentangle them systematically?

100

Bonnie Berger Lab @bergerlab.bsky.social · 03/09/2025

Excited to share our recent work: Sparse autoencoders uncover biologically interpretable features in protein language model representations now in PNAS. Thread below 🧵

172

Bonnie Berger Lab @bergerlab.bsky.social · 12/03/2025

Hello world! See the thread below for our recent work 🌿 MINT!

040