iiinova — where your digital self keeps evolving

With my wonderful lab, who mostly aren't on here (except @lukasbillera.bsky.social and @antonoresten.bsky.social ?) we've been tinkering in this space since the end of the summer, but we think this is just too cool to sit on any longer. The manuscript should be up by tomorrow and I'll drop a link.

241

Eg. here is a protein example where the model designs two domains with an intervening linker (light blue chain in vid). A regular flow model would need to know, early on, exactly how many AAs are needed in the linker, but Branching Flows can decide on-the-fly and grow or shrink it.

You can see the branching pattern in the static plots "with trails", and if you stare at them you can see where lineages delete too.

The process can be seen clearly with a QM9-trained model (continuous atom positions, discrete atom types), starting from a single atom.

Then on the internal nodes of the trees, we place "anchor" states (same space as the X1 elements). We put all that in Z. Then, given this Z, Xt evolves over the trees, sampling when (but not which) branching and deletion events occur, all constructed to terminate at X1.

120

How does this work? First, you need to understand the amazing Generator Matching (arxiv.org/abs/2410.20587). In GM you first sample Z, and then construct a stochastic process Xt that, conditioned on Z, terminates at the data distribution.

Another infix, this time a nanobody CDR3.

100

And this is "infix sampling", but where you let the model figure out how many amino acids are needed to span the gaps.

130

Another binder, which threads a tail down a groove.

For example, here is what "binder design" looks like when you start from a single amino acid.

This is our "graphical abstract":

130

And here is a small molecule example.

130

We figured out flow matching over states that change dimension. With "Branching Flows", the model decides how big things must be! This works wherever flow matching works, with discrete, continuous, and manifold states. We think this will unlock some genuinely new capabilities.

42413

We will have a preprint up about this soon. This is just a "base model" so you can't really ask it to do anything specific (but stay tuned). We've put the code and weights up, so give it a spin: github.com/MurrellGroup... If you have a GPU, it is pretty fast.

100

We can mask that head's ability to attend to non-self AAs, and this prevents the model from generating symmetric backbones, without damaging the in silico refoldability of the structures themselves.

We dug in a bit, and it turns out that, among all of the layers in this network, there is a single (!!) attention head that attends to the matching residue of any symmetric copies of a chain (and also to within-chain structural repeats).

RFdiffusion, Chroma, and others also generate symmetric structures. But, as far as we know, they only do so if some sort of symmetry constraint is imposed during generation. We aren't doing anything like that. This model just does it spontaneously!

After checking that we weren't just mistakenly looking at the training PDBs, we realized: these are, at least approximately, exhibiting symmetry. We saw dimers, dimers of heterodimers, trimers (though the model isn't as good at these) etc.

Like MultiFlow this does struct+seq, with struct similar to FoldFlow-SFM, and with seq per Meta's Discrete Flow Matching. We trained this for a week on one RTX6000 Ada, expecting sketchy helices, and maybe the odd beta sheet pairing if we were lucky. Instead, we got these:

120

With the ecosystem all in place, this is what it takes to 1) specify the entire model, and 2) the flow matching training objective. Just mixing and matching components:

BenjMurrell @benjmurrell.bsky.social · 15/05/2025

We tried to set up a simple demo/tutorial model for the protein design ecosystem we've been developing, and it turned out a bit more interesting than we expected. 🧵 This was a team effort from a few people in my lab, including @antonoresten.bsky.social and others (not sure who is on this app)

2134

Karolinska has a rich work environment, and you'll get to collaborate closely with experimentalists. If you already have a strong biology background then you'll fit right in, and if you don't then this is an excellent way to broaden your education.

120

BenjMurrell @benjmurrell.bsky.social · 15/05/2025

Coming from South Africa (via five years in the US), I can say that Stockholm is a wonderful place to live. The city itself is amazing, and, if you need some nature, there are forests and water everywhere. And you never need to speak any Swedish!

140

BenjMurrell @benjmurrell.bsky.social · 15/05/2025

My lab, at Karolinska, in Stockholm, is looking for a PhD student with a computational/quantitative background to work on probabilistic/generative models of proteins (structure and sequence). The research will involve methods development, and applications in vaccine design.

22113

BenjMurrell @benjmurrell.bsky.social · 11/12/2024

A pleasure! I had a question about your weight decay. In the Algorithm 1 box, λ appears only in the low-rank AdamW update, and not in the final W update. Is that correct? I included it in the W update in my implementation instead.

BenjMurrell @benjmurrell.bsky.social · 10/12/2024

And here is an attempt at a Julia implementation: github.com/FluxML/Optim...

140

BenjMurrell @benjmurrell.bsky.social · 09/12/2024

One option is that, since these are additive offsets, this is some token-insensitive long-range (since they rotate slowly) position sensor, which might help with length extrapolation, and would be tricky to achieve without a bias? There are similar effects in later layers, but much less pronounced.

100

BenjMurrell @benjmurrell.bsky.social · 09/12/2024

Transformer/attention folks: I saw massive activations in Qwen's keys, always towards the end of each head, especially in layer 1. Turns out this is directly driven by the key projection bias (which Qwen has but eg. Llama3 does not). These large values are where RoPE has the slowest(?) effect. Why?

250

BenjMurrell @benjmurrell.bsky.social · 30/11/2024

And one more update: I added a tweak for Qwen models, and even the 32B QwQ "reasoning" model (huggingface.co/Qwen/QwQ-32B...) runs. Painfully slow on CPU though (and I don't have enough VRAM for the full float32 model). Hat tip to @cameron.pfiffer.org for the prompt idea.

$julia> prompt = smollm2_assistant_prompt(tkn, "How many 'f' letters in the name 'Pfiffer'?"); julia> @time ts = generate(model, prompt, max_new_tokens=5000, tokenizer_for_printing=tkn, end_token = eos, sampler = top_nσ_sampler()); Let's see. The name is "Pfiffer". I need to count how many 'f's are in there. Let me look at each letter one by one. Starting with P - that's not f. Then f - that's one f. Next is i - not f. Another f - that's two f's now. Then another f - that makes three f's. Next is another f - so four f's. Wait, let's check again. P, f, i, f, f, e, r. Wait, is it Pfiffer or Phiffer? Oh, it's Pfiffer. So, P, f, i, f, f, e, r. So, the first letter after P is f, then i, then f, f, e, r. So, f's are in the second, fourth, and fifth positions. Wait, let me count again. P is the first letter. f is the second. i is the third. f is the fourth. f is the fifth. e is the sixth. r is the seventh. So, there are three f's in "Pfiffer". But I thought there were four at first. Maybe I miscounted. Wait, maybe it's Pfiffer with two f's. Let me check again. P, f, i, f, f, e, r. So, positions 2,4,5 are f's. So, three f's in total. Wait, but in English names, sometimes double letters are used, but here it seems there are three f's. Wait, maybe it's Pffiffer with four f's. Wait, the name is Pfiffer, which has three f's. But to be sure, let's look it up. Oh, I think I might be wrong. Wait, perhaps it's Phiffer, with a ph at the beginning. But the question says Pfiffer. In German names, it's common to have names starting with Pf. Like Pfister, Pfaffenberger, etc. So, in Pfiffer, it's P-f-i-f-f-e-r. So, P, f, i, f, f, e, r. Therefore, there are three f's in Pfiffer. But maybe in some variations, it has more or less. But based on the name given, Pfiffer, there are three f's. **Final Answer** \[ \boxed{3} \] 3577.734212 seconds (7.76 M allocations: 480.805 GiB, 0.07% gc time)$

251

BenjMurrell @benjmurrell.bsky.social · 26/11/2024

Which, for your example (using huggingface.co/HuggingFaceT...), gives the correct answer.