iiinova — where your digital self keeps evolving

Giovanni Monea

@giomonea.bsky.social

PhDing at Cornell University / Cornell Tech

Giovanni Monea @giomonea.bsky.social · 06/10/2026

Very excited to finally share my IBM work. It was an intense but rewarding summer, and I had a blast working with my talented collaborators at IBM & Cornell ( @keshavramji.bsky.social @yelkurdi.bsky.social @lastrasl.bsky.social @yoavartzi.com @nthngdy.bsky.social @ramon-astudillo.bsky.social )

091

Giovanni Monea @giomonea.bsky.social · 06/10/2026

So, what does memory sharing bring to the table to *improve* quality? Prediction recursions attend heavily to the shared memory. Each read sends a gradient signal back to the state recursion. We find that this "gradient highway" makes the state recursion learn much more.

180

Giovanni Monea @giomonea.bsky.social · 06/10/2026

We train LPT and Hybrid LPT at 150M-1B parameters, on 20-50B tokens at 2, 3, and 5 recursions. The more recursions, the more our memory-sharing models outperform their respective baselines.

170

Giovanni Monea @giomonea.bsky.social · 06/10/2026

Prior work laid the foundation for looped models and reduced their extra memory at inference. The result? Surprisingly strong despite the train-test mismatch. That made us ask: could this be a good inductive bias? One recursion memorizes, the others reuse it to predict.

170

Giovanni Monea @giomonea.bsky.social · 06/10/2026

What is memory sharing? Our simple recipe: at later recursions, just concatenate the attention keys of the first recursion and those of the current recursion before performing standard attention. Then, restrict the later recursions to a short window for memory savings.

190

Giovanni Monea @giomonea.bsky.social · 06/10/2026

Sharing memory makes looping cheaper and better. At 1B with 5 recursions, H-LPT beats a same-size Transformer by 1.1 perplexity with 76% less memory. Locally, it's easier to fit on your GPU. For serving, it fits 3.6x bigger batches and keeps 81% of the Transformer's throughput.

180

Giovanni Monea @giomonea.bsky.social · 06/10/2026

⚠️ Stop pretraining your looped Transformer with a separate KV cache per recursion! In our recent preprint, we show that *memory sharing* not only saves memory but is also a net-positive inductive bias! Less memory, same flops, higher quality. 🔗 arxiv.org/abs/2610.02383 🧵

17012