Ai2 @ai2.bsky.social · 05/03/2026Introducing Olmo Hybrid, a 7B fully open model combining transformer and linear RNN layers. It decisively outperforms Olmo 3 7B across evals, w/ new theory & scaling experiments explaining why. 🧵 1315
Ai2 @ai2.bsky.social · 05/03/2026Olmo Hybrid uses a 3:1 pattern—three Gated DeltaNet layers followed by one attention layer. This replaces 75% of attention with linear recurrence while keeping attention frequent enough to recover details the recurrent state compresses away. 150
