Ai2 @ai2.bsky.social · 05/03/2026Olmo Hybrid uses a 3:1 pattern—three Gated DeltaNet layers followed by one attention layer. This replaces 75% of attention with linear recurrence while keeping attention frequent enough to recover details the recurrent state compresses away. 150
Ai2 @ai2.bsky.social · 05/03/2026Key finding: hybrid models are substantially more data-efficient than transformers. We show this through rigorous theory + controlled experiments. On MMLU, Olmo Hybrid matches Olmo 3’s accuracy using 49% fewer tokens—roughly 2× efficiency. 140
Ai2 @ai2.bsky.social · 05/03/2026Olmo Hybrid’s gains hold across pretraining evals. After pretraining and mid-training, Olmo Hybrid outperforms Olmo 3 in every primary evaluation domain. It wins big on long-context too—on RULER 64k, performance jumps from 70.9% to 85.0%. 🚀 140
