Ai2 @ai2.bsky.social · 30/04/2026Recipes for teaching language models to handle long inputs don't work equally well across model families. We wanted to know why—is it the architecture, the training data, or both? 🧵 1255
Ai2 @ai2.bsky.social · 30/04/2026We trained 26 7B models on the same data with the same context extension recipe, varying only the architecture. Four design choices – each used by at least one major model family – compound to drop scores on long context evals by up to 47%. 110
Ai2 @ai2.bsky.social · 30/04/2026The four choices – QK normalization, grouped-query attention, sliding window attention, & shorter pretraining context length – all affect how a model decides which parts of its input to focus on. Combine three or more, and long-context performance falls off a cliff. 120
