Ai2 @ai2.bsky.social · 30/04/2026Recipes for teaching language models to handle long inputs don't work equally well across model families. We wanted to know why—is it the architecture, the training data, or both? 🧵 1255
Ai2 @ai2.bsky.social · 30/04/2026We trained 26 7B models on the same data with the same context extension recipe, varying only the architecture. Four design choices – each used by at least one major model family – compound to drop scores on long context evals by up to 47%. 110
