Simple beats complicated:
We show that switching to a sliding-window attention mask with attention sinks (at no cost) beats linear attention post-training.
Huge thanks to my collaborators
Rhea Sukthanker, Pashmina Cameron, and Emy Gervais.
Paper: arxiv.org/abs/2608.28444
arxiv.org
Sliding-window beats linear attention
Due to the nature of quadratic attention, Large Language Models (LLMs) consume a lot of memory and energy. Every new token costs more than the previous one. For each additional token, the keys and val...
