🧵New paper: "Lost in Backpropagation: The LM Head is a Gradient Bottleneck"
The output layer of LLMs destroys 95-99% of your training signal during backpropagation, and this significantly slows down pretraining 👇
Nathan Godey
@nthngdy.bsky.social
161 followers 29 following 65 posts
Post-doc at Cornell Tech NYC Working on the representations of LMs and pretraining methods nathangodey.github.io
