Training with more data = better LLMs, right? 🚨
False! Scaling language models by adding more pre-training data can decrease your performance after post-training!
Introducing "catastrophic overtraining." 🥁🧵👇
arxiv.org/abs/2503.19206
1/10
Jacob Springer
@jacobspringer.bsky.social
140 followers 125 following 10 posts
Machine Learning (the science part) | PhD student @ CMU