I am excited to share our new paper: “Grokking at the Edge of Numerical stability”!
We show that floating point errors in the Softmax play a surprising role in grokking, explaining among other things, why weight decay seems necessary for grokking in most cases!
🧵
