In our new preprint, we explain how some salient features of representational geometry in language modeling originate from a single principle - translation symmetry in the statistics of data.
arxiv.org/abs/2602.150...
With Dhruva Karkada, Daniel Korchinski, Andres Nava, & Matthieu Wyart.
arxiv.org
Symmetry in language statistics shapes the geometry of model representations
Although learned representations underlie neural networks' success, their fundamental properties remain poorly understood. A striking example is the emergence of simple geometric structures in LLM rep...
