Then I can't resist pointing you this paper that shows the emergence of internal representations (the chess board, the location of the pieces, etc.) in a small transformer trained solely on next character prediction (arxiv.org/abs/2403.154...)
Jacques Pesnot Lerousseau
@jacquespesnot.bsky.social
281 followers 957 following 3 posts
Postdoctoral researcher at ILCB (Aix-Marseille Université). Computational and neural mechanisms of human learning.