“For any given task, a sufficiently large neural network will contain within it a good subnetwork that can be isolate...

Rescuing a marooned audienceTechnical/scientific communication—writing, talks, day-to-day collaboration, especially d...

TL;DR: Does the next-token logit track the conditional or the joint probability of the whole sequence?I had an invisi...

A major problem in the study of large language models is to understand their inherent low-dimensional structure. We introduce an approach to study the low-dimensional structure of language models at a...

Training modern large language models (LLMs) has become a veritable smorgasbord of algorithms and datasets designed to elicit particular behaviors, making it critical to develop techniques to understa...

Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap ...

Deep sequence models are said to store atomic facts predominantly in the form of associative memory: a brute-force lookup of co-occurring entities. We identify a dramatically different form of storage...