Tokenizers govern the allocation of computation. It's a waste to spend a whole token of compute predicting the "way" in "By the way". SuperBPE redirects that compute to predict more difficult tokens, leading to wins on downstream tasks!
Jonathan Hayase
@jon.jon.ke
196 followers 39 following 1 posts
5th year PhD student at UW CSE, working on Security and Privacy for ML
