Reposted by Catherine Arnett
🚨 [Token][ization] Paper Alert 🚨
Tokenization is a wildly understudied area of language modeling despite it having effects across all of NLP. Over the past ~8 months, 32 (!) tokenizer researchers put together the most comprehensive survey of the field. Check it out!
