Reposted by Christopher Akiki
🚨 [Token][ization] Paper Alert 🚨
Tokenization is a wildly understudied area of language modeling despite it having effects across all of NLP. Over the past ~8 months, 32 (!) tokenizer researchers put together the most comprehensive survey of the field. Check it out!
![Three scatterplots of colorful points.
titles = ['Color Space', 'Text Space', 'Image Space']
subtitles = ['Embeddings of color features', 'Text embedding of color names', 'Image embeddings of color swatches']](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:s62ilc26irrbo2dndmdzboqi/bafkreieocq6xfp5xzela4ge3f4nidi4krh7ufkxftg423pjwmipz53zmpm)
