People should probably think about more ways to modify BPE tokenisers. I made this graphical representation of all the algorithms we have for this in the current literature (included in the ReBPE paper, aclanthology.org/2026.finding...)
ir. Thomas Bauwens
@bauwenst.bsky.social
9 followers 10 following 9 posts
PhD researcher in NLP at KU Leuven. Creator and maintainer of TkTkT, the largest tokenisation library in the world. Dancer, photographer and videographer in my free time (IG: @thomas__bailes)
