andrea wen-yi wang @andreawwenyi.bsky.social · 21/02/2024Tokens in different writing systems can be linearly separated with an average of 99.2 % accuracy for XLM. Even in high-dimensional space, mT5 embeddings are less separable. 101
andrea wen-yi wang @andreawwenyi.bsky.social · 21/02/2024mT5 embeddings neighborhoods are more linguistically diverse: the 50 nearest neighbors for any token represent an average of 7.61 writing systems, compared to 1.64 with XLM-R embedding. 100
andrea wen-yi wang @andreawwenyi.bsky.social · 21/02/2024The nearest neighbors of mT5 tokens are often translations. NLP spent 10 years trying to make word embeddings align across languages. mT5 embeddings find cross-lingual semantic alignment without even being asked! 140
