iiinova — where your digital self keeps evolving

Tomasz Limisiewicz @tomlim.bsky.social · 07/10/2026

I'm in San Francisco for #COLM and @tokshop.bsky.social Happy to chat about anything! pictured: me breathing AGI for the first time.

020

Tomasz Limisiewicz @tomlim.bsky.social · 04/05/2026

At the same time, the optimal compression rate varies across languages and can differ substantially from the compression rate of popular BPE tokenizers. [7/N]

110

Tomasz Limisiewicz @tomlim.bsky.social · 04/05/2026

We observe an analogous pattern of constant optimal scaling (bytes ~ parameters) for other languages than English. Optimal compression is language dependent and scales with Parity (i.e. the ratio of byte length needed to express the same content in a given language versus English). [6/N]

110

Tomasz Limisiewicz @tomlim.bsky.social · 04/05/2026

These findings hold both for latent tokenizers (BLT) and subword tokenizers (BPE variants). Interestingly, with BPE we observe that at large scale decreasing compression by choosing smaller vocabulary improves performance. [4/N]

120

Tomasz Limisiewicz @tomlim.bsky.social · 04/05/2026

We find the optimal compression rate. At each training compute budget, there is an optimal compression rate that minimizes loss. The optimal compression rate decreases slowly as the train budget increases. [3/N]

110

Tomasz Limisiewicz @tomlim.bsky.social · 04/05/2026

Scaling rule transfers in bytes. Contrary to token centered rules, we find that the optimal data/model size ratio should be in bytes/param. For EN it’s ~60 bytes/param across comp budgets and compression. When changing tokenizers, match training bytes (not tokens) to params.[2/N]

111

Tomasz Limisiewicz @tomlim.bsky.social · 04/05/2026

We present Compute Optimal Tokenization! 🔠 Common in LLM scaling works stick to one tokenizer, sweeping data/model size. But what happens when we control the tokenizer’s compression rate (bytes/token)? Here we sweep tokenizers, params, and data across compute budgets: [1/N]

2225

Tomasz Limisiewicz @tomlim.bsky.social · 18/07/2025

Check the BLT poster at @aclmeeting.bsky.social . It’s just fortaste before the main presentation at @tokshop.bsky.social next week from Artidoro Pagnoni!

0100

Tomasz Limisiewicz @tomlim.bsky.social · 16/07/2025

I'm pleased to be in Vancouver for @ICML this week 🇨🇦🤖. I'll be happy to chat about multilingual, multimodal LMs and tokenization(free).

040

Tomasz Limisiewicz @tomlim.bsky.social · 03/04/2025

So, apparently, confusing these two buttons can ignite a serious flame-war in reviewer-author discussion🔥 @aclmeeting.bsky.social

060

Tomasz Limisiewicz @tomlim.bsky.social · 31/03/2025

Excited to continue my research adventure as a postdoc at @uwnlp.bsky.social and Meta! I’ve joined @lukezettlemoyer.bsky.social’s fantastic lab. Together, we plan to rethink how LLMs perceive data to unlock their capabilities to uncharted language and, further, beyond text!

1141

Tomasz Limisiewicz @tomlim.bsky.social · 11/11/2024

#firstpost Are you working on NLP for low-resource or non-Latin script languages? If yes, I have great news for you! Our MYTE tokenizer and MyT5 models 🪲 are now easily available through🤗. It’s easy to try:

$from transformers import T5ForConditionalGeneration from transformers import MyT5Tokenizer MODEL_SIZE = "large" # small, base, or large MODEL = f"Tomlim/myt5_{MODEL_SIZE}" model = T5ForConditionalGeneration.from_pretrained( MODEL, use_safetensors=True) tokenizer = MyT5Tokenizer.from_pretrained(MODEL)$

180