iiinova — where your digital self keeps evolving

Benjamin Minixhofer @bminixhofer.bsky.social · 07/10/2026

Super excited to share that our method to convert language models to byte-level has been published in @nature.com! This was originally the approach behind Bolmo. We now generalized it to other model families. All checkpoints (including new Qwen and Llama models) and data are public!

0162

Reposted by Benjamin Minixhofer

Tiancheng Hu @tiancheng.bsky.social · 30/10/2025

Instruction tuning unlocks incredible skills in LLMs, but at a cost: they become dangerously overconfident. You face a choice: a well-calibrated base model or a capable but unreliable instruct model. What if you didn't have to choose? What if you could navigate the trade-off? (1/8)

121

Benjamin Minixhofer @bminixhofer.bsky.social · 02/04/2025

Models from our paper, including Gemma-2B and Llama-3B instruction-tunes transferred to byte-level, are up on Hugging Face 🤗 huggingface.co/collections/...

huggingface.co

ALM Transfers - a benjamin Collection

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

010

Benjamin Minixhofer @bminixhofer.bsky.social · 02/04/2025

Check out the paper for lots of details. We are also releasing our code as part of `tokenkit`, a new library implementing advanced tokenization transfer methods. More to follow on that👀 Paper: arxiv.org/abs/2503.20083 Code: github.com/bminixhofer/... w/ Ivan Vulić and @edoardo-ponti.bsky.social

arxiv.org

Cross-Tokenizer Distillation via Approximate Likelihood Matching

Distillation has shown remarkable success in transferring knowledge from a Large Language Model (LLM) teacher to a student LLM. However, current distillation methods predominantly require the same tok...

130

Benjamin Minixhofer @bminixhofer.bsky.social · 02/04/2025

2️⃣We also use ALM to directly transfer knowledge from a large teacher (with one tokenizer) to a smaller student (with another tokenizer). We test this by distilling a large maths-specialized Llama into a small Gemma model.🔢

130

Benjamin Minixhofer @bminixhofer.bsky.social · 02/04/2025

1️⃣continued: we can also transfer different base models to the same tokenizer, then ensemble them by combining their logits. This would not be possible if they had different tokenizers. We try ensembling Gemma, Llama and Qwen. They perform better together than separately!🤝

130

Benjamin Minixhofer @bminixhofer.bsky.social · 02/04/2025

We investigate two use cases of ALM in detail (but there's definitely more!) 1️⃣Tokenizer transfer: the teacher is the model with its original tokenizer; the student is the same model with a new tokenizer. Here, ALM even lets us distill subword models to a byte-level tokenizer😮

131

Benjamin Minixhofer @bminixhofer.bsky.social · 02/04/2025

Chunks of tokens with different tokenization biases are not fairly comparable!⚠️⚠️ We thus develop a method to find chunks with low tokenization bias differences (making them *approximately comparable*), then learn to match the likelihoods of those✅

120

Benjamin Minixhofer @bminixhofer.bsky.social · 02/04/2025

Our greatest adversary in this endeavour is *tokenization bias*. Due to tokenization bias, a sequence of subword tokens can leak information about the future contents of the text they encode.

120

Benjamin Minixhofer @bminixhofer.bsky.social · 02/04/2025

Most distillation methods so far needed the teacher and the student to have the same tokenizer. We lift this restriction by first identifying comparable chunks of tokens in a sequence (surprisingly, this is not so easy!), then minimizing the difference between their likelihoods.

120

Benjamin Minixhofer @bminixhofer.bsky.social · 02/04/2025

We created Approximate Likelihood Matching, a principled (and very effective) method for *cross-tokenizer distillation*! With ALM, you can create ensembles of models from different families, convert existing subword-level models to byte-level and a bunch more🧵

Image illustrating that ALM can enable Ensembling, Transfer to Bytes, and general Cross-Tokenizer Distillation.

12514

Reposted by Benjamin Minixhofer

Edoardo Ponti @edoardo-ponti.bsky.social · 12/12/2024

Two amazing papers from my students at #NeurIPS today: ⛓️💥 Switch the vocabulary and embeddings of your LLM tokenizer zero-shot on the fly (@bminixhofer.bsky.social) neurips.cc/virtual/2024... 🌊 Align your LLM gradient-free with spectral editing of activations (Yifu Qiu) neurips.cc/virtual/2024...

2468