iiinova — where your digital self keeps evolving

Tiago Pimentel @tpimentel.bsky.social · 22/05/2026

Our new paper reformulates tokenisation as a linear program (LP), which we solve to get SOTA tokenisers 😁 As a bonus, this LP tells us how close to optimal any tokeniser is! Check it out 👇 w/ J. Tempus, @philipwitti.bsky.social, @craigschmidt.com, D. Komm Paper: arxiv.org/abs/2605.22821

14410

Tiago Pimentel @tpimentel.bsky.social · 20/11/2025

Tokenisers are a vital part of LLMs, but how hard is it to find an optimal one? 🤔 Considering arbitrarily large alphabets, prior work showed this is NP-hard. But what if we use bytes instead? Or unary strings like a, aa, aaa, ...? In our new paper, we show this is still hard, NP-hard!

Screenshot of paper title: Tokenisation over Bounded Alphabets is Hard.

1173

Tiago Pimentel @tpimentel.bsky.social · 01/10/2025

LLMs are trained to mimic a “true” distribution—their reducing cross-entropy then confirms they get closer to this target while training. Do similar models approach this target distribution in similar ways, though? 🤔 Not really! Our new paper studies this, finding 4-convergence phases in training 🧵

Figure showing the four phases of convergence in LM training

1244

Tiago Pimentel @tpimentel.bsky.social · 14/07/2025

Mechanistic interpretability often relies on *interventions* to study how DNNs work. Are these interventions enough to guarantee the features we find are not spurious? No!⚠️ In our new paper, we show many mech int methods implicitly rely on the linear representation hypothesis🧵

Paper title "The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?" with the paper's graphical abstract showing how more powerful alignment maps between a DNN and an algorithm allow more complex features to be found and more "accurate" abstractions.

16612

Tiago Pimentel @tpimentel.bsky.social · 04/06/2025

A string may get 17 times less probability if tokenised as two symbols (e.g., ⟨he, llo⟩) than as one (e.g., ⟨hello⟩)—by an LM trained from scratch in each situation! Our new ACL paper proposes an observational method to estimate this causal effect! Longer thread soon!

Title of paper "Causal Estimation of Tokenisation Bias" and schematic of how we define tokenisation bias, which is the causal effect we are interested in.

1518

Tiago Pimentel @tpimentel.bsky.social · 29/05/2025

If you're finishing your camera-ready for ACL or ICML and want to cite co-first authors more fairly, I just made a simple fix to do this! Just add $^*$ to the authors' names in your bibtex, and the citations should change :) github.com/tpimentelms/...

Inline citations with only first author name, or first two co-first author names.

48322

Tiago Pimentel @tpimentel.bsky.social · 08/12/2023

Are you interested in word lengths and natural language’s efficiency? If yes, check out our new #EMNLP2023 paper! It has everything you need: drama, suspense, a new derivation of Zipf’s law, an update to Piantadosi et al’s classic word length paper, transformers... 😄 arxiv.org/abs/2312.03897

Screenshot of paper's title. Paper title is: "Revisiting the Optimality of Word Lengths"

1254