iiinova — where your digital self keeps evolving

Francois Meyer @francois-meyer.bsky.social · 27/03/2026

Tomorrow 2pm I am giving a talk at the AfricaNLP workshop about actionable interpretability for low-resource language modelling. Stop by if you're interested in the intersection of interpretability and data-efficient modelling!

020

Francois Meyer @francois-meyer.bsky.social · 23/03/2026

I am in Rabat for @eaclmeeting.bsky.social! I am giving an invited talk at AfricaNLP on Saturday about actionable interpretability for low-resource language modelling. @jumelet.bsky.social is presenting BabyBabelLM on Friday in the 11AM session on Parameter-Efficient Tuning and Training Dynamics.

010

Francois Meyer @francois-meyer.bsky.social · 19/12/2025

I am at @aaclmeeting.bsky.social in Mumbai to present our paper "The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages". I'm giving a talk tomorrow at 2pm in Room 21, in the MC-OP2 session on Multilingualism and Cross-Lingual NLP.

030

Francois Meyer @francois-meyer.bsky.social · 19/11/2025

Our model enables a new type of analysis by tracking subwords as a learnable part of LM training. Like other LM dynamics, subword learning progresses in clear stages. Optimal subwords change over time, so using fixed tokenisers like BPE/ULM might be constraining model learning.

100

Francois Meyer @francois-meyer.bsky.social · 19/11/2025

We see 4 stages of subword learning. (1) Initially, subwords change rapidly. (2) Next, learning trajectories undergo a sudden shift (around 30% in the plot below). (3) After a while, subword boundaries stabilise. (4) In finetuning, subwords change again to suit downstream tasks.

100

Francois Meyer @francois-meyer.bsky.social · 19/11/2025

We study subword learning for 3 morphologically diverse languages: isiXhosa is agglutinative, Setswana is disjunctive (morphemes space-separated), and English as a typological middle ground. Learning dynamics vary across languages, with agglutinative isiXhosa being most unstable.

100

Francois Meyer @francois-meyer.bsky.social · 19/11/2025

T-SSLM (Transformer Subword Segmental LM) marginalises over tokenisation candidates and learns which subwords optimise its training objective. We extract its learned subwords over the course of training, using metrics like fertility and productivity to track subword properties.

100

Francois Meyer @francois-meyer.bsky.social · 19/11/2025

Tokenisation is usually fixed, so research on LM learning dynamics (how grammar/knowledge emerges during training) excludes subword learning. We create an architecture that learns tokenisation during training and study how its subword units evolve across checkpoints.

100

Francois Meyer @francois-meyer.bsky.social · 19/11/2025

If a language model could dynamically optimise subword tokenisation, how would its subwords evolve during training? In our new paper we study the learning dynamics of subword segmentation: arxiv.org/pdf/2511.09197

131

Reposted by Francois Meyer

Jaap Jumelet @jumelet.bsky.social · 15/10/2025

🌍Introducing BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Data! LLMs learn from vastly more data than humans ever experience. BabyLM challenges this paradigm by focusing on developmentally plausible data We extend this effort to 45 new languages!

14416

Reposted by Francois Meyer

Francesca Padovani @frap98.bsky.social · 14/10/2025

𝐃𝐨 𝐲𝐨𝐮 𝐫𝐞𝐚𝐥𝐥𝐲 𝐰𝐚𝐧𝐭 𝐭𝐨 𝐬𝐞𝐞 𝐰𝐡𝐚𝐭 𝐦𝐮𝐥𝐭𝐢𝐥𝐢𝐧𝐠𝐮𝐚𝐥 𝐞𝐟𝐟𝐨𝐫𝐭 𝐥𝐨𝐨𝐤𝐬 𝐥𝐢𝐤𝐞? 🇨🇳🇮🇩🇸🇪 Here’s the proof! 𝐁𝐚𝐛𝐲𝐁𝐚𝐛𝐞𝐥𝐋𝐌 is the first Multilingual Benchmark of Developmentally Plausible Training Data available for 45 languages to the NLP community 🎉 arxiv.org/abs/2510.10159

24016

Francois Meyer @francois-meyer.bsky.social · 20/01/2025

Today our poster will be up at @loreslm.bsky.social Poster Session #2 (2-3pm local time Abu Dhabi). It's also available online at Whova: whova.com/portal/webap...

010

Francois Meyer @francois-meyer.bsky.social · 14/01/2025

This work was carried out by three great UCT CS Honours students - Alexis, Charl, and Hishaam.

000

Francois Meyer @francois-meyer.bsky.social · 14/01/2025

This work unites two directions of research: cognitively plausible modelling and NLP for low-resource languages. We hope more researchers pursue work at the intersection of these two subfields, since they share the goal of improving data-efficiency in the era of scaling.

100

Francois Meyer @francois-meyer.bsky.social · 14/01/2025

However, unlike in the original BabyLM challenge, our isiXhosa BabyLMs do not outperform all skylines. We attribute this to a lack of developmentally plausible isiXhosa data. The success of English BabyLMs is due to both modelling innovations and highly curated pretraining data.

100

Francois Meyer @francois-meyer.bsky.social · 14/01/2025

We pretrain two of the top BabyLM submissions (ELC-BERT and MLSM) for isiXhosa and evaluate it on isiXhosa POS tagging, NER, and topic classification. The BabyLMs outperform an isiXhosa RoBERTa and ELC-BERT even outperforms XLM-R on two tasks.

100

Francois Meyer @francois-meyer.bsky.social · 14/01/2025

The BabyLM challenge (babylm.github.io) produced new sample-efficient architectures. We investigate the potential of BabyLMs to improve LMs for low-resource languages with limited pretraining data. As a case study we use isiXhosa, a language with corpora similar in size to BabyLM strict-small.

100

Francois Meyer @francois-meyer.bsky.social · 14/01/2025

Our paper "BabyLMs for isiXhosa: Data-Efficient Language Modelling in a Low-Resource Context" will be presented at The First Workshop on Language Models for Low-Resource Languages at #COLING2025 in Abu Dhabi. Paper: arxiv.org/pdf/2501.03855

arxiv.org

111