iiinova — where your digital self keeps evolving

For more information check out the website, paper, and datasets: Website: babylm.github.io/babybabellm/ Paper: arxiv.org/pdf/2510.10159 We hope BabyBabelLM will continue as a 'living resource', fostering both more efficient NLP methods, and opening ways for cross-lingual computational linguistics!

babylm.github.io

BabyBabelLM

010

Next to our training resources, we also release an evaluation pipeline that assess different aspects of language learning. We present results for various simple baseline models, but hope this can serve as a starting point for a multilingual BabyLM challenge in future years!

100

To deal with data imbalances, we divide languages into three Tiers. This better enables cross-lingual studies and makes it possible for low-resource languages to be a part of BabyBabelLM as well.

100

With a fantastic team of international collaborators we have developed a pipeline for creating LM training data from resources that children are exposed to. We release this pipeline and welcome new contributions! Website: babylm.github.io/babybabellm/ Paper: arxiv.org/pdf/2510.10159

111

Jaap Jumelet @jumelet.bsky.social · 01/09/2025

🌍Introducing BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Data! LLMs learn from vastly more data than humans ever experience. BabyLM challenges this paradigm by focusing on developmentally plausible data We extend this effort to 45 new languages!

14416

Wij speelden als kind (in Breda) vaak "1 keer tets", waar je een voetbal maximaal 1 keer mocht laten stuiteren; ik had ook geen idee dat dat een Brabants woord was.

010

Jaap Jumelet @jumelet.bsky.social · 01/08/2025

Happening now at the SIGTYP poster session! Come talk to Leonie and me about MultiBLiMP!

1202

Reposted by Jaap Jumelet

Jelle Zuidema 🟥 @wzuidema.bsky.social · 29/07/2025

I'll be in Vienna only from tomorrow, but today my star PhD student Marianne is already presenting some of our work: BLIMP-NL, in which we create a large new dataset for syntactic evaluation of Dutch LLMs, and learn a lot about dataset creation, LLM evaluation and grammatical abilities on the way.

1111

Jaap Jumelet @jumelet.bsky.social · 02/07/2025

Congrats and good luck in Canada!

010

Reposted by Jaap Jumelet

Arianna Bisazza @arianna-bis.bsky.social · 19/06/2025

Proud to introduce TurBLiMP, the 1st benchmark of minimal pairs for free-order, morphologically rich Turkish language! Pre-print: arxiv.org/abs/2506.13487 Fruit of an almost year-long project by amazing MS student @ezgibasar.bsky.social in collab w/ @frap98.bsky.social and @jumelet.bsky.social

arxiv.org

TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs

We introduce TurBLiMP, the first Turkish benchmark of linguistic minimal pairs, designed to evaluate the linguistic abilities of monolingual and multilingual language models (LMs). Covering 16 linguis...

1112

Reposted by Jaap Jumelet

Casper Albers 🟥 @casperalbers.nl · 13/06/2025

Ik snap niet dat hier niet meer ophef over is: Het binnenhalen van Amerikaanse wetenschappers wordt betaalt door Nederlandse academici geen inflatiecorrectie op hun salaris te geven. 1/2

56535

Jaap Jumelet @jumelet.bsky.social · 12/06/2025

Ohh cool! Nice to see the interactions-as-structure idea I had back in 2021 is still being explored!

030

Reposted by Jaap Jumelet

Catherine Arnett @catherinearnett.bsky.social · 05/06/2025

My paper with @tylerachang.bsky.social and @jamichaelov.bsky.social will appear at #ACL2025NLP! The updated preprint is available on arxiv. I look forward to chatting about bilingual models in Vienna!

182

Reposted by Jaap Jumelet

Neil Renic @ncrenic.bsky.social · 28/05/2025

"A well-delivered lecture isn’t primarily a delivery system for information. It is an ignition point for curiosity, all the better for being experienced in an audience." Marvellous defence of the increasingly maligned university experience by @patporter76.bsky.social thecritic.co.uk/university-a...

thecritic.co.uk

University: a good idea | Patrick Porter | The Critic Magazine

A former student of mine has penned an attack on universities, derived from their own disappointing experience studying Politics and International Relations at the place where I ply my trade. In short...

05819

Reposted by Jaap Jumelet

Miryam de Lhoneux @mdlhx.bsky.social · 16/05/2025

Interested in multilingual tokenization in #NLP? Lisa Beinborn and I are hiring! PhD candidate position in Göttingen, Germany: www.uni-goettingen.de/de/644546.ht... PostDoc position in Leuven, Belgium: www.kuleuven.be/personeel/jo... Deadline 6th of June

uni-goettingen.de

Stellen OBP - Georg-August-Universität Göttingen

Webseiten der Georg-August-Universität Göttingen

22513

Reposted by Jaap Jumelet

BlackboxNLP @blackboxnlp.bsky.social · 15/05/2025

BlackboxNLP, the leading workshop on interpretability and analysis of language models, will be co-located with EMNLP 2025 in Suzhou this November! 📆 This edition will feature a new shared task on circuits/causal variable localization in LMs, details here: blackboxnlp.github.io/2025/task

3218

Reposted by Jaap Jumelet

Leshem (Legend) Choshen @EMNLP @lchoshen.bsky.social · 09/05/2025

Close your books, test time! The evaluation pipelines are out, baselines are released & the challenge is on There is still time to join and We are excited to learn from you on pretraining and human-model gaps *Don't forget to fastEval on checkpoints github.com/babylm/evalu... 📈🤖🧠 #AI #LLMS

0104

Jaap Jumelet @jumelet.bsky.social · 23/04/2025

Scherp geschreven en geheel mee eens, maar beetje wrang wel dat de boodschap zich achter een paywall van 450 euro bevindt :') (dank voor de screenshots!)

Jirui Qi @jiruiqi.bsky.social · 11/04/2025

Reposted by Jaap Jumelet

✨ New Paper ✨ [1/] Retrieving passages from many languages can boost retrieval augmented generation (RAG) performance, but how good are LLMs at dealing with multilingual contexts in the prompt? 📄 Check it out: arxiv.org/abs/2504.00597 (w/ @arianna-bis.bsky.social @Raquel_Fernández) #NLProc

145

Jaap Jumelet @jumelet.bsky.social · 17/04/2025

That is definitely possible indeed, and a potential confounding factor. In RuBLiMP, a Russian benchmark, they defined a way to validate this based on LM probs, but we left that open for future work. The poor performance on low-res langs shows they're definitely not trained on all of UD though!

Reposted by Jaap Jumelet

Arianna Bisazza @arianna-bis.bsky.social · 08/04/2025

✨New paper ✨ Introducing 🌍MultiBLiMP 1.0: A Massively Multilingual Benchmark of Minimal Pairs for Subject-Verb Agreement, covering 101 languages! We present over 125,000 minimal pairs and evaluate 17 LLMs, finding that support is still lacking for many languages. 🧵⬇️

37622

Reposted by Jaap Jumelet

Modern LLMs "speak" hundreds of languages... but do they really? Multilinguality claims are often based on downstream tasks like QA & MT, while *formal* linguistic competence remains hard to gauge in lots of languages Meet MultiBLiMP! (joint work w/ @jumelet.bsky.social & @weissweiler.bsky.social)

2216

Person agreement is easier to model than Gender or Number. Sentences with higher overall perplexity lead to less accurate judgements, and models are more likely to pick the wrong inflection if it is split into more tokens. Surprisingly, subject-verb distance has no effect.

130

We find that boosting specific languages works, but only if you pre-, and not post-train: EuroLLM outperforms same size Llama3 on its target languages, but Aya is not significantly better. Neither of them outperform Llama3 significantly on a language not intentionally included.

120

We evaluate 17 Language Models, among them Llama 3, Aya, and Gemma 3. Overall, Llama3 70B and Gemma 27B perform best, but the monolingual 500M Goldfish models significantly outperform them in 14 languages! Base models consistently outperform their instruction-tuned counterparts.

250

We create 125,000 pairs for 101 languages and six types of agreement, resulting in high diversity across phenomena, typological families, geography, amount of resources available, sentence length, and word frequencies. 43 of our languages are not Indo-European.

MultiBLiMP is created automatically using Universal Dependencies and Universal Morphology. We search for subject-verb or -participle pairs with our target features Number, Person, and Gender in UD, then insert the word with the opposite feature value to form a minimal pair.

120

Jaap Jumelet @jumelet.bsky.social · 01/04/2025

37622

Quite some papers in this direction recently, really fruitful direction (and shameless self-plug, did this in 2021 already): aclanthology.org/2021.finding... aclanthology.org/2024.tacl-1.... arxiv.org/abs/2407.04593 aclanthology.org/2024.emnlp-m...

aclanthology.org

Filtered Corpus Training (FiCT) Shows that Language Models Can Generalize from Indirect Evidence

Abhinav Patil, Jaap Jumelet, Yu Ying Chiu, Andy Lapastora, Peter Shen, Lexie Wang, Clevis Willrich, Shane Steinert-Threlkeld. Transactions of the Association for Computational Linguistics, Volume 12. ...

072

Jaap Jumelet @jumelet.bsky.social · 31/03/2025

Fantastic paper!! Fascinating findings, really cool to see this whole corpus modifying setup being so useful for investigating these questions.

130

Reposted by Jaap Jumelet

Qing Yao @qyao.bsky.social · 31/03/2025

LMs learn argument-based preferences for dative constructions (preferring recipient first when it’s shorter), consistent with humans. Is this from memorizing preferences in training? New paper w/ @kanishka.bsky.social , @weissweiler.bsky.social , @kmahowald.bsky.social arxiv.org/abs/2503.20850

examples from direct and prepositional object datives with short-first and long-first word orders:
DO (long first): She gave the boy who signed up for class and was excited it.
PO (short first): She gave it to the boy who signed up for class and was excited.
DO (short first): She gave him the book that everyone was excited to read.
PO (long-first): She gave the book that everyone was excited to read to him.

1177

Jaap Jumelet @jumelet.bsky.social · 13/03/2025

To make my original assumption a bit more explicit: I expected the meta-linguistic ability to be something that arises mostly from post-training, and not from a larger amount of pre-training data / larger model, whereas your result seems to suggest the latter.

120

Reposted by Jaap Jumelet

Siyuan Song @siyuansong.bsky.social · 12/03/2025

New preprint w/ @jennhu.bsky.social @kmahowald.bsky.social : Can LLMs introspect about their knowledge of language? Across models and domains, we did not find evidence that LLMs have privileged access to their own predictions. 🧵(1/8)

26116

Jaap Jumelet @jumelet.bsky.social · 13/03/2025

This is a fascinating result! I would have expected there to be a more noticeable difference between pre- and post-trained models. Did you observe any other differences in meta-linguistic performance for the post-trained model wrt to its base model?