iiinova — where your digital self keeps evolving

Kenneth @kennethenevoldsen.eurosky.social · 07/10/2026

Submitted my two accepted posters for EMNLP today, if you are there an want meet up feel free to reach out! I also have a not so @mteb.org themed poster for a workshop on socio-cultural alignment -- stay tuned for that one

Poster on MVEB: Massive Video Embedding Benchmark

Poster on the paper: One Prompt is Not Enough: Instruction sensitivity Endermines Embedding Model Evaluation.

0102 10

Kenneth @kennethenevoldsen.eurosky.social · 05/10/2026

Very happy to announce our new paper "MULTILINGUAL GSM-SYMBOLIC: What determines capability transfer across languages?" Where we construct explore how capabilities transfer between languages and show how these insights impact both evaluation and development.

152 5

Kenneth @kennethenevoldsen.eurosky.social · 23/09/2026

We do find that blame has risen notably in recent years, but also that who blames is highly determined by opposition status, a phenomenon we name 'political contrasting'.

120 2

Kenneth @kennethenevoldsen.eurosky.social · 23/09/2026

We often say that the political environment has grown more hostile in recent years, but has it? Markus Lundsfryd Jensen, Rune Egeskov Trust, Sara Kolding, and I examine this in a recent work exploring the dynamics of blame attribution in the Danish Parliament from 1997-2026.

194 9

Kenneth @kennethenevoldsen.eurosky.social · 31/07/2026

Fewer than 40% of open-weight models share training code or training data. This makes it impossible to improve the training script or iterate on the data, and reduces transparency. To combat this, we previously introduced an openness score. Today we add it to our leaderboard.

150 5

Kenneth @kennethenevoldsen.eurosky.social · 24/07/2026

MTEB reached 5000 contributions! Happy to see the package expand, as well as the community around it.

050 5

Kenneth @kennethenevoldsen.eurosky.social · 16/06/2026

Very happy to announce our new release of MVEB our new benchmark in MTEB for evaluating video embeddings! Whether you want to evaluate embeddings for clustering video, matching video with text, or using them within a retrieval system, this benchmark should cover you.

153 5

Kenneth @kennethenevoldsen.eurosky.social · 05/06/2026

Stuff is happening to the MTEB leaderboard — looking forward to sharing more soon!

030 3

Kenneth @kennethenevoldsen.eurosky.social · 25/05/2026

So what is the solution? We see multiple, but if you have one we haven't considered, do let us know!

100 1

Kenneth @kennethenevoldsen.eurosky.social · 25/05/2026

All models are affected, but some are much more stable than others. Qwen3 and KaLM show notably lower variance than the BGE family, where individual task CVs reach 30–45%

100 1

Kenneth @kennethenevoldsen.eurosky.social · 25/05/2026

The ranking implications are notable. Choosing the best prompt for your model while others use defaults can shift you up by 2 places.

100 1

Kenneth @kennethenevoldsen.eurosky.social · 25/05/2026

We evaluated 6 models across 4 task types and 11 datasets, each with 15 different prompts. The same model on the same task can land anywhere in a wide score distribution depending purely on prompt choice.

100 2

Kenneth @kennethenevoldsen.eurosky.social · 25/05/2026

Most models consistently report scores above what a naïve prompt would achieve. This isn't random noise — it's a systematic upward bias in how performance gets reported.

100 1

Kenneth @kennethenevoldsen.eurosky.social · 25/05/2026

Instruction-tuned embedding models systematically overreport their expected performance. A single reported score hides a wide distribution — and it's rarely in the middle. 🧵 #MTEB #NLP #evaluation

120 2

Kenneth @kennethenevoldsen.eurosky.social · 19/05/2026

We are very happy to introduce AfriMTEB to #MTEB 🌍 If you are looking to evaluate an embedding model for African languages, you can now do so directly with MTEB! Big thanks for Kosei Uemura and @dadelani.bsky.social for the PR! #retrieval, #evaluation, #nlp

052 5

Kenneth @kennethenevoldsen.eurosky.social · 17/05/2026

Had the pleasure to finish of #LREC2026 at the CMCL workshop, with a joint work with Sara, where we show that the choice of embedding model greatly influences your conclusion when modelling semantic associations — in follow up work we will look into what the best practices are

061 6

Kenneth @kennethenevoldsen.eurosky.social · 14/05/2026

Presented on the Dynaword today at LREC. The Dynaword is a group of continuously developed datasets that currently exists in Danish and Norwegian, with multiple languages coming up. If you are around LREC and want to discuss how to make lasting data infrastructure do reach out!

031 3

Kenneth @kennethenevoldsen.eurosky.social · 10/05/2026

One of my favourite new mteb features is the new result submission workflow. Submitting results for a model is just one line away, and once it is in, you don't have to compute it again. The results repository is a public cache that everyone can use! 💫

100 1

Kenneth @kennethenevoldsen.eurosky.social · 21/04/2026

Please welcome the Spanish benchmark to MTEB 🇪🇸 Big thanks to Clemente-H for the contribution!

020 2

Kenneth @kennethenevoldsen.eurosky.social · 11/03/2025

I especially like our dashboard, which allows the comparison of models of interest across target languages.

010 1

Kenneth @kennethenevoldsen.eurosky.social · 11/03/2025

It notably includes both high, mid, and highly low-resource languages, which allow examining generalization even in areas where the available training data is minuscule in comparison to English:

000 0

Kenneth @kennethenevoldsen.eurosky.social · 11/03/2025

Last week at #NoDaLiDa, we presented our work on 🇪🇺EuroEval, a large-scale benchmark for evaluating decoders and encoders.  The framework consists of 9 (🇬🇧🇫🇷🇩🇪🇳🇱🇸🇪🇩🇰🇳🇴🇮🇸🇫🇴) languages, with more to come, where each includes a language understanding and generation benchmark

250 5

Kenneth @kennethenevoldsen.eurosky.social · 11/03/2025

One of the features that I really like is the ability to compare specific models of interest across target languages. Here, we show an example of Dutch, English, and German, but you can try out any combination: euroeval.com/extras/radia...

100 1

Kenneth @kennethenevoldsen.eurosky.social · 11/03/2025

This notably includes low-resource languages such as Faroese and Icelandic, which are great for checking generalizations to languages in which the available data is minuscule

100 1

Kenneth @kennethenevoldsen.eurosky.social · 20/02/2025

This work resulted from a large-scale collaboration, and I would like to thank all of the authors and contributors on MTEB.

120 2

Kenneth @kennethenevoldsen.eurosky.social · 20/02/2025

This new release also comes with a whole new leaderboard, where it is possible to build benchmarks tailored to your use case using in-depth task selection.

130 3

Kenneth @kennethenevoldsen.eurosky.social · 20/02/2025

Such an extensive collection of tasks comes with a considerable computational cost. Thus, we have added multiple optimizations to ensure the benchmark is accessible and quick to run. We see notable speedups for the English benchmark while maintaining relative rank.

130 3

Kenneth @kennethenevoldsen.eurosky.social · 20/02/2025

Examining this claim, we see that the Mistral-derived models indeed perform better in languages on which the models are believed to be trained:

130 3

Kenneth @kennethenevoldsen.eurosky.social · 20/02/2025

We use this collection of tasks to propose multiple benchmarks for multilingual, code, European and Indic languages, and many more. We find that smaller multilingual models (~500M) outperform notably larger 7B models, likely due to a limited multilingual pre-training.

131 3

Kenneth @kennethenevoldsen.eurosky.social · 20/02/2025

I am delighted to announce that we have released 🎊 MMTEB 🎊, a large-scale collaboration working on efficient multilingual evaluation of embedding models. This work implements >500 evaluation tasks across >1000 languages and covers a wide range of use cases and domains🩺👩‍💻⚖️

1286 28