iiinova — where your digital self keeps evolving

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 07/10/2026

Submitted my two accepted posters for EMNLP today, if you are there an want meet up feel free to reach out! I also have a not so @mteb.org themed poster for a workshop on socio-cultural alignment -- stay tuned for that one

Poster on MVEB: Massive Video Embedding Benchmark

Poster on the paper: One Prompt is Not Enough: Instruction sensitivity Endermines Embedding Model Evaluation.

081

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 05/10/2026

Very happy to announce our new paper "MULTILINGUAL GSM-SYMBOLIC: What determines capability transfer across languages?" Where we construct explore how capabilities transfer between languages and show how these insights impact both evaluation and development.

141

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 23/09/2026

We often say that the political environment has grown more hostile in recent years, but has it? Markus Lundsfryd Jensen, Rune Egeskov Trust, Sara Kolding, and I examine this in a recent work exploring the dynamics of blame attribution in the Danish Parliament from 1997-2026.

194

Reposted by Kenneth Enevoldsen

MTEB @mteb.org · 03/09/2026

New on MTEB: google/gemini-embedding-2, which takes interleaved text, image, audio and video in one input. Its first results are on ViDoRe(v2), visual document retrieval: 62.37, first of the 13 single-vector models with complete results. Model and results by whybe-choi.

New model on MTEB: google/gemini-embedding-2 — First of the 13 single-vector models on ViDoRe(v2). Bar chart of 13 of the 13 models with complete results on ViDoRe(v2) (4 tasks), each row given with its rank on that board. 1. google/gemini-embedding-2, 62.37, size not published (this release); 2. nanovdr/NanoVDR-S-Multi, 61.93, 69M parameters; 3. jinaai/jina-embeddings-v4, 58.23, 3.9B parameters; 4. eagerworks/eager-embed-v1, 57.20, 4.4B parameters; 5. ibm-granite/granite-vision-3.3-2b-embedding, 56.92, 3.0B parameters; 6. voyageai/voyage-multimodal-3, 54.94, size not published; 7. royokong/e5-v, 46.55, 8.4B parameters; 8. google/siglip-so400m-patch14-384, 33.00, 878M parameters; 9. TIGER-Lab/VLM2Vec-Full, 31.82, 4.1B parameters; 10. jinaai/jina-clip-v2, 26.94, 865M parameters; 11. laion/CLIP-ViT-bigG-14-laion2B-39B-b160k, 19.35, 2.5B parameters; 12. openai/clip-vit-base-patch16, 11.26, 150M parameters; 13. jinaai/jina-clip-v1, 9.74, 223M parameters. Model and results by whybe-choi. Source: MTEB, leaderboard.mteb.org.

111

Reposted by Kenneth Enevoldsen

MTEB @mteb.org · 01/09/2026

New benchmark: MTEB(slk, v1) is MTEB's first Slovak board — 31 tasks across 7 task types, 17 built for it: parliamentary sentiment, fact-checking NLI, pharmacy Q&A, news clustering. Of the 8 models scored on all 31, intfloat/multilingual-e5-large-instruct leads. Contributed by andrejridzik.

New benchmark on MTEB: MTEB(slk, v1) — 31 tasks across 7 task types — 17 of them built for this board. Stacked column chart, tasks per type, new tasks above existing ones: Classification 7 (3 new), BitextMining 6 (1 new), Clustering 5 (4 new), Retrieval 5 (1 new), Reranking 3 (3 new), PairClassification 3 (3 new), STS 2 (2 new). 8 models have complete 31-task coverage so far. Six of the 17 new tasks: SlovakPharmacyDrMaxReranking (Reranking) — Pharmacists’ answers to medication and health questions, from a Slovak pharmacy site; DemagogSKNLI (PairClassification) — Fact-checkers’ evidence paired with the political claims it supports or refutes; SlovakParlaSentClassification (Classification) — Sentiment in Slovak parliamentary debate, three levels; PravdaSKTagClustering (Clustering) — News articles from Pravda.sk, grouped by their editorial tags; SMESumRetrieval (Retrieval) — Article introductions as queries over 80,000 SME news documents; SlovakSTS (STS) — A professional Slovak translation of the STS Benchmark. Benchmark by andrejridzik. Source: MTEB, leaderboard.mteb.org.

041

Reposted by Kenneth Enevoldsen

MTEB @mteb.org · 20/08/2026

New on MTEB: malteos/most-embed-de, a German retrieval model fine-tuned from Nemotron-3-Embed-1B, takes 1st of 106 models with full coverage on a custom 6-task German-retrieval subset at 58.07 — beating the runner-up at roughly 1/15th the active parameters.

New model on MTEB: malteos/most-embed-de — Highest score on German retrieval among 106 full-coverage models — 15× smaller than the runner-up. Scatter plot of German Retrieval (6 tasks, custom subset) against Active parameters for 104 German-retrieval models with complete 6-task coverage. malteos/most-embed-de scores 58.07 at 872M active parameters. Compared against codefuse-ai/F2LLM-v2-14B at 13.214B, 55.73; codefuse-ai/F2LLM-v2-8B at 6.947B, 55.22; Snowflake/snowflake-arctic-embed-l-v2.0 at 312M, 54.24; codefuse-ai/F2LLM-v2-4B at 3.634B, 53.98; codefuse-ai/F2LLM-v2-1.7B at 1.41B, 52.91; Snowflake/snowflake-arctic-embed-m-v2.0 at 113M, 51.66; hotchpotch/bekko-embedding-v1-a25m at 25M, 51.26; jinaai/jina-embeddings-v3 at 316M, 49.22; hotchpotch/bekko-embedding-v1-a8m at 8M, 48.90; codefuse-ai/F2LLM-v2-0.6B at 441M, 48.82; codefuse-ai/F2LLM-v2-330M at 198M, 48.79; GritLM/GritLM-7B at 7.109B, 47.68; intfloat/e5-mistral-7b-instruct at 6.98B, 46.69; intfloat/multilingual-e5-large-instruct at 304M, 45.96; OrdalieTech/Solon-embeddings-large-0.1 at 304M, 45.82; intfloat/multilingual-e5-large at 304M, 44.91; mteb/baseline-bm25s at 0 (static), 44.80; Lajavaness/bilingual-embedding-large at 304M, 44.71; intfloat/multilingual-e5-base at 86M, 43.43; HIT-TMG/KaLM-embedding-multilingual-mini-v1 at 358M, 43.39; codefuse-ai/F2LLM-v2-160M at 62M, 43.05; Lajavaness/bilingual-embedding-base at 86M, 42.94; intfloat/multilingual-e5-small at 22M, 42.62; aari1995/German_Semantic_STS_V2 at 304M, 40.99; ibm-granite/granite-embedding-278m-multilingual at 86M, 40.25; Lajavaness/bilingual-embedding-small at 22M, 40.17; HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1 at 358M, 39.91; codefuse-ai/F2LLM-v2-80M at 32M, 39.21; ibm-granite/granite-embedding-107m-multilingual at 11M, 37.91; jinaai/jina-embedding-b-en-v1 at 85M, 36.53; manu/sentence_croissant_alpha_v0.2 at 1.214B, 36.22; intfloat/e5-large-v2 at 304M, 35.97; sentence-transformers/static-similarity-mrl-multilingual-v1 at 0 (static), 35.55; intfloat/e5-large at 304M, 33.95; Snowflake/snowflake-arctic-embed-l at 304M, 33.86; thenlper/gte-large at 304M, 33.82; sentence-transformers/paraphrase-multilingual-mpnet-base-v2 at 86M, 33.75; WhereIsAI/UAE-Large-V1 at 304M, 33.33; BAAI/bge-large-en-v1.5 at 304M, 33.24; mixedbread-ai/mxbai-embed-large-v1 at 304M, 32.96; avsolatorio/GIST-large-Embedding-v0 at 304M, 32.36; Omartificial-Intelligence-Space/Arabic-all-nli-triplet-Matryoshka at 86M, 32.35; sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 at 22M, 31.49; ibm-granite/granite-embedding-125m-english at 86M, 31.46; intfloat/e5-base-v2 at 86M, 31.44; Snowflake/snowflake-arctic-embed-s at 22M, 31.23; Omartificial-Intelligence-Space/Arabic-labse-Matryoshka at 86M, 31.02; thenlper/gte-base at 86M, 30.82; BAAI/bge-base-en-v1.5 at 86M, 30.78; jinaai/jina-embedding-s-en-v1 at 19M, 30.61; thenlper/gte-small at 22M, 30.07; Omartificial-Intelligence-Space/Arabic-MiniLM-L12-v2-all-nli-triplet at 22M, 29.50; avsolatorio/GIST-Embedding-v0 at 86M, 29.44; Snowflake/snowflake-arctic-embed-m at 86M, 29.16; intfloat/e5-small-v2 at 21M, 29.13; sdadas/mmlw-e5-large at 304M, 28.80; BAAI/bge-small-en-v1.5 at 22M, 28.73; dwzhu/e5-base-4k at 89M, 28.61; intfloat/e5-base at 86M, 28.61; intfloat/e5-small at 21M, 28.51; sentence-transformers/LaBSE at 86M, 28.49; abhinand/MedEmbed-small-v0.1 at 22M, 28.02; minishlab/potion-multilingual-128M at 0 (static), 27.99; Snowflake/snowflake-arctic-embed-m-v1.5 at 86M, 27.97; sentence-transformers/all-mpnet-base-v2 at 86M, 27.49; Gameselo/STS-multilingual-mpnet-base-v2 at 86M, 27.15; ibm-granite/granite-embedding-30m-english at 11M, 26.60; avsolatorio/GIST-small-Embedding-v0 at 22M, 26.37; Mihaiii/Ivysaur at 11M, 26.33; avsolatorio/GIST-all-MiniLM-L6-v2 at 11M, 26.20; sdadas/mmlw-roberta-large at 304M, 25.73; sentence-transformers/all-MiniLM-L12-v2 at 22M, 25.46; manu/sentence_croissant_alpha_v0.4 at 1.214B, 25.27; shibing624/text2vec-base-multilingual at 22M, 24.79; sentence-transformers/all-MiniLM-L6-v2 at 11M, 23.43; Snowflake/snowflake-arctic-embed-xs at 11M, 23.13; sdadas/mmlw-e5-base at 86M, 23.13; sergeyzh/LaBSE-ru-turbo at 87M, 22.97; sdadas/mmlw-roberta-base at 86M, 21.39; cointegrated/LaBSE-en-ru at 87M, 20.17; Mihaiii/Bulbasaur at 6M, 19.74; Mihaiii/Venusaur at 4M, 19.58; ai-forever/ru-en-RoSBERTa at 303M, 19.45; sdadas/mmlw-e5-small at 22M, 19.17; Mihaiii/Wartortle at 6M, 18.75; Mihaiii/gte-micro-v4 at 7M, 18.46; Mihaiii/Squirtle at 4M, 18.08; deepvk/USER-base at 388M, 16.41; consciousAI/cai-lunaris-text-embeddings at 304M, 13.54; sergeyzh/rubert-tiny-turbo at 3M, 7.51; Mihaiii/gte-micro at 6M, 7.13; silma-ai/silma-embeddding-matryoshka-v0.1 at 86M, 6.40; cointegrated/rubert-tiny at 3M, 5.35; cointegrated/rubert-tiny2 at 3M, 5.35; DeepPavlov/rubert-base-cased-sentence at 15M, 5.16; Omartificial-Intelligence-Space/Arabic-mpnet-base-all-nli-triplet at 86M, 4.79; Omartificial-Intelligence-Space/Arabert-all-nli-triplet-Matryoshka at 86M, 4.75; DeepPavlov/distilrubert-small-cased-conversational at 15M, 2.29; DeepPavlov/rubert-base-cased at 1.188B, 2.14; ai-forever/sbert_large_mt_nlu_ru at 304M, 2.08; Omartificial-Intelligence-Space/Marbert-all-nli-triplet-Matryoshka at 86M, 1.87; ai-forever/sbert_large_nlu_ru at 304M, 1.87; deepvk/deberta-v1-base at 85M, 1.09. On the Pareto frontier (nothing is both smaller and higher-scoring): mteb/baseline-bm25s, hotchpotch/bekko-embedding-v1-a8m, hotchpotch/bekko-embedding-v1-a25m, Snowflake/snowflake-arctic-embed-m-v2.0, Snowflake/snowflake-arctic-embed-l-v2.0, malteos/most-embed-de. Contributed by malteos. Source: MTEB, leaderboard.mteb.org.

031

Reposted by Kenneth Enevoldsen

MTEB @mteb.org · 25/08/2026

New benchmark: BRIGHT-Pro, a new multi-aspect retrieval benchmark on real StackExchange questions across 7 domains, built for agentic search — scored on surfacing several reasoning aspects, not one passage. AQ-MedAI/Diver-Retriever-4B-1020 leads at 55.88. Contributed by yilunzhao.

New benchmark: BRIGHT-Pro — A new multi-aspect retrieval benchmark on real StackExchange questions. Bar chart of the top 10 models on BRIGHT-Pro (7 tasks), out of 19 models with complete results. 1. AQ-MedAI/Diver-Retriever-4B-1020, 55.88, 3.6B parameters; 2. AQ-MedAI/Diver-Retriever-4B, 51.97, 3.6B parameters; 3. AQ-MedAI/Diver-Retriever-0.6B, 48.99, 440M parameters; 4. AQ-MedAI/Diver-Retriever-1.7B, 48.02, 1.4B parameters; 5. yale-nlp/RTriever-4B, 46.88, 3.6B parameters; 6. Qwen/Qwen3-Embedding-8B, 41.75, 6.9B parameters; 7. Qwen/Qwen3-Embedding-4B, 40.84, 3.6B parameters; 8. Alibaba-NLP/gte-Qwen2-7B-instruct, 37.75, 6.5B parameters; 9. ReasonIR/ReasonIR-8B, 32.94, 7.0B parameters; 10. Qwen/Qwen3-Embedding-0.6B, 29.93, 440M parameters. Contributed by yilunzhao. Source: MTEB, leaderboard.mteb.org.

121

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 22/08/2026

Danish dynaword was ~4B tokens we released it. Less than a year after it is now more than doubled in size 🔥 Really shows the power of building externsible open datasets.

0111

Reposted by Kenneth Enevoldsen

MTEB @mteb.org · 18/08/2026

1/ MTEB's audio and video coverage grew substantially this month — 25 new tasks across 16 pull requests, spanning music, speech, environmental sound, and video retrieval. 10 of the 16 cite a published paper.

MTEB expands: 25 new audio & video tasks. 16 pull requests, 20–30 Jul 2026. 18 audio tasks, 7 video tasks, across 16 pull requests. 10 of 16 cite a published paper. Featured: Music — Song Describer: Text-to-music retrieval: the query is a caption and the goal is to retrieve the music track it describes from the corpus of 706 recordings. (Manco et al. (2023)) Speech — SPEECH-COCO: Queries are spoken captions and the corpus contains the images; the goal is to retrieve the image described by the spoken caption. (Havard et al. (2017)) Environmental sound — SoundingEarth: Queries are field recordings and the corpus contains the aerial images; the goal is to retrieve the image of the location where the recording was made. (Heidler et al. (2023)) Audio scene — CASTELLA: Retrieval on CASTELLA, the human-annotated long-audio benchmark used as the DCASE 2026 Task 6 evaluation set: given a caption describing a moment, retrieve the full recording (1 to 5 minutes) that contains it. The corpus is the 566 complete recordings and there are 1,347 caption queries, each with its source recording as the single relevant document. Adapted from CASTELLA's audio moment localization task, which regresses a continuous time interval, to whole-recording retrieval for the mteb ranking format. (Munakata et al. (2025)) Video retrieval — MomentSeeker: Queries combine a text question with a reference image, e.g. asking what happened around the pictured moment. (Yuan et al. (2025)) Video QA — MMVU: MMVU is an expert-level, multi-discipline video understanding benchmark with questions spanning 27 subjects across Science, Healthcare, Humanities & Social Sciences, and Engineering. Each multiple-choice example pairs a specialized-domain video with a question and 5 candidate answers. The task is formulated as multiple-choice retrieval: given the (video, question) pair, retrieve the correct candidate. Used the public validation multiple-choice subset (~625 examples). (Zhao et al. (2025)) Credit by task count: dukesun99 (14 tasks) · wissam-sib (9) · Rakshitha-Ireddi (2). Source: MTEB, leaderboard.mteb.org.

121

Reposted by Kenneth Enevoldsen

MTEB @mteb.org · 12/08/2026

New on MTEB: webAI-Official/webAI-ColVec1.1-8b, a visual document retrieval model, takes 1st of 41 on ViDoRe(v3) at 64.95. Its 4B sibling places 3rd at 63.90 — and both beat the lab's previous generation, which sits 6th and 7th.

New model on MTEB: webAI ColVec1.1 — 1st and 3rd on ViDoRe(v3) — both ahead of the lab's own prior generation. Bar chart of the top 10 models on ViDoRe(v3) (10 tasks), out of 41 models with complete results. 1. webAI-Official/webAI-ColVec1.1-8b, 64.95, 8.4B parameters (this release); 2. vultr/VultronRetrieverPrime-Qwen3.5-8B, 64.26, 8.4B parameters; 3. webAI-Official/webAI-ColVec1.1-4b, 63.90, 4.5B parameters (this release); 4. vultr/VultronRetrieverCore-Qwen3.5-4.5B, 63.57, 4.5B parameters; 5. nvidia/nemotron-colembed-vl-8b-v2, 63.42, 8.7B parameters; 6. webAI-Official/webAI-ColVec1-9b, 63.00, 9.4B parameters (same lab, previous generation); 7. webAI-Official/webAI-ColVec1-4b, 62.22, 4.5B parameters (same lab, previous generation); 8. TomoroAI/tomoro-colqwen3-embed-8b, 61.59, 8.0B parameters; 9. nvidia/nemotron-colembed-vl-4b-v2, 61.54, 4.8B parameters; 10. athrael-soju/colqwen3.5-4.5B-v3, 61.46, 4.6B parameters. Both ColVec1.1 sizes outrank the ColVec1 generation they replace. Contributed by zhanlunchang-webai. Source: MTEB, leaderboard.mteb.org.

262

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 10/08/2026

While we see many open weights model these days we are often still missing training code and training data, making it hard for the field to iterate and improve upon the works of others

070

Reposted by Kenneth Enevoldsen

MTEB @mteb.org · 07/08/2026

New on MTEB: Hanno-Labs/dinghy-law-4b-v1, a legal-domain embedding model, takes 1st on MTEB(Law, v1) at 71.22. Its 0.6B sibling places 4th at 65.83 — the smallest model in the top eight by an order of magnitude.

New model on MTEB: dinghy-law — 1st and 4th on MTEB(Law, v1). Bar chart of the top 8 models on MTEB(Law, v1) (8 tasks), out of 159 models with complete results. 1. Hanno-Labs/dinghy-law-4b-v1, 71.22, 4.0B parameters; 2. Mira190/Euler-Legal-Embedding-V1, 70.37, 7.6B parameters; 3. minetta/nemotron-3-embed-8b-legal, 69.33, 8.0B parameters; 4. Hanno-Labs/dinghy-law-0.6b-v1, 65.83, 0.6B parameters; 5. voyageai/voyage-law-2, 65.39, proprietary; 6. codefuse-ai/F2LLM-v2-14B, 64.66, 14B parameters; 7. voyageai/voyage-3, 64.13, proprietary; 8. infly/inf-retriever-v1, 63.68, 7.1B parameters. Contributed by trashhalo. Source: MTEB, leaderboard.mteb.org.

021

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 05/08/2026

Embedding models keep getting cheaper - what required ~300M parameters three years ago now requires only 11M. In practice, that means that what runs on the server today could run in the browser tomorrow.

0222

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 31/07/2026

Fewer than 40% of open-weight models share training code or training data. This makes it impossible to improve the training script or iterate on the data, and reduces transparency. To combat this, we previously introduced an openness score. Today we add it to our leaderboard.

150

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 24/07/2026

MTEB reached 5000 contributions! Happy to see the package expand, as well as the community around it.

050

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 14/07/2026

A somewhat standard reviewing practice keeps frustrating me. When a paper does not mention sharing code or data it seems common that people rate if it was shared. Giving it scores like: "Software: 4 = Useful: I would recommend [...]" "Dataset: 2 = Documentary: The new datasets will be [...]"

110

Reposted by Kenneth Enevoldsen

Vladimir Salnikov @v4ldelund.bsky.social · 05/07/2026

Not so long ago we expanded Dynaword to Swedish, Norwegian and Icelandic Now have a new addition - Faroese 🇫🇴 If you are working on Faroese NLP or know someone who works on it, we would greatly appreciate any advice regarding what datasets we should add -> huggingface.co/datasets/dan...

huggingface.co

danish-foundation-models/faroese-dynaword · Datasets at Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

2132

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 03/07/2026

Really cool work from our Danish foundation model, from the group at SDU. Do check it out!

0121

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 22/06/2026

mteb.org is now the new home of mteb -- you can now find the docs at docs.mteb.org and the leaderboard on leaderboard.mteb.org 💫

leaderboard.mteb.org

000

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 16/06/2026

Very happy to announce our new release of MVEB our new benchmark in MTEB for evaluating video embeddings! Whether you want to evaluate embeddings for clustering video, matching video with text, or using them within a retrieval system, this benchmark should cover you.

153

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 05/06/2026

Stuff is happening to the MTEB leaderboard — looking forward to sharing more soon!

030

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 31/05/2026

I don't get why sharing code and data in the review process is not yet required as a default. 86% of papers that share code and data reproduce, while it is only 33% of for studies that only share data [1] and of course lower for studies that share neither.

420

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 25/05/2026

Instruction-tuned embedding models systematically overreport their expected performance. A single reported score hides a wide distribution — and it's rarely in the middle. 🧵 #MTEB #NLP #evaluation

120

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 19/05/2026

We are very happy to introduce AfriMTEB to #MTEB 🌍 If you are looking to evaluate an embedding model for African languages, you can now do so directly with MTEB! Big thanks for Kosei Uemura and @dadelani.bsky.social for the PR! #retrieval, #evaluation, #nlp

052

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 17/05/2026

Had the pleasure to finish of #LREC2026 at the CMCL workshop, with a joint work with Sara, where we show that the choice of embedding model greatly influences your conclusion when modelling semantic associations — in follow up work we will look into what the best practices are

061

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 14/05/2026

Presented on the Dynaword today at LREC. The Dynaword is a group of continuously developed datasets that currently exists in Danish and Norwegian, with multiple languages coming up. If you are around LREC and want to discuss how to make lasting data infrastructure do reach out!

031

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 10/05/2026

One of my favourite new mteb features is the new result submission workflow. Submitting results for a model is just one line away, and once it is in, you don't have to compute it again. The results repository is a public cache that everyone can use! 💫

100

Reposted by Kenneth Enevoldsen

Vladimir Salnikov @v4ldelund.bsky.social · 21/04/2026

We are organizing Dynaword mini-hackathon Danish Dynaword is the largest continuously developed collection of openly licenced datasets for Danish language. We are currently expanding it to Swedish, Norwegian and Icelandic and looking for new contributors ! 1/2

121

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 21/04/2026

Please welcome the Spanish benchmark to MTEB 🇪🇸 Big thanks to Clemente-H for the contribution!

020

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 24/02/2026

We recently released a new audio benchmark in MTEB along with a paper. You can get all the main conclusion from our blog post: huggingface.co/blog/AdnanEl...

huggingface.co

MAEB: Evaluating Audio Embeddings at Scale

A Blog post by Adnan El Assadi on Hugging Face

021

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 12/03/2025

New postdoc position at AarhusNLP, come join us! The research includes efficient post-training, alignment, evaluation, and preference optimization, but we are very flexible for reinterpretation. So, if you think that you might be a partial fit do apply! international.au.dk/about/profil...

international.au.dk

Postdoctoral Positions in NLP Post-Training for Cultural Alignment and Preference Optimization - Vacancy at Aarhus University

Vacancy at School of Culture and Society - Center for Humanities Computing Aarhus, Aarhus University

0112

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 11/03/2025

Last week at #NoDaLiDa, we presented our work on 🇪🇺EuroEval, a large-scale benchmark for evaluating decoders and encoders.  The framework consists of 9 (🇬🇧🇫🇷🇩🇪🇳🇱🇸🇪🇩🇰🇳🇴🇮🇸🇫🇴) languages, with more to come, where each includes a language understanding and generation benchmark

250

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 20/02/2025

I am delighted to announce that we have released 🎊 MMTEB 🎊, a large-scale collaboration working on efficient multilingual evaluation of embedding models. This work implements >500 evaluation tasks across >1000 languages and covers a wide range of use cases and domains🩺👩‍💻⚖️

1286

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 11/01/2025

Multilingual MTEB is soon to be released and with it, a shining new benchmark with a zero-shot filter! However, zero-shot is quite hard to define in a time of derivative models and synthetic data. If you have an opinion on how it should define zero-shot, let us know: github.com/embeddings-b...

github.com

Defining zero-shot for MTEB · Issue #1760 · embeddings-benchmark/mteb

The next version of the MTEB leaderboard will soon be released and with it a new zero-shot filter. However, we are currently planning to use the following definition of zero-shot. This issue is to ...

031

Reposted by Kenneth Enevoldsen

Iris van Rooij 💭 @irisvanrooij.bsky.social · 24/12/2024

📣 Vacancy for Assistant Professor of Cognitive Science at Department of Linguistics, Cognitive Science and Semiotics, Aarhus University, Denmark. (Deadline January 6) international.au.dk/about/profil...

international.au.dk

Assistant Professor of Cognitive Science at the School of Communication and Culture - Vacancy at Aarhus University

Vacancy at School of Communication and Culture - Linguistics, Cognitive Science and Semiotics, Dept. of, Aarhus University

14938

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 27/12/2024

Does anyone know of good methods for tracking dataset contaminations that doesn't rely on generation? Anything would be greatly appreciated Asking as we would like to track and detect dataset contamination in MTEB: github.com/embeddings-b...

github.com

track and detect dataset contamination · Issue #1636 · embeddings-benchmark/mteb

In multiple threads tracking dataset contamination have been mentioned as multiple models do not share their training dataset. This issue is intended to link these discussions together as well as p...

002

Reposted by Kenneth Enevoldsen

Kasper Junge @kasperjunge.bsky.social · 20/12/2024

Ugens afsnit af Verbos podcast er live med @kennethenevoldsen.bsky.social og Thomas Kobber Panum til en snak om AI's udvikling med udgangspunkt i Ilya Sutskevers talk til NeurIPS 🔥 #dkai #dktech Lyt her 👇 YouTube: youtu.be/IpEla8mZHnU?... Spotify: open.spotify.com/episode/41WT...

031

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 06/12/2024

Any good reason for using (machine) translated datasets/benchmarks for evaluating language models?

000

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 05/12/2024

On my way to Neurips to present our work on the Scandinavian Embedding Benchmark (SEB) for evaluating embedding for retrieval, classification etc. for mainland Scandinavian languages. Is there anything I should see while I am there?

010