iiinova — where your digital self keeps evolving

Reposted by MTEB

Kenneth Enevoldsen @kennethenevoldsen.eurosky.social · 07/10/2026

Submitted my two accepted posters for EMNLP today, if you are there an want meet up feel free to reach out! I also have a not so @mteb.org themed poster for a workshop on socio-cultural alignment -- stay tuned for that one

Poster on MVEB: Massive Video Embedding Benchmark

Poster on the paper: One Prompt is Not Enough: Instruction sensitivity Endermines Embedding Model Evaluation.

082

MTEB @mteb.org · 08/09/2026

New on MTEB: litillabs/octen-law-8b-v1 takes 1st of 164 on MTEB(Law, v1) at 76.77, 4.19 clear of the next model. It declares two of the eight tasks as training data. Recomputed without them, the lead widens: 81.77 to 75.00. Model and results by narcolepticchicken.

New model on MTEB: litillabs/octen-law-8b-v1 — 1st on MTEB(Law, v1) at 76.77, 4.19 clear of the next model. Bar chart of 10 of the 164 models with complete results on MTEB(Law, v1) (8 tasks), each row given with its rank on that board. 1. litillabs/octen-law-8b-v1, 76.77, 7.6B parameters (this release · declares 2 of the 8 tasks as training data); 2. Hanno-Labs/dinghy-law-8b-v1, 72.58, 8.0B parameters; 3. Hanno-Labs/dinghy-law-4b-v1, 71.22, 4.0B parameters; 4. Mira190/Euler-Legal-Embedding-V1, 70.37, 7.6B parameters; 5. minetta/nemotron-3-embed-8b-legal, 69.33, 8.0B parameters; 6. Hanno-Labs/dinghy-law-0.6b-v1, 65.83, 597M parameters; 7. voyageai/voyage-law-2, 65.39, size not published; 8. codefuse-ai/F2LLM-v2-14B, 64.66, 14B parameters; 9. judicialmind/greenleaf-law-embed-tiny, 64.49, 596M parameters; 10. voyageai/voyage-3, 64.13, size not published. Model and results by narcolepticchicken. Source: MTEB, leaderboard.mteb.org.

010

MTEB @mteb.org · 03/09/2026

The other half of that board: 20 models score higher, and every one of them is a late-interaction retriever, storing a vector per page patch instead of one per document. Of 50 models with complete results, gemini-embedding-2 is 21st.

New model on MTEB: google/gemini-embedding-2 — 21st of 50 — and every model above it is a late-interaction retriever. Bar chart of 11 of the 50 models with complete results on ViDoRe(v2) (4 tasks), each row given with its rank on that board. 1. DataScience-UIBK/Argus-Colqwen3.5-9b-v0, 69.27, 8.8B parameters, late-interaction; 2. DataScience-UIBK/Argus-Colqwen3.5-9b-v0-bf16, 69.12, 8.8B parameters, late-interaction; 3. OpenSearch-AI/Ops-Colqwen3-4B, 68.66, 4.8B parameters, late-interaction; 4. vultr/VultronRetrieverPrime-Qwen3.5-8B, 68.18, 8.4B parameters, late-interaction; 5. vultr/VultronRetrieverCore-Qwen3.5-4.5B, 66.12, 4.5B parameters, late-interaction; 6. webAI-Official/webAI-ColVec1-9b, 65.82, 9.4B parameters, late-interaction; 7. TomoroAI/tomoro-colqwen3-embed-8b, 65.40, 8.0B parameters, late-interaction; 8. ApsaraStackMaaS/EvoQwen2.5-VL-Retriever-7B-v1, 65.24, 7.0B parameters, late-interaction; 9. nvidia/nemotron-colembed-vl-8b-v2, 65.16, 8.7B parameters, late-interaction; 10. TomoroAI/tomoro-colqwen3-embed-4b, 64.69, 4.0B parameters, late-interaction; ranks 11 to 20, not shown individually: all late-interaction, 64.49 down to 62.47; 21. google/gemini-embedding-2, 62.37, size not published, single-vector (this release). Model and results by whybe-choi. Source: MTEB, leaderboard.mteb.org.

000

MTEB @mteb.org · 03/09/2026

New on MTEB: google/gemini-embedding-2, which takes interleaved text, image, audio and video in one input. Its first results are on ViDoRe(v2), visual document retrieval: 62.37, first of the 13 single-vector models with complete results. Model and results by whybe-choi.

New model on MTEB: google/gemini-embedding-2 — First of the 13 single-vector models on ViDoRe(v2). Bar chart of 13 of the 13 models with complete results on ViDoRe(v2) (4 tasks), each row given with its rank on that board. 1. google/gemini-embedding-2, 62.37, size not published (this release); 2. nanovdr/NanoVDR-S-Multi, 61.93, 69M parameters; 3. jinaai/jina-embeddings-v4, 58.23, 3.9B parameters; 4. eagerworks/eager-embed-v1, 57.20, 4.4B parameters; 5. ibm-granite/granite-vision-3.3-2b-embedding, 56.92, 3.0B parameters; 6. voyageai/voyage-multimodal-3, 54.94, size not published; 7. royokong/e5-v, 46.55, 8.4B parameters; 8. google/siglip-so400m-patch14-384, 33.00, 878M parameters; 9. TIGER-Lab/VLM2Vec-Full, 31.82, 4.1B parameters; 10. jinaai/jina-clip-v2, 26.94, 865M parameters; 11. laion/CLIP-ViT-bigG-14-laion2B-39B-b160k, 19.35, 2.5B parameters; 12. openai/clip-vit-base-patch16, 11.26, 150M parameters; 13. jinaai/jina-clip-v1, 9.74, 223M parameters. Model and results by whybe-choi. Source: MTEB, leaderboard.mteb.org.

111

MTEB @mteb.org · 01/09/2026

New benchmark: MTEB(slk, v1) is MTEB's first Slovak board — 31 tasks across 7 task types, 17 built for it: parliamentary sentiment, fact-checking NLI, pharmacy Q&A, news clustering. Of the 8 models scored on all 31, intfloat/multilingual-e5-large-instruct leads. Contributed by andrejridzik.

New benchmark on MTEB: MTEB(slk, v1) — 31 tasks across 7 task types — 17 of them built for this board. Stacked column chart, tasks per type, new tasks above existing ones: Classification 7 (3 new), BitextMining 6 (1 new), Clustering 5 (4 new), Retrieval 5 (1 new), Reranking 3 (3 new), PairClassification 3 (3 new), STS 2 (2 new). 8 models have complete 31-task coverage so far. Six of the 17 new tasks: SlovakPharmacyDrMaxReranking (Reranking) — Pharmacists’ answers to medication and health questions, from a Slovak pharmacy site; DemagogSKNLI (PairClassification) — Fact-checkers’ evidence paired with the political claims it supports or refutes; SlovakParlaSentClassification (Classification) — Sentiment in Slovak parliamentary debate, three levels; PravdaSKTagClustering (Clustering) — News articles from Pravda.sk, grouped by their editorial tags; SMESumRetrieval (Retrieval) — Article introductions as queries over 80,000 SME news documents; SlovakSTS (STS) — A professional Slovak translation of the STS Benchmark. Benchmark by andrejridzik. Source: MTEB, leaderboard.mteb.org.

041

MTEB @mteb.org · 27/08/2026

On LMEB (long-horizon memory retrieval), opensearch-project/opensearch-neural-sparse-encoding-doc-v2-mini (11M active) sits on the Pareto frontier — nothing smaller scores higher, at any size up to 7.4B active. Newly scored, not newly released: one of 12 models that got LMEB results this week.

On the leaderboard: opensearch-project/opensearch-neural-sparse-encoding-doc-v2-mini — Newly scored on LMEB, and on its Pareto frontier — nothing smaller scores higher, at any size up to 7.4B active params. Scatter plot of LMEB against Active parameters for 70 models with complete 22-task LMEB coverage. opensearch-project/opensearch-neural-sparse-encoding-doc-v2-mini scores 56.03 at 11M active parameters. Compared against nvidia/Nemotron-3-Embed-8B-BF16 at 7.416B, 64.36; nvidia/Nemotron-3-Embed-1B-BF16 at 872M, 61.50; BAAI/bge-multilingual-gemma2 at 8.322B, 61.32; nvidia/NV-Embed-v2 at 7.719B, 60.00; tencent/KaLM-Embedding-Gemma3-12B-2511 at 10.759B, 59.83; codefuse-ai/F2LLM-v2-4B at 3.634B, 58.68; codefuse-ai/F2LLM-v2-8B at 6.947B, 58.64; microsoft/harrier-oss-v1-0.6b at 440M, 58.43; Salesforce/SFR-Embedding-Mistral at 6.98B, 58.13; perplexity-ai/pplx-embed-v1-0.6b at 440M, 57.92; lightonai/DenseOn at 110M, 57.61; codefuse-ai/F2LLM-v2-1.7B at 1.41B, 57.60; Alibaba-NLP/gte-Qwen2-7B-instruct at 6.526B, 57.07; Salesforce/SFR-Embedding-2_R at 6.98B, 57.06; opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte at 113M, 56.99; Snowflake/snowflake-arctic-embed-m-v2.0 at 113M, 56.95; Snowflake/snowflake-arctic-embed-l-v2.0 at 312M, 56.89; opensearch-project/opensearch-neural-sparse-encoding-doc-v2-distill at 43M, 56.53; opensearch-project/opensearch-neural-sparse-encoding-doc-v3-distill at 43M, 56.34; HIT-TMG/KaLM-embedding-multilingual-mini-v1 at 358M, 56.10; intfloat/e5-large-v2 at 304M, 55.95; Qwen/Qwen3-Embedding-8B at 6.946B, 55.94; codefuse-ai/F2LLM-v2-0.6B at 441M, 55.78; intfloat/multilingual-e5-large at 304M, 55.49; intfloat/e5-mistral-7b-instruct at 6.98B, 55.22; HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1.5 at 358M, 55.19; Alibaba-NLP/gte-Qwen2-1.5B-instruct at 1.31B, 55.18; codefuse-ai/F2LLM-v2-330M at 198M, 55.15; intfloat/multilingual-e5-large-instruct at 304M, 54.95; KaLM-Embedding/KaLM-embedding-multilingual-mini-instruct-v2.5 at 358M, 54.93; Qwen/Qwen3-Embedding-0.6B at 440M, 54.56; Snowflake/snowflake-arctic-embed-m-v1.5 at 86M, 54.53; intfloat/e5-base-v2 at 86M, 54.42; jinaai/jina-embeddings-v5-text-small at 440M, 54.22; hotchpotch/bekko-embedding-v1-a25m at 25M, 53.74; microsoft/harrier-oss-v1-270m at 100M, 53.36; Qwen/Qwen3-Embedding-4B at 3.634B, 53.30; intfloat/e5-small-v2 at 21M, 53.23; intfloat/multilingual-e5-base at 86M, 53.18; BAAI/bge-base-en-v1.5 at 86M, 52.57; ibm-granite/granite-embedding-278m-multilingual at 86M, 52.28; hotchpotch/bekko-embedding-v1-a8m at 8M, 52.28; codefuse-ai/F2LLM-v2-160M at 62M, 52.24; Snowflake/snowflake-arctic-embed-s at 22M, 52.22; ibm-granite/granite-embedding-english-r2 at 110M, 52.03; ibm-granite/granite-embedding-311m-multilingual-r2 at 110M, 51.32; LiquidAI/LFM2.5-Embedding-350M at 287M, 51.05; BAAI/bge-small-en-v1.5 at 22M, 51.01; BAAI/bge-m3 at 312M, 50.93; ibm-granite/granite-embedding-107m-multilingual at 11M, 50.76; Alibaba-NLP/gte-base-en-v1.5 at 113M, 50.47; Snowflake/snowflake-arctic-embed-xs at 11M, 50.38; mteb/baseline-bm25s at 0 (static), 50.37; intfloat/multilingual-e5-small at 22M, 50.33; ibm-granite/granite-embedding-small-english-r2 at 28M, 49.79; codefuse-ai/F2LLM-v2-80M at 32M, 49.24; ibm-granite/granite-embedding-30m-english at 11M, 48.85; ibm-granite/granite-embedding-97m-multilingual-r2 at 28M, 48.81; BAAI/bge-en-icl at 6.979B, 47.68; ibm-granite/granite-embedding-125m-english at 86M, 46.24; sentence-transformers/all-MiniLM-L12-v2 at 22M, 45.79; sentence-transformers/all-mpnet-base-v2 at 86M, 45.08; sentence-transformers/all-MiniLM-L6-v2 at 11M, 43.77; BAAI/bge-large-en-v1.5 at 304M, 42.14; sentence-transformers/static-similarity-mrl-multilingual-v1 at 0 (static), 41.49; sentence-transformers/paraphrase-multilingual-mpnet-base-v2 at 86M, 41.23; minishlab/potion-multilingual-128M at 0 (static), 39.92; TencentBAC/Conan-embedding-v1 at 304M, 38.83; sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 at 22M, 36.20. On the Pareto frontier (nothing is both smaller and higher-scoring): mteb/baseline-bm25s, hotchpotch/bekko-embedding-v1-a8m, opensearch-project/opensearch-neural-sparse-encoding-doc-v2-mini, opensearch-project/opensearch-neural-sparse-encoding-doc-v2-distill, lightonai/DenseOn, microsoft/harrier-oss-v1-0.6b, nvidia/Nemotron-3-Embed-1B-BF16, nvidia/Nemotron-3-Embed-8B-BF16. Contributed by the OpenSearch project. Source: MTEB, leaderboard.mteb.org.

010

MTEB @mteb.org · 25/08/2026

New benchmark: BRIGHT-Pro, a new multi-aspect retrieval benchmark on real StackExchange questions across 7 domains, built for agentic search — scored on surfacing several reasoning aspects, not one passage. AQ-MedAI/Diver-Retriever-4B-1020 leads at 55.88. Contributed by yilunzhao.

New benchmark: BRIGHT-Pro — A new multi-aspect retrieval benchmark on real StackExchange questions. Bar chart of the top 10 models on BRIGHT-Pro (7 tasks), out of 19 models with complete results. 1. AQ-MedAI/Diver-Retriever-4B-1020, 55.88, 3.6B parameters; 2. AQ-MedAI/Diver-Retriever-4B, 51.97, 3.6B parameters; 3. AQ-MedAI/Diver-Retriever-0.6B, 48.99, 440M parameters; 4. AQ-MedAI/Diver-Retriever-1.7B, 48.02, 1.4B parameters; 5. yale-nlp/RTriever-4B, 46.88, 3.6B parameters; 6. Qwen/Qwen3-Embedding-8B, 41.75, 6.9B parameters; 7. Qwen/Qwen3-Embedding-4B, 40.84, 3.6B parameters; 8. Alibaba-NLP/gte-Qwen2-7B-instruct, 37.75, 6.5B parameters; 9. ReasonIR/ReasonIR-8B, 32.94, 7.0B parameters; 10. Qwen/Qwen3-Embedding-0.6B, 29.93, 440M parameters. Contributed by yilunzhao. Source: MTEB, leaderboard.mteb.org.

121

MTEB @mteb.org · 20/08/2026

New on MTEB: malteos/most-embed-de, a German retrieval model fine-tuned from Nemotron-3-Embed-1B, takes 1st of 106 models with full coverage on a custom 6-task German-retrieval subset at 58.07 — beating the runner-up at roughly 1/15th the active parameters.

New model on MTEB: malteos/most-embed-de — Highest score on German retrieval among 106 full-coverage models — 15× smaller than the runner-up. Scatter plot of German Retrieval (6 tasks, custom subset) against Active parameters for 104 German-retrieval models with complete 6-task coverage. malteos/most-embed-de scores 58.07 at 872M active parameters. Compared against codefuse-ai/F2LLM-v2-14B at 13.214B, 55.73; codefuse-ai/F2LLM-v2-8B at 6.947B, 55.22; Snowflake/snowflake-arctic-embed-l-v2.0 at 312M, 54.24; codefuse-ai/F2LLM-v2-4B at 3.634B, 53.98; codefuse-ai/F2LLM-v2-1.7B at 1.41B, 52.91; Snowflake/snowflake-arctic-embed-m-v2.0 at 113M, 51.66; hotchpotch/bekko-embedding-v1-a25m at 25M, 51.26; jinaai/jina-embeddings-v3 at 316M, 49.22; hotchpotch/bekko-embedding-v1-a8m at 8M, 48.90; codefuse-ai/F2LLM-v2-0.6B at 441M, 48.82; codefuse-ai/F2LLM-v2-330M at 198M, 48.79; GritLM/GritLM-7B at 7.109B, 47.68; intfloat/e5-mistral-7b-instruct at 6.98B, 46.69; intfloat/multilingual-e5-large-instruct at 304M, 45.96; OrdalieTech/Solon-embeddings-large-0.1 at 304M, 45.82; intfloat/multilingual-e5-large at 304M, 44.91; mteb/baseline-bm25s at 0 (static), 44.80; Lajavaness/bilingual-embedding-large at 304M, 44.71; intfloat/multilingual-e5-base at 86M, 43.43; HIT-TMG/KaLM-embedding-multilingual-mini-v1 at 358M, 43.39; codefuse-ai/F2LLM-v2-160M at 62M, 43.05; Lajavaness/bilingual-embedding-base at 86M, 42.94; intfloat/multilingual-e5-small at 22M, 42.62; aari1995/German_Semantic_STS_V2 at 304M, 40.99; ibm-granite/granite-embedding-278m-multilingual at 86M, 40.25; Lajavaness/bilingual-embedding-small at 22M, 40.17; HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1 at 358M, 39.91; codefuse-ai/F2LLM-v2-80M at 32M, 39.21; ibm-granite/granite-embedding-107m-multilingual at 11M, 37.91; jinaai/jina-embedding-b-en-v1 at 85M, 36.53; manu/sentence_croissant_alpha_v0.2 at 1.214B, 36.22; intfloat/e5-large-v2 at 304M, 35.97; sentence-transformers/static-similarity-mrl-multilingual-v1 at 0 (static), 35.55; intfloat/e5-large at 304M, 33.95; Snowflake/snowflake-arctic-embed-l at 304M, 33.86; thenlper/gte-large at 304M, 33.82; sentence-transformers/paraphrase-multilingual-mpnet-base-v2 at 86M, 33.75; WhereIsAI/UAE-Large-V1 at 304M, 33.33; BAAI/bge-large-en-v1.5 at 304M, 33.24; mixedbread-ai/mxbai-embed-large-v1 at 304M, 32.96; avsolatorio/GIST-large-Embedding-v0 at 304M, 32.36; Omartificial-Intelligence-Space/Arabic-all-nli-triplet-Matryoshka at 86M, 32.35; sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 at 22M, 31.49; ibm-granite/granite-embedding-125m-english at 86M, 31.46; intfloat/e5-base-v2 at 86M, 31.44; Snowflake/snowflake-arctic-embed-s at 22M, 31.23; Omartificial-Intelligence-Space/Arabic-labse-Matryoshka at 86M, 31.02; thenlper/gte-base at 86M, 30.82; BAAI/bge-base-en-v1.5 at 86M, 30.78; jinaai/jina-embedding-s-en-v1 at 19M, 30.61; thenlper/gte-small at 22M, 30.07; Omartificial-Intelligence-Space/Arabic-MiniLM-L12-v2-all-nli-triplet at 22M, 29.50; avsolatorio/GIST-Embedding-v0 at 86M, 29.44; Snowflake/snowflake-arctic-embed-m at 86M, 29.16; intfloat/e5-small-v2 at 21M, 29.13; sdadas/mmlw-e5-large at 304M, 28.80; BAAI/bge-small-en-v1.5 at 22M, 28.73; dwzhu/e5-base-4k at 89M, 28.61; intfloat/e5-base at 86M, 28.61; intfloat/e5-small at 21M, 28.51; sentence-transformers/LaBSE at 86M, 28.49; abhinand/MedEmbed-small-v0.1 at 22M, 28.02; minishlab/potion-multilingual-128M at 0 (static), 27.99; Snowflake/snowflake-arctic-embed-m-v1.5 at 86M, 27.97; sentence-transformers/all-mpnet-base-v2 at 86M, 27.49; Gameselo/STS-multilingual-mpnet-base-v2 at 86M, 27.15; ibm-granite/granite-embedding-30m-english at 11M, 26.60; avsolatorio/GIST-small-Embedding-v0 at 22M, 26.37; Mihaiii/Ivysaur at 11M, 26.33; avsolatorio/GIST-all-MiniLM-L6-v2 at 11M, 26.20; sdadas/mmlw-roberta-large at 304M, 25.73; sentence-transformers/all-MiniLM-L12-v2 at 22M, 25.46; manu/sentence_croissant_alpha_v0.4 at 1.214B, 25.27; shibing624/text2vec-base-multilingual at 22M, 24.79; sentence-transformers/all-MiniLM-L6-v2 at 11M, 23.43; Snowflake/snowflake-arctic-embed-xs at 11M, 23.13; sdadas/mmlw-e5-base at 86M, 23.13; sergeyzh/LaBSE-ru-turbo at 87M, 22.97; sdadas/mmlw-roberta-base at 86M, 21.39; cointegrated/LaBSE-en-ru at 87M, 20.17; Mihaiii/Bulbasaur at 6M, 19.74; Mihaiii/Venusaur at 4M, 19.58; ai-forever/ru-en-RoSBERTa at 303M, 19.45; sdadas/mmlw-e5-small at 22M, 19.17; Mihaiii/Wartortle at 6M, 18.75; Mihaiii/gte-micro-v4 at 7M, 18.46; Mihaiii/Squirtle at 4M, 18.08; deepvk/USER-base at 388M, 16.41; consciousAI/cai-lunaris-text-embeddings at 304M, 13.54; sergeyzh/rubert-tiny-turbo at 3M, 7.51; Mihaiii/gte-micro at 6M, 7.13; silma-ai/silma-embeddding-matryoshka-v0.1 at 86M, 6.40; cointegrated/rubert-tiny at 3M, 5.35; cointegrated/rubert-tiny2 at 3M, 5.35; DeepPavlov/rubert-base-cased-sentence at 15M, 5.16; Omartificial-Intelligence-Space/Arabic-mpnet-base-all-nli-triplet at 86M, 4.79; Omartificial-Intelligence-Space/Arabert-all-nli-triplet-Matryoshka at 86M, 4.75; DeepPavlov/distilrubert-small-cased-conversational at 15M, 2.29; DeepPavlov/rubert-base-cased at 1.188B, 2.14; ai-forever/sbert_large_mt_nlu_ru at 304M, 2.08; Omartificial-Intelligence-Space/Marbert-all-nli-triplet-Matryoshka at 86M, 1.87; ai-forever/sbert_large_nlu_ru at 304M, 1.87; deepvk/deberta-v1-base at 85M, 1.09. On the Pareto frontier (nothing is both smaller and higher-scoring): mteb/baseline-bm25s, hotchpotch/bekko-embedding-v1-a8m, hotchpotch/bekko-embedding-v1-a25m, Snowflake/snowflake-arctic-embed-m-v2.0, Snowflake/snowflake-arctic-embed-l-v2.0, malteos/most-embed-de. Contributed by malteos. Source: MTEB, leaderboard.mteb.org.

031

MTEB @mteb.org · 18/08/2026

3/ Video (7 tasks): VSC2022, MomentSeeker, MMVU, plus FLARE (3 tasks). Contributed by dukesun99 (14 tasks), wissam-sib (9), and Rakshitha-Ireddi (2).

010

MTEB @mteb.org · 18/08/2026

1/ MTEB's audio and video coverage grew substantially this month — 25 new tasks across 16 pull requests, spanning music, speech, environmental sound, and video retrieval. 10 of the 16 cite a published paper.

MTEB expands: 25 new audio & video tasks. 16 pull requests, 20–30 Jul 2026. 18 audio tasks, 7 video tasks, across 16 pull requests. 10 of 16 cite a published paper. Featured: Music — Song Describer: Text-to-music retrieval: the query is a caption and the goal is to retrieve the music track it describes from the corpus of 706 recordings. (Manco et al. (2023)) Speech — SPEECH-COCO: Queries are spoken captions and the corpus contains the images; the goal is to retrieve the image described by the spoken caption. (Havard et al. (2017)) Environmental sound — SoundingEarth: Queries are field recordings and the corpus contains the aerial images; the goal is to retrieve the image of the location where the recording was made. (Heidler et al. (2023)) Audio scene — CASTELLA: Retrieval on CASTELLA, the human-annotated long-audio benchmark used as the DCASE 2026 Task 6 evaluation set: given a caption describing a moment, retrieve the full recording (1 to 5 minutes) that contains it. The corpus is the 566 complete recordings and there are 1,347 caption queries, each with its source recording as the single relevant document. Adapted from CASTELLA's audio moment localization task, which regresses a continuous time interval, to whole-recording retrieval for the mteb ranking format. (Munakata et al. (2025)) Video retrieval — MomentSeeker: Queries combine a text question with a reference image, e.g. asking what happened around the pictured moment. (Yuan et al. (2025)) Video QA — MMVU: MMVU is an expert-level, multi-discipline video understanding benchmark with questions spanning 27 subjects across Science, Healthcare, Humanities & Social Sciences, and Engineering. Each multiple-choice example pairs a specialized-domain video with a question and 5 candidate answers. The task is formulated as multiple-choice retrieval: given the (video, question) pair, retrieve the correct candidate. Used the public validation multiple-choice subset (~625 examples). (Zhao et al. (2025)) Credit by task count: dukesun99 (14 tasks) · wissam-sib (9) · Rakshitha-Ireddi (2). Source: MTEB, leaderboard.mteb.org.

121

MTEB @mteb.org · 14/08/2026

lightonai/mLateOn and mDenseOn are multilingual extensions of LateOn and DenseOn — eight languages plus English, same backbone. On English retrieval, both beat the English-only models they extend: 23rd of 197 vs 25th, and 28th vs 31st. Added by NohTow, results by paulomouraj.

New models on MTEB: mDenseOn & mLateOn — Going multilingual did not come at the cost of English retrieval quality (BEIR). On BEIR, English retrieval (15 tasks, 197 models with complete coverage): lightonai/mLateOn ranks 23 of 197 at 57.56, ahead of lightonai/LateOn, which ranks 25 of 197 at 57.22 (+0.34 for the multilingual model); lightonai/mDenseOn ranks 28 of 197 at 56.70, ahead of lightonai/DenseOn, which ranks 31 of 197 at 56.20 (+0.50 for the multilingual model). Added by NohTow, results by paulomouraj. Source: MTEB, leaderboard.mteb.org.

010

MTEB @mteb.org · 13/08/2026

New benchmark: MTEB(kor, v2) is now the default Korean board, superseding the 6-task v1. 20 tasks across 6 task types, adding clustering and NLI and with an increased focus on retrieval. Added by daegonYu.

New benchmark: New Korean Benchmark — 20 tasks across 6 task types, adding clustering and NLI and with an increased focus on retrieval. Grouped column chart, task count by type, v1 versus v2: Classification 1 to 1, Clustering 0 to 3 (new), PairClassification 0 to 3 (new), Reranking 1 to 1, Retrieval 2 to 9, STS 2 to 3. v1 has 116 models with complete coverage; v2 has 23. Best model per size class, restricted to the 23 models with complete coverage on both boards, v1 scoring versus v2 scoring: >5B, 4 models: Qwen/Qwen3-Embedding-8B (77.86) to Qwen/Qwen3-Embedding-8B (77.73), unchanged; 1-5B, 3 models: Qwen/Qwen3-Embedding-4B (77.44) to Qwen/Qwen3-Embedding-4B (76.94), unchanged; 500M-1B, 7 models: nlpai-lab/KURE-v1 (74.27) to codefuse-ai/F2LLM-v2-0.6B (72.16), changed; <500M, 9 models: hotchpotch/bekko-embedding-v1-a25m (69.80) to hotchpotch/bekko-embedding-v1-a25m (67.86), unchanged. Added by daegonYu. Source: MTEB, leaderboard.mteb.org.

010

MTEB @mteb.org · 12/08/2026

New on MTEB: webAI-Official/webAI-ColVec1.1-8b, a visual document retrieval model, takes 1st of 41 on ViDoRe(v3) at 64.95. Its 4B sibling places 3rd at 63.90 — and both beat the lab's previous generation, which sits 6th and 7th.

New model on MTEB: webAI ColVec1.1 — 1st and 3rd on ViDoRe(v3) — both ahead of the lab's own prior generation. Bar chart of the top 10 models on ViDoRe(v3) (10 tasks), out of 41 models with complete results. 1. webAI-Official/webAI-ColVec1.1-8b, 64.95, 8.4B parameters (this release); 2. vultr/VultronRetrieverPrime-Qwen3.5-8B, 64.26, 8.4B parameters; 3. webAI-Official/webAI-ColVec1.1-4b, 63.90, 4.5B parameters (this release); 4. vultr/VultronRetrieverCore-Qwen3.5-4.5B, 63.57, 4.5B parameters; 5. nvidia/nemotron-colembed-vl-8b-v2, 63.42, 8.7B parameters; 6. webAI-Official/webAI-ColVec1-9b, 63.00, 9.4B parameters (same lab, previous generation); 7. webAI-Official/webAI-ColVec1-4b, 62.22, 4.5B parameters (same lab, previous generation); 8. TomoroAI/tomoro-colqwen3-embed-8b, 61.59, 8.0B parameters; 9. nvidia/nemotron-colembed-vl-4b-v2, 61.54, 4.8B parameters; 10. athrael-soju/colqwen3.5-4.5B-v3, 61.46, 4.6B parameters. Both ColVec1.1 sizes outrank the ColVec1 generation they replace. Contributed by zhanlunchang-webai. Source: MTEB, leaderboard.mteb.org.

262

MTEB @mteb.org · 10/08/2026

Most embedding models publish their weights. Far fewer publish what it took to build them. Of 171 models with complete MTEB(Multilingual, v2) results: 153 open-weight, 10 fully open (weights + training code + data), 8 proprietary.

The state of the field: Open weights are the norm. Open data is not. Every model with complete MTEB(Multilingual, v2) results, by how much of it is public. 171 models have complete results on MTEB(Multilingual, v2) (131 tasks). proprietary: 8 models with complete results out of 51 with any results, best Bytedance/Seed1.6-embedding-1215 at 70.26. open weights: 153 models with complete results out of 361 with any results, best microsoft/harrier-oss-v1-27b at 74.27. open source: 10 models with complete results out of 56 with any results, best nvidia/llama-embed-nemotron-8b at 69.46. Source: MTEB, leaderboard.mteb.org.

0100

MTEB @mteb.org · 07/08/2026

New on MTEB: Hanno-Labs/dinghy-law-4b-v1, a legal-domain embedding model, takes 1st on MTEB(Law, v1) at 71.22. Its 0.6B sibling places 4th at 65.83 — the smallest model in the top eight by an order of magnitude.

New model on MTEB: dinghy-law — 1st and 4th on MTEB(Law, v1). Bar chart of the top 8 models on MTEB(Law, v1) (8 tasks), out of 159 models with complete results. 1. Hanno-Labs/dinghy-law-4b-v1, 71.22, 4.0B parameters; 2. Mira190/Euler-Legal-Embedding-V1, 70.37, 7.6B parameters; 3. minetta/nemotron-3-embed-8b-legal, 69.33, 8.0B parameters; 4. Hanno-Labs/dinghy-law-0.6b-v1, 65.83, 0.6B parameters; 5. voyageai/voyage-law-2, 65.39, proprietary; 6. codefuse-ai/F2LLM-v2-14B, 64.66, 14B parameters; 7. voyageai/voyage-3, 64.13, proprietary; 8. infly/inf-retriever-v1, 63.68, 7.1B parameters. Contributed by trashhalo. Source: MTEB, leaderboard.mteb.org.

021

MTEB @mteb.org · 05/08/2026

The best embedding score available at the end of 2023 needed a 304M-parameter model. Today the same score comes from an 11M one — 28x fewer active parameters.

The state of the field: Frontier quality keeps getting cheaper. Best score achievable at a given size, on English. Three Pareto frontiers of score against active parameters on MTEB(eng, v2), for models released by end-2023, by end-2024, and today, across 87 models with complete results. The best score available by end-2023 was 62.32 from sdadas/mmlw-e5-large at 304M active parameters. Today MongoDB/mdbr-leaf-mt matches it at 11M — 27.6 times fewer. Static embedding models appear at zero active parameters, behind an axis break. Source: MTEB, leaderboard.mteb.org.

2152

MTEB @mteb.org · 03/08/2026

Task by task, bekko-embedding-v1-a25m tracks jinaai/jina-embeddings-v3 almost exactly — 58.36 against 58.37 — with 12.7x fewer active parameters.

New model on MTEB: Bekko — Matches jina-embeddings-v3 within 0.01 at 12.7× fewer active parameters. Radar chart over 8 task types on MTEB(Multilingual, v2): Classification (43 tasks), Retrieval (18 tasks), Clustering (16 tasks), STS (16 tasks), BitextMining (13 tasks), PairClassification (11 tasks), Reranking (6 tasks), MultilabelClassification (5 tasks). hotchpotch/bekko-embedding-v1-a25m (25M active, mean 58.36): Classification 57.2, Retrieval 57.5, Clustering 43.0, STS 73.3, BitextMining 75.4, PairClassification 77.8, Reranking 61.5, MultilabelClassification 19.1. jinaai/jina-embeddings-v3 (316M active, mean 58.37): Classification 58.8, Retrieval 55.8, Clustering 45.6, STS 77.1, BitextMining 65.3, PairClassification 79.3, Reranking 57.1, MultilabelClassification 18.4. intfloat/multilingual-e5-large-instruct (304M active, mean 63.22): Classification 64.9, Retrieval 57.1, Clustering 50.8, STS 76.8, BitextMining 80.1, PairClassification 80.9, Reranking 62.6, MultilabelClassification 23.0. codefuse-ai/F2LLM-v2-80M (32M active, mean 55.23): Classification 55.5, Retrieval 50.1, Clustering 47.7, STS 68.5, BitextMining 61.9, PairClassification 75.0, Reranking 59.0, MultilabelClassification 21.5. ibm-granite/granite-embedding-97m-multilingual-r2 (28M active, mean 51.92): Classification 49.6, Retrieval 60.3, Clustering 43.6, STS 65.6, BitextMining 44.2, PairClassification 74.5, Reranking 59.4, MultilabelClassification 17.9. InstructionReranking excluded: its centered metric can be negative and has no position on a zero-based radial axis. Contributed by hotchpotch. Source: MTEB, leaderboard.mteb.org.

020

MTEB @mteb.org · 03/08/2026

New on MTEB: hotchpotch/bekko-embedding-v1-a8m is mmBERT-small with its 22 layers pruned to 4 — active parameters cut from 42M to 7.7M, vocabulary left intact. It scores 56.73 on MTEB(Multilingual, v2), against 47.21 for a static model of the same total size.

New model on MTEB: Bekko — A new Pareto frontier for parameter-efficient multilingual embeddings. Scatter plot of MTEB(Multilingual, v2) against Active parameters for 6 multilingual models ≤ 60M active params. hotchpotch/bekko-embedding-v1-a25m scores 58.36 at 25M active parameters; hotchpotch/bekko-embedding-v1-a8m scores 56.73 at 8M active parameters. Compared against codefuse-ai/F2LLM-v2-80M at 32M, 55.23; ibm-granite/granite-embedding-97m-multilingual-r2 at 28M, 51.92; sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 at 22M, 48.73; sentence-transformers/static-similarity-mrl-multilingual-v1 at 0 (static), 47.21. On the Pareto frontier (nothing is both smaller and higher-scoring): sentence-transformers/static-similarity-mrl-multilingual-v1, hotchpotch/bekko-embedding-v1-a8m, hotchpotch/bekko-embedding-v1-a25m. Contributed by hotchpotch. Source: MTEB, leaderboard.mteb.org.

1141