iiinova — where your digital self keeps evolving

Reposted by Antoine Chaffin

Tom Aarsen @tomaarsen.com · 30/04/2025

ColBERT (a.k.a. multi-vector, late-interaction) models are extremely strong search models, often outperforming dense embedding models. And @lightonai.bsky.social just released a new state-of-the-art one: GTE-ModernColBERT-v1! Details in 🧵

1122

Reposted by Antoine Chaffin

Tom Aarsen @tomaarsen.com · 30/04/2025

I'm a big fan of the PyLate project for ColBERT models, and I'm glad to see these strong models coming out. Very nice work by the @lightonai.bsky.social folks, especially @nohtow.bsky.social. Learn more about PyLate here: lightonai.github.io/pylate/

Antoine Chaffin @nohtow.bsky.social · 30/04/2025

As per usual, thanks to my dear co-maintainer @raphaelsty.bsky.social for helping me make PyLate what it is 🫶

010

Antoine Chaffin @nohtow.bsky.social · 30/04/2025

In addition to knowledge distillation, we recently added features to allow large-scale contrastive pre-training, and this model has been released upon popular demand, but we are currently doing heavier training so stay tuned!

110

Antoine Chaffin @nohtow.bsky.social · 30/04/2025

PyLate makes downstream usage easy, but also facilitate training! You can reproduce this SOTA training with <80 LoC and 2 hours of training and it'll run NanoBEIR during training, report it to W&B and create an informative model card! Link to the gist: gist.github.com/NohTow/3030f...

100

Antoine Chaffin @nohtow.bsky.social · 30/04/2025

Besides, it also comes with the 8k context window of ModernBERT, which is very useful given that late interaction models generalize very well to longer context as highlighted in the ModernBERT paper It is thus very suited to handle your very long documents!

100

Antoine Chaffin @nohtow.bsky.social · 30/04/2025

It is also the first model to outperform ColBERT-small on BEIR While it is bigger, it is still a very lightweight model and benefits from the efficiency of ModernBERT! Also, it has only been trained on MS MARCO (for late interaction) and should thus generalize pretty well!

100

Antoine Chaffin @nohtow.bsky.social · 30/04/2025

Model link: huggingface.co/lightonai/GT... GTE-ModernColBERT is trained on top of the GTE-ColBERT model using knowledge distillation on the MS MARCO dataset and is the first SOTA model trained using PyLate! Get started with PyLate using the documentation: lightonai.github.io/pylate/

huggingface.co

lightonai/GTE-ModernColBERT-v1 · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

100

Antoine Chaffin @nohtow.bsky.social · 30/04/2025

Among all those LLM releases, here is an important retrieval release: To overcome limitations of awesome ModernBERT-based dense models, today @lightonai.bsky.social is releasing GTE-ModernColBERT, the very first state-of-the-art late-interaction (multi-vectors) model trained using PyLate 🚀

160

Reposted by Antoine Chaffin

LightOn @lightonai.bsky.social · 14/01/2025

ModernBERT-embed-large is released under Apache 2.0 and is available on Hugging Face: huggingface.co/lightonai/mo... Congrats to @nohtow.bsky.social for this great work!

huggingface.co

lightonai/modernbert-embed-large · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

0102

Antoine Chaffin @nohtow.bsky.social · 14/01/2025

When I saw the release of ModernBERT-embed during the holidays, I knew I had to build the large variant, so I wanted to thank Zach Nussbaum from Nomic AI for building and sharing it (as well as all the nomic-embed tools and data) and bearing with me during the training!

020

Antoine Chaffin @nohtow.bsky.social · 14/01/2025

ModernBERT-embed-large not only enables usage of ModernBERT-large out-of-the-box, but it should also be a very good starting point for strong fine-tunings on various tasks, so I can't wait to see what the community will build on top of it!

110

Antoine Chaffin @nohtow.bsky.social · 14/01/2025

Obviously, it comes at a slightly higher cost, but it is also trained with Matryoshka capabilities to reduce the footprint of embeddings Notably, the performance with dimension 256 is only slightly worse than the base version with full dimension 768

100

Antoine Chaffin @nohtow.bsky.social · 14/01/2025

Model link: huggingface.co/lightonai/mo... ModernBERT-embed-large is trained using the same (two-stage training) recipe as its smaller sibling and expectedly increases the performance, reaching +1.22 in MTEB average

100

Antoine Chaffin @nohtow.bsky.social · 14/01/2025

ModernBERT-embed-base is awesome because it allows to use ModernBERT-base for various tasks out-of-the-box But the large variant of ModernBERT is also awesome... So today, @lightonai.bsky.social is releasing ModernBERT-embed-large, the larger and more capable iteration of ModernBERT-embed!

1122

Antoine Chaffin @nohtow.bsky.social · 14/01/2025

The multilingual version is not planned yet, there have been a work on adapting ModernBERT to other languages: www.linkedin.com/posts/fremyc... In the mean time, you could have a shot with mGTE (using xformers) or recent language-specific iteration of BERT such as CamemBERTv2!

linkedin.com

François REMY on LinkedIn: Fast and multilingual embedding models are the key to many of the world's…

Fast and multilingual embedding models are the key to many of the world's RAG pipelines. Today marks my first step into the realm of ModernBERT, and certainly…

020

Reposted by Antoine Chaffin

LightOn @lightonai.bsky.social · 19/12/2024

Today, LightOn releases ModernBERT, a SOTA model for retrieval and classification. This work was performed in collaboration with Answer.ai and the model was trained on Orange Business Cloud Avenue infrastructure. www.lighton.ai/lighton-blog...

194

Reposted by Antoine Chaffin

Benjamin Warner @benjaminwarner.dev · 22/12/2024

This week we released ModernBERT, the first encoder to reach SOTA on most common benchmarks across language understanding, retrieval, and code, while running twice as fast as DeBERTaV3 on short context and three times faster than NomicBERT & GTE on long context.

27415

Reposted by Antoine Chaffin

Calvin McCarter @calvinmccarter.bsky.social · 20/12/2024

When one evaluates log-likelihood of a sequence of length L via the chain rule of probability, the first term has missingness fraction of 1, the second has missingness of (L-1)/L, etc. So the inference-time masking rate is ~ Uniform[0, 1].

211

Antoine Chaffin @nohtow.bsky.social · 20/12/2024

Oh, I see! Having also worked a lot on causal models, I never thought of this kind of modelling because I always opposed MLM to open ended generation I guess with papers such as this one arxiv.org/pdf/2406.04823, I should more! Very interesting perspective, thanks!

arxiv.org

010

Antoine Chaffin @nohtow.bsky.social · 20/12/2024

Could you elaborate? Or give me pointers? Is it because having a fixed value bias the learning w.r.t the way we will sample downstream? (Like not mask 30% of the target?)

100

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

But there is definitely some diggings to be made to find an optimal strategy in this regard To me the logic would be to ramp up to have a kick off signal and make it harder and harder but papers seems to say otherwise Maybe random is the optimal solution!

120

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

Not really, we considered ramping up/down the masking ratio, but the findings from the literature (at least what we read at time) seemed counter-intuitive/not consensual We ended up not really digging much into this particular aspect, again because we had so much to explore

120

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

Definitely! Again, the original goal of the project (besides cool models) was to convince some researchers to spend a bit of their GPUs hours on encoders pre-training again! Hopefully we nailed it and will have the answers to a lot of questions in the future!

010

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

As you can see in the paper, we did _a lot_ of ablations (with even more not making it into the paper actually), but there is still so much to be explored, we hope ModernBERT will encourage people exploring these avenues!

110

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

We did not try to go beyond this vocab size because: First, for small encoders, this is a non negligible part of the total params count Second, we discussed training our own tokenizer but at some point lacked time and I am not sure we had discussed existing tokenizer with bigger vocab size

110

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

Thanks! As mentioned in the paper, we tested the tokenizers from OLMo, BERT RoBERTa and Llama They were all pretty much equivalent except Llama that was underperforming. We went for OLMo because of the distribution it was trained on and code tokens

110

Reposted by Antoine Chaffin

Tom Aarsen @tomaarsen.com · 19/12/2024

BERT is BACK! I joined a collaboration with AnswerAI and LightOn to bring you the next iteration of BERT. Introducing ModernBERT: 16x larger sequence length, better downstream performance (classification, retrieval), the fastest & most memory efficient encoder on the market. 🧵

1487

Reposted by Antoine Chaffin

Jeremy Howard @howard.fm · 19/12/2024

I'll get straight to the point. We trained 2 new models. Like BERT, but modern. ModernBERT. Not some hypey GenAI thing, but a proper workhorse model, for retrieval, classification, etc. Real practical stuff. It's much faster, more accurate, longer context, and more useful. 🧵

19620147

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

The project started as a will to bring some light back to the encoders Besides the models, I think we showed that there is still a lot to be done and I hope we succeed at igniting back the interest in encoder pre-training 🔥

010

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

We have conducted many ablations, and many of them did not end up being used in the final models, but we tried to share as many insights as we could in the paper. I hope you will enjoy the read!

110

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

Given that I never pre-trained a model before, there were obviously a lot of challenges and it was not always easy But I am very grateful to everyone involved in this project, I truly learned a lot and I am so proud of the models we managed to build together

110

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

Besides versatility, this also highlighted the potential of PyLate The ModernBERT-base checkpoint achieves 51.3 of BEIR average This means that we beat e5 in a <45 minutes training on MS MARCO only (using only half of the memory of our 8x100)

110

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

Starting with my beloved PyLate We have a lot of experiments on ColBERT models in the paper, with tons of different base models PyLate handled it all, even models using half-baked remote code This was a really cool stress test and I am really happy it went so smoothly

120

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

You can find the models here huggingface.co/answerdotai/... huggingface.co/answerdotai/... If you want all the details, please have a look at the nicely written blog post and the very detailed paper I'll go on with some less general and more personal information

huggingface.co

answerdotai/ModernBERT-base · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

121

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

Efficiency is not only about speed. ModernBERT is also memory-friendly and can handle larger batch sizes than previous encoders, which can come in handy for contrastive learning or running on smaller GPUs (which is an important use case for encoders)

120

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

We also carefully designed the shapes of the models to optimize inference on common hardware Coupled to unpadding through the full processing flash attention, this allows ModernBERT to be two to three times faster than most encoders on long-context on a RTX 4090

121

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

There are a lot of details about the architecture in the BP and the paper, but one key architecture design is the use of alternating attention, i.e, attending to the full input only every few layers This enables fast processing of long sequences while maintaining accuracy

120

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

It is also in a league of its own for code, thanks to its tokenizer and training data, enabling new use cases in the coding assistant domain

120

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

This is impressive because most encoders trade capabilities on some tasks to be better on other ModernBERT, on the other hand, is the first model to beat DeBERTa on GLUE, and does it while beating RoBERTa and GTE on BEIR

120

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

These various improvements coupled to a 2T tokens training results in fast and accurate models that handle long context (and code!) ModernBERT yields state-of-the-art results on various tasks, including IR (short and long context text as well as code) and NLU

131

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

Blog post: huggingface.co/blog/modernb... Paper link: arxiv.org/abs/2412.13663 With @lightonai.bsky.social folks, we collaborated with @answerai.bsky.social (and friends!) to leverage the advances from recent years of work on LLMs (architecture and training) and apply them to a BERT-style model

huggingface.co

Finally, a Replacement for BERT: Introducing ModernBERT

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

143

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

I am thrilled to announce the release of ModernBERT, the long-awaited BERT replacement! There might be a few LLM releases per week, but there is only one drop-in replacement that brings Pareto improvements over the 6 years old BERT while going at lightspeed

3383

Reposted by Antoine Chaffin

Jeremy Howard @howard.fm · 19/12/2024

Seven months ago, @bclavie.bsky.social kicked things off, and soon Benjamin Warner & @nohtow.bsky.social joined him as project co-leads. I don't think anyone quite knew what we were getting in to… It turns out that training a new, SoTA model from scratch is actually pretty hard. Who knew? 🤷

1263

Reposted by Antoine Chaffin

Jeremy Howard @howard.fm · 19/12/2024

This project is made possible by the huge effort from everyone involved, including the project leads Benjamin Warner, @bclavie.bsky.social, & @nohtow.bsky.social. And a big thanks to LightOnIO for contributing human brains AND computer brains!

1111

Reposted by Antoine Chaffin

LightOn @lightonai.bsky.social · 19/12/2024

🔗 Read the full blog article here: www.lighton.ai/lighton-blog... Congratulations to @nohtow.bsky.social, Oskar Hallström, Said Taghadouini, Iacopo Poli and all the folks at Answer.ai that made this model a reality.

052

Antoine Chaffin @nohtow.bsky.social · 26/11/2024

If you are not familiar with PyLate, this is a modular library based on sentence-transformers to train and explore around multi-vectors retrieval models You can find it there: github.com/lightonai/py... As always, thanks to @raphaelsty.bsky.social for developing this with me!

github.com

GitHub - lightonai/pylate: Late Interaction Models Training & Retrieval

Late Interaction Models Training & Retrieval. Contribute to lightonai/pylate development by creating an account on GitHub.

030

Antoine Chaffin @nohtow.bsky.social · 26/11/2024

PyLate v1.1.3 is now live! This update pushes the latest features upstream: - Loading of stanford-nlp models natively - Serving of embeddings using a FastAPI with dynamic batch processing - Trained models now include a model card with information about model and training setup

152