iiinova — where your digital self keeps evolving

Reposted by Antoine Chaffin

Tom Aarsen @tomaarsen.com · 30/04/2025

ColBERT (a.k.a. multi-vector, late-interaction) models are extremely strong search models, often outperforming dense embedding models. And @lightonai.bsky.social just released a new state-of-the-art one: GTE-ModernColBERT-v1! Details in 🧵

1122

Reposted by Antoine Chaffin

Tom Aarsen @tomaarsen.com · 30/04/2025

I'm a big fan of the PyLate project for ColBERT models, and I'm glad to see these strong models coming out. Very nice work by the @lightonai.bsky.social folks, especially @nohtow.bsky.social. Learn more about PyLate here: lightonai.github.io/pylate/

Antoine Chaffin @nohtow.bsky.social · 30/04/2025

Among all those LLM releases, here is an important retrieval release: To overcome limitations of awesome ModernBERT-based dense models, today @lightonai.bsky.social is releasing GTE-ModernColBERT, the very first state-of-the-art late-interaction (multi-vectors) model trained using PyLate 🚀

160

Reposted by Antoine Chaffin

LightOn @lightonai.bsky.social · 14/01/2025

ModernBERT-embed-large is released under Apache 2.0 and is available on Hugging Face: huggingface.co/lightonai/mo... Congrats to @nohtow.bsky.social for this great work!

huggingface.co

lightonai/modernbert-embed-large · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

0102

Antoine Chaffin @nohtow.bsky.social · 14/01/2025

ModernBERT-embed-base is awesome because it allows to use ModernBERT-base for various tasks out-of-the-box But the large variant of ModernBERT is also awesome... So today, @lightonai.bsky.social is releasing ModernBERT-embed-large, the larger and more capable iteration of ModernBERT-embed!

1122

Reposted by Antoine Chaffin

LightOn @lightonai.bsky.social · 19/12/2024

Today, LightOn releases ModernBERT, a SOTA model for retrieval and classification. This work was performed in collaboration with Answer.ai and the model was trained on Orange Business Cloud Avenue infrastructure. www.lighton.ai/lighton-blog...

194

Reposted by Antoine Chaffin

Benjamin Warner @benjaminwarner.dev · 22/12/2024

This week we released ModernBERT, the first encoder to reach SOTA on most common benchmarks across language understanding, retrieval, and code, while running twice as fast as DeBERTaV3 on short context and three times faster than NomicBERT & GTE on long context.

27415

Reposted by Antoine Chaffin

Calvin McCarter @calvinmccarter.bsky.social · 20/12/2024

When one evaluates log-likelihood of a sequence of length L via the chain rule of probability, the first term has missingness fraction of 1, the second has missingness of (L-1)/L, etc. So the inference-time masking rate is ~ Uniform[0, 1].

211

Reposted by Antoine Chaffin

Tom Aarsen @tomaarsen.com · 19/12/2024

BERT is BACK! I joined a collaboration with AnswerAI and LightOn to bring you the next iteration of BERT. Introducing ModernBERT: 16x larger sequence length, better downstream performance (classification, retrieval), the fastest & most memory efficient encoder on the market. 🧵

1487

Reposted by Antoine Chaffin

Jeremy Howard @howard.fm · 19/12/2024

I'll get straight to the point. We trained 2 new models. Like BERT, but modern. ModernBERT. Not some hypey GenAI thing, but a proper workhorse model, for retrieval, classification, etc. Real practical stuff. It's much faster, more accurate, longer context, and more useful. 🧵

19620147

Antoine Chaffin @nohtow.bsky.social · 19/12/2024

I am thrilled to announce the release of ModernBERT, the long-awaited BERT replacement! There might be a few LLM releases per week, but there is only one drop-in replacement that brings Pareto improvements over the 6 years old BERT while going at lightspeed

3383

Reposted by Antoine Chaffin

Jeremy Howard @howard.fm · 19/12/2024

Seven months ago, @bclavie.bsky.social kicked things off, and soon Benjamin Warner & @nohtow.bsky.social joined him as project co-leads. I don't think anyone quite knew what we were getting in to… It turns out that training a new, SoTA model from scratch is actually pretty hard. Who knew? 🤷

1263

Reposted by Antoine Chaffin

Jeremy Howard @howard.fm · 19/12/2024

This project is made possible by the huge effort from everyone involved, including the project leads Benjamin Warner, @bclavie.bsky.social, & @nohtow.bsky.social. And a big thanks to LightOnIO for contributing human brains AND computer brains!

1111

Reposted by Antoine Chaffin

LightOn @lightonai.bsky.social · 19/12/2024

🔗 Read the full blog article here: www.lighton.ai/lighton-blog... Congratulations to @nohtow.bsky.social, Oskar Hallström, Said Taghadouini, Iacopo Poli and all the folks at Answer.ai that made this model a reality.

052

Antoine Chaffin @nohtow.bsky.social · 26/11/2024

PyLate v1.1.3 is now live! This update pushes the latest features upstream: - Loading of stanford-nlp models natively - Serving of embeddings using a FastAPI with dynamic batch processing - Trained models now include a model card with information about model and training setup

152