iiinova — where your digital self keeps evolving

Reposted by Benjamin Warner

Ted Underwood @tedunderwood.com · 08/08/2026

initially seems good, but the more you use a tool like this, the more you lose your own ability to forecast cylones

20717113

Benjamin Warner @benjaminwarner.dev · 27/10/2025

Some personal news: I've joined sophont.med to help build the next generation of open medical foundation models. We've relaunched medarc.ai, our open science research community. Join us if you want to help advance open medical AI. And we are hiring.

131

Reposted by Benjamin Warner

mr. TIM @timkellogg.me · 13/09/2025

counterpoint: GPT-5 does this, it says it doesn’t know rather than hallucinate, the world hasn’t fallen apart

3204

Reposted by Benjamin Warner

Tom Aarsen @tomaarsen.com · 09/09/2025

ModernBERT goes MULTILINGUAL! One of the most requested models I've seen, @jhuclsp.bsky.social has trained state-of-the-art massively multilingual encoders using the ModernBERT architecture: mmBERT. Stronger than an existing models at their sizes, while also much faster! Details in 🧵

1146

Benjamin Warner @benjaminwarner.dev · 06/09/2025

ChatGPT has been the best technical search engine since o4-mini. Thinking Mini still makes for a good faster search if you don’t need the extra reasoning ability.

000

Benjamin Warner @benjaminwarner.dev · 24/08/2025

Good LLMs do know and/or can reason about these things. Small, cheap, and often free LLMs are the models which cannot. Remember the glue on pizza Reddit post that the subpar Google AI cited uncritically? Bing’s then integration of GPT 3.5 recognized the Reddit post as sarcasm.

100

Reposted by Benjamin Warner

Sung Kim @sungkim.bsky.social · 24/08/2025

Writing Speed-of-Light Flash Attention for 5090 in CUDA C++ by Thien Tran He walkthrough how he learned to implement Flash Attention for 5090 in CUDA C++. The main objective is to learn writing attention in CUDA C++,

1133

Reposted by Benjamin Warner

Eugene Vinitsky 🍒 @eugenevinitsky.bsky.social · 21/08/2025

Microsoft made a useful LLM copilot tool that could summarize text in spreadsheets. They provided clear instructions about how to use it and not to use it. In response, journalists are now mocking them for doing exactly the right thing and showing how to use and not use the tools.

1111120

Benjamin Warner @benjaminwarner.dev · 18/07/2025

Reports of AI eating entry level jobs are greatly exaggerated. My guess is current and near-future LLMs are more likely to increase the demand for programmers, not decrease demand (Jevons Paradox).

110

Benjamin Warner @benjaminwarner.dev · 10/02/2025

One of the questions we debated while training ModernBERT was whether a modern trained encoder would unlock zero-shot reasoning using only it's generative head? Spoilers: the answer is yes.

$from transformers import pipeline model_name = "answerdotai/ModernBERT-Large-Instruct" fill_mask = pipeline("fill-mask", model=model_name, tokenizer=model_name) text = """You will be given a question and options. Select the right answer. QUESTION: If (G, .) is a group such that (ab)^-1 = a^-1b^-1, for all a, b in G, then G is a/an CHOICES: - A: commutative semi group - B: abelian group - C: non-abelian group - D: None of these ANSWER: [unused0] [MASK]""" results = fill_mask(text) answer = results[0]["token_str"].strip() print(f"Predicted answer: {answer}") # Answer: B$

3286

Reposted by Benjamin Warner

Maria Antoniak @mariaa.bsky.social · 27/01/2025

If you want to quickly catch up on all the open modeling things (DeepSeek, ModernBERT, etc.), this was a great overview, by @natolambert.bsky.social. I somehow got into an argument last week with someone who was insisting that all models are industrial blackboxes... and I wish I'd had this on hand.

interconnects.ai

The latest open artifacts (#6): Reasoning models, China's lead in open-source, and a growing multimodal space

Artifacts log 6 The open LM ecosystem yet again accelerates.

05310

Benjamin Warner @benjaminwarner.dev · 23/01/2025

In addition to being the best retrieval model under 300M params on METB (without extra work), and top 10 for under 1B, here's a fun tidbit from Alibaba's GTE ModernBERT model card: gte-modernbert-base beats gte-qwen1.5-7b on LoCo long context retrieval with 7B less parameters.

130

Reposted by Benjamin Warner

Tom Aarsen @tomaarsen.com · 14/01/2025

The newest extremely strong embedding model based on ModernBERT-base is out: `cde-small-v2`. Both faster and stronger than its predecessor, this one tops the MTEB leaderboard for its tiny size! Details in 🧵

1317

Reposted by Benjamin Warner

Antoine Chaffin @nohtow.bsky.social · 14/01/2025

ModernBERT-embed-base is awesome because it allows to use ModernBERT-base for various tasks out-of-the-box But the large variant of ModernBERT is also awesome... So today, @lightonai.bsky.social is releasing ModernBERT-embed-large, the larger and more capable iteration of ModernBERT-embed!

1122

Benjamin Warner @benjaminwarner.dev · 10/01/2025

ModernBERT is officially released on Transformers v4.48.0. You no longer need to install from git to use. If you are plugging ModernBERT into an existing encoder finetuning pipeline, try increasing the learning rate. We've found that ModernBERT tends to prefer a higher LR than older models.

Transformers v4.48.0: ModernBERT, Aria, TimmWrapper, ColPali, Falcon3, Bamba, VitPose, DinoV2 w/ Registers, Emu3, Cohere v2, TextNet, DiffLlama, PixtralLarge, Moonshine

1113

Benjamin Warner @benjaminwarner.dev · 07/01/2025

The good: 32GB The bad: $2,000 The Ugly*: PCIe 5 without NVLink

100

Reposted by Benjamin Warner

John West @johnwest.bsky.social · 01/01/2025

Via @simonwillison.net's excellent blog, I found this great quote about AI models, from @benjaminwarner.dev et al. www.answer.ai/posts/2024-1... It seems to me that AI will be most relevant in people's lives because the Honda Civic is ubiquitous, not so much because everyone is driving a Ferrari.

Basically, a frontier model like OpenAI’s O1 is like a Ferrari SF-23. It’s an obvious triumph of engineering, designed to win races, and that’s why we talk about it. But it takes a special pit crew just to change the tires and you can’t buy one for yourself. In contrast, a BERT model is like a Honda Civic. It’s also an engineering triumph, but more subtly, since it is engineered to be affordable, fuel-efficient, reliable, and extremely useful. And that’s why they’re absolutely everywhere.

121

Reposted by Benjamin Warner

Tom Aarsen @tomaarsen.com · 31/12/2024

That didn't take long! Nomic AI has finetuned the new ModernBERT-base encoder model into a strong embedding model for search, classification, clustering and more! Details in 🧵

23710

Benjamin Warner @benjaminwarner.dev · 22/12/2024

This week we released ModernBERT, the first encoder to reach SOTA on most common benchmarks across language understanding, retrieval, and code, while running twice as fast as DeBERTaV3 on short context and three times faster than NomicBERT & GTE on long context.

27415

Reposted by Benjamin Warner

Mark J. Nelson @mm-jj-nn.bsky.social · 19/12/2024

Great blog post (by a 15-author team!) on their release of ModernBERT, the continuing relevance of encoder-only models, and how they relate to, say, GPT-4/llama. Accessible enough that I might use this as an undergrad reading.

huggingface.co

Finally, a Replacement for BERT: Introducing ModernBERT

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

17519