Ben Trent @benwtrent.bsky.social · 01/10/2025The @hf.co community is awesome. Real work that moves everyone forward: huggingface.co/blog/rtebhuggingface.coIntroducing RTEB: A New Standard for Retrieval EvaluationWe’re on a journey to advance and democratize artificial intelligence through open source and open science. 021
Ben Trent @benwtrent.bsky.social · 19/09/2025Apache Lucene 10.3.0 is released! 40% faster lexical search is absolutely crazy for a project that has been doing lexical search for a quarter of a century lucene.apache.org/core/corenew...lucene.apache.orgLucene™ Core NewsApache Lucene is a high-performance, full-featured search engine library written entirely in Java. It is a technology suitable for... 030
Ben Trent @benwtrent.bsky.social · 27/08/2025Storing floating point values as a big 'ole JSON blob is silly, so we stopped doing that. Great stuff from Jim on making vector search in Elasticsearch substantially cheaper! www.elastic.co/search-labs/...elastic.coElasticsearch vector search: Excluding vectors from source - Elasticsearch LabsElasticsearch now excludes vectors from source by default, saving space and improving performance while keeping vectors accessible when needed. 030
Reposted by Ben TrentDoug Turnbull @softwaredoug.bsky.social · 21/05/2025Next in the series of building a search engine from scratch - we focus on hybrid retrieval with @benwtrent.bsky.socialof Elastic. How do you add filtering to a vector search index? I'll code. He'll yell at me. maven.com/p/430592/hyb...maven.comHybrid search live coded from scratchRAG systems all use vector databases. HNSW (Hierarchical Navigable Small Worlds) is the most common algorithm. If you want to build RAG, you should appreciate how this algorithm works (Missed previous... 021
Reposted by Ben TrentAdrien Grand @jpountz.bsky.social · 12/04/2025It's time to redo benchmarks! #Lucene 10.2 was just released, with - huge speedups to non-scoring boolean queries, range queries and filtered vector search, - better merging defaults for faster search, - much faster merging of vectors And more... lucene.apache.org/core/corenew...lucene.apache.orgLucene™ Core NewsApache Lucene is a high-performance, full-featured search engine library written entirely in Java. It is a technology suitable for... 161
Ben Trent @benwtrent.bsky.social · 08/04/2025Indexing and merging times are getting better for #Apache #Lucene vector search. Lucene has a read-only segment architecture. One of the drawbacks of this approach is throwing away previously completed work when merging HNSW graphs. Well, this got better :) 121
Ben Trent @benwtrent.bsky.social · 28/02/2025Filtered vector search is crazy important. So we made HNSW filtered search in Apache Lucene better. At similar recall, it can be 3-5x faster! 151
Reposted by Ben TrentPhilipp Krenn @xeraa.net · 13/02/2025"elasticsearch: 15 years of indexing it all, finding what matters": www.elastic.co/search-labs/... we turned it into a proper blog post with shay :)elastic.coElasticsearch history: 15 years of indexing and searching - Elasticsearch LabsElasticsearch just turned 15-years-old! Take a look back at the last 15 years of indexing and searching, and turn to the next 15 years of relevance. 031
Ben Trent @benwtrent.bsky.social · 13/02/2025I really enjoyed this talk by @elasticmark.bsky.social. He is back at finding crazy & interesting ways to explore data (I guess he never stopped). Clustering with binary vectors & vector search with Elasticsearch www.youtube.com/watch?v=sJU_...youtube.comBinary Vectors & Fuzzy Facets: Clustering Results in a Browser Using Binary VectorsYouTube video by Official Elastic Community 020
Ben Trent @benwtrent.bsky.social · 07/02/2025It's wonderful to see practical & important programming work. Debugging concurrent programs is incredibly difficult, here is a bug found in Apache Lucene by the CMU Pasta Lab using their new Fray testing framework www.elastic.co/search-labs/...elastic.coConcurrency bugs in Lucene: How to fix optimistic concurrency failures - Elasticsearch LabsThanks to Fray, a deterministic concurrency testing framework from CMU’s PASTA Lab, we tracked down a tricky Lucene bug and squashed it 121
Ben Trent @benwtrent.bsky.social · 15/01/2025The number of improvements in Lucene here are crazy. Pretty much every count and boolean query gets a nice boost and some of the count improvements are hilarious 🚀🚀🚀. 150
Ben Trent @benwtrent.bsky.social · 10/01/2025It's so cool to see #Apache #Lucene going strong after about a quarter of a century 🤯. 2025 is gonna be a fun year for Lucene. www.elastic.co/search-labs/...elastic.coLucene Wrapped 2024 - Elasticsearch Labs2024 has been another major year for Apache Lucene. In this blog, we’ll explore the key highlights. 030
Ben Trent @benwtrent.bsky.social · 07/01/2025Early termination for vector search can be more than just "gathering K candidates" my colleague Tommaso gives a small overview of basic early termination strategies for vector index search. www.elastic.co/search-labs/...elastic.coEarly termination in HNSW for faster approximate KNN search - Elasticsearch LabsLearn how HNSW can be made faster for KNN search, using smart early termination strategies. 030
Ben Trent @benwtrent.bsky.social · 06/01/2025My team wrote a new backing algorithm for our BBQ indices, called Optimized Scalar Quantization. Here is a high level overview of its implementation in Elasticsearch (and soon Apache Lucene). www.elastic.co/search-labs/... for the math nerds, skip to Tom's blog: www.elastic.co/search-labs/...elastic.coOptimized Scalar Quantization: Even Better Binary Quantization - Elasticsearch LabsHere we explain optimized scalar quantization in Elasticsearch and how we used it to improve Better Binary Quantization (BBQ). 021
Reposted by Ben TrentAdrien Grand @jpountz.bsky.social · 21/12/2024Lucene has been evaluating disjunctive queries by loading (windows of) postings into a bit set and or-ing these bit sets for 20+ years. It started using the same approach for conjunctive queries a few days ago. benchmarks.mikemccandless.com/CountAndHigh... (annotation HS)benchmarks.mikemccandless.comLucene CountAndHighHigh queries/sec 121
Ben Trent @benwtrent.bsky.social · 27/12/2024Something a little different from my typical blogs. This line of code in Apache Lucene took me 3 days to write. For fixing bugs, it's about the journey, not necessarily the destination. www.elastic.co/search-labs/... (the cover art was provided by one of my kids :))elastic.coLucene bug adventures: Fixing a corrupted index exception - Elasticsearch LabsSometimes, a single line of code takes days to write. Here, we get a glimpse of an engineer's pain and debugging over multiple days to fix a potential Apache Lucene index corruption. 141
Ben Trent @benwtrent.bsky.social · 20/12/2024Our Better Binary Quantization (BBQ) index in Elasticsearch has a new backing algorithm. Better(er) recall & query speed for vector search. Its a natural evolution of our scalar quantization. Shipping soon. It's pretty neat www.elastic.co/search-labs/...elastic.coUnderstanding optimized scalar quantization - Elasticsearch LabsIn this post we explain a new form of scalar quantization we've developed at Elastic that achieves state-of-the-art accuracy for binary quantization 031
Ben Trent @benwtrent.bsky.social · 19/12/2024Elasticsearch just got more powerful. Now, semantic, hybrid, and vector retrieval with custom rules for pinning and bubbling results to the top! Now you have multi-phased, hybrid retrieval in combination with business rules :D www.elastic.co/search-labs/...elastic.coEnsuring business rules work seamlessly with semantic search - Elasticsearch LabsHarness the power of query rules combined with semantic search and rerankers. 010
Ben Trent @benwtrent.bsky.social · 11/12/2024It was so much fun talking #Elasticsearch with Steve Mayzak on “You Know, For Search”. I could nerd out for hours, but we kept it down to just 1 hour (maybe even that is too long....). Give it a listen, if nothing else, for Steve's dulcet tones: open.spotify.com/episode/7HLH...open.spotify.comQuantization: The Important BitsYou know, for search, an Elastic podcast · Episode 023
Ben Trent @benwtrent.bsky.social · 05/12/2024Be prepared to learn more about semantic rerankers than you ever thought you needed to know. Another awesome analysis from my colleagues at Elasticsearch www.elastic.co/search-labs/...elastic.coExploring depth in a 'retrieve-and-rerank' pipeline - Elasticsearch LabsSelect an optimal re-ranking depth for your model and dataset. 070
Ben Trent @benwtrent.bsky.social · 04/12/2024More magic from chef Chris Hegarty. How better binary quantization vector ops are accelerated with Java SIMD in Elasticsearch vector search www.elastic.co/search-labs/...elastic.coSmokin' fast BBQ with hardware accelerated SIMD instructions - Search LabsHow we optimized vector comparisons in BBQ with hardware accelerated SIMD (Single Instruction Multiple Data) instructions. 040
Ben Trent @benwtrent.bsky.social · 02/12/2024I cannot adequately express how proud I am of the #Elasticsearch team for delivering this. It is a humungous engineering achievement and the results of (metaphorical) blood, sweat, and (maybe real ;) ) tears. go.es.io/3CVo82Xgo.es.ioDo less with serverless: Elastic Cloud Serverless — Now GAElastic Cloud Serverless is the easiest way to start and scale your capabilities in search, observability and security. Built on a reimagined Elasticsearch architecture, it ensures low-latency queryin... 020
Ben Trent @benwtrent.bsky.social · 26/11/2024We have seen this idea played out nicely with Tantivy and Apache Lucene. Benchmarking between each other and lovingly borrowing ideas between the projects. 032
Ben Trent @benwtrent.bsky.social · 25/11/2024I work with some pretty smart folks. We are shipping a new cross-encoder reranking models. Some pretty great numbers at a moderately sized 183M params. www.elastic.co/search-labs/...elastic.coIntroducing Elastic Rerank: Elastic's new semantic re-ranker model - Search LabsLearn about how Elastic's new re-ranker model was trained and how it performs. 030
Ben Trent @benwtrent.bsky.social · 22/11/2024Honestly, Grant Sanderson (3blue1brown) is one of (if not the best) mathematics communicator of my generation. If you want an overview of how LLMs work at a deep & intuitive level, I whole heartedly recommend this lecture: youtu.be/KJtZARuO3JY?...youtu.beVisualizing transformers and attention | Talk for TNG Big Tech Day '24YouTube video by Grant Sanderson 030
Ben Trent @benwtrent.bsky.social · 20/11/2024Here is a talk I gave on Better Binary Quantization (BBQ) for vector search in Elasticsearch. For those that don't like reading blogs, or just want to know how bad the puns can really get, this talk is for you: www.youtube.com/watch?v=04Nz...youtube.comBetter Binary Quantization (BBQ): From Bytes to BBQ, The Secret to Better Vector Search by Ben TrentYouTube video by Official Elastic Community 051
Ben Trent @benwtrent.bsky.social · 18/11/2024Better Binary Quantization (BBQ) is now in 8.16. When it comes to index time, query time, and recall, we found it way better than the old faithful Product Quantization (PQ) in Elasticsearch vector search. Here is how we compared them when developing BBQ: www.elastic.co/search-labs/...elastic.coBetter Binary Quantization vs. Product Quantization - Search LabsWhy we chose to spend time working on better binary quantization instead of production quantization in Lucene and Elasticsearch. 092
Ben Trent @benwtrent.bsky.social · 14/11/2024Did you know Elasticsearch added the "retrievers" framework back in 8.12? As a further extension of _search, it gives extra flexibility that our traditional "query" API does not. Like RRF, global reranking, and multi-phased retrieval. It's now GA: elastic.co/search-labs/... [🧵 begin]elastic.coElasticsearch retrievers are generally available with Elasticsearch 8.16.0! - Search LabsElasticsearch retrievers have gone through a significant revamp and are now generally available for all to use. Learn all about their architecture and use-cases. 160
Ben Trent @benwtrent.bsky.social · 11/11/2024We have shipped Better Binary Quantization for vectorsearch in Elasticsearch. Product Quantization is sooo slow. BBQ is 30x faster to quantize, >2x faster queries, and you still get the 95+% reduction in memory requirements. Here is an overview of how it works: www.elastic.co/search-labs/...elastic.coBetter Binary Quantization (BBQ) in Lucene and Elasticsearch - Search LabsHow Better Binary Quantization works in Lucene and Elasticsearch. 030
Ben Trent @benwtrent.bsky.social · 08/11/2023Fast HNSW and 1/4 the memory cost? Yes please! New int8 quantization in Apache Lucene!elastic.coIntroducing Scalar Quantization in Lucene — Elastic Search LabsHow did we introduce scalar quantization into Lucene 030
