Hybrid Search: BM25 Is Not Dead
Pure vector search fails on part numbers, error codes, and other exact identifiers. We explain why, define BM25 and dense retrieval precisely, show how Reciprocal Rank Fusion combines both without score normalization, and review measured gains from hybrid retrieval and early rerankers such as Cohere Rerank — with numbers from 2023 benchmarks.
Why pure vector search misses exact identifiers
In 2023 one retrieval stack became the default for RAG applications: split documents into chunks, embed each chunk with a model such as OpenAI's text-embedding-ada-002, index the vectors in an ANN structure, retrieve by cosine similarity. This works well for paraphrased questions. It fails predictably on queries users type every day: part numbers, error codes, invoice IDs, function names, references to standards.
The failure is structural, not a tuning problem. An embedding compresses a chunk into a vector of fixed length — 1,536 dimensions for ada-002. Rare identifiers such as DIN 4108-2 or a specific SKU contribute almost nothing to that vector. Microsoft quantified this in September 2023: on keyword-style queries, pure vector retrieval scored 11.7 NDCG@3 while plain keyword search scored 79.2. The exact match the user asked for is simply not represented.
What BM25 still does better
BM25 is a ranking function from the Okapi system, first evaluated at TREC-3 in 1994. It scores a document by term frequency, inverse document frequency and document length, controlled by two parameters, k1 and b. It matches exact tokens, its scores are explainable, it needs neither training data nor a GPU, and the inverted index is cheap to build and to update.
It is also hard to beat out of domain. The BEIR benchmark (Thakur et al., 2021) compared retrieval systems zero-shot across 18 datasets and found BM25 to be a strong baseline that many dense retrievers trained on MS MARCO failed to beat on unfamiliar corpora. What BM25 does not do: it has no notion of synonyms or paraphrase. "Kündigungsfrist" and "notice period" share no tokens. That gap is real — and it is exactly the gap embeddings close.
What dense embeddings add
Dense retrieval maps queries and documents into the same vector space with a bi-encoder. Relevance becomes geometric proximity. This closes the vocabulary gap: paraphrases, synonyms and — with multilingual models — queries across language boundaries find the right passages without shared tokens. Since December 2022, text-embedding-ada-002 has been the pragmatic default; in 2023 open models such as E5, GTE and BGE reached comparable quality on the MTEB benchmark.
The limits are equally concrete. Embeddings degrade on domains far from their training data. Scores are cosine similarities without a calibrated meaning. And exact identifiers remain a blind spot, because the encoder was never trained to preserve them. Dense retrieval complements lexical search. It does not replace it.
Reciprocal Rank Fusion in one formula
Reciprocal Rank Fusion was published by Cormack, Clarke and Büttcher at SIGIR 2009 as a two-page paper. The rule: RRF-score(d) = Σ 1/(k + r(d)), summed over all result lists, where r(d) is the document's rank in a list and k = 60. In their TREC experiments this simple rule beat Condorcet Fuse and the best individual system by 4 to 5 percent.
The decisive property: RRF consumes ranks, not scores. BM25 scores are unbounded; cosine similarities live in [-1, 1]. RRF never has to normalize either. It requires no training and no tuning; the paper found k = 60 near-optimal but not critical. By late 2023 it is productized: Elasticsearch 8.8 (May 2023) ships RRF in the search API, Weaviate has offered hybrid queries since v1.17 (December 2022), and Azure Cognitive Search fuses keyword and vector results with RRF.
What RRF does not do: it ignores score magnitudes, so a confident top hit and a marginal one count the same at the same rank. A weak retriever in the mix dilutes a strong one. Fusion window size and any per-list weighting remain design decisions.
Measured gains from hybrid retrieval
Microsoft published the most cited hybrid numbers of the year in September 2023, measured on Azure Cognitive Search with ada-002 vectors, 512-token chunks and RRF fusion. The pattern matters more than the absolute values: every stage adds relevance, and each stage adds it for a different reason.
Hybrid beats both single-method configurations on every benchmark in the study. The reranker delivers the largest single step on customer data. One caveat is due: this is a vendor benchmark using the vendor's own reranker. But the direction matches what we at Blue IT Systems measure in client projects: fusing lexical and dense retrieval is the cheapest relevance win available — it costs one extra query against an index you probably already operate.
| Configuration | Customer datasets (NDCG@3) | BEIR (NDCG@10) |
|---|---|---|
| Keyword (BM25) | 40.6 | 40.6 |
| Vector (ada-002) | 43.8 | 45.0 |
| Hybrid (RRF) | 48.4 | 48.4 |
| Hybrid + semantic reranker | 60.1 | 50.0 |
Rerankers as a second stage
A reranker is a cross-encoder: it reads query and candidate document together and outputs one relevance score. That joint attention is more accurate than any bi-encoder comparison — and far too slow to run over a full corpus. Hence the two-stage pattern: retrieval (BM25, dense or hybrid) selects 50 to 100 candidates; the reranker reorders them.
In 2023 this became an API call. Cohere launched Rerank on May 1, 2023, with rerank-english-v2.0 and rerank-multilingual-v2.0. In Cohere's own evaluation, lexical search placed a relevant result in the top 3 for about 44% of queries, embedding-based search for 65%, and reranking for 72%. The trade-offs are plain: added latency per query, cost per call, and a hard ceiling — a reranker cannot rescue a document the first stage never retrieved. Recall is decided before it runs.
Outlook from December 2023
Writing in December 2023, we expect three developments. First, hybrid retrieval moves from expert feature to default; search engines already ship fusion as a one-line query parameter. Second, learned sparse models such as SPLADE and Elastic's ELSER blur the boundary — they produce term weights like BM25 but learn expansion like an embedding model. Third, rerankers get smaller and cheaper and will increasingly run self-hosted next to the index.
Our prediction: BM25 will still sit in production retrieval pipelines in five years, next to whatever embedding model is current then. Rank fusion of heterogeneous retrievers is a more durable idea than any single model. If you build RAG today, keep the inverted index — and measure every stage on your own queries, not on someone else's benchmark.
Sources
- Cormack, Clarke & Büttcher: Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods (SIGIR, July 2009)
- Thakur et al.: BEIR — A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models (arXiv, April 2021)
- Cohere: Say Goodbye to Irrelevant Search Results — Cohere Rerank Is Here (May 1, 2023)
- Elastic: Elasticsearch 8.8 — ELSER and hybrid scoring with Reciprocal Rank Fusion (May 25, 2023)
- Microsoft: Azure Cognitive Search — Outperforming vector search with hybrid retrieval and ranking capabilities (September 18, 2023)