The 2025 Retrieval Baseline: Hybrid Search Plus Rerankers
A field guide to the retrieval stack that became the 2025 default: BM25 and dense vectors merged with reciprocal rank fusion, then reordered by a cross-encoder reranker. We cover the measured lifts from Anthropic and Cohere, a realistic latency budget, three production rerankers, and the query classes this baseline still cannot answer.
Why One Retriever Is Not Enough
Retrieval-augmented generation stands or falls with its retriever, and every single retriever has a blind spot. BM25 matches exact tokens. It finds error codes, part numbers and legal citations, but it does not see that "terminate an agreement" and "cancel a contract" mean the same thing. Dense embeddings capture exactly that paraphrase, yet they blur rare identifiers and domain jargon into fuzzy neighborhoods. The failure modes are complementary.
The BEIR benchmark made this measurable. Thakur et al. (2021) evaluated retrievers zero-shot across 18 datasets: most dense models fell behind BM25 outside their training domain, while BM25 followed by a cross-encoder ranked best on average. That result quietly defined the architecture the industry now treats as a baseline.
Anatomy of the 2025 Baseline
The 2025 baseline has four stages. One: BM25 retrieves the top 50 to 200 chunks. Two: an approximate-nearest-neighbor search over dense embeddings retrieves its own top 50 to 200. Three: reciprocal rank fusion merges both lists into one. Four: a cross-encoder reranker scores the fused top 50 to 100 candidates against the query and keeps the best 5 to 20 for the prompt.
The division of labor is deliberate. The first three stages optimize recall and are cheap. The reranker optimizes precision and is expensive, so it only sees a short candidate list. Note what the design does not do: a reranker cannot rescue a document the first stage never retrieved. Recall errors are unrecoverable downstream.
Reciprocal Rank Fusion Does the Merging
RRF assigns each document the sum of 1/(k + rank) over all input rankings, with k = 60. Cormack, Clarke and Büttcher published it as a two-page SIGIR paper in 2009. In their TREC experiments the fused ranking beat Condorcet fusion, CombMNZ and the best individual system by 4 to 5 percent on average. It needs no training data and no score calibration — decisive, because BM25 scores and cosine similarities live on incompatible scales.
RRF has known limits. It uses ranks only, so a barely-first document counts exactly like a confidently-first one. The paper itself reports that the choice of k is not critical. Weighted variants let you favor one retriever, but the unweighted form remains the default because it has nothing to tune.
Three Rerankers Worth Knowing
A cross-encoder reads query and document together through one transformer and outputs a relevance score. That joint attention is why it outranks bi-encoders — and why its scores cannot be precomputed. Three models dominate our project work in mid-2025; all three are interchangeable at the same pipeline position.
Cohere Rerank 3.5 covers more than 100 languages and handles multi-constraint queries well; the price is an API dependency with undisclosed weights. bge-reranker-v2-m3 is the standard self-hosted choice under Apache 2.0. Jina's April 2025 release of jina-reranker-m0, a 2.4B multimodal model that ranks page images across 29 languages, shows where the category is heading.
| Model | Released | Parameters | Max context | Access |
|---|---|---|---|---|
| Cohere Rerank 3.5 | Dec 2024 | not disclosed | 4,096 tokens | API (Cohere, Amazon Bedrock) |
| jina-reranker-v2-base-multilingual | Jun 2024 | 278M | 8,192 tokens | API; open weights (non-commercial) |
| BAAI bge-reranker-v2-m3 | Mar 2024 | 568M | 8,192 tokens (fine-tuned to 1,024) | Self-hosted, Apache 2.0 |
What the Measurements Show
Anthropic's contextual retrieval study from September 2024 is the cleanest public measurement of the stack. Hybrid retrieval — contextual embeddings plus contextual BM25 — cut the top-20 retrieval failure rate from 5.7 to 2.9 percent, a 49 percent reduction. Adding a reranker cut it to 1.9 percent, 67 percent below baseline. Each stage stacked on the previous one.
Cohere's internal benchmarks from December 2024 report Rerank 3.5 at 23.4 percent higher nDCG@10 than hybrid search and 30.8 percent higher than BM25 alone on financial-services datasets. Treat vendor numbers as upper bounds on foreign domains. The only lift that matters is the one you measure on your own labeled query set.
Budgeting the Latency
Lexical search and ANN search each return in single- to low-double-digit milliseconds on corpora of a few million chunks; RRF is arithmetic. The reranker dominates the budget. Anthropic's cookbook puts reranking at roughly 100 to 200 milliseconds per query, depending on candidate count. A realistic end-to-end retrieval budget is 150 to 300 milliseconds — small next to the seconds an LLM spends generating.
Cost scales linearly with candidates, so the main lever is how many documents you rerank. Scoring 100 instead of 50 doubles latency for a recall gain that flattens quickly. Truncating documents helps: bge-reranker-v2-m3 was fine-tuned at 1,024 tokens and BAAI recommends capping input there. Self-hosting on a GPU removes the API round-trip but adds operational work.
When the Baseline Is Not Enough
The baseline retrieves passages that resemble the query. Whole query classes fall outside that contract. Aggregations — how many contracts expire this quarter — are questions about the corpus, not about any passage; no top-k answers them. Multi-hop questions need evidence chained across documents that a single query never co-retrieves. Comparisons and trend questions scatter their answer over dozens of chunks.
The remedies sit beside the baseline, not inside it. Contextual chunk enrichment attacks ambiguity at indexing time. Query decomposition and iterative retrieval let a model issue several targeted queries. Aggregation belongs in structured stores you query with SQL, and metadata filters beat semantic similarity wherever a hard constraint exists. The baseline is a floor, not an architecture.
Outlook From July 2025
Three developments look likely from where we stand in July 2025. Rerankers are going multimodal: jina-reranker-m0 already ranks page images alongside text, and we expect API rerankers to follow within a year. Listwise reranking with small LLMs is getting cheap enough to compete with cross-encoders on quality-critical paths. And fusion plus reranking may collapse into a single learned stage.
The larger shift is agentic. Models increasingly issue their own queries, inspect results and refine — retrieval becomes a loop, not a stage. That does not retire the hybrid baseline; it multiplies how often it runs, which makes the latency budget stricter, not looser. Growing context windows change how much you retrieve, not whether. We at Blue IT Systems are building on that assumption.
Sources
- Cormack, Clarke, Büttcher: Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods (SIGIR, 19 Jul 2009)
- Thakur et al.: BEIR — A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models (arXiv, 17 Apr 2021)
- Anthropic: Introducing Contextual Retrieval (19 Sep 2024)
- Cohere: Introducing Rerank 3.5 — Precise AI Search (2 Dec 2024)
- BAAI: bge-reranker-v2-m3 model card (Hugging Face, Mar 2024)
- Jina AI: jina-reranker-m0 — Multilingual Multimodal Document Reranker (8 Apr 2025)