26 Feb 2024 AI Engineering 5 min read

Gemini 1.5 Pro: Does a Million-Token Context Kill RAG?

Google's Gemini 1.5 Pro accepts up to one million tokens of context. Does that make retrieval-augmented generation obsolete? We run the numbers: per-query cost, latency, multi-needle recall, freshness, and access control. The conclusion from February 2024: long context changes how retrieval pipelines are built — it does not replace them.

One Million Tokens Change the Question

In February 2024 Google announced Gemini 1.5 Pro. The headline number is the context window: up to 1 million tokens in a private preview — enough for roughly 700,000 words, one hour of video, or more than 30,000 lines of code. The largest previously available window was Claude 2.1 at 200,000 tokens. The reaction in many engineering discussions was immediate: if the model can read the entire corpus, why build a retrieval pipeline at all?

The question deserves arithmetic, not sentiment. We at Blue IT Systems build retrieval-augmented systems for production use. Our answer as of February 2024: long context changes how we retrieve. It does not remove the reasons why we retrieve. This article states those reasons with numbers.

A request arrives — the router classifies its difficulty. 1/4

What Gemini 1.5 Pro Actually Ships

Gemini 1.5 Pro is a mid-size Mixture-of-Experts model. Google states that it reaches the quality of Gemini 1.0 Ultra with less training compute. The standard context window is 128,000 tokens. The 1-million-token window is an experimental feature, available in private preview via AI Studio and Vertex AI. In internal research, Google reports successful tests up to 10 million tokens.

Two caveats stand in the announcement itself. First, pricing: tiers starting at the standard 128,000-token window and scaling up to 1 million tokens are announced but not priced. Second, latency: Google tells early testers to expect longer response times and calls the full window computationally intensive. Both caveats decide the RAG question, so we take them seriously.

Needle Tests Are Not Production Workloads

The technical report shows near-perfect single-needle recall: 100% up to 530,000 tokens, above 99.7% at 1 million, and 99.2% at 10 million. That is a real result. The lost-in-the-middle effect that Liu et al. documented in July 2023 — accuracy dropping when relevant text sits mid-context — does not appear in this test.

But a needle test asks for one planted fact. The same report also plants 100 needles in one context: recall falls to about 70% at 128,000 tokens and above 60% at 1 million. GPT-4 Turbo averages around 50% at its 128,000-token limit. Real queries — "list every liability clause across these contracts" — are multi-needle aggregation, not single-fact lookup. A system with 60% recall does not ship unassisted.

The Cost Arithmetic Favors Retrieval

Gemini 1.5 Pro has no published price. We therefore calculate with the closest priced reference: GPT-4 Turbo, announced on November 6, 2023 at $0.01 per 1,000 input tokens — $10 per million. Assume a knowledge base of 700,000 words, roughly 1 million tokens, and 1,000 questions per day against it.

The token volume differs by a factor of 200. No plausible price difference between models closes that gap. If long-context input tokens cost five times less, the full-context approach still spends $2,000 per day where retrieval spends $50. Sending the whole corpus with every request pays for the same reading over and over. Retrieval pays once per index update.

Approach	Input tokens per query	Cost per query	Cost per day (1000 queries)
Full corpus in context	~1,000,000	~$10.00	~$10,000
Retrieval (top-k chunks)	~5,000	~$0.05	~$50

Latency Is a Product Constraint

Attention cost grows with input length, and Google says so plainly: early testers of the 1-million-token window should expect longer latency, and optimizations are ongoing. Interactive products have latency budgets of a few seconds. An assistant that re-reads the entire manual before every answer can be correct and still unusable.

Latency also compounds. Agent-style systems issue several model calls per user task. Five long-context calls in sequence turn seconds into minutes. Retrieval keeps each call small, each step fast, and the end-to-end budget predictable. For batch workloads — analyzing one contract, one codebase, one video — tens of seconds are acceptable. That is exactly where the 1-million-token window fits today.

What Retrieval Does That Context Cannot

Four properties of retrieval sit outside any context window. Freshness: an index is updated incrementally and reflects a document change within minutes; a stuffed prompt is a snapshot that must be rebuilt per request. Access control: different users may see different documents; a retrieval filter enforces permissions per query, a shared mega-prompt cannot.

Attribution: retrieved chunks carry document identity and position, which makes citations and audits possible; an answer distilled from a million undifferentiated tokens does not. Scale: 1 million tokens is roughly 1,500 pages. Corpora we see in practice are gigabytes — thousands of times larger than the window. There, retrieval is not an optimization. It is the only way in.

Long Context and Retrieval Combined

Long context does not replace retrieval. It relaxes retrieval's tightest constraints. With 100,000 or more tokens of headroom, chunking becomes coarse: whole sections or whole documents instead of 300-token fragments. Ranking errors cost less because more candidates fit. Sensitivity to chunk boundaries and to where a fact lands in the prompt drops — the needle results support that.

The architecture we expect to standardize: retrieval as a coarse filter that selects the relevant fraction of a corpus, the long-context model as a thorough reader of that fraction. Cheap where the volume lives, careful where the reasoning lives. That is a simpler and better RAG — not the end of RAG.

Outlook From February 2024

Our predictions from February 2024, stated so they can be checked later. Context windows keep growing; the 10-million-token research result reaches production within two years. Per-token prices fall, but query volumes rise with adoption, so the cost asymmetry between full context and retrieval remains. Providers will ship a mechanism to amortize repeated context — some form of prefix or prompt caching — because the economics demand one.

RAG does not die in 2024. It gets simpler: fewer chunking heuristics, less aggressive reranking, more trust in the model to read what retrieval hands it. The engineering question shifts from "how do we squeeze a corpus into 4,000 tokens" to "which 100,000 tokens deserve the model's attention". We consider that progress.