The Economics of Prompt Caching
Anthropic's prompt caching beta changes the unit economics of long prompts: cache writes cost 25 percent more than base input tokens, cache reads 90 percent less. We walk through the pricing, the break-even arithmetic, the limits of the five-minute TTL, and what the numbers mean for RAG pipelines and agent loops.
The Cost of Repeating Yourself
Large language model APIs are stateless. Every request must carry its full context: system prompt, tool definitions, few-shot examples, reference documents. The provider bills all of it at the full input rate on every single call — even when 95 percent of the prompt is identical to the previous request. For applications with long static prompts, this repetition dominates the bill.
The numbers are concrete. A 100,000-token context on Claude 3.5 Sonnet costs $0.30 per request at the base input price of $3 per million tokens. At 1,000 requests per day, that is $300 per day for tokens the model has processed hundreds of times before. Until now there was no way to tell the API: this part is already known.
What Anthropic Shipped
In August 2024, Anthropic released prompt caching as a public beta on the Anthropic API, initially for Claude 3.5 Sonnet and Claude 3 Haiku, with support for Claude 3 Opus announced to follow. The feature is enabled per request via the HTTP header anthropic-beta: prompt-caching-2024-07-31 and cache_control markers on individual content blocks.
The mechanics are strictly prefix-based. The cacheable prefix spans tools, system prompt, and messages, in exactly that order, up to a cache breakpoint; up to four breakpoints are allowed per request. Cached segments live for five minutes, and every cache hit resets that timer. Minimum cacheable length is 1,024 tokens on Claude 3.5 Sonnet and 2,048 tokens on Claude 3 Haiku.
Write and Read Pricing
Prompt caching introduces two new token prices. Writing a prefix into the cache costs 25 percent more than the base input rate. Reading it back costs 10 percent of the base input rate — a 90 percent discount. Output tokens are unaffected. Anthropic states cost reductions of up to 90 percent and latency reductions of up to 85 percent for long prompts.
The structure is deliberate. The 25 percent premium prices the storing of the prefix; the 90 percent discount prices its reuse. There is no separate storage fee and no capacity charge — access frequency alone determines whether caching pays. That makes the model easy to reason about, and easy to get wrong for low-traffic endpoints.
| Model | Base input ($/MTok) | Cache write ($/MTok) | Cache read ($/MTok) |
|---|---|---|---|
| Claude 3.5 Sonnet | 3.00 | 3.75 | 0.30 |
| Claude 3 Haiku | 0.25 | 0.30 | 0.03 |
| Claude 3 Opus (announced) | 15.00 | 18.75 | 1.50 |
The Break-Even Arithmetic
The arithmetic is simple. A prefix used N times inside the TTL window costs 1.25 + (N − 1) × 0.1 in units of the uncached price, instead of N. At N = 2 that is 1.35 versus 2.0 — the second call already amortizes the write premium and saves 32 percent. At N = 10 the saving reaches 78 percent, converging toward 90 percent.
The inverse also holds. An application that reuses a prefix less than once per five minutes pays the 25 percent write premium on every call and gets nothing back. Simon Willison put it plainly on the day of the release: apps prompting less than once every five minutes lose money. Caching is a bet on request frequency.
What Prompt Caching Does Not Do
Prompt caching is exact-prefix matching, not semantic caching. A single changed byte early in the prompt — a timestamp, a user ID, a reordered tool definition — invalidates everything after it. Volatile content therefore belongs at the end of the prompt, never at the beginning. Nothing about the model's context window changes; cached tokens still count toward it in full.
It is also not a persistence layer. The five-minute TTL makes the cache an optimization for bursts of activity, not a stored knowledge base. It saves nothing on output tokens, which remain the most expensive part of every response. Prompts below the minimum thresholds are not cached at all. And the feature is a beta: pricing and semantics may still change.
Comparison with Gemini Context Caching
Google shipped context caching for Gemini 1.5 Pro and 1.5 Flash in June 2024 with a different model: a 75 percent discount on cached input tokens plus a storage fee of $4.50 per million tokens per hour for 1.5 Pro ($1 for Flash), a minimum of 32,768 tokens, and explicitly managed cache objects with configurable TTL.
The designs target different workloads. Gemini's hourly storage fee favors a few very large, long-lived caches that are queried continuously. Anthropic's scheme has no storage fee, a low 1,024-token minimum, and a short self-refreshing TTL — it favors high-frequency access patterns such as chat sessions and agent loops. Neither is strictly cheaper; the workload decides.
Implications for RAG and Agents
For RAG systems the pricing shifts a boundary. Reading 200,000 cached tokens on Claude 3.5 Sonnet costs $0.06 per request. For small and mid-sized corpora that fit into the context window, placing the entire document set into a cached prefix now competes with chunk retrieval — no retriever, no index, no chunking artifacts. Retrieval remains necessary for large or fast-changing corpora, and retrieved chunks belong after the last cache breakpoint.
For agents the effect is larger. An agent loop resends its growing conversation history on every tool call, and tool calls typically arrive seconds apart — well inside the five-minute TTL. Caching the history prefix cuts the input cost of each iteration by up to 90 percent. Long tool definitions and many-shot examples, previously a per-call tax, become nearly free after the first write. We at Blue IT Systems consider this the strongest argument for the feature.
Outlook: Caching as Default Infrastructure
Our expectations, written in August 2024: caching will become default infrastructure rather than a beta flag. Competing APIs will converge on some form of it, and TTL options will grow beyond five minutes where workloads demand it. Billing models will differentiate further — write premiums, storage fees, or both.
Prompt design will change accordingly. We expect prompts to be engineered cache-first: a frozen prefix of instructions, tools, and examples, with all volatility pushed to the suffix. That discipline weakens one argument for fine-tuning and strengthens long-context architectures against retrieval for mid-sized corpora. Predictions age quickly in this field; the direction — paying once for static context instead of on every call — seems safe.