Prompt Caching: How to Cut LLM Costs by 70% on Repeated Context

If you're running a system with a large system prompt or substantial static context that gets repeated across many requests, you're likely paying full price for the same tokens over and over. Prompt caching is the mechanism that stops that — and it's one of the most impactful cost optimizations available without changing your prompts or your model.

This post explains how caching works on Anthropic and OpenAI, which usage patterns benefit most, and a worked example showing what 70% cost reduction looks like in practice.

What prompt caching is

When you send an LLM request, the model processes all your input tokens before generating a response. For large system prompts or injected documents, this processing step is the dominant cost driver. Prompt caching stores the computed KV (key-value) cache for a portion of your input, so that subsequent requests reusing the same prefix skip the expensive recomputation.

The result: cached input tokens are billed at a fraction of the standard rate. Non-cached tokens are billed at standard rate. If 80% of your prompt is a repeated prefix, 80% of your input cost drops dramatically.

How Anthropic caching works

Anthropic's prompt caching requires explicit opt-in — you mark cache breakpoints in your message structure using cache_control:

response = anthropic.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": your_large_system_prompt,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[{"role": "user", "content": user_message}]
)

Key details:

• Cache TTL: 5 minutes. If no request reuses the cached prefix within 5 minutes, the cache expires. For real-time applications with steady traffic, this is usually fine. For batch jobs with large gaps between requests, you'll miss the cache.
• Minimum cacheable size: ~1,024 tokens (varies by model). Caching is not applied to shorter prefixes.
• Pricing: cached input tokens are billed at 10% of standard input cost on Claude Sonnet 5. Non-cached input tokens and all output tokens are billed at standard rates. There is a small cache write cost on the first request that creates the cache.

You can place up to 4 cache breakpoints in a single request, which allows caching different sections of your context independently.

How OpenAI caching works

OpenAI caching is automatic — no code changes required. For requests where the first 512+ tokens of the prompt match a cached prefix in their system, they automatically apply cache pricing.

Key details:

• Automatic: no cache_control flags or configuration needed
• Minimum cacheable prefix: 512 tokens
• Pricing: cached tokens are billed at a reduced rate on GPT-5 (compared to Anthropic's 10% cache rate)
• No explicit TTL: OpenAI manages cache expiration internally; the specifics aren't publicly documented

The tradeoff: OpenAI's caching is easier to implement (nothing to do) but the discount is smaller (50% vs 90%). For the highest-volume use cases, Anthropic's explicit caching with the 90% discount produces larger savings.

Which patterns benefit most

Not all usage patterns benefit equally from caching. The impact is proportional to: (cached tokens) / (total tokens) and how frequently the cache is actually hit.

High benefit patterns:

Large static system prompts — if your system prompt is 10,000+ tokens of instructions, few-shot examples, and domain knowledge, caching it saves 90% on those tokens for every subsequent request. The static content is 80–90% of a typical request's input; caching it produces dramatic cost reduction.

RAG context injection — when you retrieve the same documents repeatedly (a knowledge base that most queries reference), caching the retrieved context avoids re-billing the same chunks. This works best when you structure the cache breakpoint after the documents and before the user query.

Multi-turn conversations with persistent context — if your system injects user profile data, account history, or policy documents at the start of every conversation turn, that static prefix is ideal for caching.

Low benefit patterns:

• Short system prompts under 1,024 tokens (below the minimum cache size)
• Highly variable context where the cached prefix changes frequently
• Low-volume applications where the 5-minute cache TTL expires between requests

Worked example: 100K token system prompt

A RAG-based customer support system with:

• 100,000-token system prompt (detailed instructions + 80+ few-shot examples)
• 500-token average user query
• 800-token average response
• 10,000 requests per day

Without caching:

| Component | Tokens | Daily volume | Rate (Claude Sonnet 5) | Daily cost | |---|---|---|---|---| | Input: system prompt | 100,000 | 10,000 requests | $3.00 per 1M | $3,000 | | Input: user query | 500 | 10,000 requests | $3.00 per 1M | $15 | | Output | 800 | 10,000 requests | $15.00 per 1M | $120 | | Total daily | | | | $3,135 | | Monthly | | | | $94,050 |

With prompt caching (system prompt cached):

| Component | Tokens | Daily volume | Rate | Daily cost | |---|---|---|---|---| | Input: system prompt (cached reads) | 100,000 | ~9,950 requests | $0.30 per 1M | $298.50 | | Input: system prompt (cache writes) | 100,000 | ~50 requests (TTL misses) | $3.75 per 1M | $18.75 | | Input: user query | 500 | 10,000 requests | $3.00 per 1M | $15 | | Output | 800 | 10,000 requests | $15.00 per 1M | $120 | | Total daily | | | | $452.25 | | Monthly | | | | $13,568 |

Monthly savings: $80,482. Cost reduction: 85.6%.

The exact savings depend on cache hit rate, which depends on request volume and consistency. At lower volume with more cache TTL misses, savings are smaller. At higher volume with steady traffic, savings are larger.

When caching doesn't help

Dynamic context that changes every request. If every request has a unique system prompt or unique injected documents, the cache never hits. The cache write cost adds expense without benefit.

Very short prompts. Under 1,024 tokens (Anthropic) or 512 tokens (OpenAI), caching doesn't apply. For simple, short-prompt applications, other optimizations (model tier selection, output length reduction) will have more impact.

Sporadic traffic. If requests come in intermittently with gaps longer than the cache TTL, every request is effectively a cache miss. Consider whether your traffic pattern supports caching before investing engineering time in it.

Implementation checklist

1. Identify the static prefix in your prompts (what's the same across requests?)
2. Measure its size in tokens — is it above the minimum cache threshold?
3. For Anthropic: add cache_control at the breakpoint between static and dynamic content
4. For OpenAI: verify your prompt structure puts static content first in the prefix
5. Monitor cache hit rates in your provider's usage dashboard
6. Calculate actual savings vs projected savings — validate the impact

Prompt caching is one of the fastest cost wins available for any production LLM system with repeated context. For systems running at scale, it's not optional — it's a standard part of the cost architecture.