Token Cache - a practical introduction

Adam Wrobel · Jan 12, 2026

What is the KV Cache?

When a transformer-based LLM processes a sequence of tokens, it computes key-value (KV) pairs for each token at every attention layer. These KV pairs represent the "memory" of the conversation—Keys encode "what information is available" at each position, while Values encode "what to retrieve" when attending to that position. Together, they capture how each token relates to all previous tokens. Without caching, every new token generation requires recomputing these KV pairs for the entire context, which is computationally expensive and time-consuming. The KV cache stores these pre-computed attention states, allowing the model to reuse them when generating subsequent tokens. For API-based LLMs, providers extend this concept to cache KV states across requests, enabling prompt prefix reuse between separate API calls.

Benefits: Cost, Speed, and Repeatability

Token caching delivers three compelling advantages:

Cost reduction is often the most immediate benefit—Anthropic charges only 10% of the base input token price for cache hits, meaning a 100K token system prompt costs 90% less on subsequent requests.

Latency improvements are equally significant: skipping KV computation for cached tokens can reduce time-to-first-token (TTFT) by 50-85% depending on the cached prefix length, transforming multi-second waits into sub-second responses.

Repeatability and consistency emerge as a subtler but important benefit for agentic systems—when an agent makes multiple tool calls within a conversation, cached context ensures each call builds on identical foundational computations, reducing the risk of subtle inconsistencies that can occur when models "drift" during recomputation. For batch processing and evaluation pipelines, caching provides a necessary (though not sufficient) foundation for reproducible results—full determinism also requires controlling temperature, sampling parameters, and in some cases batch configuration.

Metric	Without Cache	With Cache	Improvement
Cost (Anthropic, per MTok)	$3.00	$0.30	90% savings
Cost (OpenAI, per MTok)	$2.50	$1.25	50% savings
Time to First Token (50K context)	~2.5s	~0.4s	85% faster
Time to First Token (10K context)	~0.8s	~0.3s	60% faster

How It Works in Practice (Gotchas)

Understanding the mechanics of token caching prevents frustrating debugging sessions.

Prefix matching is exact: the cache only hits when your prompt starts with a byte-for-byte identical prefix to a previously cached request—even a single whitespace difference invalidates the cache.

Tool definitions and subagent definitions are part of the prompt agent and subagents receive. Even a slight difference in allowed tools like Bash(glob:*) vs Bash(glob:/tmp/*) will cause prefix to NOT MATCH.

Model versions must match: a cache created with claude-3-5-sonnet-20241022 won't hit for claude-3-5-sonnet-20240620—different model weights produce different KV values even for identical input tokens, making cached states incompatible. Some providers also invalidate caches during model updates.

Subagents don't inherit parent caches: when using frameworks like Claude Code or LangChain that spawn child agents, each agent typically maintains its own cache namespace, meaning your carefully warmed system prompt cache won't benefit tool-calling subagents.

Single agent apps accumulate cache automatically: in multi-turn chat or workflow, each message appends it's output to the cached context, so ie. turn N benefits from all previous turns 0...N-1 without explicit configuration.

Direct API usage requires explicit markers: when calling LLM APIs directly (not through chat interfaces), you must use provider-specific mechanisms like Anthropic's cache_control blocks or OpenAI's cached parameter to designate which content should be cached.

Cache Hit Decision Flow

Agent vs. Subagent Cache Isolation

Provider Implementations

Anthropic Claude

Anthropic's prompt caching is explicit and granular. You mark cacheable content using cache_control blocks with ephemeral type (named "ephemeral" because the cache is short-lived and server-managed, not a persistent user-controlled store), which signals the API to cache that content for approximately 5 minutes (extendable with activity). The cache operates at the message level—you can cache system prompts, tool definitions, or even specific user messages independently. Anthropic returns detailed cache metrics in the response: cache_creation_input_tokens shows tokens written to cache (charged at 1.25x base rate), while cache_read_input_tokens shows tokens read from cache (charged at 0.1x base rate). A key nuance: cache blocks must be at least 1,024 tokens for Claude 3.5 Sonnet or 2,048 tokens for Claude 3 Opus to be eligible for caching.

from anthropic import Anthropic

client = Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are an expert code reviewer..." + large_context,  # Must be 1024+ tokens
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[{"role": "user", "content": "Review this PR"}]
)

# Check cache performance
print(f"Cache write: {response.usage.cache_creation_input_tokens}")
print(f"Cache read: {response.usage.cache_read_input_tokens}")

OpenAI

OpenAI implements automatic prompt caching for all API requests with zero configuration required—if your prompt shares a prefix of at least 1,024 tokens with a recent request, the cache kicks in automatically. Cached tokens are billed at 50% of input token cost (compared to Anthropic's 10%, making it less aggressive but still significant). The cache has a TTL of 5-10 minutes during off-peak hours and may be shorter during high demand. OpenAI exposes cache metrics through cached_tokens in the usage response. For predictable caching behavior, structure your prompts with static content (system instructions, reference documents) at the beginning and dynamic content (user queries) at the end.

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": large_system_prompt},  # 1024+ tokens, auto-cached
        {"role": "user", "content": "Analyze this data"}
    ]
)

# Check cache hits
if hasattr(response.usage, 'prompt_tokens_details'):
    print(f"Cached tokens: {response.usage.prompt_tokens_details.cached_tokens}")

Google Gemini

Gemini offers explicit context caching through a dedicated CachedContent API, designed for large static contexts like entire codebases or document corpora. You create a named cache object with a configurable TTL (minimum 1 minute, no maximum), then reference it in subsequent requests. This model differs from Anthropic/OpenAI in that you're explicitly managing cache lifecycle—useful for cost control but requiring more orchestration code. Gemini charges for cache storage per hour plus a reduced input token rate for cache reads. The minimum cacheable size is 32,768 tokens, making it suited for large-context applications rather than conversational caching.

from google import genai
from google.genai import types
from datetime import timedelta

client = genai.Client()

# Create a persistent cache
cache = client.caches.create(
    model="gemini-1.5-pro",
    config=types.CreateCachedContentConfig(
        system_instruction="You are a codebase expert...",
        contents=[large_codebase_content],  # 32K+ tokens
        ttl=timedelta(hours=1),
        display_name="codebase-cache"
    )
)

# Use the cache in requests
response = client.models.generate_content(
    model="gemini-1.5-pro",
    contents="Explain the authentication flow",
    config=types.GenerateContentConfig(cached_content=cache.name)
)

DeepSeek

DeepSeek implements automatic context caching similar to OpenAI but with more aggressive pricing—cached input tokens are priced at just 0.1x the base rate, matching Anthropic's discount. The cache activates for prompts sharing a 64+ token prefix with recent requests (the lowest threshold among major providers), making it effective even for shorter system prompts. DeepSeek's cache persists for several hours under typical usage patterns. When using DeepSeek for agentic applications, the automatic caching means multi-turn conversations naturally benefit without code changes, but you should still structure prompts with static content first to maximize hit rates.

from openai import OpenAI  # DeepSeek uses OpenAI-compatible API

client = OpenAI(
    api_key="your-deepseek-key",
    base_url="https://api.deepseek.com"
)

response = client.chat.completions.create(
    model="deepseek-chat",
    messages=[
        {"role": "system", "content": system_prompt},  # 64+ tokens for caching
        {"role": "user", "content": user_query}
    ]
)

# Cache metrics in usage response
print(f"Cached tokens: {response.usage.prompt_cache_hit_tokens}")
print(f"Uncached tokens: {response.usage.prompt_cache_miss_tokens}")

Measuring Cache Utilization and Cost Savings

Effective cache monitoring requires tracking three key metrics:

cache hit rate (percentage of input tokens served from cache)
cost savings (actual spend vs. theoretical uncached spend)
latency improvement (TTFT with vs. without cache hits)

Build a simple logging wrapper that captures these from API responses and aggregates them over time.

For Anthropic, calculate hit rate as cache_read_input_tokens / (cache_read_input_tokens + input_tokens). For cost savings, multiply cached tokens by the discount differential (e.g., for Anthropic: cached_tokens × input_price × 0.9).

Set up alerts for cache hit rate drops—a sudden decline often indicates prompt changes that broke prefix matching, model version updates, or TTL misconfigurations. For production systems, consider dashboards that visualize cache performance per endpoint or use case, enabling you to identify which prompts benefit most from optimization.

Key Metrics to Track

Modern Caching Techniques

LLM providers implement several caching strategies to balance memory usage with performance.

TTL-based caching automatically expires cached KV states after a time window (typically 5-10 minutes for Anthropic, up to an hour for extended sessions), making it ideal for interactive applications where users may pause between messages.

Session-based caching ties the cache lifetime to a conversation session, automatically clearing when the session ends—this is common in chat APIs where each conversation maintains its own cache space.

Context-based cache clearing triggers invalidation when the conversation context changes significantly, such as when system prompts are modified or when the conversation is "forked" into multiple branches. Some providers also offer prefix-based deduplication, where identical prompt prefixes across different requests share the same cached KV states, reducing infrastructure costs for high-volume applications with standardized system prompts.

Typical TTL values: Anthropic ~5 min, OpenAI 5-10 min, DeepSeek several hours, Gemini configurable.

Provider Comparison

Provider	Min Cache Size	TTL	Cache Read Discount	Activation	Metrics Field
Anthropic	1,024 tokens (Sonnet) / 2,048 (Opus)	~5 min (extends with use)	90% off (0.1x)	Explicit (`cache_control`)	`cache_read_input_tokens`, `cache_creation_input_tokens`
OpenAI	1,024 tokens	5-10 min	50% off (0.5x)	Automatic	`prompt_tokens_details.cached_tokens`
Google Gemini	32,768 tokens	Configurable (1 min+)	~75% off + storage fee	Explicit (`CachedContent` API)	Separate cache API
DeepSeek	64 tokens	Several hours	90% off (0.1x)	Automatic	`prompt_cache_hit_tokens`, `prompt_cache_miss_tokens`

Technical insights from Grid Dynamics engineers