Prompt Caching Across LLM Providers at the Gateway Layer

Prompt caching can cut input costs by half to 90% and shrink time-to-first-token by 3 to 10 times on repeat traffic. It's also the number most teams leave on the table once they're calling more than one model provider. Every major provider implements caching differently, and none of those implementations line up. Get this wrong and you're not losing a few percentage points, you're paying full price on traffic that should be nearly free.

Start with the mechanism, because it explains everything downstream. When a model processes a prompt, it computes attention key-value tensors (KV tensors) for every token in that prompt. That step, called prefill, is expensive, and it scales quadratically with prompt length: double the tokens, and the compute roughly quadruples. Caching stores those KV tensors server-side, so the next time the same prefix shows up, the model skips prefill for that portion and picks up where it left off. The output is byte-for-byte identical to a fresh run. Nothing about quality changes, only the path the compute takes to get there.

The other half of inference, decode (generating tokens one at a time after the prompt is processed), is bound by memory bandwidth, not compute, and inference engines have already squeezed most of the slack out of it. Caching targets prefill, which still has room to give.

So why don't caches just live forever? Because KV state takes up a lot of room. A 32,000-token context on a 70-billion-parameter-class model needs something like 10 gigabytes of storage for its KV tensors, and GPUs run out of space fast. Most providers default to a 5-minute cache window, roughly the point where memory pressure forces eviction. Getting to hours or days means writing the cache to disk instead of GPU memory. DeepSeek's MLA architecture is built around exactly that trade.

The workloads that benefit most all share one trait: a big, stable chunk of prompt that repeats. Chatbots reuse a system prompt and growing conversation history turn after turn. RAG pipelines (retrieval-augmented generation) reuse the same retrieved document block across many queries. Agent loops resend a 40,000-token context on every step of a multi-turn task. In every case, that repeated block is what caching is built to skip.

Realizing the savings depends entirely on how a given provider exposes the feature. And that's where things get messy, because no two providers agree on how it should work.

How the major providers implement prompt caching differently

Two philosophies split the market. Anthropic and Google use explicit caching: the developer has to mark which part of the prompt is cacheable using a parameter in the request. OpenAI and DeepSeek use automatic caching, where the provider notices a repeated prefix on its own and applies the discount without anyone flagging anything.

Anthropic charges 1.25x the base input rate for a 5-minute cache write, or 2.0x for a 1-hour write. A cache read costs just 0.10x, a 90% discount off standard input pricing. Do the math and the break-even point lands after two reads: everything after that is 90% cheaper than running uncached. Getting there requires adding explicit cache_control markers, and the minimum token count to qualify shifts across the Opus 4.x line: 1,024 tokens for Opus 4.8, 2,048 for Opus 4.7, and 4,096 for Opus 4.5 and 4.6.

The sharper trap sits in the lookback window. Anthropic only scans the last 20 content blocks of a message for prior cache entries. A conversation running past 20 turns without fresh cache_control breakpoints roughly every 15 blocks lets the oldest cached context slide outside that window. It silently reverts to full price. Nobody gets an error. The bill just goes up.

Stacking the Batch API's flat 50% discount on top of cache reads lets a team hit roughly 95% savings on the repeated portion of a prompt, about as good as caching gets anywhere right now.

OpenAI detects the repeated prefix automatically, no markers or setup required, provided it's at least 1,024 tokens, then in increments of 128 tokens after that. TTL policy has moved around: GPT-5.5 and gpt-5.5-pro extended the cache window to 24 hours. Cached input runs at roughly a tenth of standard price ($0.50 per million tokens against $5.00 per million standard, with the implied write cost around $6.25 per million). On August 20, OpenAI shipped a dashboard reporting cache hit rate, reads per write, and the split between cached, write, and uncached tokens. It's a real tool, but it only sees OpenAI traffic. Anything routed elsewhere is invisible to it.

Google Gemini uses explicit caching again, with a 32,000-token minimum block, but the pricing model is structurally different from either of the above. Gemini bills caching as storage: $0.40 per million tokens per hour for caches under 200,000 tokens, $4.50 per million tokens per hour above that, stacked on top of standard token rates. The bill scales with wall-clock time, not request volume, while OpenAI and Anthropic both charge per request. The break-even point on Gemini depends on how often the app is actually calling the model, in addition to the cache hit rate.

AWS Bedrock's Converse API only supported a 5-minute TTL before January 26, 2026, full stop. On that date, AWS added 1-hour TTL support, but only for the Claude 4.5-plus family (Sonnet 4.5, Haiku 4.5, Opus 4.5), set through a new ttl: "1h" field on the cache point. Amazon's own Nova models use caching with no TTL field. Older Claude models on Bedrock do not support the 1-hour TTL option, a quiet breakage risk for any team running mixed model versions on the same account.

DeepSeek's caching is fully automatic and disk-backed thanks to the MLA architecture mentioned earlier, so the cache survives idle periods measured in hours instead of minutes. Cache granularity runs well below OpenAI's 1,024-token floor, so partial prefix matches still earn a discount instead of needing an exact long match. On DeepSeek V4-Flash, a cache hit drops price from $0.14 to $0.003 per million tokens, roughly 47 times cheaper.

Lining all five up shows that nothing maps cleanly across them: different parameters, different TTL menus, different minimum token floors, per-request pricing sitting next to per-hour storage pricing, different lookback rules, different (or absent) dashboards. Anyone calling more than one of these providers is juggling five separate mental models just to keep caching working. That's not a footnote; that's the whole problem this piece is about. That's the whole problem this piece is about.

The effect of provider pricing fragmentation on a team's actual bill

The clearest illustration happened with Anthropic on August 10. Anthropic made Claude Sonnet 5's $2 per million input price permanent, cancelling a scheduled bump to $3 that had been set for September 1. Output pricing was set to rise from $10 to $15, and Anthropic cancelled that increase too on the same day, holding output at $10 for good.

Sounds like good news, and for uncached traffic it was. But Anthropic prices every cache tier as a multiple of the base input rate, so changing the base repriced the entire cache stack automatically: 5-minute writes moved from $2.50 to $3.75 per million, 1-hour writes from $4 to $6 per million, cache reads from $0.20 to $0.30 per million. Nobody touched a line of code. Cached workloads got 50% more expensive overnight because the multiplier structure propagates any base-price change straight through every tier above it.

Ranking providers by headline input price is a trap for anyone running cache-heavy workloads. Analysis of the actual per-provider billing shows two deployments of the same model can differ by a wide margin on the actual invoice once cache reads are billed, because providers charge cache reads at very different fractions of their listed input price. Same sticker price on the model card, wildly different bill once caching enters the picture.

Gemini's storage-based pricing adds a wrinkle neither OpenAI nor Anthropic has. A team optimizing hard for cache hit rate can still end up paying more overall if requests come in slowly enough that storage fees outrun the discount from cached reads. And on Bedrock, the silent TTL rejection on older Claude models means a team that upgraded some services but not others can lose the 1-hour cache with no error message, quietly falling back to 5-minute caching and eroding hit rates on long agent sessions with nothing in the logs to flag it.

None of this is an edge case. It's the default operating condition for any team calling more than one provider. And the logic for handling it does not belong scattered across a dozen services, each guessing at a different provider's rules.

Why application code is the wrong place to manage this complexity

Caching at the SDK layer, inside each service's own code, feels fine at first. Fast to ship, tight feedback loop. It falls apart once an organization has ten or more services touching different providers, because every one of those teams has to independently remember the right markers, the right TTL, the right minimum token count. As documented across provider pricing guides, drift at that scale is constant.

Drift looks boring in practice. That is why it's dangerous. One service sets a 5-minute TTL on a workload that would clearly benefit from an hour. Another forgets the cache_control marker on Anthropic. A third never accounts for the 20-block lookback limit and starts paying full price on long conversations without anyone noticing. None of these failures throw an error. They just quietly cost more, month after month, and nobody's dashboard is built to catch them.

TrueFoundry documented a production incident that shows how bad this gets under failover. Anthropic's API was caching correctly on its own traffic, but a significant share of requests had been rerouted to a different provider during a brief outage. The translation layer handling that failover dropped the cache directive silently. The bill for that traffic came in around $30 on requests that should have cost about $3, a tenfold miss traced to one dropped header. When failover happens inside application code, there's no single place that knows to re-apply the cache directive in whatever format the new provider expects, because the code that set the original marker was written with only the first provider in mind.

Agent loops make this worse in a different way. Getmaxim.ai's analysis finds a gateway that adds 40 milliseconds of latency per call contributes 200 milliseconds of pure proxy overhead across five sequential calls in an agent loop. That's a real cost, but it's dwarfed by what happens when cache directives get dropped on those same turns: the compute cost compounds far faster than the latency ever could.

A governance gap sits behind all of it, too: finance can't see the aggregate cache hit rate, and platform teams can't tell how much a cache miss costs, because no shared view of caching logic exists across the scattered services. Most large organizations now have caching logic scattered across dozens of services with no shared view. Finance can't see the aggregate cache hit rate. Platform teams can't tell how much a cache miss is actually costing across the org, because the data lives in five different dashboards that don't talk to each other.

Drift, silent failover breakage, fragmented visibility: all three point to the same fix. Pull the logic up to one layer that sits above every provider.

The unified control surface a gateway provides for prompt caching

A gateway sits between the application and every model provider it talks to. For caching specifically, TrueFoundry's description of gateway architecture explains that it translates cache directives into whatever format the target provider expects, handles fallback cleanly when a provider doesn't support caching at all, and reports hit and miss metrics through one consistent interface instead of five separate ones.

The architecture breaks into a few working parts. A routing engine decides which model and provider handle a given request. A policy layer enforces cache-write settings centrally, instead of leaving that decision to whichever team wrote a given service. A caching layer cuts down on redundant calls and can be tuned per request through headers, TTL settings, and cache-control rules. An observability stack logs hits, misses, and cost attribution across every provider in one place.

LLM Gateway, as of August 21, 2026, treats cache-write control as a three-way setting rather than an all-or-nothing switch. Client-managed mode forwards whatever cache markers the client already sends, without the gateway adding its own on top. That matters because a single API key might serve a coding agent that manages its own caching logic right alongside general traffic that shouldn't be paying a cache-write premium at all, and the gateway needs to treat those two streams differently without anyone hand-coding the split.

Unified visibility is the real payoff. Per-request and per-model spend tracking, cache hit rate by team or project or key, cost attribution that doesn't require someone reconstructing it from an invoice at the end of the month. Braintrust Gateway also encrypts cached responses with AES-GCM and scopes them to the requesting user by default, a security requirement that would otherwise have to be rebuilt independently inside every service.

Most production gateways plug into existing OpenAI or Anthropic SDK calls with nothing more than a base_url change, so none of this requires rewriting application code to get the benefit. This is arriving at a moment when it's needed for reasons beyond cost: the EU AI Act's high-risk obligations become enforceable starting December 2027, and SOC 2 and GDPR audit expectations are rising in parallel. Structured logs of cache behavior, cost events, and routing decisions are becoming a compliance requirement.

Cache-aware routing: how a gateway optimizes provider selection for cached workloads

Ranking providers by list price picks the wrong winner for any workload that leans on caching. LLM Gateway's analysis found two providers can share the exact same headline input price and still bill a cached workload very differently: one charging 56% of its input rate per cache read, the other 16%.

The two kinds of routing mistakes aren't symmetric, either. Treating a one-shot large prompt as though it were cached brings a worst case of roughly an 11% cost penalty. Rank a genuinely cache-heavy workload using plain list prices instead, and the penalty can run up to 160%. Getting this wrong in one direction is a rounding error. Getting it wrong in the other is a budget problem, and the two shouldn't be weighed as if they're the same size of mistake.

LLM Gateway's cache-aware routing, shipped September 5, 2026, handles this by scoring providers differently once a prompt crosses 5,000 tokens. The token-price score blends each provider's actual cached input rate, assuming a 70% cache-hit rate, and weights output cost by an output-to-input ratio, after first resolving which context-length pricing tier the prompt falls into. The system learns from a project's last 24 hours of traffic once enough usage data exists, tracking cache hit rate separately per provider. Sessions stay pinned to whichever provider is already running well, so the cache stays warm. If that provider goes down, routing picks a replacement based on the current score rather than starting over from a cold cache, unless a full restart turns out to be unavoidable.

Before there's enough history to learn from, the system falls back on sensible defaults: general API traffic assumes a 10% cache-hit rate, recognized coding clients assume 90%, chat-style organizations assume 50%. Enterprise accounts can override any of this with explicit thresholds for cache hit rate and output-to-input ratio, set per project.

None of this is something application code can replicate on its own. The application layer doesn't know the actual observed cache hit rate per provider in real time, doesn't have live access to each provider's current cached-pricing tier, and has no way to persist which provider a session is pinned to across separate services. That state has to live somewhere central. The gateway holds that state because it is the only part of the stack with the view to hold it.

Cache-aware price routing isn't the only lever available, either. Inworld Router, published March 24, 2026, supports conditional routing through CEL expressions, meaning requests can be routed by user tier, query complexity, region, or any custom metadata a team wants to key on. That's a different axis than cache-aware pricing, and the two aren't in competition. A gateway can route on request characteristics and on cache economics at the same time.

Workload-specific caching strategy at the gateway layer

Different workloads want genuinely different cache settings, and a gateway is the only place that can apply the right one automatically instead of leaving it to whoever wrote a given service.

Chatbots are the easy case. The system prompt and the growing conversation history form a stable prefix that naturally repeats across turns, so a 5-minute TTL usually matches the rhythm of an active session just fine, since the cache stays warm across the typical gap between one message and the next.

RAG pipelines look different. The retrieved document block is often large and reused across many queries, which makes the case for a longer TTL, closer to an hour where the provider supports it, rather than the shorter default built for chat.

Agent loops are their own category entirely, resending a large prefix, sometimes 40,000 tokens or more, on every step of a multi-turn task. That repetition is precisely the pattern caching was built to catch, and it's also where a dropped cache directive during failover does the most damage: the same expensive prefix gets recomputed at full price on every single turn until someone notices the bill.

The pattern across all three holds regardless of provider: the right TTL, the right minimum token threshold, and the right fallback behavior are workload properties, not provider properties. A gateway that understands the shape of the traffic passing through it can apply the setting that matches, provider by provider, without asking any application team to keep five different vendor rulebooks straight in their heads.