Prompt Cache Hit Rate as a Routing Optimization Signal

Prompt cache hit rate is usually treated as something you check after the fact, a line on a cost dashboard. That's backwards: at the gateway layer, hit rate can work as an input that decides which provider and model a request goes to before it's ever sent, and this piece lays out how that mechanism works, when it pays off, and how to build it.

Cache hit rate and routing

More cache capacity does nothing for you if the routing layer keeps sending requests somewhere that can't use it. Round-robin routing and load-first routing both scatter requests across replicas by design, so the second or third turn of a conversation rarely lands back on the instance that still holds its prefix in memory. Hit rates stay flat even as the cache infrastructure grows, because the thing deciding where a request goes has no idea the cache exists.

The standard routing playbook makes this worse. Load balancing and cost-based cascades both treat per-model price as fixed, a number you look up rather than a number that depends on what happened a moment ago. The Unified AI Gateway paper names the gap directly: gateways route, and they manage cache, but they don't coordinate the two. A routing switch that looks smart on paper, say, moving a request to a cheaper or faster model, can throw away a warm cache and force a full prefill from scratch. That full prefill can cost more than whatever the switch was supposed to save.

The result is a strange kind of lock-in. Teams either let routing run blind to cache state and accept the hit rate that comes with it, or they freeze their routing policy just to protect the cache they've built, treating it as something fragile. Neither path uses the cache as what it should be: an input to the routing decision, not a side effect routing happens to damage.

Prompt caching and prefix cacheability

Prompt caching works by reusing the key-value tensors a model already computed for a chunk of stable text sitting at the front of a prompt. Instead of running that chunk through the model again, the provider charges a much lower cache-read rate and skips the computation for that part. Cost drops and latency drops together, because they come from the same saved step.

The catch is that the match has to be exact. One changed character before the cache boundary, and the hit disappears. That means how you structure a prompt matters as much as which provider you pick. The PwC study on agentic caching shows that naive caching of a full context, tool calls, tool results, everything, can actually slow a system down, because dynamic content inside the cached block forces constant cache writes for text that will never be reused. System-prompt-only caching, where only the stable instructions at the top get cached and everything dynamic sits outside that block, held up as the more reliable approach across both cost and speed.

There's a second structural piece worth building into this picture early: TTL, or how long a cached prefix stays warm before it expires. A five-minute TTL works fine for a high-traffic workload firing requests constantly. But a slower, longer-running agent needs a longer TTL, or it never benefits at all, because the cache goes cold between calls. TTL turns request frequency into something the routing layer needs to know about, not just a detail for capacity planning.

TTL, minimum token thresholds, and effective cost at a given hit rate

Comparing providers on base input price alone misses the point. The number that actually matters is effective cost: base price adjusted for the hit rate you're actually getting, given that provider's discount depth and TTL. Two providers with identical list prices can produce very different bills for the same workload, because one holds a cache warm twice as long as the other.

Anthropic needs either a single automatic cache_control field at the top of a request or explicit per-block markers, and it charges a premium on the cache write itself. Its standard TTL is five minutes, with a deeper, longer-lived option available if the caller opts in on the prompt side. Hit-rate predictability on Anthropic depends partly on how carefully the markers are placed.

OpenAI caches automatically, with no markers required from the caller, and offers extended retention for workloads that run on a slower cadence. The PwC evaluation found cost reductions of 41 to 80 percent across providers on multi-turn agentic benchmarks, with real improvements in time to first token, and it flagged something easy to miss: the caching approach that saves the most money isn't always the one that improves latency the most. Those two gains can come apart depending on provider.

Google's implicit caching on Gemini 2.5 models needs no setup at all, it's on by default, but it carries its own discount depth distinct from the other two. Google also offers explicit context caching with hourly storage fees, and whether that's worth paying depends on how often the cached content actually gets called.

None of this ranks one provider above another. It establishes that TTL and request cadence have to line up. A workload whose requests come in slower than a provider's TTL window will miss the cache no matter how well the prompt is structured, and routing that ignores this fact pays full price for tokens that a longer-TTL provider would have cached for free.

Why cache-affinity routing conflicts with load balancing, and what that tradeoff costs in practice

Cache-affinity routing and load balancing want opposite things. Sending a request back to the instance that already holds its prefix maximizes reuse, but it also means that instance gets hit repeatedly while other instances sit idle. Spread requests out evenly instead, and every node stays busy, but almost nothing gets reused.

The Unified AI Gateway paper treats this collision as the central design problem facing any system that wants to route on cache state. To get high hit rates, requests need to go to the instances that already hold the right prefix, even when those instances are already loaded. Most schedulers today split the difference badly: cache-affinity logic handles some slice of requests, load balancing handles the rest, and the system ends up with mediocre results on both fronts.

The vLLM Semantic Router project's WRP research places this same tension inside a broader structure, one built around three dimensions: workload type, routing strategy, and pool topology. In that framework, how the KV cache is laid out across a pool of instances counts as a property of the pool itself, something the routing policy has to account for, not a detail to sort out after the routing decision is already made.

The gap between ignoring this tradeoff and solving it is large. The Unified AI Gateway's modeled simulation, run across eight workload profiles, reports time-to-first-token speedups and input-cost reductions spanning a wide range, from modest gains to substantial multiples, when cache state is coordinated with routing. These are modeled results across simulated workloads, not a single measured production run, so they describe the scale of what's possible rather than a number any specific system is guaranteed to hit. Even as a range, the spread is wide enough to make clear that cache hit rate belongs in the routing decision itself, not in a report generated after the request has already gone out.

Cache hit rate as a real-time routing signal

Cache-aware routing starts with the gateway watching hit rate per provider, per model, and per prefix group, and feeding that number back into the decision about where a request goes next, before the request is sent. That's the shift: hit rate stops being something you report and starts being something the routing logic reads.

The architecture the Unified AI Gateway paper points toward involves a joint control layer. At the moment a request arrives, that layer picks a target model, a place to run it, and what to do with the KV cache, all at once, under constraints on quality, latency, cost, and available resources. Separate from that, background processes keep managing where cached data lives and how long it survives, so the next request has a better shot at landing somewhere warm.

Three situations change what the gateway does once hit rate is part of the input. First, if a provider's hit rate for a given prefix group drops below a working threshold, whether because the TTL expired or because recent requests got scattered across instances, that provider loses the cost edge that made it the default, and the gateway shifts the request to wherever the prefix is still warm. Second, if a workload's requests come in slower than a provider's TTL window allows, the gateway sends that workload to a provider with a longer cache lifetime, so it doesn't eat the full input cost on every call. Third, if an incoming request shares a prefix with a session served moments ago, the gateway pins it to the same instance or provider, accepting some imbalance in load in exchange for the cache hit.

Reading the hit rate signal correctly also depends on knowing which caching strategy is active for a given workload. The PwC evaluation on DeepResearch Bench found that controlling where dynamic content sits in a prompt, pushing it to the end and keeping tool results out of the cached block, produced cost cuts of 41 to 80 percent across providers, well ahead of what naive full-context caching delivered. For a gateway reading hit rate, a low number might signal a routing problem, or it might just mean the workload's cache strategy was never set up to produce a hit. The gateway has to know which one it's looking at.

Workload types for cache-aware routing

Cache-aware routing earns its keep where a large, stable prefix gets reused across many requests packed into a short window. Three workload types fit that description: agentic multi-step tasks, RAG pipelines running against a fixed document set, and multi-turn chat conversations with long running history.

Agentic tasks sit at the top of that list. The PwC study ran its evaluation on DeepResearch Bench, testing multi-turn agent sessions built around system prompts in the thousands of tokens, spread across hundreds of sessions, and it found consistent cost and latency gains across providers once caching strategy was handled deliberately. The Unified AI Gateway simulation points the same direction: the longest-running coding agents produced some of the largest modeled speedups of any workload tested, which lines up with a simple pattern, the longer a task runs, the more stable its shared prefix tends to become.

RAG pipelines face a sharper version of the same problem. Retrieved documents often get reordered between calls, and even a small shuffle in the middle of a prompt breaks the cache hit for everything after it. Fixing that means you put retrieved references at the end of the prompt and keep chunk ordering consistent, but that fix only works if the gateway knows which ordering rule is active, since routing on hit rate without that context gives you a confusing signal.

Short, high-variance, single-turn queries sit at the other end. There's little stable content to cache in the first place, so no amount of routing sophistication changes the outcome. Plain load-balanced routing is the right call there, and building cache-awareness into that path adds complexity without buying anything back. The agentic and RAG cases justify building this system: long-running, prefix-heavy, and repetitive enough that a warm cache actually means something.

Operationalizing cache-aware routing in a production gateway

Turning this into a working system starts with logging. The gateway needs to record cache hit rate per request, tagged by provider, model, prefix group, and session, and that log has to feed directly into the routing policy rather than sitting in a dashboard that gets checked once a week.

None of that works if the prompt itself is unstable. Before a gateway can route on hit rate with any confidence, the application generating the prompt needs to guarantee a stable prefix: fixed content up front, anything dynamic appended at the end, and tool results kept out of the cacheable block, the same structure the PwC evaluation found necessary for consistent gains, because a prompt that breaks its own cache boundary every call turns the hit-rate signal into noise.

With that foundation in place, the gateway handles cache checks and provider selection as one coordinated step. A cache hit at the exact-match or semantic layer skips the model call outright, so routing logic only needs to run on a miss. When a miss happens, the gateway weighs base input price against current hit rate and how well each provider's TTL fits the workload's request cadence, comparing expected effective cost across providers, and sends the request to whichever provider is most likely to produce a hit on the next call. The Unified AI Gateway framework ties this together with background cache-management actions, placement, replication, lifecycle decisions, that run continuously and in parallel with routing, so that whatever the routing policy decides, the cache is actually there waiting when the request arrives.