Semantic Caching to Reduce LLM Routing Costs

Most teams assume their LLM bill climbs because more people are using the product. The real cause is a routing default: every query, whether it's a one-word classification task or a hard multi-step reasoning problem, gets sent to the same expensive frontier model. That default is a structural choice, not a fact of nature, and it's the first thing to understand before touching semantic caching.

Look at what production traffic for a typical LLM application actually contains. FAQ lookups. Simple classification. These tasks make up the bulk of real traffic, and none of them need a frontier model's full reasoning power. The cost difference between a frontier model and a smaller, capable model on these routine tasks is large enough that fixing just this one default often cuts spend more than any amount of user growth would have added to it.

A second layer of waste stacks on top of the first, visible not as a tiering problem but in repeated queries the system treats as new. Research from Carnegie Mellon and Microsoft Research Asia found that more than a third of LLM queries in production are semantically similar to queries the system already served. That means a large share of inference spend pays, again, for an answer the system already generated once. Two users asking "what's your refund policy" and "how do I get my money back" are asking the same question in different words, and a system with no concept of semantic repetition treats them as two separate, full-price inference calls.

Put the two problems together: traffic volume isn't the driver behind either one. Both are fixable, and semantic caching is the fix for the second.

What semantic caching does and does not do

Semantic caching cuts inference cost by matching what a query means, not what it says word for word. It's a distinct mechanism from both exact-match caching and model routing, and the three get confused constantly.

Exact-match caching only returns a stored answer when the incoming prompt is the identical string to one seen before. That works fine for a fixed set of button clicks or API calls, but it's close to useless for natural language, where two people ask for the same thing in a dozen different phrasings. An IT helpdesk bot that gets "how do I install the VPN app?" one minute and "guide me through company VPN setup" the next sees two unrelated strings under exact matching. A human support agent sees the same request twice.

Semantic caching closes that gap by replacing the string key with a vector embedding, a numerical representation of what the query is actually asking for. When a new query comes in, the system compares its embedding against embeddings already stored, and if the two are close enough, it returns the cached response instead of calling the model again. As vectors, they land close together, and one stored response serves both.

Four pieces make this work. Get any one of these wrong and the whole mechanism either misses obvious duplicates or serves the wrong answer to a query that only looked similar.

Atalar et al. found that semantic caching works best on queries chasing factual, non-user-specific information. (2026). A question about store hours or a refund window has one correct answer no matter who asks it, which makes it an ideal caching candidate. A question about "my account balance" does not, because the correct answer depends entirely on who's asking.

Semantic caching has nothing to do with choosing a model. Model selection, routing simple queries to cheap models and hard ones to frontier models, happens after caching, and only for the queries caching doesn't catch. A cache hit means no model gets called at all, cheap or expensive. Keeping that boundary clear matters for everything that follows, because the two mechanisms solve different problems and sit in different places in the pipeline.

Where semantic caching sits in the LLM routing stack

Semantic caching belongs at the gateway, which sits between application code and the model providers themselves. An application calls one API, and the gateway behind it handles routing, caching, failover, cost tracking, and governance without the application needing to know any of that is happening. Skip the gateway, and each of these concerns, caching included, ends up bolted on separately, half-built, and scattered across the codebase instead of living in one place where it can be managed and audited.

Placing the cache upstream of routing is what makes it valuable rather than redundant with routing. Here's the request flow: a query comes in, an embedding model converts it to a vector, and the vector store checks for a close match against everything already cached. If the match clears the similarity threshold, the gateway returns the stored response immediately. If there's no match, the query moves on to the routing logic, which decides which model should handle it, and the new response gets stored with its own embedding so the next semantically similar query can hit the cache too.

That sequencing is the whole point. A cache hit is cheaper than even the cheapest possible routed model call, because a cache hit requires no inference. Routing to a smaller model still costs something. A cache hit costs an embedding lookup, a fraction of a cent and a few milliseconds.

Prompt caching, sometimes called prefix caching, is a related mechanism that providers offer for repeated prompt prefixes like system instructions, retrieved context, or tool definitions. Prompt caching discounts the token cost of that repeated prefix. It doesn't eliminate the inference call the way semantic caching does. The two aren't competing approaches. They run in parallel and cover different parts of the cost, so a well-built system uses both rather than treating one as a substitute for the other.

Placing semantic caching at the gateway also means one policy, one threshold strategy, one invalidation approach, covers every application and every team drawing from the same models.

Which queries to cache

Knowing that caching saves money doesn't answer the harder question: which responses earn a spot in the cache. Cache space is finite. The cost of serving any given query and the likelihood it'll come up again are both unknown ahead of time. And real LLM traffic doesn't come from a short, fixed list of possible questions, it comes from an open-ended, continuous space of phrasings that keeps expanding as the user base grows.

Earlier caching approaches sidestepped this by assuming a known, finite set of discrete queries, which made the math tractable but doesn't hold up against how people actually talk to these systems. Atalar et al. built the first rigorous theoretical framework for semantic caching that treats queries as points scattered across an infinite, continuous embedding space rather than entries on a list, combining dynamic n-net discretization with Kernel Ridge Regression, a pairing that lets the system estimate the cost and value of caching a query based on its neighborhood in that space, rather than needing to have seen that exact query before.

Two factors decide whether a query-response pair deserves a cache slot. The other is serving cost: a query that would otherwise demand a long context window, a frontier model, or a high token count saves more every time it's served from cache, so expensive queries deserve priority when cache space is tight.

A cache policy fixed at launch will underperform one that keeps adjusting to what traffic actually looks like. The online algorithm in this framework achieves sublinear regret against the best possible caching decisions made with perfect hindsight, meaning its choices get measurably better as it accumulates experience with real queries, without ever needing a full list of what users might ask ahead of time. Switching which responses are cached isn't free either. There's computational overhead every time the cache population changes, and systems that don't account for that cost will thrash, swapping cache slots in and out faster than the swaps pay for themselves.

Similarity threshold tuning: the decision that determines whether semantic caching helps or hurts

If there's one implementation decision that decides whether semantic caching helps or actively hurts a product, it's the similarity threshold. Setting it too high or too low makes the system fail in a different, specific way.

When the threshold is set too low, queries that are only superficially similar get treated as matches. The system returns a cached answer that's wrong for the new question, and that failure doesn't look like a cost or latency problem on a dashboard. It looks like a wrong answer delivered confidently; catching it requires a dedicated eval suite checking cached responses against what the model would have said fresh.

The right threshold isn't a single number for the whole system. It depends on the query type. A factual, FAQ-style question like store hours or a return policy can tolerate a fairly wide similarity window, because the range of phrasings that mean the same thing is broad and the correct answer doesn't shift. A query that depends on personal context, like an account balance or a specific order status, needs a tight window or no caching at all: two queries that look similar on the surface can require completely different answers underneath.

Some categories of query should have caching turned off entirely rather than tuned. And anywhere a slightly wrong answer carries real consequences, medical guidance, legal interpretation, financial decisions, the risk of a bad cache hit outweighs the savings.

Calibrating a threshold means running precision and recall analysis on a representative sample of real production queries, broken out by category, and measuring two error rates: how often the cache serves a wrong answer (false positive) and how often it misses a valid match (false negative). Pairing threshold decisions with an eval suite that checks cached responses stay correct for the similarity window in use catches the problem routing without measurement causes elsewhere in the stack: savings on paper, and wrong answers in production.

Cache invalidation: TTL, event-based, and staleness strategies

A cached response that was correct the day it was stored can become wrong the moment the fact behind it changes. No single invalidation method covers every situation, so the right setup combines a few mechanisms, matched to how fast each kind of content actually changes.

Four strategies cover most cases. Time-to-live, or TTL, expires a cached entry after a set period and suits content that updates on a predictable schedule, a policy document revised quarterly, pricing refreshed nightly. The TTL duration should match how often the underlying content actually changes, not some arbitrary default.

Event-based invalidation purges a cached entry the moment its source document or data record changes, which is more precise than TTL when content is tied to one specific authoritative source. It requires the cache to be wired into the same pipeline that updates the content, so the two systems know about each other.

Version-based invalidation ties cached responses to a content version identifier. When the version number moves, every response built from the old version gets invalidated at once. This fits knowledge-base and product documentation workloads well, where content changes in discrete releases rather than continuously.

None of these four needs to stand alone. Without that record, there's no way to know what to invalidate when the source changes.

For any team operating under compliance requirements, the invalidation log becomes part of the audit trail. What matters is a record of when a given response was replaced and why, which turns cache management from a performance tool into part of the governance record.

Compounding savings from semantic caching and model routing

Semantic caching and model routing solve two different problems, and the savings from combining them add up rather than overlapping. Caching removes redundant inference on queries that repeat in meaning. Routing right-sizes which model handles the queries that are actually new.

A cache hit bypasses routing entirely: no model at any price tier gets called. Routing only ever applies to the queries that miss the cache, the ones confirmed to need fresh inference. Once a query reaches that stage, routing decides whether it's routine enough for a cheap model or complex enough to justify a frontier model.

Several routing strategies apply to that cache-miss traffic. Complexity-based routing classifies each request and sends routine ones to a low-cost model and hard ones to a frontier model, which tends to be the highest-leverage approach for most products. Domain-based routing sends code questions to a coding-specialized model and vision questions to a multimodal one.

The size of the savings depends on two splits: how much of total traffic the cache absorbs, and how much of what's left gets routed to cheaper models instead of the frontier default. Both numbers come from a team's actual traffic distribution rather than a headline percentage borrowed from somewhere else. Aggressive routing only pays off if an eval suite confirms the cheaper model actually handles what it's assigned correctly, since routing without that check saves money on paper while quietly losing customers on answer quality.

Routing also serves as a resilience layer, not just a cost one. When a primary model provider is rate-limited or down, the same routing logic that picks a cheap model for a simple task can redirect a request to an equivalent model on another provider, turning what would be an outage into a brief handoff instead.