Idempotency in LLM Gateway Request Handling

Every retry in an LLM-integrated system originates from one of two places, and they require completely different responses. Conflating them is where the quiet, expensive bugs live.

The first intent is: "I never got a usable response." Network failure, timeout, 429, 5xx. The model may or may not have processed the request; you genuinely do not know. The goal is to obtain a response, and the right mechanism is a cached response keyed on the prompt and parameters, so that if the model already ran the inference, you are not re-billing the provider for work it already completed.

The second intent is: "I got a response, but the downstream action failed." The model responded. The tool was called. The external system timed out on the write. Re-calling the LLM here is wasteful at best and, depending on the tool, actively harmful. The goal is to resume, not restart. The correct mechanism is an idempotency key scoped to the logical operation: the booking, the charge, the record write, not the individual API call.

Mixing these two intents produces a failure mode that is genuinely hard to catch. If you cache a failed parse attempt under an idempotency key, every subsequent retry returns the same broken output. The system looks like it is working. It is not. For sampling retries, include a retry count or a hash of the previous error in the key, so those attempts are treated as distinct operations. Network retries, by contrast, should still resolve to the cached result, because the underlying operation is genuinely the same and re-executing it accomplishes nothing.

Three Specific Failure Modes That Turn Retry Logic Into a Liability Without Idempotency Enforcement

Start with the automatic retry on transient failure. An agent retries a network timeout on a booking call. The original call succeeded before the timeout fired. The result is two bookings. This is not a hypothetical; it is the default behavior of most HTTP retry libraries applied naively to stateful operations, and it happens constantly in production.

Then there is LLM re-planning after context loss. Context truncation, missing tool results, or an interrupted multi-step plan causes the model to re-issue a tool call it already made. Without explicit deduplication at the gateway, the tool has no record of the prior call. The model is not malfunctioning; it is doing exactly what it was designed to do given incomplete information. The failure is architectural, not behavioral.

The parallel tool execution race is the third, and it is particularly insidious. Two concurrent branches of an agent call the same tool with identical arguments. Both execute. The same side effect fires twice. Both branches return success independently, leaving the inconsistency buried in a downstream system where it sits undetected for days.

The severity gradient across these three scenarios is real and worth sitting with. A duplicate Slack message is annoying. A duplicate payment is a customer service incident with chargeback exposure. Contradictory database records can require weeks of debugging before someone connects the symptoms to the cause. Token costs compound on top of all of this in ways that catch teams off guard: a single turn retrying on a misconfigured tool can spend an order of magnitude more than the request should cost. Retry budgets in production are a hard requirement, not an optional guardrail.

How Idempotency Keys Should Be Structured and Scoped at the Gateway Layer

The core pattern is straightforward: a stable unique identifier sent with every mutating operation; the server stores the result of the first execution against that key; retries return the cached result without re-executing. The complexity lives almost entirely in scope decisions, and teams get the scope wrong far more often than they get the mechanics wrong.

Keying to the individual API call is the wrong scope. A single agent orchestration step involves several LLM calls and several tool calls. Keying each call separately misses the actual unit of duplication risk.

The right scope is the logical operation, meaning the action with external side effects, regardless of how many inference calls it took to reach that point. For network retries, a stable key per logical operation works well: session ID combined with action type and action target is a reasonable starting composition. For sampling retries, append a retry count or a hash of the previous error to that key. Same logical operation, distinct attempt, distinct result slot.

The gateway must store the result, including whether it was a success or a typed failure, the key itself, and a TTL appropriate to the operation's risk window. A payment confirmation key needs a materially longer TTL than a read result. That distinction matters and should be explicit in the configuration, not an afterthought.

One clarification that prevents significant confusion: provider-level prompt caching, prefix caching, and semantic caching reduce cost and latency. They are not idempotency mechanisms. They do not enforce the "execute once" guarantee. Conflating them with idempotency produces systems that feel protected but are not, and the gap only becomes visible after something has already gone wrong.

Semantic Deduplication for Agentic Tool Calls Where Content Hashing Is Insufficient

Hash-based key matching works when parameters are byte-for-byte identical. Agentic re-planning breaks that assumption constantly. A model re-issuing a tool call after context loss generates parameters that differ in wording but express identical intent: "submit invoice #4421" versus "process invoice number 4421." The hash fails. A naive key lookup treats it as a new operation and executes it again.

Semantic deduplication addresses this directly. Embed the proposed tool call, check that embedding against recent stored calls from the same agent session, and if the cosine similarity exceeds a defined threshold, return the cached result rather than executing. In practice, thresholds around 0.9 capture near-identical phrasings reliably. Dropping to 0.85 starts to capture equivalent intent with more wording variation, and whether that is useful or dangerous depends entirely on the tool.

That last point is not a minor caveat. A 0.85 similarity threshold that correctly deduplicates invoice submissions is dangerously aggressive for code execution, where small parameter differences are semantically significant and absolutely not interchangeable. The overhead of semantic deduplication is also real, so apply it selectively. High-consequence tool calls: payments, emails, record writes. These justify the cost. Lower-consequence reads typically do not.

State Machine Modeling for Long-Running Operations That Can Be Interrupted Mid-Execution

An idempotency key tells you whether an operation started. It does not tell you where it stopped if the process was interrupted partway through a multi-step sequence. State machine modeling closes that gap.

The pattern: model each long-running tool as a series of named checkpoints. PENDING, VALIDATED, SUBMITTED, CONFIRMED. Each state transition is itself idempotent; arriving at a state you are already in is a no-op, not an error. On retry, the agent checks the current state and resumes from there rather than restarting from the beginning.

Consider a "process invoice" tool interrupted while in SUBMITTED state. On resume, the tool checks the accounting system for a confirmation record before attempting re-submission. If confirmation exists, it is returned immediately. If not, the submission completes. The tool never submits twice, because the state check precedes any action.

The gateway's role is to persist checkpoint state across agent turns and surface the current state on resume, rather than requiring the agent to reconstruct that state from context alone. This matters more than it appears. Agentic workflows already consume substantially more tokens per task than standard chat interactions, and an interrupted long-running operation that restarts from scratch rather than resuming from its last checkpoint compounds an already elevated cost profile in ways that are genuinely difficult to explain to stakeholders after the fact.

The Retry Policy That Must Wrap Idempotency to Make It Safe End-to-End

Idempotency prevents duplicate execution. It does not prevent an unbounded retry loop from burning tokens and hammering downstream systems. A complete production approach requires both, and one without the other is not actually safe.

Retry classification comes first: retryable errors (429s, transient 5xxs) versus fast-fail errors (authentication failures, policy violations, malformed requests). Never retry the latter. Exponential backoff with jitter comes next, and it needs to be provider-aware rather than generic. A 429 from a rate-limited provider needs materially longer backoff than a transient 503.

A hard cap on retries per agent turn is non-negotiable: not a soft guideline, a hard cap. Without it, a misconfigured tool call can exhaust the token budget in a loop while the system logs successes and nobody notices. Circuit breaking isolates a failing provider or tool and prevents cascading failures across unrelated workflows. A dead letter queue ensures that operations exhausting their retry budget without resolution are routed for manual review or scheduled reprocessing rather than silently dropped.

Timeout tiers need to vary by use case. Interactive requests warrant short timeouts and fast-fail behavior. Batch operations warrant longer timeouts and more retry headroom. A single global timeout policy is wrong for both simultaneously, and teams that apply one anyway end up with either frustrated users or failed batch jobs. Enforcing all of this at the gateway, rather than distributing retry logic across application code, produces consistent behavior and makes the policy visible to cost tracking infrastructure.

What Idempotency Enforcement Reveals About Real AI Spend, and How the Gateway Makes That Visible

Tokens spent on duplicate inference calls do not appear as a separate line item on provider invoices. Without gateway-level logging, they are completely invisible. That invisibility is a more serious problem than it sounds when you look at what AI spend actually looks like right now.

Enterprise AI spend reached an average of $85,521 per month in 2025, according to CloudZero's State of AI Costs 2025 report, a 36 percent increase year over year. That same report found that only 34 percent of companies had mature cost management processes in place. Benchmarkit and Mavvrik's 2025 research found that 57 percent of companies were still tracking AI spend via spreadsheets. Agentic workflows sit on top of that already strained picture, consuming substantially more tokens per task than standard chat. Duplicate tool calls in those workflows do not cost twice a chat retry; they cost multiples of an already expensive operation.

Gateway-level idempotency logging changes what is visible and actionable. Tagging every request with its idempotency key makes deduplication events auditable rather than silent. Surfacing retry rates by model, tool, provider, and team exposes misconfigured tools and unreliable providers before they become budget incidents. Attributing token spend to sessions and agent turns, rather than raw API calls, aligns the cost unit with the business logic unit, which is where decisions actually get made.

When a pipeline restarts and the gateway returns a cached response for an already-resolved inference call, the team pays nothing for that token operation. Real-time, per-key spend tracking at the gateway makes retry behavior actionable before the damage accumulates.

What Separates a Prototype Integration From a Production-Grade One When It Comes to Idempotency

The prototype pattern is recognizable to anyone who has debugged one. Retry logic lives in application code. No idempotency keys. Provider errors surface as unhandled exceptions. Spend is reconstructed from invoices after the fact, usually when someone notices a budget anomaly that is already weeks old and the root cause has long since disappeared from logs.

The production-grade pattern centralizes the enforcement. The gateway handles idempotency keys, retry classification, circuit breaking, and dead letter routing. Application code calls the gateway and trusts the policy. Retry behavior is consistent across every workflow because it is defined in one place, not scattered across a codebase where three different teams each implemented their own version with subtly different assumptions about what "retry" means.

The build-versus-buy question for this layer has a clear shape. A self-hosted gateway means the team owns the idempotency store, the retry policy, the key TTL management, and the circuit breaker state. That is a meaningful operational surface that grows with traffic and demands sustained engineering attention. Managed infrastructure shifts idempotency enforcement, retry policy, and spend attribution from something the team builds to something the team configures.

Concentrate provides this as a managed layer, with a unified API across more than 130 providers and request handling, spend visibility, and policy enforcement built in.

The production readiness signal worth watching: if retry logic lives in more than one place in the codebase, idempotency guarantees are already inconsistent across workflows, regardless of how correct any individual implementation happens to be. Agentic systems make this more urgent as workflows grow longer and tool call counts increase. A single missing idempotency key protecting a ten-step operation creates more exposure than the same gap in a two-step one. The blast radius scales with the operation's length, and the cost of that scaling is rarely linear.