Rate Limiting Strategies for LLM API Traffic

A request to an LLM endpoint can cost 50 tokens or 50,000 tokens, and both are counted identically in a request-per-minute count. That gap in cost between two calls that look the same on a dashboard is why traditional rate limiting breaks down on AI traffic. A chat completion, an embedding call, and a RAG retrieval all register as "one request," but they draw wildly different amounts of compute and spend. Counting requests tells you almost nothing about what you're actually spending. Rate limiting built for LLM traffic has to measure something else entirely: tokens.

What token-aware rate limiting controls

Token-aware rate limiting works by watching three numbers at once: requests per minute, tokens per minute, and tokens per day. None of the three can be dropped without opening a hole the other two don't cover.

Requests per minute catches a different problem than tokens per minute. RPM limits stop connection-level pile-ups, the kind of burst where thousands of calls hit an endpoint at once regardless of size. TPM limits cap how fast money actually gets spent, because that depends on token volume, not call count. TPD limits hold the line on a budget over the course of a day, catching slow leaks that never spike hard enough to trip an RPM or TPM limit but add up by evening. Remove any one of the three, and traffic finds the gap. If a system has only RPM limits, a single oversized prompt gets through, because the limit never looks at size. A system with only TPM limits lets a flood of tiny, rapid-fire calls through, because the limit never looks at count. If a system has no daily ceiling, a slow drip can run all day and quietly drain a month's budget.

What actually drives token consumption per call comes down to a short list: total tokens processed (input plus output together), the size and architecture of the model handling the request, and the type of request itself. A chat completion, an embedding call, and a RAG retrieval each carry a different token profile, which is part of why a single LLM request can run anywhere from 50 to 50,000 tokens depending on these variables; Zuplo puts the range at exactly that.

Billing adds another layer that a naive token count misses. Providers log and bill input tokens and output tokens separately, so a rate limit that treats them as one shared pool will misattribute cost between the two. A system also needs to account for price differences between models: a completion token from a flagship model costs far more than one from a smaller model, so a dollar-denominated budget needs a cost multiplier attached to each model. Without that multiplier, a team that switches from a cheap model to an expensive one can blow through its real budget while staying comfortably inside its token count. Teams and workloads also don't all deserve the same ceiling. A mission-critical production agent needs a higher-priority allowance than a developer poking at the API in a sandbox, and that tiering has to happen at the point of enforcement, not inside each application.

The four algorithms and the failure mode each one targets

Every rate-limiting algorithm was built to stop one specific kind of failure. Picking the wrong one for LLM traffic brings back the failure it was meant to prevent.

A fixed window counter resets at strict calendar intervals. A caller who knows the reset time can spend a full budget right at the end of one window, then spend a full budget again the instant the next window opens, effectively doubling the allowed rate for a few seconds around the boundary. This is a particularly bad fit for LLM traffic, where bursts are already unpredictable: a prompt loop can time itself deliberately to land its bursts right at the window edge.

Sliding window fixes the boundary exploit because it never resets at all in the fixed-window sense. It continuously recalculates usage from recent activity, rolling the measurement period forward with each new request instead of snapping back to zero on a clock. That makes it a strong match for LLM workloads specifically, where bursts arrive with no predictable rhythm: the rolling calculation absorbs irregular traffic without the cliff-edge reset that fixed windows create.

Token bucket works differently again. A bucket fills with tokens at a steady rate, and each request draws down tokens in proportion to its actual cost, not a flat count of one. The bucket can absorb short bursts, but it still holds the line on a long-run average rate. For generative AI applications where inference time varies widely from one call to the next, this keeps high-priority tasks from getting stuck behind a pile of smaller requests, since the bucket drains by cost rather than by number of calls.

Cost-based limiting works at a different layer than a window or a bucket. It weights tokens by what they actually cost from the provider, so a dollar-denominated budget reflects real spend no matter which model handled the request. Once routing spans more than one provider, you need this, because a token budget that ignores the cost differences between models will consistently undercharge for the expensive ones. Cost-based limiting isn't a replacement for a window or a bucket. It works alongside them: the window or the bucket sets the rate, and cost weighting sets what each unit inside that rate is actually worth. Production systems for LLM traffic tend to need more than one of these running together, since each one closes a gap the others leave open.

Enforcement belongs at the gateway, not in application code

Spreading rate-limit logic across individual application services recreates the exact fragmentation it was supposed to fix. Without a shared enforcement point, every application has to understand each provider on its own: its own API keys, its own rate limits, its own retry logic, its own usage reports. Visibility ends up broken into pieces, with no governance that holds across the whole organization.

This fragmentation causes a deeper problem: if two services each track their own token consumption against the same shared provider quota, neither one has an accurate live view of how much headroom is actually left. The limits get duplicated, they drift apart from reality, or they directly conflict, and nobody finds out until a request fails somewhere downstream. This is the noisy-neighbor problem in its purest form: one team's runaway agent, left unchecked by any service that can see across the whole system, can consume quota that was meant for another team entirely, and no single application has the vantage point to stop it.

A gateway sitting in front of all provider traffic solves this because it's the one place that sees everything. It works like a traditional API gateway, a single front door handling authentication, rate limiting, and audit logging, with three additions that LLM traffic specifically demands: metering tokens on every call, screening prompts and responses against policy, and routing requests across models based on the task at hand. None of those three additions can live safely inside individual application code, because each one needs to see traffic that no single application has access to.

Partitioning quotas by tenant or workload type at this layer is what actually stops the noisy-neighbor scenario from happening in production. Done well, enforcement also doesn't cost latency: production gateway architectures keep the control plane, authentication, authorization, rate limiting, separate from the data plane, processing those checks in memory so overhead stays in the sub-millisecond range no matter how complex the governance rules get underneath. A further piece of this architecture matters for operational safety: virtual keys issued by the gateway stand in for real provider credentials inside application code. Each application or service account holds a scoped key, and you can revoke or throttle it instantly, without touching a deployment.

Partitioning quotas by team, project, and key simultaneously

Enterprise gateways support multi-tier budget enforcement: a per-key limit for each application or service account, a per-team limit for each business unit or product team, a per-project limit for each use case or deployment, and a global hard ceiling for the organization as a whole.

If you tie these budgets to API key metadata, the system can tell consumers apart automatically, with no code changes needed in any application. A developer on a free tier experimenting with the API gets a small daily ceiling. An enterprise production agent gets a ceiling orders of magnitude larger, enforced purely by the metadata attached to the key rather than by anything the application itself has to check. When one of these limits gets hit, the response matters as much as the limit itself: the gateway should return a 429 with a configurable retry-after header, rather than letting the request fail unpredictably once it's already reached the provider.

None of this works as a once-a-day report. The metering pipeline needs to run as a stream processor, not a batch job: token counts arrive per call and get aggregated into per-team, per-tenant, and per-model totals within seconds. Budgets get checked against those live totals, which is what makes a hard stop possible in the moment rather than an alert that arrives after the damage is already done. Comparing locally estimated token counts against what the provider actually reports, and flagging any drift past a meaningful threshold, is what catches a silent quota breach before a user ever notices anything wrong.

How routing decisions interact with rate limit headroom

Rate limit headroom isn't a fixed number sitting untouched until traffic arrives. Routing decisions actively shape how much of a team's token budget gets used and which provider's limits get consumed first. Sending a simple classification task to a lightweight model instead of defaulting to the most capable one available uses a fraction of the tokens, and a fraction of the cost, that the same request would cost on a flagship model. That difference preserves headroom for the tasks that genuinely need the stronger model.

Provider limit structures aren't uniform, and the differences between them can be used to a team's advantage. Anthropic does not count cached input tokens toward input TPM limits for most Claude models, so if you have effective prompt caching in place, a nominal TPM limit can carry far more effective input volume per minute than its raw number suggests. You can also spread load across multiple API keys or multiple deployments of the same model to extend headroom, because each key draws against its own independent quota pool. A gateway can manage that key rotation behind the scenes, so the application itself still sees one single endpoint.

Semantic caching at the gateway layer works as a multiplier on top of all of this. A semantically similar prompt, one that means roughly the same thing as a prior request even if the wording differs, can return a cached response without touching the provider's quota. That stretches effective throughput without raising a single limit. Real semantic caching, compared with simple exact-match caching, can cut redundant model calls by 40 to 70 percent on real-world workloads, Flotorch's analysis found.

The RouteLLM research presented at ICLR 2025 makes the underlying point concrete: a matrix-factorization router that sent only a small share of queries to a strong model achieved a steep cost reduction while holding onto most of the quality, proving that routing decisions and rate-limit headroom are the same lever viewed from two different angles: get routing right, and the budget stretches further without anyone touching the limits themselves.

Retry logic and the thundering-herd failure pattern

A rate limit hit by a single caller is a minor event. A rate limit hit by hundreds of callers that all retry on the same fixed interval is a different kind of problem entirely, known as the thundering herd. Every one of those requests waits the same fixed delay, then retries at exactly the same moment, immediately triggering the rate limit again, and the cycle repeats until the whole system degrades under its own retry traffic.

The fix is random jitter added to each retry interval. A small randomized delay spreads the retries out across time instead of letting them land in sync, breaking the stampede and giving the provider's capacity room to recover between waves of requests. Fallback routing adds a second layer on top of jitter: when one provider is rate-limited or unreachable, the gateway can automatically shift traffic to an alternate provider. That comes with a cost implication many teams underestimate. Failover to a different provider can carry a meaningfully different price per token, and that price difference compounds for as long as the incident runs.

One honest limit applies to all of this: a router can route around a provider failure, but it cannot route around its own failure. If the gateway layer itself goes down, every fallback path built on top of it goes down with it. Critical workloads need a fallback path that bypasses the gateway entirely, direct API calls as a last resort, because no routing layer can back up itself.

Observability that makes rate limiting governable, not just operational

A rate limit that nobody can see in real time is enforcement without accountability. Teams responsible for spend and for compliance need to know what's being consumed, by whom, and against which limit, as it happens, in real time rather than in a report that lands weeks later. The same gateway that enforces the per-key, per-team, per-project, and global ceilings has to surface those numbers to the people who answer for them: finance teams tracking spend against budget, engineering teams debugging why a workload got throttled, and compliance teams confirming that usage policies actually held under real traffic.

The enforcement layer and the audit layer turn out to be the same layer. A gateway that meters tokens per call, screens prompts against policy, and routes across models already has, by necessity, a complete record of every request that passed through it. If you make that record visible in near real time, instead of locking it inside logs nobody checks until an invoice arrives, you turn rate limiting from a technical safeguard into something an organization can actually govern. Concentrate AI is one gateway built around this idea, centralizing token metering and spend limits by team, project, key, model, and provider so that enforcement and visibility sit in the same place instead of being reconstructed after the fact.

Tokens are invisible until the invoice arrives, and usage is invisible until an audit turns it up, unless the system enforcing the limits is also the system reporting on them. That's the gap a token-aware gateway is built to close.