HTTP 429 Errors From LLM Provider APIs

HTTP 429 from an LLM provider API is rarely what it first appears to be. One status code carries several distinct conditions: rate limits, exhausted billing budgets, and provider-side capacity failures. Each demands a different response, and the most common production mistake is treating them all as the same problem.

Why HTTP 429 from LLM APIs is harder to handle than it looks

A single 429 from OpenAI can indicate six separate conditions. The request rate may be too high, traffic may have increased too quickly, the credit balance may be empty, the organization spend limit may be reached, the project spend limit may be reached, or the organization usage limit may be exhausted. Only the first two are rate-limit problems. The other four are billing states. Exponential backoff will never clear them. The error looks identical at the HTTP layer, but the correct response is completely different.

Anthropic complicates the picture further. It enforces three independent rate-limit dimensions per model class: requests per minute, input tokens per minute, and output tokens per minute. A 429 can fire even when your request count is low. One large prompt can saturate the token budget on its own. On top of that, Anthropic has a separate 529 overloaded_error. It fires when Anthropic's own infrastructure is saturated, regardless of your tier. A naive handler sees a retryable error and backs off, but the correct response is failover.

The core mistake is assuming the status code is self-explanatory. It is a container, not a diagnosis. Reading the error body, the headers, and the surrounding context is the only way to know which 429 you are dealing with.

How rate-limit windows actually work across providers

Rate-limit windows are continuous token buckets, not hard resets. A 60 RPM limit is effectively one request per second with burst headroom. It is not a cliff that resets at the top of each minute. Hitting the limit mid-burst does not mean the full window must expire before you retry. The bucket replenishes continuously, and a small pause may be enough.

Anthropic ties rate limits to cumulative credit purchases through its Build Tier system. Three tiers exist: Start, Build, and Scale, plus a Custom tier. Your ceiling is set by spending history, not just your current plan. At Tier 1, Opus 4.x, Sonnet 4.x, and Haiku 4.5 each begin at 50 RPM with tightly bounded token ceilings. At Tier 4, Opus 4.x and Haiku 4.5 reach dramatically higher RPM and ITPM limits. The gap between tiers is substantial.

One detail catches teams by surprise: Opus 4.x limits are pooled across all Opus 4.x versions. Opus 4.8, 4.7, 4.6, and 4.5 share one bucket. There is no fresh limit per version. If you assume version-level isolation, you will overcommit and hit 429s without understanding why.

Prompt caching offers a real lever for raising your effective ceiling. For most current Anthropic models, cache_read_input_tokens do not count toward ITPM. Claude Haiku 3.5 is the exception. There, cached input tokens still count. But for supported models, heavy use of prompt caching can expand throughput without a tier upgrade.

Every provider returns rate-limit headers on 429 responses. These tell you exactly where you stand: remaining requests, remaining tokens, reset timestamps. Reading them before retrying is the difference between informed backoff and a blind retry storm. OpenAI returns headers like x-ratelimit-limit-requests and x-ratelimit-remaining-tokens. Anthropic returns a richer set, including anthropic-ratelimit-requests-remaining and anthropic-ratelimit-input-tokens-reset. The Retry-After header is not universal. OpenRouter documents it on 429, 503, and 402 when the limit source is an in-flight budget. Anthropic SDKs honor it when present. OpenAI guidance is to respect it when it exists. Your retry logic needs a sensible default for when the header is absent.

One more trap: failed requests still consume quota at some providers. A request rejected with a 429 may have already consumed one request slot and the tokens it carried. Repeated naive retries actively worsen your position. Each failed attempt costs quota while producing no output.

Why billing-triggered 429s never clear on their own

A 429 that reflects an exhausted credit balance or a breached spend limit will not clear from waiting. No backoff schedule restores service. The error persists until the billing condition is resolved. OpenAI surfaces six distinct 429 conditions. Four of them, credit balance exhausted, organization spend limit reached, project spend limit reached, and organization usage limit reached, are billing states, not throughput states.

Retrying a billing-triggered 429 is actively harmful. Each request potentially consumes tokens toward other quota dimensions. You add cost to an account that is already blocked. The standard retry playbook does not just fail here. It makes the situation worse.

Cloudflare AI Gateway added spend limits in June 2026. These are dollar budgets over rolling or fixed windows, scoped by model, provider, or custom metadata dimensions. When a rule trips, requests are blocked at 429. A gateway-layer spend limit that fires 429 is distinguishable from a provider-layer 429 by the caller's routing context. The difference matters for diagnosis.

Detection requires active monitoring. Count 429s by error subtype or message. Watch token spend per tenant in real time. Read the error body. A spend-limit 429 carries a different message string than a rate-limit 429. Treating them the same leads to alert fatigue and invisible outages.

How the wrong retry strategy turns a brief limit into a cascade

The thundering-herd problem is the classic failure mode. Multiple clients all receive a 429 and retry at the same interval. They hit the limit again simultaneously at the next window boundary. The result is a self-reinforcing cycle, not a recovery. Every round of retries lands on the same exhausted window.

LiteLLM's default configuration illustrates how easily this happens. As of v1.40+, it retries failed requests up to 2 times by default, using exponential backoff and a 5-second cooldown on 429 responses. It does not enforce a fixed per-model send rate. Standard OpenAI Tier 1 accounts cap at 3 requests per second. LiteLLM's out-of-the-box settings exceed that ceiling. The proxy does not query each provider's rate-limit endpoint before sending. It assumes unlimited capacity until it hits a wall. The result is a 429 storm for a large share of new deployments using defaults.

Retries also inflate cost. If the provider is genuinely constrained, every retry is a billable request that fails. You stack spend on top of a blocked account. The wrong retry strategy does not just waste time. It costs money and deepens the outage.

Exponential backoff with jitter for genuine rate-limit 429s

For a genuine rate-limit 429, the correct pattern is exponential backoff with randomized jitter, anchored to the Retry-After header when it is present. This prevents thundering-herd synchronization and respects the provider's window. The two-part rule is simple. Honor Retry-After as the floor wait time. Add random jitter on top so concurrent clients do not synchronize their retry moments even when they all received the same header value.

Anthropic 529 overloaded_error needs a different approach. It has no guaranteed Retry-After. Use pure exponential backoff with jitter and a cap on the retry count. Then fail over. Retrying indefinitely blocks the request and delays the user-visible response. The failover is the actual fix.

For LiteLLM specifically, switching the retry policy from the default exponential to token_bucket prevents request pile-up. Set explicit rpm and tpm per model at a conservative fraction of the actual API key quota. Add max_retries and retry_after_timeout caps as a baseline. The token_bucket policy paces requests before they leave the proxy.

Proactive rate-limit tracking goes further. Count request timestamps and token usage in a local sliding window before sending. This prevents 429s rather than reacting to them. In high-throughput applications, it eliminates the wasted round-trip latency of a failed request entirely. You do not pay for a rejection you never trigger.

Why rate-limit 429s call for provider failover, not more retries

When a rate-limit 429 signals that a provider's quota window is exhausted, more retries waste time that other providers could fill immediately. Failover is faster and more reliable than waiting. Rate limits cascade. A burst of traffic exhausts quota on one provider, and every subsequent request fails even though other providers have available capacity at the same moment. Your application is not out of options. It is out of options on a single endpoint.

LLM providers are materially less reliable than traditional cloud SaaS infrastructure. OpenAI experienced a major recorded outage in June 2025 spanning many hours across ChatGPT, the API, and Sora. When one provider is down or exhausted, a fallback route is not a luxury. It is the only path to availability.

The quality gap in fallbacks is an underplanned risk. A fallback that succeeds at the HTTP level is not automatically a good outcome if the fallback model produces noticeably shorter or lower-quality responses. Teams need to select and validate fallback models on output quality, not just availability. Failover that silently degrades your product is still a failure.

What a gateway does that application-level retry logic cannot

Application-level retry code handles one connection at a time. A gateway handles all provider connections simultaneously and can make routing decisions, failover, load balancing, spend blocking, before a 429 ever reaches application code. Put a gateway in front of multiple providers, and a 429 from one provider triggers an immediate route to another without a code change. The event is logged in the same place as every other request. Visibility and recovery happen in one operation.

Spend-limit 429s specifically require gateway-layer enforcement. A budget rule that fires a 429 at the gateway when a team or project hits its dollar ceiling stops runaway spend before the provider invoice arrives, not after. Centralized retry and fallback policy removes per-team guesswork. Every application key on the gateway inherits the configured backoff curve, fallback chain, and provider health checks. Behavior stays consistent across all services, so no team has to build its own retry logic.

The gateway also surfaces the data needed to distinguish 429 types. Request counts by error subtype, token spend per tenant, quota utilization by model and provider, all in real time rather than reconstructed from end-of-month invoices. That visibility is the foundation of the monitoring baseline the next section describes.

The managed versus self-hosted decision carries its own 429-relevant risk. A self-hosted gateway that is itself misconfigured, as LiteLLM's defaults illustrate, can generate 429 storms on behalf of every application behind it. Self-hosted options like LiteLLM support a wide range of providers and are a strong fit for teams that want full control. The trade-off is that correct per-model rpm and tpm configuration and retry policy selection fall entirely on the operator. Bifrost by Maxim AI is a Go-based open-source gateway benchmarked at very low overhead and high throughput with automatic failover and semantic caching. Portkey, now part of Palo Alto Networks' Prisma AIRS portfolio, offers managed and hybrid gateway options with conditional routing, guardrails, and organization-wide governance.

Cloudflare AI Gateway provides edge-routed management with spend limits added in June 2026, caching, and analytics. For teams already on Cloudflare, it adds zero additional ops. A fully managed gateway, one you connect to by swapping a base URL with no proxy to deploy or maintain, lets engineers ship product instead of operating retry infrastructure. Provider-specific rate-limit knowledge stays in the infrastructure layer rather than scattered across application codebases.

Building a 429-aware monitoring baseline in production

Catching 429 conditions early requires an active monitoring setup, not reactive log review. The signals are available at request time. They should drive dashboards and alerts, not post-incident analysis. Three categories of signal matter in real time.

First, capture HTTP 429 response counts segmented by error subtype: rate-limit versus spend-limit versus capacity, and by model and provider. A spike in billing-triggered 429s is then immediately distinguishable from a rate-limit burst. Second, watch token spend per tenant, project, and key. An anomalous spend trajectory should trigger an alert before a hard cap is hit. Third, track quota utilization as a percentage of each provider tier's limits. A team operating well into their ITPM headroom knows a traffic spike will produce 429s before it happens.

Circuit breakers complement backoff and failover. After a configurable number of consecutive 429s from a single provider, stop sending to that provider for a defined window. Open the circuit. Do not keep accumulating failed retries against an exhausted endpoint. This is one more signal-driven response, but the monitoring baseline is the primary piece.

The practical outcome of getting this right is a shift in posture. Teams that monitor 429 by subtype and maintain a real-time spend view stop treating provider outages and billing surprises as unpredictable events. They start treating them as observable, manageable infrastructure conditions. A 429 becomes a signal with a known response, not a mystery error.