LLM Provider Outage Response Playbook for Production Teams

LLM provider outages break production systems in ways that ordinary API failures don't, because the failure is often partial, sometimes correlated across providers that were supposed to be independent, and frequently invisible until users are already affected. On September 3, 2026, OpenAI, Anthropic, and xAI all showed degraded service within the same few-hour window, while Google's Gemini had reported issues without a confirmed outage. That single afternoon is worth building a playbook around, because it shows that redundancy across providers doesn't guarantee independence when those providers share cloud regions, network paths, and the same accelerator supply chains.

LLM provider outages hit harder than other API failures

Most API incident playbooks assume a clean signal: the endpoint is up, or it's down. LLM providers don't fail that cleanly. Partial degradation, such as elevated p95 latency, intermittent tool-call failures, or silently truncated responses, is harder to detect than a clean 503, and teams relying on binary health checks miss it entirely. Latency creeps upward, tool calls fail intermittently, and none of it trips a binary alarm until a customer notices first.

Partial failures also leave the application mid-state: a REST read can be retried without consequence, but an agentic workflow that stalls halfway through a multi-step chain leaves the application in a state a simple retry doesn't repair. Tool-calling adds a second layer of failure: the retry logic that covers the LLM call itself says nothing about the tool call embedded in that response, the one that executes against the deployer's own systems.

Scaled across a fleet of agents, this cadence stops looking like an edge case. Anthropic logged 114 incidents across a 90-day window in early 2026, and OpenAI's 99.76% uptime figure works out to roughly 21 hours of downtime a year. At that frequency, every production team running on these providers will hit an outage. Whether the response is planned or improvised is the difference that decides how the incident goes.

The four phases every outage runbook must cover

A working runbook breaks into four phases, run in order: detection, failover execution, stakeholder communication, and post-mortem, and skipping one or letting two blur together leaves improvisation to fill the gap.

Detection has to fire before failover can execute, because a system can't route around a problem it hasn't identified yet. Failover needs to happen without engineers hand-editing application code mid-incident. Communication runs in parallel with both, keeping stakeholders aligned instead of guessing. Post-mortem closes the loop, turning what just happened into a sharper runbook for next time.

Teams that fold detection and failover into a single fuzzy step, or that skip communication because "the engineers already know," tend to run longer incidents. That's not a coincidence: engineers working without shared visibility into what's already been tried start making conflicting interventions, and each one adds recovery time rather than cutting it. Each phase below gets its own treatment, because each one has its own specific failure modes and its own fixes.

Detection: how to know a provider is down before your users do

Asking "is the endpoint reachable" answers the wrong question. Production LLM systems fail through quality loss long before they fail through unreachability, and a binary health check built around uptime alone won't catch that slide.

The metrics that matter are p95 latency, error rate, and functional output integrity, beyond HTTP status codes. Functional output integrity means actually checking whether a response is coherent and complete, not just whether the status code came back clean, because a provider can return 200 while handing back a truncated or malformed answer.

Sustained elevation in p95 latency should serve as an early warning. Error rate crossing a set threshold should trigger failover. Total unresponsiveness should trip an emergency fallback. Treating these as one undifferentiated "provider is having problems" alert means either overreacting to noise or underreacting to a slow slide into failure.

Agentic pipelines need tighter thresholds than a single-turn chatbot, because one user action there can trigger a whole chain of LLM calls, and latency that creeps upward at each hop compounds across the chain.

Status pages lag behind reality, sometimes badly. Anthropic's outage on April 6, 2026 ran for ten hours and stalled enterprise workloads worldwide, and teams that waited on a status page update for confirmation absorbed the entire duration before acting. Detection has to run on active synthetic probes hitting each provider at a short interval, independent of live user traffic and independent of whatever a provider's own status page says. Once that detection is running, it needs somewhere to send its signal: a failover path that's already configured and ready to take traffic.

Failover execution: routing traffic away from a degraded provider without touching application code

Recovery speed comes down to whether the fallback chain was built and tested before the incident started rather than improvised during it. That work belongs at the gateway layer, sitting between the application and the providers, not scattered across application code that has to be edited under pressure.

Retries handle transient errors inside a single provider, things like a network blip or a momentary rate-limit hit, and should stop after two or three attempts with exponential backoff. Fallbacks exist for a different situation entirely: the primary provider is confirmed degraded, and traffic needs to move to a different one. Retrying indefinitely against a provider that's actually down delays the moment someone realizes a fallback is needed, and every one of those retries extends the time users spend looking at a broken feature.

A correct fallback chain pre-specifies the primary provider, an ordered list of fallback providers, circuit-break conditions for each, and a non-AI fallback path for mission-critical processes where no LLM substitute is acceptable. Without a non-AI fallback path, mission-critical processes have no safety net: a support workflow that can drop to a cached response, a simplified rule-based answer, or a human escalation degrades gracefully. One with no such path just breaks.

That regional and network diversity isn't a nice-to-have. The September 2026 correlated outage showed that fallback targets sharing infrastructure with the primary, even under different provider names, can fail together.

Tool-calling adds a second layer that's easy to overlook. Fallback logic needs to cover both the LLM call and the tool execution sitting inside it, and a gateway that only handles LLM failover while leaving tool-call retries to application code leaves a gap right where things break. And weighted load balancing, sending some live traffic to fallback providers under normal conditions, keeps those paths warm, so the first real request they see during an incident isn't also the first request they've seen all month.

None of this holds up unless it's been tested. A fallback chain that looks correct in a config file and has never actually been triggered is an assumption, not a safeguard, and a post-mortem needs to check that gap.

Monitoring at the gateway layer for trustworthy failover decisions

A failover decision made on thin signal can do as much damage as no failover. Routing away from a provider that's only mildly degraded wastes capacity for no reason. Stay on one that's quietly returning bad output and users feel it for longer than they should.

Token-level observability makes reliable failover decisions possible, since it supplies the per-token counts and provider-level attribution those decisions depend on. Input tokens, output tokens, cache-read, cache-write, and reasoning tokens each need their own count, because collapsing them into a single combined number erases the cost and performance detail that routing decisions actually depend on. Attribution matters just as much: breaking usage down by provider, model, and virtual key means a degradation event on one provider doesn't get buried inside an aggregate dashboard alongside providers that are running fine.

Real-time dashboards and end-of-month invoices answer different questions. Live cost figures from an observability layer are estimates built from token counts and published rates, useful for making a routing call in the moment, but they're not a substitute for finance-grade reconciliation after the fact. The runbook should say which signals drive automated decisions and which ones need a human to sign off first.

OpenTelemetry and Prometheus integration connects gateway-level signals into existing APM stacks so incident responders are not context-switching between dashboards during an outage. Spend-rate anomaly detection deserves its own alert too, separate from latency and error rate, because a runaway retry loop can burn through a month's budget in minutes before anyone notices the pattern. And every routing decision made during the incident, the timestamp of each circuit-break, which fallback served which requests, needs to land in an audit log. None of this can be reconstructed from memory once the incident's over, which is exactly the data the post-mortem depends on later.

Communication during an outage

Technical failover can work perfectly and the incident can still run long, because engineers debugging in isolation, product teams escalating through separate channels, and executives getting conflicting updates all add friction of their own. That's a coordination failure sitting on top of a technical one, and it extends recovery time independent of whether the gateway did its job.

Three stakeholder groups need three different kinds of update, not one broadcast message sent to everyone. Engineering needs the technical state: which provider is active, what failover mode the system is in, whether any manual step is required. Product and customer-facing teams need to know which user-facing features are degraded and what to tell customers who are asking. Finance and security need to know whether the incident touches spend or compliance, since an unexpected reroute to a different provider can brush up against data residency rules or chew through a budget allocation faster than planned.

Draft templates for each of these audiences ahead of time, parameterized by severity, so the incident commander isn't composing messages from scratch while also managing the technical response. One person should own external communication during the event, named by role rather than by individual, so the runbook still works after someone changes teams.

Cadence carries as much weight as the content of the message. Stakeholders who stop hearing updates tend to assume the worst and escalate on their own, and a short "still investigating, next update in 30 minutes" beats silence every time. Finance and security are often left out of the loop entirely during a technical incident, but a failover that reroutes sensitive prompts to an unfamiliar provider can raise a residency or compliance concern that needs eyes on it immediately, not after the fact. Once the provider recovers, a separate after-action summary, written in plain language for non-engineering stakeholders, should cover what happened, what it affected, how long it ran, and what changes will prevent it next time.

Post-mortem: what to review after the provider comes back online

A post-mortem earns its keep by making the runbook faster the next time, not by producing a record of blame. That means checking the specific decisions and configurations that got tested for real during the incident, not writing a generic retrospective.

Four questions need answers. Did detection fire before users reported the problem, or after, and if it fired late, what threshold change would have caught it sooner? Did the fallback chain execute the way it was configured? If it didn't, what blocked it, stale credentials, a fallback provider that was cold, or a hole in the tool-call retry logic? Was the fallback provider genuinely independent of the one that degraded, or did the September 2026 pattern repeat, with shared infrastructure dragging both providers down together? And did each stakeholder audience get the right communication template at the right moment?

Gateway audit logs carry most of the weight in answering these questions. Routing decisions, circuit-break events, provider switch timestamps, and token spend by provider during the incident window build a timeline that's far more precise than whatever engineers remember or whatever got typed into a Slack thread in the moment. Chaos engineering closes the remaining gap: the only way to know a fallback chain actually works before a real incident is to trigger a circuit-break deliberately in staging and watch whether the fallback behaves the way it's documented to.

Every post-mortem should produce a dated revision of the runbook, with updated thresholds, revised fallback chains, and sharper communication templates, so the document reflects where the team actually stands rather than where it stood when the runbook was first written. If a mission-critical process served a degraded-mode response during the incident, that response deserves a second look: was it acceptable, and were users told clearly that they were getting it?

The gateway layer turns these four phases from manual to managed

The gap between a team that recovers in minutes and one that burns hours usually has nothing to do with how well-written their runbook document is. It comes down to whether the infrastructure was already configured to execute that runbook the moment its conditions were met.

Detection stops depending on an engineer noticing a latency graph once health checks are degradation-aware and wired directly to circuit-break triggers at the gateway. Failover stops requiring a code deploy once the fallback chain lives at the gateway layer, since the application keeps sending requests to the same endpoint while the gateway makes the routing call underneath it. Communication gets easier to keep accurate once the audit trail, provider switches, circuit-break timestamps, spend by provider, comes from the same gateway logs rather than from someone's recollection of what happened at 2 a.m. And the post-mortem gets sharper once that data already exists, rather than needing to be reconstructed after the fact.

None of the four phases disappears. What changes is who's doing the work in the moment: the gateway handles the mechanical decisions it was configured to make, and the humans on call spend their attention on the judgment calls the runbook couldn't anticipate.