API Gateway Timeout Configuration for LLM Streaming Requests

The timeout defaults baked into that world assume the same thing every time: compute a full response and return it in one shot. LLM inference doesn't follow that script, and it breaks the old assumptions in three ways. First, duration: a single prompt can take anywhere from a couple of seconds to several minutes to finish generating, depending on how long the output runs and how much reasoning the model does along the way. The response itself is a live stream of bytes that has to move chunk by chunk, and any gateway or proxy that buffers it quietly turns the experience back into a full wait, no matter how the backend behaves. Third, a model can sit silently "thinking" before it produces a single token, and that silence looks, on the wire, exactly like a dead connection. AWS's own framing of this problem is direct: before it added response streaming to API Gateway, users were waiting tens of seconds for AI agent responses, an experience AWS itself describes as disconnected and unnatural. Gateway timeout errors appear in production on a fraction of requests once a team deploys behind a gateway, because the gateway's default timeout is shorter than how long the model actually takes to finish any moderately complex prompt.

Streaming is not a flag you flip, it is an end-to-end property of the entire request path

A backend that streams its tokens correctly does not guarantee that a user ever sees a stream. Any layer sitting between the model and the browser, a gateway, a proxy, a CDN, a web application firewall, a load balancer, can quietly buffer the whole response and release it as one block, and nothing in that chain has to throw an error to do it. A typical production path runs from client to CDN to WAF to gateway to load balancer to application to model, and every hop in that chain can buffer the response, enforce its own timeout, or close the connection early.

There are four specific places where streaming tends to die without anyone noticing right away. The gateway itself might default to buffered mode: AWS API Gateway, for instance, ships with BUFFERED as its default transfer mode, and a team has to explicitly switch it to STREAM to get incremental delivery at all. A reverse proxy or CDN sitting in front of the gateway might inspect, compress, or cache the response on its own terms, which reintroduces buffering even after the gateway has been configured correctly. A timeout can fire somewhere in the middle of the stream, and because there are several different timeout clocks running at once, the one that fired is usually not the one an engineer would guess. And sometimes the backend was never really streaming to begin with: it buffers the full response internally and then drains it out in one motion, which produces total latency numbers that look completely normal in a dashboard while quietly destroying the thing that actually matters to a user, time to first token.

The practical conclusion is that fixing streaming is not a single setting a team flips once and forgets. Every layer in the path, from CDN to WAF to gateway to load balancer, has to be checked and explicitly configured to preserve the stream, because any one of them left on default behavior can undo the work done everywhere else.

The four timeout clocks that govern a streaming request

The most common mistake teams make when debugging streaming failures is treating "timeout" as a single setting, when in fact four separate clocks run at once on every request, each measuring something different. If you operate an LLM gateway in production, this is the single most useful mental model you can have.

The first clock is connection timeout: how long the gateway waits to open a TCP connection to the model provider. This one should stay short, because a slow connection is a networking or routing failure, not a sign that the model is working hard. The second clock is the integration timeout, sometimes called the backend timeout, which measures how long the gateway waits for the upstream to start responding. This is usually what people mean when they say "the gateway timed out," and on several platforms it's a hard ceiling that no configuration change can raise. The third clock, and the one that matters most for streaming specifically, is the idle timeout: how long the gateway will tolerate silence between chunks once a stream is already open. If a model is reasoning quietly before it produces its first token, this is the clock that fires, even though generation is proceeding exactly as it should. The fourth clock is the total request timeout, an end-to-end ceiling running from the moment the connection opens to the moment it closes. Not every gateway enforces this one. A documentation audit of one open-source gateway's codebase found that its advertised end-to-end ceiling wasn't enforced anywhere in the code: the limits that did apply were scoped per upstream attempt rather than per session.

For a streaming request, the idle timeout is the clock that should govern behavior, not the total timeout. A stream should stay open as long as tokens keep arriving, and it needs an idle clock that trips only on genuine silence, not on a model that's mid-thought. For endpoints that don't stream, the total timeout is what matters, and it needs to be set generously for standard models and more generously still for reasoning models, which pause before they start producing visible output. If an endpoint streams, the idle gap between chunks should run between 30 and 60 seconds, so it can survive a model's thinking pause but still catch a connection that's actually dead. The backend's own timeout has to be set higher than the gateway's timeout, because if the gateway gives up first, the model keeps running and the provider keeps billing for a response the client will never receive.

Gateway platform enforcement and ceilings

Timeout behavior is not standard across providers, and several platforms enforce hard ceilings that no amount of configuration can move. The first question any team should ask before building around a gateway is what that gateway's hard limit actually is, since that answer decides whether the fix is a configuration change or a change in architecture.

| Platform | Streaming support | Timeout ceiling | Notes | |---|---|---|---| | AWS Application Load Balancer | Supported | Hard idle-timeout ceiling at 4,000 seconds | Any session longer than the ceiling is cut regardless of configuration, which makes an application-level heartbeat necessary | | AWS API Gateway | Supported, but not on by default | Defaults to BUFFERED transfer mode | Must be explicitly switched to STREAM mode to deliver tokens incrementally | | NGINX (self-hosted) | Supported, fully configurable | No hard ceiling | Requires explicit changes: proxy_buffering off and proxy_read_timeout set to 600 seconds or higher, longer still for reasoning models, because default buffering behavior silently turns a stream back into request-response | | Managed LLM gateway (per-attempt model) | Supported | Streaming completions get a substantially longer window per attempt than non-streaming calls; every retry or fallback opens a fresh window | No separate end-to-end timer sits on top of the per-attempt limits |

The per-attempt model used by some managed gateways works out well for agentic systems. A coordinator making many sequential completions over the course of hours is unaffected by per-attempt limits, since only a single attempt that itself exceeds the limit will fail. Tool execution time doesn't count against the clock either: the time between a completion that returns tool_calls and the next completion isn't part of the timeout window, because the HTTP request itself ends the moment the completion finishes. That distinction matters for anyone building multi-step agents on top of a gateway with per-attempt limits, since it means long tool calls don't silently eat into a budget meant for model generation time.

NGINX and self-hosted proxy configuration for LLM streaming

NGINX sits in front of more LLM backends than any other reverse proxy, and its out-of-the-box settings will buffer a streaming response and fire a timeout on a long-running request unless someone goes in and changes them. Two changes matter more than any others. The first is proxy_buffering off. Without it, NGINX collects the entire response before passing any of it along, so even if the backend is streaming tokens correctly, the user sees nothing until generation finishes entirely. The second is raising proxy_read_timeout well past its default, since this is the directive that governs the idle gap between bytes arriving from the upstream, and it's the clock that actually matters for a streaming connection, while proxy_connect_timeout governs connection establishment to the upstream and should stay short, around the same order as a typical TCP handshake. NGINX exposes three distinct timeout directives that need to be set separately: proxy_connect_timeout, proxy_send_timeout, and proxy_read_timeout.

And proxy_read_timeout, the idle clock on bytes coming back from the upstream, is the one that needs the most generous value: an NGINX+LLM guide uses 600 seconds as its minimal streaming-safe value, with other LLM proxy guides recommending 3,600 seconds, since 300 seconds is cited in generic, non-LLM examples and as a floor rather than the recommended value for reasoning-model workloads.

A location block for an LLM streaming endpoint typically sets proxy_buffering off, proxy_read_timeout to 600 seconds or more, proxy_send_timeout to match, and proxy_connect_timeout to a short value measured in single-digit seconds. The two changes that matter most. Both are configuration problems, so they aren't signs that the model itself is behaving badly.

Graceful shutdown and rolling deploys: the timeout problem that only surfaces at scale

Everything covered so far assumes a gateway that stays running. A gateway that's been configured correctly can still cut a long-running stream in the middle of a response during a routine rolling deploy, if the pod's termination grace period is shorter than the longest stream the gateway is supposed to support. This is a failure mode that stays invisible until a team is running real deploys against real production traffic, because nothing about it appears in a staging environment with light load.

A pull request from NVIDIA's NVCF gateway project tackles this shutdown behavior and lays out the failure chain in detail. On receiving SIGTERM, the gateway drained in-flight requests for at most the duration of its WriteTimeout setting. The Kubernetes chart deploying it never set terminationGracePeriodSeconds, so Kubernetes used its 30-second default, killing the pod before the 60-second drain window could complete. On top of that, several dependencies, the auth client, rate-limit sync, and a component called Olric, were stopped through shutdown hooks that ran at the same time as the drain itself. When rate-limit sync was turned on, a request that finished during that drain window tried to publish an event to a synchronizer channel that had already closed, which caused a panic that crashed the whole gateway and took down every other in-flight request along with it.

The general rule is that terminationGracePeriodSeconds must be at least as long as the longest stream the gateway permits, plus the time needed for dependency teardown, and managed gateway documentation states the same requirement for self-hosted deployments: load balancer backend timeout and Kubernetes grace period must both be raised in lockstep when streaming timeout limits are raised.

Why raising timeouts alone makes the underlying problem worse

Setting every timeout to a very large number gets rid of 504 errors, but it trades them for a different set of production problems: connections pile up, failures take longer to surface, and retries start compounding on top of each other. Every request that's still pending holds a connection open, so longer timeouts mean more concurrent connections sitting in memory, and that eats into connection pool capacity under real load. If a provider genuinely goes down, a 300-second timeout means 300 seconds of a user staring at a hung request before anything fails visibly, compared to a short, correctly tuned timeout paired with a fallback route that kicks in fast. Clients that get a timeout error tend to retry automatically, and a request that hangs for a long stretch before finally timing out invites a retry that doubles the load on a provider that was already struggling, turning one slow request into a retry storm. In multi-step agent systems, a single timed-out step can fail an entire workflow, and the tokens spent getting to that failed step are still billed by the provider even though the client never got to see the output.

LLM inference breaks this in at least three distinct ways. It takes streaming so the user can watch progress happen, idle-timeout logic that can tell silence from genuine generation, and fallback routing so a slow or failed provider never gets to hold a connection open indefinitely.