Observability Architecture for Production LLM Traffic

On a Wednesday, a customer-support copilot running at 9,200 requests per second got a prompt template change pushed to it. The P99 latency dashboard stayed green for sixteen hours straight. The regression stayed invisible until Thursday afternoon, when a customer escalated, because the problem lived in one tenant, one model fallback, and one tool call, and the aggregate dashboard had no way to see any of that.

Bolting observability onto individual LLM services leaves production teams blind

The failure in that incident wasn't a missing dashboard. It was a missing capture point. Every service that calls an LLM builds its own logging, its own retry logic, its own sense of what happened on a given request. None of those services can see what the others are doing, and none of them can reconstruct a request's full path once a fallback or a retry sends it somewhere unexpected.

Three gaps make the point concrete. Cost attribution is the first: when each service tracks its own token usage independently, a simple question like "what is our actual monthly LLM cost by feature" requires a forensic investigation instead of a dashboard query. Routing visibility is the second. When retries and provider switches happen inside each service's own retry loop, the calling application has no way to learn which model actually answered; it only knows a response came back. Failure pattern correlation is the third. If a rate-limit spike or a provider degradation event hits multiple services at once, it looks, from inside any single service's logs, like unrelated incidents rather than one upstream cause.

None of these gaps get fixed by a better dashboard. The signals are already scattered before they reach any monitoring layer, so no amount of visualization on top recovers what was lost at the point of capture. Fixing it means moving the capture point itself, not decorating the output of a broken one.

Gateway Control Plane Capture

Position in the request path determines what can ever be observed. A gateway sits between application services and LLM providers, so it sees every prompt going out and every completion coming back, before either one disperses into a provider-specific log that nobody downstream can query. ArchitectureDiagram.ai's 2026 reference describes exactly this placement: a centralized layer handling routing, load balancing, fallbacks, authentication, rate limiting, cost tracking, caching, and observability for all LLM traffic, in one place. The capture point sits at the only location through which all traffic must pass, so the question "what happened on this request" has a single, authoritative answer.

Five structured signals get captured at that layer on every request. Input tokens, cached tokens, and output tokens form the basis for all cost attribution. The model and provider actually used, not the model the application asked for, matters because fallbacks silently substitute one for the other. Latency breaks down into gateway processing time and provider response time, separately. The route taken records which rule applied and which fallback chain fired. Attribution metadata, service ID, user ID, team, and feature, ties all of the above to the organizational unit that produced it.

Almost every gateway speaks the OpenAI-compatible API format, almost every framework expects it, and providers have converged on it as the common contract, making centralized capture feasible without requiring per-provider instrumentation in application code. That convergence is what makes centralized capture technically feasible rather than a theoretical nicety: one layer can intercept all provider traffic without requiring custom instrumentation wired into every application service.

The alternative is instructive. Without a gateway, retry logic, cost controls, model fallbacks, and observability all get implemented independently in every service that calls a model. ArchitectureDiagram.ai frames this as the real tradeoff: route those cross-cutting concerns through one managed layer, and application services get thinner while LLM operations become something a team can actually observe and control. Engineers who lived through the microservices era already know this shape. An API gateway unified HTTP traffic management back then, and the AI gateway is the equivalent standard pattern for production LLM applications in 2026.

Per-request cost telemetry turns token spend from a mystery into a measurable signal

Without per-request cost telemetry, a team's only view of spend is the invoice at the end of the month, by which point a runaway workflow has already run up the bill. Capturing cost at the gateway changes the timeline completely.

Attribution becomes immediate rather than archaeological: by team, project, API key, model, and provider, in real time, not reconstructed weeks later from scattered logs. Anomaly detection becomes possible in the same way application performance monitoring catches latency regressions. A cost spike on a specific route appears in gateway data within hours of happening, not buried inside a monthly invoice that arrives after the damage is done. And enforcement stops being a policy document that nobody checks. The 2026 LLM Gateway Design Guide describes virtual API keys and per-team or per-feature quotas enforced directly at the gateway as the mechanism that keeps a runaway agent loop from turning into an overnight bill catastrophe.

The stakes rise because models within the same family can price out very differently. Intuition Labs' routing analysis found Anthropic's Opus 5 costs five times its Haiku 4.5 on both input and output tokens. A fivefold cost gap between two models from the same provider means the model that actually served a request determines whether a given feature is operating within budget or burning through it. Capturing that fact per request, rather than inferring it after the invoice arrives, is what turns token spend from something a finance team discovers in arrears into something an engineering team manages in real time.

Routing decisions as first-class observability events, not infrastructure side effects

The same per-request capture that makes cost attribution possible also makes routing auditable, and routing needs that audit trail more than cost does, because routing decisions change the shape of the response itself. A fallback that fires without being logged means the application receives output from a different model than it expected, carrying different quality, different latency, and a different cost profile, with no trace left behind that a switch happened.

Production systems route requests through three strategies, and each one carries its own observability burden. Rules-based routing, the kind that says "use Haiku for classification, Sonnet for chat," is deterministic and easy to debug, but only if the rule that actually applied gets logged alongside the request. Dynamic routing, which picks the cheapest model that still clears a quality bar, adds a layer of unpredictability: the same request can get served by different models across separate calls, so the routing decision itself has to be captured as a variable, not assumed constant. Hybrid routing combines both, so when a team traces a regression, it needs to know which layer made the final call, the fixed rule or the dynamic selection, before it can fix anything.

Fallbacks make this concrete. Digital Applied's 2026 reference identifies three distinct fallback categories: general fallbacks triggered by timeouts or 5xx errors, content-policy fallbacks, and context-window fallbacks. Each one routes to a different provider list, so you get different output characteristics from each. Without a log of which category triggered and which provider actually answered, a quality regression doesn't look like a fallback. It looks like unexplained output drift, a problem a team chases for days because nothing in the logs points to the actual cause.

The objection that comes up here is familiar: "we know which model we're calling, it's in the code." But that holds only until a fallback fires, or a provider swaps the model behind an existing version tag, or a routing rule changes without a full deployment. At that point the code is no longer a reliable record of what happened. The gateway log is the only place that records what actually served each request, as opposed to what the application assumed would.

Latency observability at the gateway layer, separating provider slowness from application problems

An aggregate latency number mixes together two problems that have nothing to do with each other: the provider is slow, or the application has a regression. Those two problems call for completely different responses, and a single end-to-end latency metric cannot tell a team which one it's looking at. Only decomposing latency at the gateway separates them.

Time to first token from the provider measures the provider's own responsiveness, independent of anything the application is doing. Gateway processing overhead, the time spent on routing decisions, cache lookups, and PII redaction, should stay negligible, and it's measurable as exactly that: the DEV Community's 2026 gateway comparison cites published benchmarks showing added gateway latency at the microsecond level even under high concurrency. Cache hit versus cache miss is its own distinct signal. Digital Applied's 2026 reference reports a Cloudflare finding that cache hits cut latency dramatically by serving from the edge instead of round-tripping to the upstream provider, so a cache hit rate that drops is a latency regression with a cause that has nothing to do with the provider's performance that day. Streaming responses add a fourth layer: server-sent-event passthrough has a different latency profile than a synchronous response, and gateway instrumentation built to handle streaming correctly captures time-to-first-chunk as its own metric, separate from total completion time.

MLflow's 2026 observability guide names distributed tracing with span hierarchies as the foundation: without spans wrapping each LLM call, each tool invocation, each retrieval, and each internal reasoning step, a team cannot reconstruct what happened during a multi-step task, and traditional log-based debugging simply isn't built for non-deterministic LLM failures, which is what the Wednesday incident needed and didn't have. The regression lived in one tenant, one model fallback, one tool call, and the only artifact that could have surfaced it that same afternoon was a span tagged with those attributes alongside an evaluator score that had dropped below threshold.

None of this is a hypothetical edge case that teams can deprioritize. BuildMVPFast's fallback guide reports that average API uptime across all providers fell between Q1 2024 and Q1 2025, and that rate limits accounted for roughly a third of LLM errors in production telemetry tracked by Datadog. Provider latency spikes and rate-limit-triggered retries happen so often that a team without gateway-level decomposition flies blind regularly, not just now and then.

Turning gateway signals into real-time alerts rather than post-incident archaeology

Capturing cost, routing, and latency as structured signals at the gateway makes the system visible, but a signal sitting quietly in a log store doesn't help anyone until someone or something is watching it.

Turning capture into alerting means treating each signal type the way the sections above describe it: cost anomalies checked against per-team and per-feature baselines in hours rather than at the next invoice, fallback events fired as named occurrences rather than buried inside an aggregate error rate, and latency decomposition watched continuously so a dropping cache hit rate or a rising time-to-first-token gets flagged the moment it happens, not the moment a customer escalates. For sixteen hours, that dashboard stayed green, and for all sixteen of them, the data needed to catch the regression already existed somewhere in the system. It just wasn't wired to anything that would say so.

Concentrate AI approaches this by positioning itself as the single point through which prompts, completions, token counts, and routing decisions all pass, including which model actually served a request when a fallback fires, so that cost attribution and real-time alerting have a structural foundation to sit on rather than requiring per-service instrumentation stitched together after the fact. That's one option among a field of gateway products built around the same architectural premise: that observability for production LLM traffic has to be captured at a single point in the request path, because any other location leaves gaps that no dashboard built afterward can close.