LLM Gateway Architecture Fundamentals

The moment a team starts calling more than one AI provider, the work of managing that access grows faster than any group of engineers can track by hand. One model handles customer support, another handles coding assistance, a self-hosted model takes on sensitive workloads, and a cheaper provider runs high-volume classification, and each of those relationships comes with its own pricing, its own latency, its own rate limits, and its own way of failing.

Without something tying these together, every team ends up managing its own provider keys, its own SDK integrations, its own retry logic, its own rate-limit handling, and its own usage reporting, each in isolation from the next. Nobody above the team level can see total spend. No two teams handle a provider outage the same way. Governance becomes a patchwork of whatever each team happened to build.

The real trigger for change is the point where real money starts moving through AI calls, because that is exactly when cost attribution, rate enforcement, and reliability stop being nice-to-haves and start being requirements. Once budgets are on the line, "each team does it their own way" is no longer an acceptable answer. That is the problem a gateway exists to solve.

An LLM Gateway

An LLM gateway sits between applications and the AI providers they call. People often describe it as a reverse-proxy layer, and technically it is, but that description undersells the job it does: it's the point where routing, authentication, policy, cost, and security decisions actually get made, rather than a wire that passes requests through unchanged.

The basic shape of the architecture is simple: application talks to gateway, gateway talks to the AI provider or a self-hosted model. Applications see one stable endpoint, often OpenAI-compatible, regardless of which model or provider sits on the other side. Swapping providers becomes a configuration change at the gateway. Provider complexity lives in the infrastructure layer, where it can be managed once, instead of being copied into every application that needs a model.

The requests flowing through a gateway in 2026 are rarely a single call to a single model. A request might chain Claude for complex reasoning, a smaller model for fast classification, an MCP server for data retrieval, and a second model to verify the output before it goes back to the user. The gateway governs that entire chain: every hop, every model switch, every retrieval step, under one set of rules.

That shift in how applications are built is also why gateway adoption is accelerating. Gartner projects that a large majority of software engineering teams building multimodel applications will use AI gateways by 2028, up from roughly a quarter in 2025. The jump reflects how quickly multimodel became the default way production AI gets built.

How a request moves through a gateway: the component anatomy

A request entering a gateway passes through a sequence of checks, and each one is a point where the request can be stopped, logged, or changed before it reaches a model. A documented production deployment on Azure shows this pipeline in six stages, each doing a distinct job.

The call first hits ingress carrying a team key or a managed identity token. Ingress authenticates the caller and figures out which tenant the request belongs to before anything else runs. Nothing downstream happens until identity is settled.

Next comes policy evaluation. The policy engine checks the request against that tenant's rules: which models the caller is allowed to use, what the current budget looks like, whether the caller is within its rate limit. Any violation stops the request right here, before a single token gets spent. This is the stage that turns "we have a budget" into "the budget actually holds."

Then the prompt gets screened. Before the request goes anywhere near a provider, a screener checks the text for personal data, injection attempts, or anything that breaks content policy. This happens before the provider ever sees the prompt, which is the only point in the pipeline where that's still possible.

Only after those checks pass does routing happen. The router picks a model and an endpoint based on cost, latency, the type of task, or explicit policy rules. This is where model selection actually gets decided, not somewhere buried in application code that nobody remembers writing.

The call executes, and the response that comes back gets screened again before it's handed to the application. Then metering and audit close the loop: the metering pipeline logs tokens consumed, latency, which model ran, which tenant and team made the call, and the audit log records every decision made at every earlier stage.

One architectural choice underlies the whole pipeline: provider credentials live in the gateway, not in applications. Applications never hold live API keys. Instead, teams get virtual keys that can be scoped to specific permissions, rotated on a schedule, and revoked instantly, all without touching a single application deployment.

The rate limiting itself also works differently than people expect. Request-based limits, the kind that count calls per minute, don't actually control AI spending, because a single LLM request can consume anywhere from dozens of tokens to hundreds of thousands, or even millions. Counting requests tells you almost nothing about cost. Only token-level enforcement, counting what's actually consumed, gives you a limit that means something financially.

Routing logic: how the gateway decides which model handles each request

Routing gets reduced in casual conversation to "pick the cheapest model" or "pick the fastest one." Neither is the real job. The real job is sending each task to the cheapest model that reliably clears the quality bar that task needs, and only reaching for a more expensive model when the task genuinely requires deeper reasoning.

Research from LMSYS and UC Berkeley on RouteLLM backs this up directly: routing simple prompts to smaller models can cut inference costs sharply while keeping most of the output quality intact. That routing logic belongs at the gateway, where it can see every request across every provider, not inside each individual application making its own guess.

No application-level routing can see what the gateway sees. A single application knows its own traffic. The gateway sees cost data, latency data, and task patterns across every team and every provider at once, which is exactly the vantage point routing decisions need.

Naive routing, the kind that always defaults to whatever model the team already has a key for, leaves real savings on the table and sometimes leaves quality on the table too. A provider silently swapping in a different model without telling the caller isn't routing intelligence, it's a loss of control, and teams should insist on explicit visibility into where their requests actually go rather than accepting opaque substitution. Good routing strategies stack multiple approaches at once: load balancing across deployments of the same model, latency-based routing that picks whichever provider responds fastest at that moment, cost-based routing for workflows where quality requirements have some slack, and fallback chains that kick in automatically when a provider starts erroring.

Routing benchmarks are useful but incomplete. They measure whether a router picked the model that scores best on that particular benchmark. They don't measure whether that choice matches what real users actually want, given the specific cost, quality, and latency tradeoffs of a particular workload. Teams should test routing logic against their own traffic.

Cost attribution and spend control as a structural requirement, not a reporting feature

The common failure in AI cost management is not knowing, until the invoice arrives, which team, project, or workflow actually generated that spend. By the time finance sees the number, the money is already gone and there's no way to trace it back to a decision anyone could have made differently.

Shadow AI makes the problem worse. Enterprise licenses sit underused while employees quietly run their own personal subscriptions instead, and a meaningful share of total AI spend ends up invisible to any budget process trying to reconcile it. Nobody is hiding anything on purpose. The spend is just scattered across places nobody is looking.

Part of the confusion comes from measuring the wrong thing. Cost per token sounds precise, but it actually rewards using AI less, which defeats the point of using it at all. The metric that actually matters is cost per task, per workflow, or per unit of work finished, measured against what the process cost before AI touched it.

A gateway changes this from a problem finance discovers after the fact into one that gets stopped before it happens. Budgets get enforced per key, per team, per project, and per provider, checked at request time. A bug in a retry loop that would otherwise burn through a month's budget in minutes gets stopped cold at the gateway, without anyone needing to notice and intervene manually.

The industry has started treating this as infrastructure, not bookkeeping. The Tokenomics Foundation had its intent to launch announced by the Linux Foundation on June 9, 2026 at the FinOps X conference, with the formal launch following on August 4, 2026 alongside the FinOps Foundation, aimed at building standards for measuring token consumption and showing the business value of AI spend. That a foundation exists to standardize this work says the problem lives in the infrastructure layer, not in a finance team's spreadsheet.

Real-time visibility into spend, broken down by team, project, key, model, and provider, is what lets engineering and finance actually look at the same numbers and agree on what they mean. An invoice at the end of the month can't do that. By the time it arrives, the decisions that shaped it are long past being changeable.

Security and compliance enforcement at the request layer

Personal data, health information, payment data: none of it should ever reach a model provider without being checked first, and the gateway is the only place in the stack where that check can be applied consistently across every application and every team. An individual application might remember to screen for this. Another might forget. A gateway sitting in front of all of them removes that inconsistency.

This is the same pipeline already described, just doing a second job. Policy evaluation, the stage that checks budget and rate limits, is also where access control gets enforced: which teams can call which models, which regions are allowed to serve a request, which endpoints meet zero-data-retention requirements. Prompt screening, the stage that catches injection attempts, is also where PII gets caught and redacted before it ever leaves the building. Metering and audit, the stage that logs tokens and latency, is also where every one of these security decisions gets recorded with its reason.

A compliant gateway rests on three technical pillars: masking personal data dynamically before it's transmitted anywhere, processing everything statelessly in memory so prompt content never touches disk, and logging audit trails that capture the decision made without storing the sensitive content that triggered it. A screener built this way checks every prompt in real time, catches a wide range of sensitive data types, and either redacts or blocks the content before it leaves the organization's infrastructure, all without the screener itself becoming a new place where sensitive data sits around waiting to leak.

These controls aren't something only large, regulated enterprises need. Any team handling regulated data, at any size, needs redaction, access control, and audit logging from the first day it touches that data. Building and maintaining this kind of screening in-house takes dedicated security engineering effort that most teams don't have the headcount for. Concentrate AI, for instance, handles PII redaction at the gateway layer so that sensitive content is stripped before it reaches any provider. A managed gateway gets this in place without demanding that investment up front.

Reliability engineering: how the gateway keeps production AI running through provider outages

Provider outages are routine operational events that a production system has to plan for, and the gateway is the only layer positioned to respond to them automatically, without anyone rewriting application code in the middle of an incident. OpenAI's API went down for thousands of users in December 2025, a reminder that even major providers have bad days that land directly on production traffic.

No single provider's uptime, measured on its own, gives a team the reliability it actually needs over a full year. Automatic failover across multiple providers changes that math: the odds of every provider going down at the same moment drop to close to zero once failover is actually wired in.

The mechanism that makes this work is a circuit breaker, built per provider-and-model combination rather than per provider alone. That way, an outage on one specific model doesn't block requests to a different model running on the same provider. While a breaker is open, it stops the system from hammering a provider that's already struggling, which only makes recovery slower. Periodic half-open probes check whether the provider has come back, without flooding it with retries while it's still recovering.

Fallback chains sit on top of the breaker and get defined by policy, not improvised during an incident. A team can specify that a request to its primary model should fall back to a named alternative, and then to a self-hosted model if that alternative also fails, with every step in that chain logged as it happens.

The gateway itself becomes a single point of failure if nothing backs it up, and that risk needs its own answer. Application code should keep a fallback path that can talk directly to a primary provider if the gateway itself becomes unreachable. Choosing a managed gateway with real SLA guarantees matters for the same reason: the gateway is carrying reliability for everything behind it, so its own reliability has to be accounted for too.

Centralizing provider relationships pays off in practice. Whether a team builds this layer itself or adopts a managed one, the underlying requirement doesn't change: reliability has to be engineered into the gateway, because by the time an outage happens, it is too late to decide how the system should respond.