Multi-Provider LLM Routing Without Running Your Own Proxy
Managed gateways eliminate the infrastructure tax of self-hosted routing.
Running a self-hosted proxy like LiteLLM to route across LLM providers trades one operational burden for another: teams stop depending on a single vendor's API and start depending on their own uptime, patching, and scaling of that proxy. A managed LLM gateway gives you the same multi-provider routing, fallback logic, and spend visibility, but without the infrastructure to maintain, so more teams move provider access out of their own stack entirely and into a service built to run it.
Why calling model providers directly stops scaling
A single model, single provider setup is simple, but it doesn't stay that way. A team adds a second model, a second provider, or a second team sharing the same AI infrastructure, and the first crack appears in the independently written retry logic, credential stores, and cost reports that follow. Each addition brings its own retry logic, its own credential store, its own cost report, its own way of failing, and all of it gets written independently inside application code. None of this looks like a problem on day one. It looks like a problem six months later, when three services each have their own half-working version of the same retry logic and nobody remembers which one actually works.
Vendor lock-in builds up in three specific ways. When business logic gets wired to provider-specific formats, things like tool-calling schemas and streaming event shapes, swapping providers means you have to rewrite the parts of the app that were never supposed to care which provider answered the call.
Providers also change constantly. Models get retired, pricing shifts, API behavior changes without much warning, rate limits tighten, and regional availability gaps open up. Without a gateway, every one of these events is an application-level fire drill. With a gateway, it's a configuration change.
The typical AI stack in 2026 runs in layers: application, then an orchestration framework like LangGraph, CrewAI, or LlamaIndex, then an LLM gateway, then the providers themselves. The pattern that triggers the need for a control layer is simple: the second of anything. A second compliance requirement needs centralized keys and an audit trail.
Gateway capabilities beyond direct integration and bare proxies
Three terms get used loosely in this space, and they describe three different things. A proxy forwards requests behind a unified API, nothing more. A router decides which model or provider should handle a given request, based on cost, latency, content, or a rule a team wrote. A gateway does both of those jobs and adds identity, budgets, rate limits, guardrails, audit trails, and observability on top.
A gateway is built to answer both.
The piece holding all of it together is the OpenAI-compatible API. Nearly every gateway speaks it, nearly every orchestration framework expects it, and providers have converged on it for their own SDKs. That shared contract is what makes gateway adoption a configuration change instead of a rewrite, which matters for the migration discussion later on.
The infrastructure cost of self-hosting a gateway like LiteLLM
Self-hosting a gateway solves the routing and abstraction problem. It also hands the team a new infrastructure problem that grows right alongside the scale it was meant to handle.
LiteLLM is a legitimate, widely used tool. But running it in production means also running PostgreSQL and Redis alongside it, and getting SSO or audit logs working means paying for an enterprise license. Prompt versioning and built-in A/B testing are absent out of the box, and the governance features that matter most to a security team, full SSO, org and team-level RBAC, and audit logs, live in the commercial tier, reserved for paying customers.
And when the controls that actually matter, RBAC, audit logs, PII redaction, sit behind a contract upgrade, teams tend not to pay for them. They work around them instead. This is a recurring failure mode across gateways where governance ships as an add-on rather than a default: the feature exists, but the team never turns it on, because turning it on costs money nobody budgeted for.
The build-versus-buy math has shifted. Outside of those specific cases, you no longer win on cost or control by self-hosting. If you self-host specifically for data residency, you can often get the same guarantee from a managed gateway that supports in-VPC deployment or zero-data-retention routing, so the residency concern gets resolved without anyone having to own the gateway process.
How multi-provider routing works in practice
Routing is a range of strategies, and most teams should start at the simple end rather than building toward the complex end before they have the usage data to justify it.
The common strategies include cost-based routing, latency-based routing, capability-based routing, policy-based routing, and semantic or intent-based routing, often combined into a hybrid. Picture these on a spectrum. At the other end sit learned routers, which add meaningful extra savings, but they need labeled data, an evaluation pipeline, and retraining every time the model lineup shifts.
A gateway should make routing a config change, not a code change. A team should be able to roll out a new rule, watch the metrics, and roll it back without a deploy. A managed gateway like Concentrate AI sits below the orchestration framework for exactly this reason, handling provider abstraction, credential management, and policy enforcement as configuration, so adding a second model, team, or provider becomes a gateway setting.
A tiered model approach works well as a starting structure: free-tier or open-source models for extraction and local workloads, budget models for batch jobs and speed-sensitive agentic loops, mid-tier models for most production tasks, and frontier models reserved for complex reasoning and planning. Routing also covers reliability. When a provider degrades, the gateway shifts traffic to an equivalent model on a healthy provider, and the application code never needs to know a failover happened.
Why provider reliability makes multi-provider fallback an architectural requirement
LLM provider outages happen often enough that betting a production system on a single provider is a real risk, and no vendor's terms of service give you the SLA backstop that would make that bet safer.
Every major LLM provider had downtime in 2026. If you look across a broad set of tracked API categories, AI and ML APIs come out as the least reliable, with more incidents per service than any other category tracked.
This is why fallback chains need to account for correlation between providers, not just independence. If a backup request routes to a secondary provider that happens to sit in the same cloud region or the same underlying infrastructure layer, you don't get the independence the math assumes. The mechanism that handles this at the gateway level is per-provider health tracking paired with circuit-breaker failover: the gateway continuously measures each provider's error rate, p95 latency, and rate-limit frequency, trips a breaker once a provider crosses a degradation threshold, and reroutes new requests to an equivalent model on a healthy provider, returning metadata so the calling application can react if it needs to.
How spend control works structurally inside a gateway
Knowing what was spent after the fact only solves half the problem. What matters is this: does the budget check happen before the provider gets called, or after the invoice lands?
A gateway is the one point in the stack where spend control can be structural. It meters every call with a team and feature tag and enforces budgets inline, which produces a real-time ledger: tokens in and out, cost broken down by feature, savings from cache hits. That takes real-time enforcement, not reverse-engineering a provider invoice after the month closes.
The budget check has to run before the call reaches the provider. If a team has used up its monthly allocation, it gets rejected at the gateway, preventing overspend structurally. A gateway that only meters usage and never blocks a request is a reporting tool wearing a governance costume. The dashboard shows the damage after it's already done.
A bug in a retry loop can chew through a month's budget in minutes. Those events flow into a durable store, get rolled up into views by model, tenant, and user, and feed budget counters with alerts set before and at the monthly limit for each team and model.
Caching is one of the more direct levers on cost. This isn't a nice-to-have bolted onto a gateway for marketing purposes. Caching cuts token usage directly, so it lowers cost and latency at the same time, making it a structural part of cost control.
Security and compliance controls that belong in the gateway, not in application code
Centralizing provider keys and data handling in one place shrinks the attack surface and makes compliance something a team can actually audit. Doing either of those things at the application layer means every service is a potential breach point, and every audit turns into a forensic reconstruction project.
When each service holds its own provider API key, a single leak gives an attacker direct, unthrottled access to the provider's entire quota. This is the mechanism behind the LLMjacking attack class, where stolen keys get drained for inference or resold. Compare that to a setup where provider keys live only in the gateway's vault, and applications hold short-lived gateway tokens scoped to a specific team and budget. Concentrate AI works this way: provider keys stay centralized, and applications hold Concentrate AI Universal API keys instead of direct provider credentials, which limits the blast radius of a leak without requiring the team to build or run that vault itself.
PII redaction belongs at the gateway for the same reason. The audit log should record the redaction event itself, the data category detected and redacted, the guardrail triggered, the model, the team, the trace ID, without storing the raw PII in the log.
A compliance-grade audit log is immutable, complete, tamper-evident, and exportable to a SIEM. Zero data retention routing works as a per-route policy: requests that require ZDR get rejected at any gateway that doesn't enforce it, so sensitive traffic never touches a non-compliant endpoint.
Evaluating a self-hosted gateway against a managed one
The choice between self-hosting and going managed comes down to a short list of concrete constraints, and the set of teams for whom self-hosting is genuinely the better call is smaller in 2026 than it was two years earlier. For almost everyone else, the operational cost of running that infrastructure ends up outweighing what it buys.
A production gateway, self-hosted or managed, can be evaluated against six criteria. And observability covers request logs, OpenTelemetry support, audit logs, and where logs can be exported.
Deployment model isn't a binary choice between fully self-hosted and fully managed. If a managed gateway supports in-VPC or private-link deployment, it resolves most data-residency concerns without a team having to own the gateway process. And governance shouldn't be treated as a premium feature. If RBAC, audit logs, and PII redaction sit behind a contract upgrade, teams at any size will find a way around them rather than pay for them, so those controls stop serving their purpose. What matters is whether a gateway offers these controls at the tier the team will actually be using, not just somewhere in its product line.
How Concentrate AI fits the managed gateway model
Concentrate AI is a managed LLM gateway built to remove the operational burden of running multi-provider AI infrastructure, so engineering teams get routing, governance, and observability through a single API, with no self-hosted process to maintain. It connects to 168 models across 21 providers, including OpenAI, Anthropic, and Google, through one unified, OpenAI-compatible API, with no separate provider keys, no bespoke integration code, and no infrastructure to stand up.
Spend visibility runs in real time and breaks down by team, project, key, model, and provider, with budget enforcement that blocks a request before it overspends. Routing stays transparent: Concentrate AI doesn't swap models on a team without their knowledge, and every routing decision stays visible, with teams setting the cost, speed, and quality criteria that govern where requests go.
Security and compliance controls, PII redaction, audit logs, RBAC, SSO, are available to teams of any size starting on day one. Concentrate AI serves teams across the size spectrum, from seed-stage startups to large enterprises, and it has particular relevance for fast-growing teams where AI usage is expanding faster than internal governance can keep pace with.
Moving from direct provider calls to a managed gateway
Because nearly every major gateway, Concentrate AI included, exposes an OpenAI-compatible API, moving off direct provider calls is a configuration change for most teams. An application already calling the OpenAI SDK can point at the gateway endpoint by changing a base URL, with no provider-specific integration code needing to change on day one.
Not everything needs to happen at once. The right place to start is the unified API, basic routing, fallback, and logging, with cost telemetry added early, since almost every other improvement depends on being able to measure what's actually happening first.
The rollout proceeds in stages: route all traffic through the gateway and strip provider keys out of application code, then add per-team virtual keys with basic budget alerts, then build fallback chains for each critical route, then layer in PII redaction and audit logging for whatever compliance requirements apply, and finally add routing rules once there's enough usage data to calibrate them properly. With it, you turn a spend spike into a five-minute query against a dashboard instead of a week spent tracing invoices back to root cause.
Teams that treat the gateway as infrastructure they consume, rather than software they have to run, get back the engineering time that was going into credential rotation, retry logic, and usage reconciliation, and put it into building the product the team set out to build.
