Self-Hosted LiteLLM vs Managed LLM Gateway

Most teams find out they need a gateway the hard way. The tell is usually small and annoying at first: retry logic gets copied into a third service this month, or someone in a budget review asks what the LLM bill breaks down to by feature and nobody can answer. Once an app talks to more than one model provider, or spends real money on tokens, six jobs need a home: provider abstraction, routing, retries and fallback, caching, observability, and rate limits. Left without a home, those jobs get scattered across application code, where every team writes its own version and none of them agree, which is exactly the point where they stop being enforceable. A gateway handles those six concerns in one place. Concentrate AI delivers this as a fully managed service, so a team doesn't have to own the deployment and patching surface just to get provider abstraction, routing, and spend visibility in one place, while still keeping full control over routing decisions, and that's the baseline. The real decision starts once a team accepts it needs this layer: build and run it themselves, or hand the operational weight to someone else.

What self-hosting LiteLLM means in production

The most common misreading of "self-hosted LiteLLM" is assuming it buys more than it does. Self-hosting means controlling the gateway's control plane: credentials, routing policy, logs, and keys. It does not mean controlling model execution or data privacy from end to end. Requests still leave the building and travel to OpenAI, Anthropic, Bedrock, or whichever provider is configured, unless the gateway is pointed at a local inference server. Running the proxy yourself only pays off as a privacy or sovereignty move when it sits in front of local models or approved private endpoints; otherwise the data path looks the same as it would through any other gateway.

LiteLLM also ships in two forms that get conflated often enough to cause confusion. One is the Python SDK, a library that runs inside a single application, and it never becomes shared infrastructure. The other is the proxy server, sometimes called the AI Gateway, deployed as a centralized service that a whole team or organization routes through. The build-versus-buy decision only applies to the proxy. That distinction matters because the proxy is also where the open-core business model appears: the free, self-hosted version covers routing, keys, and basic spend tracking, but several governance features that larger teams eventually want sit behind a paid enterprise tier. That ceiling becomes a bigger part of the story later, but it starts here, at the moment of picking which LiteLLM a team is even talking about.

What a production-grade self-hosted LiteLLM deployment requires

Running LiteLLM as a demo takes one process and a config file. Running it in production is a different commitment: it needs an owner, a private data layer, a release policy, and proof that a failure in the gateway doesn't take down every AI feature in the product at once. That gap between demo and production is where most of the real cost of self-hosting actually lives.

A single Docker Compose file on one box is not a credible production setup. The minimum shape that holds up looks more like a small distributed system. It starts with at least two gateway replicas, pinned to a specific version, sitting behind a load balancer that terminates TLS. Those replicas need a private, managed Postgres instance to persist keys, teams, spend records, and configuration, with backups running automatically rather than as an afterthought. Once there's more than one replica, Redis 7 or newer has to sit alongside it, because without a shared cache, each replica enforces rate limits on its own and caches independently, which quietly breaks the guarantees the gateway is supposed to provide. And schema migrations need their own job, run separately, so a database change never races an autoscaling event mid-deploy.

Virtual keys deserve particular attention in this setup, because they're the main lever teams have for controlling cost. They require Postgres to function, and they do the real work of cost governance: scoping which models a key can call, setting a budget per team, and attributing every token back to a specific key, user, team, or organization. The latest stable release as of September 2026, version 1.103.0, documents a config-file-plus-Docker-Compose path that its own docs describe as the reviewable, version-controlled way to run this. The minimum credible shape here, stateless replicas, managed Postgres, Redis, secret storage, health probes, a separate migrations job, is ongoing operational weight. A managed alternative like Concentrate AI takes this weight off a team's plate entirely, so engineers can work on routing logic and governance instead of keeping a database cluster healthy.

The security and patching obligation that comes with operating the gateway yourself

If you self-host LiteLLM, you own a security surface that needs continuous maintenance; it isn't a box that gets hardened once at launch and left alone. That distinction is the whole argument of this section.

The open-source tier currently carries 8 known GHSA vulnerabilities, and in March 2026 it was the center of a confirmed supply-chain incident. Attackers compromised PyPI credentials through an unpinned, rolling Trivy GitHub Action tag inside LiteLLM's own CI/CD pipeline, then used those credentials to ship a backdoor. That incident sets the baseline for what "operating the gateway yourself" actually requires at launch, not as optional hardening after the fact: exact version pinning, such as locking to v1.83.0 rather than tracking a moving tag; verifying image signatures with cosign; and restricting network access around the gateway. Rolling tags like main-latest are flagged explicitly as unsafe in production for exactly this reason.

Governance features that larger organizations tend to assume are standard, JWT authentication, SSO beyond five users, role-based access control, and audit logs, sit behind the enterprise tier. The audit log piece carries its own obligation. NIST SP 800-92 guidance calls for logs detailed enough to reconstruct what happened: actor, action, timestamp, and outcome, without logging raw prompt content, so the audit trail itself doesn't turn into a repository of sensitive data. None of this means LiteLLM is insecure by design. So if a team runs it themselves, they're on the hook for every patch cycle, every credential rotation, and every incident response, starting the moment it goes live.

How the operational surface affects reliability, latency, and fallbacks

A gateway sits on the critical path of every feature that touches an LLM. If it goes down, every one of those features goes down at the same moment. That single fact is what turns operational burden into a reliability question, not just a maintenance one.

Latency overhead from a well-designed gateway is usually small, typically under 50 milliseconds, and it barely registers next to the multi-second response times of the LLM call itself. But the risk that matters more becomes visible under sustained concurrency, not in normal request latency. One production performance report tested LiteLLM v1.80.15 under load and found throughput through the proxy fell to roughly half of direct-call throughput at 500 concurrent requests on a 4vCPU, 8GB virtual machine. But it's exactly the kind of finding a team needs before it commits to self-hosting at real scale, because it shows the failure mode isn't hypothetical.

Fallback design adds another layer to this picture. LiteLLM supports three tiers of failover sophistication. The simplest is a static fallback chain, try Anthropic, then OpenAI, then Google, which is predictable but has no awareness of output quality. Because the gateway can become the single point of failure for an entire product, the sound mitigations are to keep a direct fallback path in application code that talks to a primary provider without going through the gateway, to run multiple gateway instances behind a load balancer with real health checks, or to use a managed gateway that carries its own SLA guarantees.

The real cost structure of self-hosting, infrastructure, engineering time, and the crossover point

Most teams pick a side of this decision on gut feeling rather than a spreadsheet, and the actual crossover point is more nuanced than the easy line that self-hosting gets cheaper at scale. Running a self-hosted, LiteLLM-style proxy in production costs more than the provider bill alone: compute for the replicas, a managed database, storage for logs and metrics, and observability tooling all sit on top of whatever OpenAI, Anthropic, or Bedrock charge for tokens.

Where that lands depends mostly on how much a team spends on LLM calls each month. At the low end, a managed platform's free tier or per-call pricing tends to be cheaper in practice than the engineer-hours it takes to run anything self-hosted properly, even before counting the security and reliability work described above. At the high end, somewhere above the order of $10,000 a month in LLM spend, self-hosting can come out ahead on raw dollars, but only if the team genuinely has DevOps capacity to spare and the managed alternative it's being compared against is priced with a markup rather than a flat fee. Those are rough thresholds meant to show the shape of the tradeoff, not line items from an invoice.

Reaching feature parity with what an existing gateway already does, routing, fallback, caching, observability, rate limiting, virtual keys, takes something closer to two engineer-years of work, and keeping that parity as providers change their APIs costs roughly another half an engineer-year every year after that. Most teams don't budget for this cost when the decision gets framed as "self-host versus pay for a platform," because it's really the cost of a third, usually unstated option: build it all yourself.

What a managed LLM gateway takes off the table

A managed gateway is a trade. A team gives up some control over deployment topology and the exact path data takes between the app and the provider, and in exchange gets operations, upgrades, security response, and SLA guarantees handled by someone else. Patching cadence, supply-chain monitoring, database management, replica coordination, credential rotation, and on-call for gateway uptime all move off the team's plate and onto the vendor's.

Governance and observability are where this trade becomes most visible. Real-time spend broken down by team, project, key, model, and provider; role-based access control; SSO through an enterprise identity provider; audit logs, these are things managed platforms tend to include by default, while a self-hosted LiteLLM stack either pays for the enterprise tier to get them or builds equivalents in-house. Concentrate AI is one example of what this looks like in practice: it connects engineering teams to more than 21 LLM providers through a single unified API, without needing separate provider keys, custom integration code, or any self-hosted infrastructure, and it gives teams real-time spend visibility by team, project, key, model, and provider without layering a per-token platform fee on top of what providers already charge.

None of that erases the cases where self-hosting is the right call. Self-hosting earns its keep specifically where private networking or custom controls are mandatory, where local model inference needs to run right alongside the gateway, or where infrastructure control matters more to the organization than how fast it can ship. A fair comparison has to hold both of these facts at once: managed platforms remove a lot of real operational weight, and a narrow set of requirements still points toward running the gateway in-house.

Matching the choice to the team, a decision framework by actual situation

The right answer here isn't fixed, and it isn't decided by a default belief that open source is automatically cheaper or that managed is automatically easier. It comes down to who ends up owning the operational surface, and whether the control gained from that ownership is worth what it costs to hold.

Self-hosted LiteLLM tends to be the stronger fit when several of these conditions occur together. If data-sovereignty rules block any third-party network hop, that points toward self-hosting, and so does a team routing requests to local inference servers like vLLM or Ollama alongside public provider APIs. Proprietary routing logic that doesn't fit into a standard framework's config is another signal, and so is having a dedicated DevOps owner in place already, someone for whom infrastructure control is a first-order job requirement rather than a side task. Self-hosting also makes more sense once a team has already budgeted for the enterprise tier, because SSO, audit logs, and RBAC don't come free on the open-source version.

A managed gateway tends to be the stronger fit in a different cluster of situations. It fits well when AI usage across an organization is growing faster than internal governance can keep pace with, and when engineering time is the scarce resource, the team would rather spend its hours shipping product than keeping a Postgres cluster and a Redis instance healthy. It fits when finance, security, or compliance stakeholders need real-time visibility into spend and access without someone having to build a dashboard on top of Prometheus by hand, and it fits when the organization isn't yet staffed to respond to a supply-chain incident on a 24-hour clock.

One situation sits outside this framework entirely rather than falling to one side of it: a single provider at low volume. In that case, calling the provider's SDK directly skips both the proxy hop and the supply-chain surface altogether, and neither self-hosting nor a managed gateway is actually warranted yet. The mistake most teams make is skipping the arithmetic altogether and defaulting to whichever option happened to be top of mind when the question came up.