Total Cost of Ownership for LLM Infrastructure Options

The choice of infrastructure path is really a decision about where operational burden lives. Each path distributes that burden differently, and understanding the distribution matters more than understanding the feature list.

Direct provider integration means the team calls each provider's API independently, manages separate credentials and SDKs for each, and owns every adapter layer themselves. The team writes the integration code, maintains it, and responds when something breaks. No intermediary absorbs the friction.

Self-hosted open-source gateways, with LiteLLM being the most widely deployed example, introduce a proxy layer that normalizes provider interfaces behind a single surface. The team no longer writes per-provider adapters, but they now own the infrastructure running the gateway: deployment, uptime, configuration, scaling, and every upgrade cycle.

Managed gateways, with Concentrate as a representative example, move the proxy layer off the team's infrastructure entirely. A unified API handles provider routing, key management, fallback logic, and observability. The team owns its application code; the vendor owns the operational surface underneath.

Four questions cut through the noise. Who owns the integration code? On direct integration, the team does, entirely. On a self-hosted gateway, the open-source project owns the normalization logic but the team owns deployment and configuration. On a managed gateway, the vendor owns it. Who owns uptime and incident response? On direct integration, the team does. On a self-hosted gateway, the team does, for both the gateway and the provider layer beneath it. On a managed gateway, the vendor does, subject to its SLA. Who owns compliance and audit capability? On direct integration, the team builds it from scratch or accepts the gap. On self-hosted, the team configures whatever the project exposes, then builds what it does not. On a managed gateway, the vendor provides a baseline that teams configure rather than construct. What does adding a new provider actually cost? Direct integration is a sprint of engineering work per provider. Self-hosted is configuration work, assuming the gateway already supports that provider. Managed is typically an API call or a configuration toggle.

Most teams start with direct integration because it is the lowest-friction path to a working prototype — think of it as building a house by nailing boards together one at a time: fast to start, painful to scale. The problem is that the inflection point, where direct integration shifts from scrappy solution to active liability, arrives faster than anyone expects. Production AI teams in the mid-2020s commonly route across several providers and multiple model tiers simultaneously. The operational surface of direct integration scales non-linearly with provider count, and teams that fail to model that trajectory before committing routinely find themselves refactoring infrastructure under product pressure. That is the worst possible time to be doing it.

The integration engineering cost that direct provider connections carry

Every major provider has its own API schema. OpenAI's request format, Anthropic's format, and the wrapper layers that cloud platforms like AWS Bedrock and Google Vertex AI add on top of foundation models are not compatible out of the box. Each requires its own adapter logic, written, tested, and maintained by someone on your team.

Initial integration for a single provider is genuinely manageable. A capable engineer can produce a working, production-ready integration in a reasonable sprint. What changes is provider count, and costs do not scale linearly when that happens. They scale faster, because each new provider introduces its own idiosyncrasies across several dimensions simultaneously: schema translation, credential management, error handling, retry logic, and streaming behavior. These are not one-time problems. Every provider update, a new model, a changed parameter, a deprecated endpoint, propagates as an engineering task across every integration that touches it. Managing four providers is not four times the work of managing one — it is more like juggling four torches while someone keeps handing you more.

The reason teams routinely undercount this is structural. The labor is distributed across sprints rather than appearing as a discrete line item. It looks like regular engineering work because it is mixed in with regular engineering work. It does not appear on a provider invoice. Moreover, it does not show up in infrastructure spend reports. But it is real, it is recurring, and it compounds. A team running integrations with four providers that each release model updates several times a year is absorbing a continuous maintenance load that most budget models fail to capture.

The supposedly free path of calling providers directly carries a real engineering cost. Model it, with realistic assumptions about provider count and update frequency, before committing to this approach at scale.

What self-hosted gateways shift onto the engineering team

Self-hosted gateways solve a real problem. The multi-provider normalization burden that makes direct integration expensive is substantially reduced when a gateway handles schema translation and credential abstraction centrally. LiteLLM provides a genuine improvement over writing per-provider adapters independently. But "self-hosted" means the team runs the gateway on their own infrastructure, and the implications of that are broader than the name suggests.

The team owns deployment and containerization. The team owns uptime, and this is where it gets serious: the gateway is not a peripheral service. It is a critical dependency for every LLM call the product makes, which means gateway uptime is a product reliability concern requiring the same operational discipline as any other critical infrastructure. Monitoring, alerting, on-call coverage. The team also owns version upgrades and security patches, configuration updates as the provider landscape shifts, and capacity planning as AI usage grows.

Observability is a particular gap that surprises teams. Self-hosted gateways often lack the logging, tracing, and spend attribution capabilities that production workloads actually require. Filling that gap means building a second layer of infrastructure on top of the first: dashboards, log aggregation pipelines, cost attribution tooling, all of which must be deployed and maintained. Teams that discover this gap after deployment, which most teams do, frequently end up with more infrastructure to manage than they started with.

Self-hosted is not free infrastructure. It is a transfer of operational burden from a vendor to the engineering team — you are not escaping the weight, you are just changing who carries it. For teams with abundant platform engineering capacity and a genuine preference for control, it is a credible path. For teams where platform engineering capacity is constrained, that operational burden competes directly with product work, and the trade-off is sharper than it initially appears.

How per-token platform fees compound as usage scales

Some managed gateway platforms charge a percentage markup or per-token fee on every call they proxy, layered on top of the underlying provider cost. At low usage volumes, this fee is easy to dismiss. At production scale, it is not.

Token consumption in enterprise environments grows dramatically faster than budget planning cycles can track. A fee structure that looked inconsequential at pilot scale can become a significant line item within a single fiscal quarter. The fee does not grow with features or value delivered; it grows with every token the organization processes, regardless of what those tokens are accomplishing.

Agentic workloads amplify this considerably. Autonomous agents resend full conversation history with each step in a multi-step task, meaning token counts per task can be many times higher than in standard conversational use. A per-token platform fee that seemed manageable for a customer-facing chatbot can look very different when applied to an internal agent handling complex, multi-step workflows at volume. The math changes fast.

A rigorous TCO model for this cost category requires modeling provider base cost per token, any platform markup applied to that volume, a realistic volume growth trajectory rather than a static snapshot of current usage, and the delta between real-time and batch pricing where workloads allow. Some providers offer meaningful discounts for batch processing; capturing that delta requires infrastructure capable of distinguishing and routing accordingly.

Some managed gateways do not apply a per-token markup on top of provider pricing. That is a specific architectural stance, not a universal default across managed gateways. It matters because buyers who overlook fee structure before integrating often discover it afterward. When modeling TCO across managed gateway options, fee structure is as important as the feature set, and it should be modeled against a realistic growth trajectory, not against current usage.

The governance and compliance overhead that every infrastructure path carries, but assigns differently

A large share of enterprise AI interactions involve sensitive data. PII, financial records, protected health information, internal personnel data: these categories routinely appear in prompts, in retrieved context, and in conversation history. Much of this flows to provider APIs without any systematic redaction layer, because building one requires deliberate engineering work that is easy to defer and difficult to prioritize against product features.

Provider default data retention is not zero. Standard API plans at major providers retain request data for periods that vary by provider, and the specific terms are not always prominent in onboarding documentation. Zero-data-retention agreements require negotiated enterprise contracts unavailable on standard plans. Teams that assume their data is unretained because they have not configured retention should verify that assumption explicitly, preferably before a compliance review rather than during one.

The regulatory frameworks applying to LLM pipelines are not theoretical. GDPR and equivalent privacy regimes require data minimization and purpose limitation, with the ability to demonstrate both to a regulator on demand. HIPAA applies to any pipeline handling electronic protected health information, including logging requirements with extended retention floors. The NIST AI Risk Management Framework expects audit records capturing model version, policy version, redaction actions, and timestamps. The OWASP LLM Top Ten, updated in 2025, elevated sensitive information disclosure in priority, reflecting the widening exposure surface as LLMs gain access to more organizational data and external tools.

The infrastructure path determines who does the work to satisfy these requirements. Direct integration means the team builds everything from scratch: redaction logic, immutable audit log storage, role-based access controls, SSO integration, and whatever tooling is needed to produce documentation for a compliance review. Self-hosted gateways partially address some of these through the open-source project, but the team configures, maintains, and keeps current everything the project exposes. Managed gateways with these capabilities built in shift the work from construction to configuration.

Shadow AI creates a separate compliance exposure that no infrastructure choice fully eliminates. Employees handling organizational data through personal accounts and unsanctioned LLM tools is the norm in most organizations, not the exception. Gateway-level controls are the primary mechanism for containing it: enforced routing through a central surface, API key-based access control, audit logs capturing what was sent and to which provider. Those controls require a gateway layer to exist. Teams relying on direct integrations deployed per-team or per-product have no centralized surface through which to enforce them.

Regulatory enforcement in this area is accelerating. It belongs in any honest TCO model as an active and growing cost pressure with a real probability of arriving as an incident rather than a planning item.

Incident response and reliability costs that don't show up until something breaks

Major LLM providers experience outages. Not rarely. The more common pattern is frequent degradation: elevated latency, partial availability, elevated error rates on specific models, capacity constraints during peak periods. These events affect production workloads regularly and are invisible in advance.

Without a gateway layer, teams have no automated fallback. A provider incident becomes a product incident, and the response is entirely manual. Someone has to detect the degradation, diagnose whether it is a provider issue or a local one, decide what to do, implement a workaround, and communicate status. That is not a hypothetical cost. It is a recurring one, absorbed every time a provider has a bad week, and providers have bad weeks. Across a year of normal provider variability, those incident response cycles accumulate into a meaningful labor investment that rarely gets attributed to the infrastructure choice that generated it.

Fallback routing, automatically redirecting requests to a secondary provider when the primary degrades below a threshold, is the primary reliability lever available. Implementing it on a direct integration path requires bespoke engineering: degradation detection, alternate provider selection logic, schema translation to make that provider compatible with the existing integration, and ongoing maintenance as provider behaviors evolve. Managed gateways typically provide this as a platform capability rather than a build exercise.

Self-hosted gateways can support fallback configuration, and for teams that invest in it properly, it works. The complication is that the gateway itself is now a dependency in the reliability chain. If the gateway has an availability event, fallback routing cannot save the workloads that depend on it. That loops directly back to the operational overhead discussed in the previous section, gateway uptime treated with the same rigor as any other critical infrastructure.

Latency overhead from the gateway layer is worth addressing because it comes up in every evaluation conversation. Well-engineered gateway implementations add overhead in the sub-millisecond to low-millisecond range per request, which is negligible relative to inference latency that typically runs hundreds of milliseconds or more — worrying about gateway latency here is like fussing over the width of a doorframe while moving a grand piano. Validate this against any specific gateway you evaluate; it is not universal. But for reputable implementations, gateway latency is not a meaningful performance concern in practice.

How spend visibility, or the absence of it, becomes a cost in itself

The pattern repeats consistently enough across enterprises to treat as the default outcome: token consumption grows faster than budget planning cycles, and teams discover overruns at end-of-month invoice time rather than in time to intervene. By the time the invoice arrives, the spend has already happened. The only remaining action is explanation, not correction.

The structural problem with direct integration and most self-hosted paths is that spend data lives in separate provider dashboards, each with its own format, refresh rate, and attribution granularity. Constructing a unified view of AI spend by team, by project, by model, or by business unit is a manual reconstruction exercise that typically falls to an engineer or a finance analyst, done after the fact, with data that does not align cleanly across sources. It produces a historical record rather than an actionable signal. There is a meaningful difference between those two things.

Real-time, granular spend visibility enables something qualitatively different from what end-of-month reporting enables. Budget limits per team, project, or API key can cut off spend before it breaches a ceiling. Routing decisions can respond to live cost signals, shifting workloads to cheaper models or batch processing when a budget tier is approaching. Anomalous usage, a runaway agent, a misconfigured prompt, an unauthorized integration, can be detected while it is still generating incremental spend. These are not marginal improvements; they change what decisions are even possible.

Token costs also exist across three distinct accounting categories: capital investment, operating expense, and cost of goods sold embedded in product margin. A team building internal tooling needs to see AI spend as an operating cost with departmental attribution. A team whose product charges customers based in part on AI consumption needs to see it as COGS with per-customer or per-feature granularity. These are different reporting requirements, and meeting them requires visibility infrastructure that most direct integration and self-hosted paths do not provide out of the box.

The cost of missing visibility is measurable in two ways: overspend that could have been intercepted, and engineering time spent reconstructing spend data across provider dashboards after the fact rather than building product. Cost control decisions are made daily by engineers: model selection, prompt design, caching strategy, batch versus real-time routing. Visibility tools that route spend information only to finance, on a monthly cadence, arrive too late and too bluntly to change any of those decisions.

Putting the TCO model together across all three infrastructure paths

Seven cost categories apply to every infrastructure path, distributed differently across each, which is what makes a path appear cheap on one dimension while being expensive on another.

Initial integration engineering. Direct integration carries the highest cost here, borne entirely by the team: adapter code, credential management, schema normalization. Self-hosted gateways reduce this meaningfully, though deployment and configuration still require real work. Managed gateways reduce it to API integration against a unified surface.

Ongoing maintenance. Direct integration cost grows with provider count and is never done. Self-hosted gateways reduce the per-provider burden but still require version management and configuration updates as the provider landscape shifts. Managed gateways move most of this to the vendor.

Infrastructure and operational overhead. This is where self-hosted gateways carry the highest cost of any path: the team owns the gateway's infrastructure entirely, including uptime, scaling, and security patching. Direct integration has no equivalent layer to manage, though it absorbs infrastructure costs differently through per-provider complexity. Managed gateways carry minimal overhead here.

Compliance and security build cost. Direct integration requires the team to build everything from scratch. Self-hosted gateways partially address some requirements through the open-source project, but the team builds what the project does not cover. Managed gateways with compliance capabilities built in shift construction to configuration.

Incident response and reliability engineering. Direct integration means all of it is manual and absorbed entirely by the team. Self-hosted gateways allow fallback configuration but leave the team responsible for gateway uptime as well. Managed gateways treat provider fallback and gateway uptime as vendor responsibilities.

Spend visibility and governance tooling. Direct integration requires manual reconstruction across provider dashboards or custom tooling. Self-hosted gateways typically require additional tooling beyond what the gateway itself provides. Managed gateways typically include real-time attribution and configurable budget controls as platform capabilities.

Platform fees on token volume. Direct integration: zero. Self-hosted: zero. Managed gateway: varies by vendor. Some managed gateways apply a meaningful per-token markup; others do not.

The practical implications are clear. Direct integration carries the lowest platform fee cost and the highest costs across integration engineering, maintenance, compliance, incident response, and spend visibility. The labor is entirely internal, distributed across sprints, and never appears on a vendor invoice, which is precisely why it gets missed. Self-hosted gateways reduce integration and maintenance costs relative to direct integration but shift infrastructure and reliability ownership onto the team in ways that are more demanding than teams anticipate, particularly for those without dedicated platform engineering capacity. Managed gateways carry the highest potential platform fee exposure, which can be material at scale and must be modeled against realistic volume projections; for managed gateways that carry no per-token markup, that cost drops to zero, and the TCO comparison shifts considerably.

The path that appears cheapest in a token-price comparison is rarely the cheapest path when all seven categories are modeled with realistic assumptions about engineering labor, provider count, compliance requirements, usage growth, and incident frequency. The hidden costs do not stay hidden once the system is in production. They just show up as something other than a line item on a provider invoice, which is exactly why they keep catching teams off guard.