When to Build Custom LLM Infrastructure In-House

Ninety-two percent of Fortune 500 companies now run generative AI somewhere in their workflows, and almost none of them sat down and decided how the plumbing underneath it should work. Most teams didn't choose to build their own LLM infrastructure. They just never stopped to ask whether they should, and by the time the question comes up, the answer is usually already "yes, by accident."

"We know our requirements best, so we'll build it ourselves" sounds reasonable. It's also almost never tested against anything, which is the real problem. Building custom LLM infrastructure is the wrong call for the vast majority of teams that end up doing it. This piece lays out why, and the narrow set of cases where building is actually the right move, not the default one.

What LLM infrastructure actually involves once you move past a single provider

One provider is simple. OpenAI has a schema, an API key, a rate limit, and that's the whole surface area. The trouble starts the moment a second provider shows up: Anthropic's schema doesn't match OpenAI's, and Mistral, Google, Groq, Cohere, and Perplexity each bring their own auth format, error codes, and pricing model on top of that. Format fragmentation is the first wall every team hits, and it's the reason companies end up managing several providers at once instead of picking a favorite and moving on.

An AI gateway earns its keep right here, because a custom build has to replicate all of it, not just the flashy routing part:

• Routes requests to whichever provider sits behind a single endpoint, so the calling application never has to know or care which model answered
• Isolates provider API keys at the infrastructure level, handing applications scoped virtual keys instead of the real thing
• Runs fallback chains, so a 429 or 500 from one provider quietly routes to a backup instead of crashing the request
• Enforces token-level rate limits and budgets per key, per team, per project, per organization
• Logs per-request telemetry (provider, model, tokens in and out, latency, cost, who called it, what policy fired) without anyone touching application code
• Runs semantic caching, matching prompts that ask the same thing in different words against vector similarity, and serving the cached answer
• Detects and redacts PII before a request ever reaches a model provider
• Handles audit logs, role-based access, and single sign-on for compliance teams

The tell that a team has crossed into needing this layer is usually small and specific: retry-with-backoff code copy-pasted into a third service, or nobody in the building able to answer "what's our monthly LLM spend, broken down by feature." Industry research projects that 70% of software engineering teams building multimodel applications will use an AI gateway by 2028, up from roughly a quarter in 2025. That convergence isn't a trend so much as an admission: fragmentation compounds, and it compounds the same way everywhere. Gateways have also stopped being just a language-model thing. They're expanding into vision models, embedding APIs, and orchestration for agentic workflows, so "LLM infrastructure" now covers more ground than it did two years ago.

The operational costs that accumulate when this infrastructure lives in application code

A large share of AI teams report costs blowing past forecasts by 40% or more while scaling. That's not a rounding error, and it's not a one-time miss you fix with a better spreadsheet. It moves fast enough that finance teams have noticed: SpendHound's 2026 AI Spend Report found 46% of finance and procurement leaders went over their AI budget in 2025, against 37% who went over on traditional software.

Without a gateway acting as a single point of enforcement, the failures aren't edge cases. They're the default outcome, specific and repeatable:

• Finance gets one invoice, not a breakdown, because there's no cost attribution by team, feature, or model
• PII redaction logic gets written per service, drifts between services, gets skipped entirely in whatever microservice shipped last sprint, and leaves no unified audit trail behind
• Retry and fallback logic gets reimplemented by every team that calls a model, each slightly differently
• Shadow AI usage never shows up in centralized tracking at all. Torii's 2025 dataset found that 26 of the top 50 unsanctioned tools in use were pure-play AI products, running entirely outside whatever governance exists on paper
• A cheap model's output format shifts, schema validation starts failing, and the router silently escalates every request to the expensive model. Nobody gets an alert. The bill just goes up, quietly, on every single call

None of this is one team's mistake. It's what happens when five teams each build their own version of the same governance logic, so the policy enforced in one service stays invisible to the next. Inconsistency piles up in exactly the spot everyone thought redundancy was being avoided.

What the managed gateway market actually offers today

The GenAI infrastructure layer has grown substantially, with the managed gateway market now offering real depth across open-source and fully managed options. That's a market with real depth now, and the options span from fully open-source to fully managed. Treating them as interchangeable is the mistake; they solve different problems for different teams.

LiteLLM sits at the open-source end: support for over 100 models behind one OpenAI-compatible schema, deployable as a standalone server or dropped in as a Python SDK, with logging, retries, and cost tracking built in and compatible with LangChain and the OpenAI SDK out of the box. The Admin UI ships free in the open-source tier, though SSO and audit logs sit behind the enterprise version, and some rough edges show up in day-to-day developer experience.

OpenRouter runs as a managed service that abstracts the provider layer through one central endpoint and handles billing itself, applying its own internal price weighting when it routes. Its Auto Router adds a cost-versus-quality tradeoff dial at no extra charge, and it tends to get fast access to newly released models. The tradeoff is limited visibility into how its internal routing actually makes decisions, which matters if a debugging session ever needs to answer "why did this request go there."

Helicone works as a drop-in proxy across OpenAI and a broad range of other provider APIs, with monitoring and observability built into the same layer. TrueFoundry's LLM Gateway leans toward MLOps teams already running Kubernetes, offering multi-LLM abstraction built for people comfortable with infrastructure work.

Two tools get mistaken for gateways and aren't quite one, and confusing them wastes real engineering time. LangServe wraps LangChain applications as RESTful services, but it isn't a gateway by default; turning it into one takes extra work and custom security hardening. Pomerium handles identity-aware policy enforcement in front of gateways or services, which complements a gateway rather than replacing it. Access control and routing are different problems, and teams that treat them as the same thing end up with gaps in both.

There's also a category built specifically for teams that don't want to run infrastructure at all: a single unified API connecting to a wide range of providers, no separate keys to manage, no self-hosted footprint required, with spend visible by team, project, key, model, and provider from day one. That model tends to fit fast-growing teams whose AI usage is outrunning their internal governance, which describes a lot of teams right now.

The deployment spectrum runs from fully managed SaaS, which carries no operational burden, to self-hosted open-source, which hands back full control along with full responsibility for keeping it running. Which end fits depends on what a team is actually equipped to operate day to day, not on which one sounds more impressive in a planning doc.

The conditions that genuinely justify building custom LLM infrastructure

Building is the right call when an off-the-shelf option can't meet a specific, documented requirement. It is the wrong call when building just feels more controllable, or when a team wants to avoid depending on a vendor as a matter of principle. Those are two different motivations, and only the first survives scrutiny. Most teams that build are acting on the second one and calling it the first, which is exactly how "we know our requirements best" turns into eighteen months of maintenance nobody budgeted for.

A handful of conditions actually clear that bar. Some regulatory environments have no certified managed option at all: certain defense, government, and healthcare contexts require infrastructure that meets specific certification or data residency requirements no managed option currently satisfies. If nothing on the market holds the certification a contract requires, building or adapting an open-source base is a legitimate answer, not a workaround.

Latency can justify it too, but rarely as often as teams assume. Purpose-built implementations can push proxy overhead into the microsecond range, and that only matters when an application's latency budget genuinely has no room left for a network hop. Most production LLM applications are bottlenecked by model inference itself, which runs in seconds, not by proxy overhead, which runs in the microsecond-to-millisecond range. If the math doesn't require that headroom, this reason doesn't apply, no matter how good it sounds in a design review.

Routing logic that is itself the product, rather than the plumbing that delivers the product, is another real case. If the specific way requests get classified, weighted, and dispatched is the competitive edge, a generic gateway's routing model can turn into an architectural constraint instead of a shortcut. This one is rare. It's worth checking against actual business requirements rather than assuming it's true because it flatters the team proposing to build it.

Air-gapped environments, where outbound connections to any managed service are impossible and every dependency has to be self-contained and auditable, are a legitimate case too, and they show up in some intelligence, military, and critical infrastructure work. Teams that already built a mature internal platform, one that genuinely covers key management, spend attribution, and fallback logic across the board, may find little value in migrating away from it. Verifying that the existing system covers the full picture, not just the parts that got painful enough to fix first, is the catch.

The test that separates a real requirement from a preference is simple: can the specific gap be written down? "We want more control" is a preference, not a requirement, and a preference is a weak reason to fund a significant engineering project. Wanting to dodge per-token platform fees doesn't qualify either, since those costs are quantifiable and often negotiable. General unease about vendor dependency doesn't qualify, since open-source options exist precisely to manage that. "We're different enough to need our own" doesn't qualify without evidence behind it, and evidence, not instinct, is what this whole exercise is supposed to produce.

What teams consistently undercount when estimating the cost of building in-house

The initial build is the cost everyone sees. The cost that gets missed is everything after, and it's the larger number by a wide margin.

Every new provider added means new schema mapping, new auth handling, new error parsing, over and over, indefinitely. New compliance rules stack on top: the EU AI Act's high-risk obligations are set to land around December 2027, with fines reaching €15 million or 3% of global annual turnover, and audit logging bolted on after the fact costs far more than the same feature built in from day one. Observability has to get instrumented service by service without a shared layer, so per-model cost dashboards, per-team attribution, and latency percentiles all get rebuilt from scratch wherever they're needed.

OWASP raised Sensitive Information Disclosure to LLM02 in its 2025 Top Ten list, and PII redaction logic written separately in every service is exactly the architecture that produces the failure OWASP describes there. It's a risk that already exists by default. It's the predictable result of the pattern most in-house builds fall into.

Agentic workloads make the math worse. A single user request in a 2026 production agent stack can trigger several model calls, calls to MCP servers, and verification steps, all chained together, and fallback logic written for a simple single-request flow tends to break down the moment that kind of concurrency shows up. Provider data retention adds its own layer of upkeep too: OpenAI retains API data for 30 days by default, and Anthropic cut its standard log retention to 7 days in September 2025, with reduced or zero-data retention options available at the enterprise tier. Teams building their own infrastructure have to track those policies themselves, provider by provider, instead of having them enforced automatically at a shared layer.

None of this is free, and the real cost is opportunity cost. Every hour spent keeping gateway infrastructure alive is an hour not spent on whatever actually makes the product worth using. The comparison teams run is "cost of building" against "the managed service's fee." The comparison that actually matters is the total cost of building and maintaining, indefinitely, against a managed fee plus whatever limitations that service carries. The first comparison is the one that gets built into the planning doc. The second is the one that gets ignored until year two.

How to run the build vs. buy evaluation as a structured decision rather than a gut call

Start by writing down where things actually stand, not where they're assumed to stand. How many providers are in use right now, and how many are planned over the next year? Can anyone answer, within a day, what the organization spends on LLMs broken down by team, feature, and model? Where does PII redaction logic actually live, and is it enforced the same way across every service that calls a model? How many engineering hours went into retry logic, provider updates, and key rotation last quarter?

Then run every build justification through the specific-requirement test. Write the exact requirement that no available gateway meets. If it can't be written down precisely, it isn't a justification, it's a feeling wearing a justification's clothes. Check whether an open-source option already covers the gap before assuming "build" means starting from a blank file, and be honest about the difference between a requirement (documented, contractual, architectural) and a preference (familiarity, comfort, a general dislike of vendor relationships).

Finally, scope the real cost of building, including everything that shows up after launch. This includes not just initial implementation, but also the maintenance, the provider integrations that never stop arriving, and the compliance work that gets more expensive the longer it waits. Weighed against a managed fee, that full picture is what actually decides this. Which option feels more in control never should be the deciding factor, and if it is, that's the tell the decision was made before the evaluation started.