What is an LLM gateway?

An LLM gateway is a single control point in front of multiple model providers. Applications call one endpoint with one request shape, and the gateway handles routing, failover, caching, key management, cost tracking and request logging behind it. The property that distinguishes it from an ordinary API gateway is model-awareness: it understands tokens, per-model cost and provider health, and can make routing decisions on all three.

What is the difference between an LLM gateway and an AI gateway?

Most vendors use the terms interchangeably. TrueFoundry's own FAQ states that an LLM gateway and an AI gateway are generally considered the same thing, and LiteLLM markets one product under both labels. Some vendors treat AI gateway as the broader term, covering non-LLM models, MCP traffic and agent workflows alongside model calls. Read the capability list rather than the noun.

Do I need an LLM gateway?

Yes, once more than one team is calling more than one provider, or once nobody owns the aggregate spend and keys are living in three places. A gateway solves provider outages, opaque cost attribution and key sprawl directly. A single application on a single provider does not need one, and adding a hop for it buys latency without buying control.

Does an LLM gateway give me an audit trail?

It gives you a record of model traffic, which is a different thing. What is captured, how long it is retained, whether it is exportable and who can read it vary widely across vendors, and several advertise audit logging without specifying any of it. More importantly, the record covers what an agent asked a model, not what the agent then did with the answer. That action usually happens outside the gateway entirely.

What is the difference between an LLM gateway and an API gateway?

Model-awareness. An API gateway handles HTTP concerns: authentication, rate limiting, load balancing and SSL termination. OpenRouter notes it has no understanding of tokens, model capabilities or inference cost. An LLM gateway tracks token usage, per-model spend and provider health, and routes on those signals. The two solve different problems and are commonly run together.

LLM Gateway: What It Governs, and What It Cannot

Every LLM gateway page sells governance and none of them specify it. Here is what a gateway actually controls, the RBAC and credential-scoping questions to ask before you buy, and the class of risk that no gateway can see.

Rahul RamakrishnanAugust 28, 2026

Diagram showing an LLM gateway logging model traffic, with agent actions on business systems falling outside its field of view and governed by scoped permissions and audit logs.

Key takeaways

An LLM gateway is a single control point in front of multiple model providers, giving one API shape plus routing, failover, caching, key management, cost tracking and request logging.
Most vendors treat "LLM gateway" and "AI gateway" as the same product. TrueFoundry says so directly in its own FAQ.
Unified API, routing, failover, observability and cost tracking are near-universal. Caching, credential scoping and audit detail are not.
Governance is the most-claimed and least-specified capability in this category. Ask for mechanism, not posture.
A gateway sees the request and the response. It does not see what an agent does next, which is where the consequential risk sits.

What an LLM gateway actually is

An LLM gateway sits between your applications and the model providers they call. One endpoint, one request shape, many vendors behind it. TrueFoundry describes it as middleware that offers a single API, routes requests, standardizes formats, enforces authentication, tracks costs and supports failover. Portkey calls it a centralized control plane. An LLM gateway is model-aware: it understands tokens, model capabilities, provider health and inference cost.

LLM gateway vs AI gateway vs API gateway vs MCP gateway

Most vendors use LLM gateway and AI gateway interchangeably. OpenRouter offers a useful four-way distinction. An API gateway handles general HTTP concerns. An LLM gateway understands models and tokens. An agent gateway handles multi-step workflows and tools. An MCP gateway controls access to MCP tools and data sources.

API gateway: HTTP authentication, rate limits, load balancing and TLS. Concern: service traffic.
LLM gateway: models, tokens, cost and provider health. Concern: model routing.
Agent gateway: workflows, tools and handoffs. Concern: multi-step execution.
MCP gateway: MCP tools and data sources. Concern: tool access.

Why this matters now

Organizations call several model providers from several teams. Keys spread across services, spend becomes difficult to assign, and agents increasingly take actions after inference. A gateway standardizes model traffic, while the action still needs controls at its own layer.

What does an LLM gateway do?

The common capability set is a unified API, routing, failover, observability and cost tracking. Caching and key management vary by vendor, so verify the mechanism rather than trusting a feature label.

Unified API: one request shape across providers. Vendors consistently support this.
Routing: choose a provider or model using policy. Vendors consistently support basic routing.
Failover: retry or switch providers during an outage. Support is common, but retry safety varies.
Observability: record requests, responses, latency and usage. Support is common, while retention varies.
Cost tracking: attribute token use and spend. Support is common, while pricing data can lag.
Caching: reuse prior results. Exact-match caching is common; semantic caching is less consistent and can return a stale or mismatched answer.
Key management: issue and rotate scoped credentials. Support ranges from basic storage to virtual keys and secret-manager integrations.
Tenant isolation: separate projects, teams or customers. Verify isolation guarantees and administrative boundaries.

Five sub-concepts to get right

Routing and failover

Routing can be static configuration or a runtime policy based on model capability, provider health, cost or latency. Failover needs a defined fallback chain and circuit-breaking behavior. Retry safety matters when a request can trigger an external side effect. Compare these choices with routing strategies and their limits.

Caching, and what it costs you

Exact-match caching is easier to reason about because the request must match a stored key. Semantic caching compares meaning, which can reduce repeated inference while increasing the chance of returning an answer that fits the topic but misses current context. Set freshness rules, exclusions and evaluation tests before enabling it.

Key management and credential scoping

Ask whether keys are issued per application, team and environment, then check rotation, revocation and secret storage. LiteLLM documents virtual keys, budgets and integrations with Vault, AWS Secrets Manager and Azure Key Vault. A page that says control who can access models without describing those mechanisms is describing intent, not enforcement.

Audit logging and RBAC

An audit record should identify the caller, application, provider, model, timestamp, request identifiers, token counts, decision and outcome. Ask how long records remain, who can read them, how sensitive fields are redacted and whether roles map to your identity provider. A log that records a model call is useful, but it is not automatically a record of the agent's downstream work. See observability that isn't a tracing feature.

Guardrails

Gateways may inspect prompts and responses for PII, secrets or injection patterns. Those checks create false positives and false negatives, so teams need an escalation path, test corpus and policy for blocking versus warning. Review the agent threat model before treating a filter as a security boundary.

The governance questions to ask before you buy

Use this checklist to test mechanism-level governance:

Can you issue, rotate and revoke credentials per app, team and environment?
Can permissions be scoped by model, provider, project and action?
Which fields appear in each audit record?
How long are audit records retained, and can we export them?
Which administrators and teams can read the logs?
What is redacted per request, and can policies differ by data class?
How is multi-tenant isolation implemented and tested?
Does RBAC map to our identity provider and lifecycle controls?
How are retries made safe for requests with side effects?
What happens when the gateway is unavailable?

Where gateway governance stops

A gateway sees a model request and response. It generally does not see what an agent does next. If an agent reads a customer record, asks a model whether to issue a refund and then updates the system of record, the gateway can log the inference call. It cannot, by itself, prove which record changed, under whose permission or why the write was allowed. Read why control must move to the point of action.

Common misconceptions

A gateway makes AI usage governed

A gateway governs model traffic. Governance of the resulting business action needs controls where that action executes.

Cost dashboards are governance

Spend attribution is operational control. Governance also needs identity, permissions, retention, redaction and an accountable action record.

LLM and API gateways are interchangeable

They overlap on authentication and rate limits, but an LLM gateway understands models, tokens and provider health while an API gateway handles general HTTP traffic.

Model-call logs are an agent audit trail

The log covers the request and response. It may omit tool calls, record changes, messages sent and approvals that followed.

What this article does not cover

This article explains the gateway boundary and the questions to ask vendors. It does not provide a production configuration, benchmark latency across providers or rank gateways. Vendor latency claims are not methodologically comparable, and adding a hop can add latency, so measure your own traffic.

What we're building at Major in response

Major is not a gateway and does not replace one. If you have three teams on three providers, get a gateway. The layers are different and a team can reasonably run both.

Major is the enterprise platform where agents build the software they run on. When an agent works out how to handle a repeatable piece of work, it builds an app for that part: real code, with scoped credentials, role-based access, its own managed database and its own logs. What the agent did, to which record, under whose permissions, is inspectable because it happened in software rather than inside a prompt. Reason once, run forever.

Frequently asked questions

What is an LLM gateway?: An LLM gateway is a single control point in front of multiple model providers. Applications call one endpoint with one request shape, and the gateway handles routing, failover, caching, key management, cost tracking and request logging behind it. The property that distinguishes it from an ordinary API gateway is model-awareness: it understands tokens, per-model cost and provider health, and can make routing decisions on all three.
What is the difference between an LLM gateway and an AI gateway?: Most vendors use the terms interchangeably. TrueFoundry's own FAQ states that an LLM gateway and an AI gateway are generally considered the same thing, and LiteLLM markets one product under both labels. Some vendors treat AI gateway as the broader term, covering non-LLM models, MCP traffic and agent workflows alongside model calls. Read the capability list rather than the noun.
Do I need an LLM gateway?: Yes, once more than one team is calling more than one provider, or once nobody owns the aggregate spend and keys are living in three places. A gateway solves provider outages, opaque cost attribution and key sprawl directly. A single application on a single provider does not need one, and adding a hop for it buys latency without buying control.
Does an LLM gateway give me an audit trail?: It gives you a record of model traffic, which is a different thing. What is captured, how long it is retained, whether it is exportable and who can read it vary widely across vendors, and several advertise audit logging without specifying any of it. More importantly, the record covers what an agent asked a model, not what the agent then did with the answer. That action usually happens outside the gateway entirely.
What is the difference between an LLM gateway and an API gateway?: Model-awareness. An API gateway handles HTTP concerns: authentication, rate limiting, load balancing and SSL termination. OpenRouter notes it has no understanding of tokens, model capabilities or inference cost. An LLM gateway tracks token usage, per-model spend and provider health, and routes on those signals. The two solve different problems and are commonly run together.