Guide
What is an LLM gateway?
Short answer
An LLM gateway is a service between an application and model providers that centralizes access to their APIs. It can translate request formats, route calls and apply policies. Budget enforcement, logging, retries and failover are optional capabilities to verify for the product, edition and endpoint you use.
What an LLM gateway replaces
Request path: application → gateway policies and routing → model provider → gateway → application. A separate logging or tracing destination may receive a copy; hosting the gateway yourself does not automatically keep inference local.
Without a gateway, your application or SDK adapter handles provider authentication, model identifiers and response differences. A gateway can centralize that work, but a new provider may still need application changes for tool calls, streaming or provider-specific options. API compatibility needs a contract test, not just a base-URL change.
A gateway can attribute spend for traffic that passes through it. It cannot see direct calls that bypass it, and application instrumentation, provider billing exports and cloud billing can also supply accounting evidence. Reconcile those sources before treating a gateway dashboard as the complete bill.
It is also a control point, and control points fail. A gateway you self-host is a service you now operate, and a managed gateway is a dependency in front of your dependency — its outage is your outage even when the model provider is healthy. That trade is the substance of most of the choosing, and it is where the shortlist guide starts.
Gateway, router, proxy, API gateway, MCP gateway
These five are used interchangeably in marketing copy and mean different things. The distinction that does the most work is the third column: whether the component understands tokens, model names and per-call cost, or is simply moving HTTP around.
| Term | What it decides | Token-aware | Example |
|---|---|---|---|
| LLM gateway | Terminates the call. One request format in, a normalised response out, with optional accounting, limits and failover. | Yes | OpenRouter — the whole product is this |
| Model router | Which model or provider serves this particular request — by cost, by latency, by availability, or by a rule you wrote. | Yes | a feature inside most gateways here, not a separate product |
| Reverse proxy | Nothing about the payload. Forwards bytes, terminates TLS, balances load. Does not know it is carrying a completion. | No | generic nginx or Envoy proxy deployments; distinct from the Envoy AI Gateway project |
| API gateway | HTTP-level policy: authentication, quotas and routing. AI-specific policies can add token and model awareness. | With AI policies | Kong AI Gateway — in scope only because of its AI-specific plugins |
| MCP gateway | Which tools a model may reach, and under whose credentials. Fronts MCP servers rather than model endpoints. | No | Bifrost — ships one alongside its LLM gateway |
The BYOKBYOK — bring your own key: You keep your own accounts and contracts with OpenAI, Anthropic and the rest, and the gateway routes through your keys. You keep your negotiated rates and any committed-spend discounts; the gateway charges you for the plumbing, not the tokens. question cuts across all five and is worth separating out: whether the component holds your provider keys or you hold them. That is a commercial and security decision rather than an architectural one, and it is covered in the pricing guide.
Do you need one
You probably want a gateway if
- You call more than one model provider, or expect to within the year.
- You cannot currently answer "which team or feature spent this money".
- Provider keys are pasted into more than one service or environment.
- A provider outage takes a user-facing feature down with it.
- You need per-team budgets, or rate limits you set rather than inherit.
- Someone has asked for a log of prompts and responses for review or audit.
You probably do not, yet, if
- One provider, one application, one key, and no plans to change that.
- Spend is small enough that a monthly invoice is sufficient attribution.
- Your latency budget genuinely cannot absorb an extra hop.
- You have nobody to operate it and are not willing to take on a managed dependency in the request path.
If accounting is the immediate need, compare a gateway with application instrumentation and existing billing exports first. Adopt the extra dependency when its central policies and operations justify the cost.
What this site tracks
This curated catalogue currently includes 31 products; 17 record a self-hosting option. It includes purpose-built gateways and API infrastructure with AI-specific extensions. Envoy AI Gateway is included. This is a researched sample, not a complete list of qualifying products.
Azure API Management has documented LLM token metrics and token-limit policies (reviewed September 16, 2026). Its absence from this catalogue is a coverage gap, not evidence that it lacks model-level accounting.
Inclusion requires an assessed product in the model request path. Tools that only manage prompts or evaluate saved outputs do not qualify on that basis alone. Consult the maintained scope policy for coverage limits; an absent profile is not a negative product verdict.
Every figure on a provider page carries the date it was checked and a link to the vendor page it came from. Where a vendor publishes a number we cannot verify — most latency and throughput claims — it is labelled as the vendor's own rather than presented as a measurement. The methodology explains how the scoring works and what it deliberately refuses to score.
Common questions
Is an LLM gateway the same thing as a model router?
No. Routing is one decision a gateway can make — which model or provider should serve this request. A gateway is the component that terminates the request, can authenticate it, translate its schema, account for spend and return the response. You can have a gateway that never routes, because you always name the model yourself, and routing logic that is not a gateway, because it is a library call inside your own process.
Do I need a gateway if I only ever call one provider?
Usually not for the API translation, which is the part you do not need, but often yes for the accounting. The reasons single-provider teams adopt one are spend visibility per team or per feature, key rotation without a redeploy, rate limiting before the provider does it for you, and a retry and fallback path for the hour the provider is down. If none of those is a live problem, the provider SDK is fine.
Does an LLM gateway add latency?
It adds a network hop, so yes, some. How much depends far more on where the gateway runs relative to your application and the model than on the gateway software. Self-hosting it next to your app is the low-latency configuration; calling a managed gateway in another region is the slow one. Treat published overhead figures with suspicion: they are vendor-measured on vendor hardware and this catalogue marks them as such rather than comparing them.
Is an AI gateway the same as an LLM gateway?
In practice yes — the two names are used for the same category, and several products in this catalogue call themselves an AI gateway while doing exactly what is described above. "AI gateway" is sometimes stretched to cover embeddings, image and speech endpoints as well as chat, which is a difference of coverage rather than of kind.
Can I use the API gateway I already run instead?
Only if it has been taught the model-level concepts. A conventional API gateway manages HTTP: authentication, quotas, request rates, paths. Without AI-specific policies it need not understand tokens, model names or inference cost. Verify those extensions separately. This is why the established gateway vendors ship separate AI-specific plugins rather than telling you to use the generic product.
What is an MCP gateway, and do I need one as well?
An MCP gateway brokers tools, not models. Model Context Protocol servers expose capabilities a model can call — a database, a search index, an internal service — and an MCP gateway sits in front of those servers to centralise their auth, discovery and limits. It answers a different question from an LLM gateway, and some products in this catalogue now do both. You need one when you have more MCP servers than you can govern by hand, which is unrelated to how many models you call.