Guide
What is an LLM gateway?
Short answer
An LLM gateway is a single API endpoint that sits between your application and the model providers behind it. You send one request format; the gateway authenticates it, forwards the call to whichever model is named or chosen, and returns a normalised response — while logging the spend, enforcing your limits, retrying failures and falling back when a provider breaks.
What it actually replaces
Without one, every model provider you adopt is a separate integration: its own SDK, its own auth, its own error shapes, its own idea of what a streamed response looks like. The second provider costs about as much work as the first. The gateway collapses that into one interface, so adding a model becomes a configuration change rather than a code change.
The part teams underestimate is the accounting. A gateway is the only place in the request path that sees every call your organisation makes, which makes it the only honest place to answer who spent what, which feature is expensive, and which key should be revoked. That is why gateways with no routing intelligence at all still earn their place, and why several products in this catalogue are really observability tools that grew a gateway.
It is also a control point, and control points fail. A gateway you self-host is a service you now operate, and a managed gateway is a dependency in front of your dependency — its outage is your outage even when the model provider is healthy. That trade is the substance of most of the choosing, and it is where the shortlist guide starts.
Gateway, router, proxy, API gateway, MCP gateway
These five are used interchangeably in marketing copy and mean different things. The distinction that does the most work is the third column: whether the component understands tokens, model names and per-call cost, or is simply moving HTTP around.
| Term | What it decides | Token-aware | Example |
|---|---|---|---|
| LLM gateway | Terminates the call. One request format in, a normalised response out, with spend, limits and failover applied on the way through. | Yes | OpenRouter — the whole product is this |
| Model router | Which model or provider serves this particular request — by cost, by latency, by availability, or by a rule you wrote. | Yes | a feature inside most gateways here, not a separate product |
| Reverse proxy | Nothing about the payload. Forwards bytes, terminates TLS, balances load. Does not know it is carrying a completion. | No | nginx, Envoy — infrastructure, not in this catalogue |
| API gateway | HTTP-level policy: authentication, quotas, request rate, routing by path. No concept of a token or a model. | No | Kong AI Gateway — in scope only because of its AI-specific plugins |
| MCP gateway | Which tools a model may reach, and under whose credentials. Fronts MCP servers rather than model endpoints. | No | Bifrost — ships one alongside its LLM gateway |
The BYOKBYOK — bring your own key: You keep your own accounts and contracts with OpenAI, Anthropic and the rest, and the gateway routes through your keys. You keep your negotiated rates and any committed-spend discounts; the gateway charges you for the plumbing, not the tokens. question cuts across all five and is worth separating out: whether the component holds your provider keys or you hold them. That is a commercial and security decision rather than an architectural one, and it is covered in the pricing guide.
Do you need one
You probably want a gateway if
- You call more than one model provider, or expect to within the year.
- You cannot currently answer "which team or feature spent this money".
- Provider keys are pasted into more than one service or environment.
- A provider outage takes a user-facing feature down with it.
- You need per-team budgets, or rate limits you set rather than inherit.
- Someone has asked for a log of prompts and responses for review or audit.
You probably do not, yet, if
- One provider, one application, one key, and no plans to change that.
- Spend is small enough that a monthly invoice is sufficient attribution.
- Your latency budget genuinely cannot absorb an extra hop.
- You have nobody to operate it and are not willing to take on a managed dependency in the request path.
The middle case is the common one: the accounting is worth it before the multi-provider abstraction is. That is the point at which a hosted gateway you can adopt in an afternoon beats a self-hosted one you have to staff.
What this site tracks
20 products, of which 11 can be run on your own infrastructure. Two of them — Kong AI Gateway and Apache APISIX AI Gateway — are conventional API gateways, in scope only because they ship LLM-specific plugins that make them token-aware. The rest were built for this job.
Out of scope by definition: Azure API Management, which is a general-purpose API gateway with no model-level accounting and therefore belongs in the fourth row of the table above rather than in the catalogue. Products that only switch between models inside one vendor's own catalogue are out for the same reason, as are hosted prompt-management and evaluation tools that never sit in the request path.
Separately, and worth stating plainly because an absence looks like a verdict: some products qualify and are simply not published yet. Envoy AI Gateway has been generally available since June 2026 and is the strongest open-source, Kubernetes-native gateway currently missing here. Martian runs a live OpenAI-compatible endpoint, but the company has publicly refocused on interpretability research, so we want its commercial status confirmed before putting a row behind it. Opper we have not assessed at all. None of those three is a judgement on the product.
Every figure on a provider page carries the date it was checked and a link to the vendor page it came from. Where a vendor publishes a number we cannot verify — most latency and throughput claims — it is labelled as the vendor's own rather than presented as a measurement. The methodology explains how the scoring works and what it deliberately refuses to score.
Common questions
Is an LLM gateway the same thing as a model router?
No. Routing is one decision a gateway can make — which model or provider should serve this request. A gateway is the component that terminates the request, authenticates it, translates the schema, accounts for the spend and returns the response. You can have a gateway that never routes, because you always name the model yourself, and routing logic that is not a gateway, because it is a library call inside your own process.
Do I need a gateway if I only ever call one provider?
Usually not for the API translation, which is the part you do not need, but often yes for the accounting. The reasons single-provider teams adopt one are spend visibility per team or per feature, key rotation without a redeploy, rate limiting before the provider does it for you, and a retry and fallback path for the hour the provider is down. If none of those is a live problem, the provider SDK is fine.
Does an LLM gateway add latency?
It adds a network hop, so yes, some. How much depends far more on where the gateway runs relative to your application and the model than on the gateway software. Self-hosting it next to your app is the low-latency configuration; calling a managed gateway in another region is the slow one. Treat published overhead figures with suspicion: they are vendor-measured on vendor hardware and this catalogue marks them as such rather than comparing them.
Is an AI gateway the same as an LLM gateway?
In practice yes — the two names are used for the same category, and several products in this catalogue call themselves an AI gateway while doing exactly what is described above. "AI gateway" is sometimes stretched to cover embeddings, image and speech endpoints as well as chat, which is a difference of coverage rather than of kind.
Can I use the API gateway I already run instead?
Only if it has been taught the model-level concepts. A conventional API gateway manages HTTP: authentication, quotas, request rates, paths. It has no notion of a token, a model name, a streamed completion or a cost per million tokens, so it cannot attribute spend or fail over from one model to another. This is why the established gateway vendors ship separate AI-specific plugins rather than telling you to use the generic product.
What is an MCP gateway, and do I need one as well?
An MCP gateway brokers tools, not models. Model Context Protocol servers expose capabilities a model can call — a database, a search index, an internal service — and an MCP gateway sits in front of those servers to centralise their auth, discovery and limits. It answers a different question from an LLM gateway, and some products in this catalogue now do both. You need one when you have more MCP servers than you can govern by hand, which is unrelated to how many models you call.