31 products can sit between your app and the models — LLM gateways,
model routers, and the cloud platforms that act like both — and their headline pricing
hides the part that actually costs you money:
top-up feesCredit or top-up fee: A cut taken when you add money to a prepaid balance, typically around 5%. Easy to miss because it is not a markup on tokens — but you pay it on every dollar you load.Credit or top-up feeA cut taken when you add money to a prepaid balance, typically around 5%. Easy to miss because it is not a markup on tokens — but you pay it on every dollar you load., per-seat charges, retention surcharges,
and markupMarkup: A percentage the gateway adds on top of what the model actually costs. Some charge none at all and make money elsewhere.MarkupA percentage the gateway adds on top of what the model actually costs. Some charge none at all and make money elsewhere. on tokensToken: Roughly three quarters of a word. Billing is per million tokens, and output tokens usually cost several times more than input.TokenRoughly three quarters of a word. Billing is per million tokens, and output tokens usually cost several times more than input.. This is a
side-by-side LLM gateway comparison covering every one of them, with each figure
linked to the vendor's own page.
Almost every product here can call almost every model. What separates them is
who runs the software, what they add to the price, and how much control you get
over the data path. Sort out those three and the shortlist writes itself.
01
Who operates it
A managed service is running before lunch and costs a percentage forever. A
self-hostedSelf-hosted: You operate the software on your infrastructure and own its uptime, patching and upgrades. Model hosting, telemetry and control planes may still involve third parties; commercial editions can have licence fees.Self-hostedYou operate the software on your infrastructure and own its uptime, patching and upgrades. Model hosting, telemetry and control planes may still involve third parties; commercial editions can have licence fees.gatewayGateway: A single endpoint you send all your AI requests to, which then forwards them to whichever model you asked for. One integration instead of one per vendor.GatewayA single endpoint you send all your AI requests to, which then forwards them to whichever model you asked for. One integration instead of one per vendor. has
no vendor fee and costs you an on-call rotation. There
is no third answer, and picking wrong is the most expensive mistake on this page.
"Zero markupMarkup: A percentage the gateway adds on top of what the model actually costs. Some charge none at all and make money elsewhere.MarkupA percentage the gateway adds on top of what the model actually costs. Some charge none at all and make money elsewhere." and "no fees" mean different things at
different vendors. One
charges 5.5% when you add funds. One is free until you want your data deleted.
One is free per-request but $20 per developer per month. At scale these are not
rounding errors.
A SOC 2SOC 2: An audit of a company’s security controls. Type 1 checks the design at a point in time; Type 2 checks they actually operated over months. Procurement teams ask for this constantly.SOC 2An audit of a company’s security controls. Type 1 checks the design at a point in time; Type 2 checks they actually operated over months. Procurement teams ask for this constantly. report, a signed HIPAAHIPAA BAA: A signed contract required before you can put US health data through a vendor. Without one you cannot legally use the service for protected health information, regardless of how secure it is.HIPAA BAAA signed contract required before you can put US health data through a vendor. Without one you cannot legally use the service for protected health information, regardless of how secure it is.
agreement, EU-only data residencyData residency: A guarantee about which countries your data is processed in. Usually matters for EU obligations or public-sector contracts.Data residencyA guarantee about which countries your data is processed in. Usually matters for EU obligations or public-sector contracts., or a
guarantee that nothing is retainedZDR — zero data retention: A commitment that your prompts and responses are not stored after the request completes. Frequently a paid add-on or an enterprise-only option rather than the default.ZDR — zero data retentionA commitment that your prompts and responses are not stored after the request completes. Frequently a paid add-on or an enterprise-only option rather than the default.. These are pass-or-fail
requirements, and they
eliminate options far more decisively than any feature comparison.
A hyperscaler surface offering several vendors models under one contract, one bill, and one identity system. Strong compliance story, limited to that cloud.
A deliberate selection rather than a ranking: one product from each category,
all available to customers in the United States, chosen to show how differently
the five kinds are priced. How these were picked →
Open-source AI gateway and Python SDK that puts one OpenAI-compatible API in front of many LLM providers.
US company
Cost above the model bill
Self-hosted
OSS is free (MIT); you pay only for your own containers plus Postgres and Redis. Enterprise features (SSO, RBAC, JWT auth, SCIM, audit logs, support SLAs) require a paid LiteLLM commercial license whose price is not published.
Models
—
count not published
Failover
Spend limits
Logs
Semantic cache
Guardrails
Best for Platform teams that want a free, self-hosted, maximally broad provider abstraction with virtual keys and budgets, at moderate request volumes.
Oct 6 Choosing a gateway for realtime voice agents Evaluate live voice gateways for session authentication, interruptions, tool calls, reconnection and cost. Includes a conversational latency and failure-test worksheet.
A few questions, one shortlist
The wizard rules out anything that cannot meet your hard requirements, then ranks
what is left against what you said matters. It shows you why each one placed where
it did, and what is wrong with your top pick.
GatewayScore would like to record which comparisons, guides and tools people use, so the
research goes where it is actually read. No advertising, no data sold, no session recording,
and nothing is loaded until you choose. What gets recorded.