What the words mean

This corner of the industry runs on jargon, some of it deliberately vague. Here is the plain version — first the vocabulary, then a definition of every single field this site tracks, so you always know exactly what a column is claiming.

You should not have to come here. Wherever one of these words appears in the text elsewhere on the site it is marked with a dotted underline — hover it, or tab to it, and the definition appears in place. Following the link brings you back here.

The vocabulary

Gateway: A single endpoint you send all your AI requests to, which then forwards them to whichever model you asked for. One integration instead of one per vendor.
BYOK — bring your own key: You keep your own accounts and contracts with OpenAI, Anthropic and the rest, and the gateway routes through your keys. You keep your negotiated rates and any committed-spend discounts; the gateway charges you for the plumbing, not the tokens.
Markup: A percentage the gateway adds on top of what the model actually costs. Some charge none at all and make money elsewhere.
Credit or top-up fee: A cut taken when you add money to a prepaid balance, typically around 5%. Easy to miss because it is not a markup on tokens — but you pay it on every dollar you load.
Token: Roughly three quarters of a word. Billing is per million tokens, and output tokens usually cost several times more than input.
Failover: When the provider you asked for is down or rate-limiting you, the gateway automatically retries somewhere else. The single most valuable reliability feature these products offer.
Load balancing: Spreading traffic across several keys or providers so no single one hits its rate limit. Different from failover, which only reacts to failure.
Conditional routing: Rules that pick the model per request — cheap model for simple work, expensive one for hard work, a specific model for one customer.
Semantic caching: Returning a stored answer when a new question means the same thing as an old one, even if worded differently. Saves more than exact-match caching but can return a subtly wrong answer, so it needs care.
Virtual keys: Separate scoped keys you issue per team, app, or customer, each with its own budget and limits, without handing out your real provider credentials.
Observability: Logs, traces, and dashboards showing what was sent, what came back, what it cost, and how long it took. The thing you desperately want the first time a bill surprises you.
Guardrails: Automated checks on requests and responses — blocking prohibited content, enforcing output formats, catching prompt injection.
ZDR — zero data retention: A commitment that your prompts and responses are not stored after the request completes. Frequently a paid add-on or an enterprise-only option rather than the default.
SOC 2: An audit of a company’s security controls. Type 1 checks the design at a point in time; Type 2 checks they actually operated over months. Procurement teams ask for this constantly.
HIPAA BAA: A signed contract required before you can put US health data through a vendor. Without one you cannot legally use the service for protected health information, regardless of how secure it is.
Data residency: A guarantee about which countries your data is processed in. Usually matters for EU obligations or public-sector contracts.
OpenAI-compatible: Speaks the same API shape as OpenAI’s, so existing client libraries work by changing a base URL. This is what makes switching gateways cheap — and worth insisting on.
Self-hosted: You run the software on your own infrastructure. No third party sees your traffic, and there is no vendor fee — but you own the uptime, the patching, and the upgrades.
Air-gapped: Runs with no outbound internet access at all. A hard requirement in some defence and regulated environments, and a genuine constraint on which products are even eligible.
Latency overhead: The delay the gateway itself adds, separate from the model’s own thinking time. Usually single-digit milliseconds and irrelevant next to a model taking several seconds — treat large claimed differences with suspicion.

The five kinds of product

These are not interchangeable, and comparing across categories on price alone will mislead you badly.

Managed marketplace: One account and one key gets you hundreds of models from dozens of providers. Fastest way to start, widest catalog, least control over the data path.
Managed gateway: A hosted control plane in front of the models: routing, caching, guardrails, spend limits, and logs. You trade a little simplicity for governance.
Open source: You run the gateway yourself. Full control over the data path and no per-request vendor fee, in exchange for operating it.
Cloud platform: A hyperscaler surface offering several vendors models under one contract, one bill, and one identity system. Strong compliance story, limited to that cloud.
Inference provider: Hosts open-weight models on its own hardware. Often the cheapest or fastest route to a specific open model, but it is one source, not a router.

Every field we track

113 fields in 8 groups. Fields marked technical are hidden until you switch the header toggle to Technical.

Overview

Category: Marketplaces resell many providers behind one key. Gateways add governance on top. Open-source projects you run yourself. Cloud platforms are hyperscaler surfaces. Inference providers host models on their own hardware. Shown in simple mode as “What kind of product”
Deployment model: Managed means the vendor operates it. Self-host means you run it on your own infrastructure. Both means you can choose. Shown in simple mode as “Who runs it”
Licence: Proprietary products cannot be inspected or forked. Open licences such as MIT and Apache-2.0 let you audit, modify, and run the code without permission.
Company: The organisation that maintains the product.
Vendor status: Whether the product is still an independent company, has been acquired, is a large cloud vendor’s product line, is run by a software foundation, or has been put into maintenance mode. Maintenance mode means bug fixes and security patches only — no new features. Shown in simple mode as “Who you would be signing with”
Latest release: The date of the most recent release or version tag. A product that has not shipped in a year is a different risk from one that shipped last week, regardless of what its marketing site says. Shown in simple mode as “Last shipped an update”
GitHub stars technical: A rough proxy for community size on open-source projects. Not a quality measure.

Cost

Token markup %: How much the product adds on top of what the underlying model provider charges. Zero means you pay the same per-token price you would pay the model provider directly. Shown in simple mode as “Markup on model prices”
Credit purchase fee %: A percentage charged when you top up your balance, separate from token prices. It is easy to miss because it does not appear on the per-token price list. Shown in simple mode as “Fee to add funds”
Seat fee /user/mo: A recurring per-user platform charge that applies regardless of how much you use the models. Shown in simple mode as “Monthly cost per person”
BYOK supported: Bring Your Own Key: you keep direct contracts with OpenAI, Anthropic and others, and the gateway only routes traffic. This preserves negotiated rates and committed-spend discounts. Shown in simple mode as “Can use your own provider accounts”
BYOK terms: What the product charges to route traffic through your own provider keys. Shown in simple mode as “Cost of using your own accounts”
Free tier: What you can do without paying, useful for evaluation.
Enterprise plan from /yr: Annual entry price for the enterprise tier, where one is published or credibly reported.
Self-host cost technical: What self-hosting actually costs once you account for infrastructure and any paid tier. Shown in simple mode as “Cost to run it yourself”
Pricing model: The shape of the vendor’s bill: does the routing layer charge a percentage on top of tokens, a flat monthly fee, both, neither (because inference is the product), or nothing at all (open source with no paid tier). Shown in simple mode as “How the vendor makes money”
Pricing model detail: A one-paragraph description that covers the caveats a pricing category cannot: introductory rates, per-feature meters, tier gating, and pricing that resets on a specific date. Shown in simple mode as “How pricing works, briefly”
Minimum commitment: Whether the vendor requires a minimum contract term, a minimum spend, or a provisioned-capacity purchase to get its published rate.
Overage terms: The line items that scale with usage after an included allowance is exhausted — log storage, extra requests, per-feature meters, data export — which is where cost estimates usually go wrong. Shown in simple mode as “Charges that fire after you go over an allowance”
Cache mechanism: Whether the gateway offers its own response cache, what kind of match it does (exact request, prefix, semantic), or simply passes provider caching through unchanged. Shown in simple mode as “Prompt cache offered”
Cache-read discount %: How much cheaper cached tokens are than fresh input, when the vendor publishes a single figure. Bundled-inference clouds usually price this per model instead of as one number. Shown in simple mode as “Discount on cached input”
Cache-write premium %: How much more the first write of a cached prefix costs versus a plain input token. A high write premium and a low hit rate can leave you paying more than you save, so this matters as much as the read discount. Shown in simple mode as “Premium on cache writes”
Cache economics: Whether the customer keeps the full saving from caching or the vendor captures part of it — and any conditions attached (write premium, storage fees, best-effort hits). Shown in simple mode as “Who captures the cache saving”
Cost attribution: The dimensions the vendor documents for splitting spend — per key, per user, per team, per tag, per customer. Matters if you need to chargeback internally or bill an end customer. Shown in simple mode as “What you can split spend by”
Cost export: The mechanisms the vendor publishes for exporting cost and usage data: CSV, an API, webhooks, S3, a data warehouse, or nothing at all. Any per-unit price is included. Shown in simple mode as “How you get cost data out”
BYOK mode: Whether you bring your own provider accounts (BYOK), buy inference from this vendor, or can do either. This is the single biggest commercial difference between these products: it decides who holds the contract with the model provider and who carries the spend. Shown in simple mode as “Who pays the model bill”

Catalog

Models available: How many models you can call, shown as a range because several vendors publish different totals on different pages. Vendor-reported either way, so counts are not directly comparable — some count every provider variant of the same model separately.
Upstream providers: How many distinct model providers or labs you can reach, shown as a range where the vendor’s own pages disagree. More providers usually means better redundancy when one has an outage. Shown in simple mode as “Model providers reachable”
OpenAI-compatible API: If yes, you can usually switch to it by changing one base URL, and switch away just as easily. This is the main defence against lock-in. Shown in simple mode as “Works with standard OpenAI code”
POST /v1/chat/completions: The endpoint almost every application ports first. "Not documented" means the vendor never states it, which is different from a documented no. Shown in simple mode as “OpenAI chat endpoint”
POST /v1/messages: Whether Anthropic-shaped calls work without rewriting them. Several products support this only as SDK compatibility or provider passthrough rather than a native endpoint — the detail page says which. Shown in simple mode as “Anthropic messages endpoint”
POST /v1/responses: The newer stateful OpenAI surface. Support is much thinner across this market than chat completions. Shown in simple mode as “OpenAI Responses endpoint”
POST /v1/embeddings: Whether you can generate vectors through the same gateway, or need a second integration for retrieval workloads. Shown in simple mode as “Embeddings endpoint”
POST /v1/images/generations: Whether image models are reachable through the same surface as text. Shown in simple mode as “Image generation endpoint”
POST /v1/audio/*: Speech-to-text and text-to-speech. Frequently the first gap in an otherwise complete gateway. Shown in simple mode as “Audio endpoints”
POST /v1/batches: Asynchronous bulk processing, usually at a discount. Commonly undocumented, and commonly the reason a migration stalls late. Shown in simple mode as “Batch jobs endpoint”
Requires a vendor-specific SDK: A proprietary client library spreads through your codebase and has to be torn out again if you leave. “No” is the better answer here, and it means the standard OpenAI client works. Shown in simple mode as “Needs the vendor’s own code library”
Logs / usage data export: Whether you can get your own request logs, traces, or usage records back out — through an API, a bulk export, or a download. Decides whether you leave with your history or abandon it. Shown in simple mode as “You can export your request history”
Declarative config-as-code technical: Whether routing, fallback, and budget rules can be declared in a file you keep in Git, rather than existing only as settings clicked into a hosted dashboard. Shown in simple mode as “Settings can live in version control”
Embeddings technical: Text-to-vector models, needed for search and retrieval features.
Image generation technical: Whether image models are reachable through the same interface.
Speech and audio technical: Text-to-speech or transcription models through the same interface.
Video generation technical: Whether video models are reachable through the same interface.
Batch processing technical: Submitting large jobs for cheaper, slower processing. Often 50% off for work that is not time-sensitive.

Routing & reliability

Contractual SLA uptime: The uptime percentage in a published, contractual service level agreement. A public status page is not an SLA — it reports what happened, it does not promise anything or pay you back when it breaks. Shown in simple mode as “Uptime it promises in writing”
Automatic failover: When a model provider goes down or rate-limits you, traffic moves to a backup automatically instead of returning errors to your users.
Load balancing: Spreads requests across several providers or keys to raise your effective rate limit.
Conditional routing: Send different requests to different models based on rules — for example a cheap model for free users and a strong model for paying ones. Shown in simple mode as “Rule-based routing”
Response caching: Reuses the answer when the exact same request comes in again, which cuts both cost and latency.
Semantic cache: Reuses an answer when a new question means roughly the same thing as an earlier one. Saves far more than exact-match caching but can return subtly wrong answers if tuned loosely. Shown in simple mode as “Similar-question caching”
Request timeout surface: Where a request timeout can be set: per request, in a config file, in the vendor dashboard, or nowhere. Nine of the twenty products documented here do not describe a request timeout at all, so the worst case of a hung upstream call is unknowable from the docs. Shown in simple mode as “Where you set the timeout”
Retry policy surface: Where retry count and backoff are configured. Worth knowing alongside billing: a retried streaming call can be charged more than once. Shown in simple mode as “Where you set retries”
Fallback surface: Where the fallback chain is defined. Almost every product claims fallback; the useful question is whether you can change it from code or only by hand in a dashboard. Shown in simple mode as “Where you set fallbacks”
Fallback shape technical: An ordered list tries targets in sequence; a weighted split sends a percentage of traffic to each, which is what you need to trial a new model on 5% of requests. Weighted splits are much rarer than the marketing implies. Shown in simple mode as “Shape of the fallback chain”
Health checks / circuit breaking: Whether the product notices a failing upstream and stops sending traffic to it, and whether you can tune the thresholds. This is what turns a provider outage into a blip rather than a sustained error rate, and only four of the twenty expose it. Shown in simple mode as “Upstream health tracking”
Multi-region failover surface: Whether you can define what happens when a region degrades. A vendor running many regions is not the same as a vendor letting you configure failover between them; only three document a user-controlled mechanism. Shown in simple mode as “Cross-region failover you control”
Load balancing surface: Where traffic distribution across upstreams or keys is configured. Shown in simple mode as “Where you set load balancing”

Operations

Observability: Built-in visibility into what was sent, what came back, what it cost, and how long it took. Shown in simple mode as “Usage dashboards and logs”
Budget controls: Hard caps that stop spend before it becomes a surprise invoice. The single most valuable control for a small team. Shown in simple mode as “Spending limits”
Rate limits: Caps on request volume per key or per user, useful for protecting against abuse and runaway loops.
Virtual keys: Issue scoped keys with their own budgets and permissions so you can attribute cost and revoke access without rotating everything. Shown in simple mode as “Separate keys per team or app”
Prompt management technical: Store and version prompts outside your code so they can be changed without a deploy. Shown in simple mode as “Prompt versioning”
Evals technical: Built-in tooling to score model output against test cases, so you can tell whether a model swap made things better or worse. Shown in simple mode as “Quality testing”
MCP support technical: Native support for the Model Context Protocol, the emerging standard for connecting models to external tools.
Logged content: Whether full prompts and responses are stored, only metadata, or nothing. Full-body logging is the most useful debugging feature here and the one most likely to need a conversation with your compliance team. Shown in simple mode as “What gets logged”
Body-logging opt-out: Whether prompt and response bodies can be suppressed while still keeping usage metrics. Nineteen of the twenty document a way to do this; the mechanisms range from a per-request header to an organisation-wide setting. Shown in simple mode as “You can turn logging off”
Distributed tracing: Whether the product emits OpenTelemetry, a proprietary format, or nothing. OpenTelemetry means the traces land in the tooling you already run instead of only in the vendor’s dashboard. Shown in simple mode as “Traces you can take elsewhere”
Export destinations: Documented sinks for logs and metrics. This is a good proxy for how replaceable the vendor’s own dashboard is: around twenty destinations means you never have to depend on it, while a CSV download means you do. Shown in simple mode as “Where telemetry can go”
Feedback capture API technical: Whether there is an API to attach a rating or score to a logged request, which is what lets production traffic feed quality work later. Shown in simple mode as “Can record user feedback”
Online eval hooks technical: Whether automated scorers can run against real production requests, rather than only against a test set you assemble yourself. Shown in simple mode as “Scores live traffic”

Performance

Proxy overhead ms: Extra time the product itself adds to each request, on top of however long the model takes. Usually irrelevant next to multi-second model latency, but it matters for high-volume or streaming-sensitive workloads. Shown in simple mode as “Delay it adds”
Throughput technical rps: Published sustained request rate before the product becomes the bottleneck. Only relevant at genuinely high volume. Shown in simple mode as “Requests per second ceiling”
Architecture class: The comparable way to talk about latency here. An edge worker, a compiled Go or Rust binary, and a Python proxy have different overhead floors no matter which figures each vendor publishes. Products that never disclose their runtime are recorded as undisclosed rather than assumed. Shown in simple mode as “What the request path runs on”
Self-hostable data plane: Whether the component that actually carries your prompts can run on your own infrastructure. Distinct from a vendor offering a self-hosted control plane while still proxying traffic through their network. Shown in simple mode as “You can run the request path yourself”
Streaming support: Whether token-by-token streaming is documented. The caveats matter more than the yes: some products cannot cancel a stream without still being billed, and several timeout and fallback mechanisms stop applying once the first token has been sent. Shown in simple mode as “Streaming responses”

Security & compliance

Prompt transits vendor: Whether the text you send passes through this company’s own infrastructure. If it does, every other promise on this page is a policy commitment rather than a physical impossibility. Self-hosted products can answer no outright. Shown in simple mode as “Does your prompt reach their servers”
Logging default: Defaults matter more than options. A product that stores full prompts and replies unless you find the right header will have stored them by the time you read the docs. Shown in simple mode as “What they keep if you change nothing”
Default content retention (days): Default retention for request content, in days. Zero means nothing is kept. Read the note: several products keep nothing as a rule but make timed exceptions for abuse review or specific models. Shown in simple mode as “How long they keep it”
Training on customer data: Whether the vendor may use your prompts and outputs to train models. “Not published” means we could not find any position, which is not the same as a no — ask for it in writing. Shown in simple mode as “Could they train on your prompts”
Region and residency control: Which regions are offered and whether you can force processing to stay in one. A global endpoint that silently picks a region is a different compliance story from an endpoint you pin yourself. Shown in simple mode as “Where it runs, and what you can pin”
Guardrail execution location: A filter that strips personal data only helps if it runs before the data leaves your boundary. If guardrails execute in the vendor’s cloud, the vendor has already received whatever you wanted redacted. Shown in simple mode as “Where safety filters run”
Subprocessor list: The published list of third parties the vendor passes your data to. No list means you cannot know the full chain, which most data-protection agreements require you to. Shown in simple mode as “Who else touches the data”
SOC 2 audited: An independent audit of security controls. Enterprise buyers and their procurement teams routinely require it.
HIPAA BAA: Required before you may send protected health information through the service. Without a signed BAA, healthcare data is off limits. Shown in simple mode as “Will sign a HIPAA agreement”
GDPR commitments: Published data processing terms for handling personal data of people in the EU and UK.
EU data residency: Requests can be processed inside the EU rather than routed to US infrastructure. Often the deciding constraint for European customers. Shown in simple mode as “Can keep data in the EU”
Zero data retention: Prompts and responses are not stored after the request completes. Sometimes a paid add-on rather than the default. Shown in simple mode as “Does not retain your data”
PII redaction: Detects and removes identifiers such as names, emails, and card numbers before the request reaches the model provider. Shown in simple mode as “Strips personal data”
Content guardrails: Policy checks on inputs and outputs — blocking unsafe content, enforcing formats, or catching prompt-injection attempts.
Air-gapped deployment technical: Can be deployed in a network with no internet access, which some regulated and defence environments require. Shown in simple mode as “Runs fully disconnected”
PII / DLP enforcement: Whether personal data detection sits on the request path and can stop the call, merely inspects and forwards it, or is not documented. A control that only reports is a logging feature, not a policy control. Shown in simple mode as “Blocks personal data in prompts”
Injection / jailbreak enforcement: Whether injection and jailbreak detection can stop a request. Most products offering this call a partner classifier rather than shipping their own. Shown in simple mode as “Blocks prompt injection”
Toxicity / moderation enforcement: Whether hate, violence, sexual and self-harm categories are checked inline and can stop a request, in either direction. Shown in simple mode as “Blocks harmful content”
Custom policy hooks: Whether you can add your own rule — a regex, a webhook, or your own classifier — rather than choosing from the vendor library. Shown in simple mode as “Your own policy rules”
Guardrail execution location: Whether guardrail evaluation happens inside your infrastructure or on the vendor’s servers. This decides whether the prompt you are trying to protect leaves your network in order to be checked. Shown in simple mode as “Where guardrails run”
Guardrail failure mode: What happens when the guardrail service times out or errors: does the request proceed unchecked, or is it blocked? This is the worst-documented field in the entire catalogue — only two vendors state it plainly, which means most teams are running a control whose failure behaviour they cannot know. Shown in simple mode as “If the guardrail itself fails”
Guardrail integrations technical: Named external guardrail services the product can call. A long list means the product is a router for policy engines rather than a policy engine itself — which also means another vendor bill and another hop. Shown in simple mode as “Third-party guardrail vendors”

Fit & integration

Evaluation work shape: The shape of the work on the vendor’s own quickstart, from swapping one base URL through to deploying infrastructure. An ordinal class rather than a duration, because elapsed time depends on accounts and quota we cannot see. Shown in simple mode as “Work to try it”
Production work shape: The same scale applied to the vendor’s recommended production path. For several products this is much heavier than the quickstart, which is exactly why both are recorded. Shown in simple mode as “Work to run it”
Numbered quickstart steps technical: A literal count of numbered steps on the vendor’s quickstart, recorded as evidence beside the work shape. Zero means the page publishes no numbered procedure at all. Large counts usually mean interleaved language tracks rather than more work. Shown in simple mode as “Steps on the quickstart”
Self-host install documentation: Whether an install command is actually published. Several products advertise self-hosting while publishing no command to start from, which a plain yes/no would hide. Shown in simple mode as “Can you self-host it today”
OpenAI SDK drop-in: Whether an existing OpenAI-compatible client can be pointed at it by changing the base URL and key. Shown in simple mode as “Works with the OpenAI SDK”
AI SDK provider package: Whether a first-party AI SDK provider package exists, or only a community package, a documented workaround, or the generic OpenAI provider pointed at a custom base URL. Shown in simple mode as “Vercel AI SDK support”
Documented Python frameworks: LangChain, LangGraph, LlamaIndex and similar orchestration frameworks with a documented integration. Shown in simple mode as “Python framework integrations”
Cloudflare Workers support: Whether the docs show calling this product from your own Worker. Deliberately separated from the several products whose own gateway runs on Workers, which is a fact about their infrastructure and not about your edge compatibility. Shown in simple mode as “Callable from Cloudflare Workers”
Helm chart availability technical: Whether a named, published Helm chart exists, versus Helm being referenced with no chart named, versus generic cluster documentation that has nothing to do with this product. Shown in simple mode as “Kubernetes install”
Terraform provider or modules technical: Whether you can declare this in code: an official provider, official modules, resources inside a hyperscaler’s provider, a community provider, or only Terraform code shipped in a repo. Shown in simple mode as “Terraform support”
Cloud IAM reuse technical: Whether you can authenticate with IAM roles, workload identity or managed identities instead of another long-lived API key. Distinguished from products that only accept static upstream provider credentials. Shown in simple mode as “Reuses your cloud identity”
API gateway integration technical: Whether AI traffic can go through a gateway you already run, and who documented that — the product’s vendor or the gateway’s. Shown in simple mode as “Fits behind your API gateway”
MCP surface shape: Which kind of MCP support this is: a gateway that governs many MCP servers, a hosted MCP server you connect to, MCP tools accepted inside the completion API, or client tooling. These are different products behind one acronym. Shown in simple mode as “MCP support”
Upstream provider key required: Whether an upstream provider account and key must exist before your first call works. A prerequisite rather than a step, and it can differ between the hosted and self-hosted forms of the same product. Shown in simple mode as “Needs your own provider key”
Model access gate: Whether an enablement click, a quota grant, a paid tier or an approval form stands between a valid key and a working model call. Shown in simple mode as “Gate before models work”
First-party client SDK languages technical: Languages with a first-party client library. An empty list can still mean the product is usable from any language via an OpenAI-compatible SDK. Shown in simple mode as “Official client languages”