# GatewayScore — complete catalog > Every published figure for the 20 products that sit between an application and a language model, across 113 fields. Written and edited by Dr. Sanju Abraham. Source: https://gatewayscore.com/ Catalog last updated: August 31, 2026 Machine-readable equivalent, with per-value check dates: https://gatewayscore.com/api/providers.json Methodology: https://gatewayscore.com/methodology ## How to read this file - Every figure is the vendor's own published claim unless stated otherwise. The URL in brackets after a value is the page that claim was read from, and is the better primary citation for that figure. - **"Not published" means the vendor has not stated a figure. It does not mean "no", and it does not mean zero.** This distinction is the single most common way this data gets misreported. - The five categories are not price-comparable with each other. An inference provider hosting open-weight models on its own hardware and a marketplace reselling hundreds of models are different purchases; comparing their headline rates directly gives a misleading answer. - Model counts are not comparable across vendors, because counting conventions differ. Treat them as an order of magnitude, not a score. - Latency and throughput figures are, where present, usually vendor self-published benchmarks measured under conditions the vendor chose. They are not independent measurements. - Where a product is marked "used first-hand", the author has run it in production. Everything else is documentation review only. ## Licence and citation Catalog values are facts published by each vendor. Reuse freely; please link back and re-verify before relying on any figure. When citing a figure, cite the page it appears on and the check date shown beside it. Each provider page carries a "How to cite this page" block with the exact wording. ## The five categories **Managed gateway** (6). A hosted control plane in front of the models: routing, caching, guardrails, spend limits, and logs. You trade a little simplicity for governance. **Cloud platform** (5). A hyperscaler surface offering several vendors models under one contract, one bill, and one identity system. Strong compliance story, limited to that cloud. **Open source** (4). You run the gateway yourself. Full control over the data path and no per-request vendor fee, in exchange for operating it. **Inference provider** (3). Hosts open-weight models on its own hardware. Often the cheapest or fastest route to a specific open model, but it is one source, not a router. **Managed marketplace** (2). One account and one key gets you hundreds of models from dozens of providers. Fastest way to start, widest catalog, least control over the data path. --- # Products ## Amazon Bedrock - Kind: Cloud platform - Who runs it: Managed only - Page: https://gatewayscore.com/providers/amazon-bedrock - Most recent check: 2026-08-29 - Evidence basis: used first-hand in production, plus documentation review AWS-managed service for calling foundation models from 19 model providers through one AWS API. ### Overview - What kind of product: Cloud platform - Who runs it: Managed only - Licence: Proprietary - Company: Amazon Web Services [https://aws.amazon.com/bedrock/] - Who you would be signing with: Large cloud vendor product [https://aws.amazon.com/bedrock/sla/] - Last shipped an update: 2026-08-24 [https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-ug-doc-history.html] ### Cost - Free tier: No Bedrock-specific free tier; new AWS customers get up to $200 in AWS credits ($100 immediately plus up to $100 earned, over 6 months). [https://aws.amazon.com/free/] - Cost to run it yourself: Managed only, no self-host option. Bedrock sets its own per-token prices rather than marking up a third party: Standard, Flex and Batch (both 50% below Standard) and Priority (75% above Standard) tiers, plus Provisioned Throughput billed per model unit-hour with no-commitment, 1-month or 6-month terms. [https://aws.amazon.com/bedrock/pricing/] - How the vendor makes money: Bundled — inference is the product - How pricing works, briefly: Pay-for-own-inference, bundled routing: gateway/API layer carries no separately stated fee. Per-token (on-demand), per batch token, or per provisioned model unit hour. Service tiers: Priority +75% to Standard; Flex −50%; Batch −50%. - Minimum commitment: None for on-demand. Provisioned Throughput uses 1-month and 6-month commitments (e.g. Cohere Command $39.60/hr/model unit for 1 mo vs $23.77 for 6 mo; Llama 2 70B $21.18 vs $13.08); pricing requires contacting your account team. - Charges that fire after you go over an allowance: No log/trace retention overage, request-volume overage or data-export charge. Adjacent charges are usage-metered: model customization, storage, retrieval, guardrails, evaluations, Advanced Prompt Optimizer (charged for on-demand Standard-tier tokens used in optimization). - Prompt cache offered: Both exact and semantic - Discount on cached input: 90% - Premium on cache writes: 25% - Who captures the cache saving: Implicit and explicit prompt caching (prefix-based, not semantic). Explicit uses cache checkpoints (up to 4 per request); TTLs commonly 5 min (Anthropic 1h option, GPT-5.6 default 30m). Implicit caching is best-effort — identical prompts don't guarantee a hit. Cache reads billed at 90% discount for GPT-5.6 (Claude 3.5 Sonnet v2 cache read $0.60 vs $6.00 input = 90% off on the pricing page). Cache writes billed at 1.25× uncached input rate (+25%). Cache saves money conditionally: writes cost 25% more up front and hits are never guaranteed, so a low-hit-rate workload can pay more than it saves. AWS captures nothing extra beyond the write premium. - What you can split spend by: Not stated on the pricing page. Cost shaping levers Bedrock does publish: service tier (Flex/Priority), batch, Provisioned Throughput commitments. - How you get cost data out: Not stated on the pricing page. - Who pays the model bill: Not applicable ### Catalog - Models available: 100 [https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html] - Model providers reachable: 19 [https://aws.amazon.com/bedrock/pricing/] - Works with standard OpenAI code: Yes [https://docs.aws.amazon.com/bedrock/latest/userguide/doc-history.html] - OpenAI chat endpoint: Yes - OpenAI Responses endpoint: Yes - Batch jobs endpoint: Yes - Needs the vendor’s own code library: No [https://docs.aws.amazon.com/bedrock/latest/userguide/inference-chat-completions-mantle.html] - You can export your request history: Yes [https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html] - Settings can live in version control: Yes [https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-bedrock-intelligentpromptrouter.html] - Image generation: Yes [https://aws.amazon.com/bedrock/pricing/] - Video generation: Yes [https://aws.amazon.com/bedrock/pricing/] - Batch processing: Yes [https://aws.amazon.com/bedrock/pricing/] ### Routing & reliability - Uptime it promises in writing: 99.9% [https://aws.amazon.com/bedrock/sla/] - Load balancing: Yes [https://docs.aws.amazon.com/bedrock/latest/userguide/bedrock-regions.html] - Rule-based routing: Yes [https://aws.amazon.com/bedrock/pricing/] - Response caching: Yes [https://aws.amazon.com/bedrock/pricing/] - Cross-region failover you control: Fixed, cannot change - Where you set load balancing: Fixed, cannot change ### Operations - Quality testing: Yes [https://aws.amazon.com/bedrock/pricing/] - What gets logged: Nothing stored - You can turn logging off: Yes - Traces you can take elsewhere: OpenTelemetry - Where telemetry can go: CloudWatch Logs,S3,CloudTrail - Can record user feedback: No - Scores live traffic: No ### Performance - What the request path runs on: Cloud provider API - You can run the request path yourself: No - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Yes [https://aws.amazon.com/bedrock/faqs/] - What they keep if you change nothing: Nothing stored by default [https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html] - How long they keep it: 0 [https://docs.aws.amazon.com/bedrock/latest/userguide/abuse-detection.html] - Could they train on your prompts: No [https://aws.amazon.com/bedrock/faqs/] - Where it runs, and what you can pin: Content is encrypted and stored at rest in the region where you use Bedrock. Cross-region inference moves retained data to the processing region. PrivateLink reaches into your VPC, but the service itself is not customer-hosted. [https://aws.amazon.com/bedrock/faqs/] - Where safety filters run: In the vendor’s cloud [https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html] - Who else touches the data: https://aws.amazon.com/compliance/sub-processors/ [https://aws.amazon.com/compliance/sub-processors/] - SOC 2 audited: Yes [https://aws.amazon.com/bedrock/] - Will sign a HIPAA agreement: Yes [https://aws.amazon.com/compliance/hipaa-eligible-services-reference/] - GDPR commitments: Yes [https://aws.amazon.com/bedrock/] - Does not retain your data: Yes [https://aws.amazon.com/bedrock/] - Strips personal data: Yes [https://aws.amazon.com/bedrock/pricing/] - Content guardrails: Yes [https://aws.amazon.com/bedrock/pricing/] - Blocks personal data in prompts: Can block the request - Blocks prompt injection: Can block the request - Blocks harmful content: Can block the request - Your own policy rules: Can block the request - Where guardrails run: On the vendor's servers ### Fit & integration - Work to try it: Set it up in a cloud console - Work to run it: Set it up in a cloud console - Steps on the quickstart: 5 - Can you self-host it today: No self-hosting - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Official provider package - Python framework integrations: LlamaIndex,Strands - Callable from Cloudflare Workers: Documented by Cloudflare, not the vendor - Kubernetes install: Only generic cluster docs - Terraform support: Resources in the cloud’s provider - Reuses your cloud identity: Reuses your cloud identity - Fits behind your API gateway: The cloud ships its own AI gateway - MCP support: MCP tools in the API - Needs your own provider key: Not needed - Gate before models work: Approval required - Official client languages: C++,Go,Java,JavaScript,Kotlin,.NET,PHP,PowerShell,Python,Ruby,Rust,SAP ABAP,Swift ### Not published by this vendor Amazon Bedrock has not published a figure for: GitHub stars; Markup on model prices; Fee to add funds; Monthly cost per person; Can use your own provider accounts; Cost of using your own accounts; Enterprise plan from; Anthropic messages endpoint; Embeddings endpoint; Image generation endpoint; Audio endpoints; Embeddings; Speech and audio; Automatic failover; Similar-question caching; Usage dashboards and logs; Spending limits; Rate limits; Separate keys per team or app; Prompt versioning; MCP support; Delay it adds; Requests per second ceiling; Can keep data in the EU; Runs fully disconnected; Where you set the timeout; Where you set retries; Where you set fallbacks; Shape of the fallback chain; Upstream health tracking; If the guardrail itself fails. --- ## Apache APISIX AI Gateway - Kind: Open source - Who runs it: Self-host only - Page: https://gatewayscore.com/providers/apache-apisix-ai-gateway - Most recent check: 2026-08-29 - Evidence basis: documentation review only AI proxy plugins for the Apache APISIX API gateway, adding LLM routing and token limits to an OpenResty data plane. ### Overview - What kind of product: Open source - Who runs it: Self-host only - Licence: Apache-2.0 [https://github.com/apache/apisix] - Company: Apache Software Foundation [https://github.com/apache/apisix] - Who you would be signing with: Run by a software foundation [https://apisix.apache.org/] - Last shipped an update: 2026-08-20 [https://github.com/apache/apisix/releases/tag/3.18.0] - GitHub stars: 16,800 [https://github.com/apache/apisix] ### Cost - Markup on model prices: None [https://github.com/apache/apisix] - Fee to add funds: None [https://github.com/apache/apisix] - Monthly cost per person: None [https://github.com/apache/apisix] - Can use your own provider accounts: Yes [https://apisix.apache.org/docs/apisix/plugins/ai-proxy/] - Cost of using your own accounts: Provider credentials are configured in the plugin; the project charges nothing. - Free tier: Entire project is free under Apache-2.0, including the ai-proxy and ai-proxy-multi AI gateway plugins. [https://github.com/apache/apisix] - Cost to run it yourself: No license cost at all; you run APISIX and its etcd control store on your own infrastructure. The docs are co-branded with API7, a commercial vendor offering a supported distribution, whose pricing is not published on these pages. [https://apisix.apache.org/docs/apisix/plugins/ai-proxy/] - How the vendor makes money: Open source, no paid tier - How pricing works, briefly: Apache 2.0 OSS project — no ASF pricing page, tier list or paid SKU exists. Commercial support is sold by a separate vendor, API7.ai: API7 Cloud Standard is $2 per 1M API calls (floored to $0.01) + $250/gateway group/month + $10/service/month; API7 Enterprise is annual licensing by API gateway CPU core with a custom quote. - Minimum commitment: None for OSS. API7 Enterprise is annual per-CPU-core licensing. - Charges that fire after you go over an allowance: None for OSS. API7 Cloud is pay-as-you-go 'with no usage limitation', so cost grows with volume rather than being an overage penalty. - Prompt cache offered: No gateway-owned cache - Who captures the cache saving: The AI Gateway overview and ai-rate-limiting plugin docs do not document a response cache or provider-cache passthrough — usage fields like cached_tokens are surfaced but no gateway-owned cache is described. APISIX never prices tokens, so any caching saving is 100% the customer's. - What you can split spend by: Token consumption trackable by Route, Service, Consumer, Consumer Group or custom attributes. Dollar cost attribution not documented. - How you get cost data out: None stated. Observability comes from the plugin ecosystem. - Who pays the model bill: Your keys only ### Catalog - Model providers reachable: 10 [https://apisix.apache.org/docs/apisix/plugins/ai-proxy/] - Works with standard OpenAI code: Yes [https://apisix.apache.org/docs/apisix/plugins/ai-proxy/] - OpenAI chat endpoint: Yes - Anthropic messages endpoint: Yes - OpenAI Responses endpoint: Partly - Embeddings endpoint: Yes - Needs the vendor’s own code library: No [https://apisix.apache.org/docs/apisix/3.17/plugins/ai-proxy/] - You can export your request history: Yes [https://apisix.apache.org/docs/apisix/plugins/http-logger/] - Settings can live in version control: Yes [https://apisix.apache.org/docs/apisix/deployment-modes/] - Embeddings: Yes [https://apisix.apache.org/docs/apisix/plugins/ai-proxy/] ### Routing & reliability - Automatic failover: Yes [https://apisix.apache.org/docs/apisix/plugins/ai-proxy-multi/] - Load balancing: Yes [https://apisix.apache.org/docs/apisix/plugins/ai-proxy-multi/] - Rule-based routing: Yes [https://apisix.apache.org/docs/apisix/plugins/ai-proxy-multi/] - Where you set the timeout: In config - Where you set retries: In config - Where you set fallbacks: In config - Shape of the fallback chain: Weighted split - Upstream health tracking: In config - Where you set load balancing: In config ### Operations - Usage dashboards and logs: Yes [https://apisix.apache.org/docs/apisix/plugins/ai-proxy/] - Rate limits: Yes [https://apisix.apache.org/docs/apisix/plugins/ai-proxy/] - MCP support: Yes [https://github.com/apache/apisix] - What gets logged: Metadata only - You can turn logging off: Yes - Traces you can take elsewhere: OpenTelemetry - Where telemetry can go: OpenTelemetry,Prometheus,Kafka,HTTP logger,Loki,Datadog,Splunk,Google Cloud Logging - Can record user feedback: No - Scores live traffic: No ### Performance - Delay it adds: 0.2 ms [https://github.com/apache/apisix] - Requests per second ceiling: 18,000 rps [https://github.com/apache/apisix] - What the request path runs on: Interpreted proxy - You can run the request path yourself: Yes - Streaming responses: Partly ### Security & compliance - Does your prompt reach their servers: No [https://apisix.apache.org/ai-gateway/] - What they keep if you change nothing: Not applicable — you own the logs [https://apisix.apache.org/ai-gateway/] - Could they train on your prompts: Not applicable - Where it runs, and what you can pin: Anywhere you run it. There are no vendor-managed regions. [https://apisix.apache.org/ai-gateway/] - Where safety filters run: In your own infrastructure [https://apisix.apache.org/docs/apisix/plugins/ai-prompt-guard/] - Content guardrails: Yes [https://github.com/apache/apisix] - Blocks prompt injection: Can block the request - Blocks harmful content: Can block the request - Your own policy rules: Can block the request - Where guardrails run: Either, your choice - Third-party guardrail vendors: AWS Comprehend ### Fit & integration - Work to try it: Run something locally first - Work to run it: Deploy it on your infrastructure - Steps on the quickstart: 0 - Can you self-host it today: Install command published - Works with the OpenAI SDK: Partly - Vercel AI SDK support: Via the OpenAI provider - Python framework integrations: LangChain,LangGraph,LlamaIndex - Kubernetes install: Official Helm chart - Reuses your cloud identity: Reuses your cloud identity - Fits behind your API gateway: It is the API gateway - MCP support: MCP gateway or registry - Needs your own provider key: Required ### Not published by this vendor Apache APISIX AI Gateway has not published a figure for: Enterprise plan from; Discount on cached input; Premium on cache writes; Models available; Image generation endpoint; Audio endpoints; Batch jobs endpoint; Image generation; Speech and audio; Video generation; Batch processing; Uptime it promises in writing; Response caching; Similar-question caching; Spending limits; Separate keys per team or app; Prompt versioning; Quality testing; How long they keep it; Who else touches the data; SOC 2 audited; Will sign a HIPAA agreement; GDPR commitments; Can keep data in the EU; Does not retain your data; Strips personal data; Runs fully disconnected; Cross-region failover you control; Blocks personal data in prompts; If the guardrail itself fails; Callable from Cloudflare Workers; Terraform support; Gate before models work. --- ## Azure AI Foundry - Kind: Cloud platform - Who runs it: Managed or self-host - Page: https://gatewayscore.com/providers/azure-ai-foundry - Most recent check: 2026-08-29 - Evidence basis: used first-hand in production, plus documentation review Microsoft's Azure platform for deploying models from its own and partner catalogs, now branded Microsoft Foundry. ### Overview - What kind of product: Cloud platform - Who runs it: Managed or self-host [https://learn.microsoft.com/en-us/azure/ai-foundry/what-is-azure-ai-foundry] - Licence: Proprietary - Company: Microsoft [https://learn.microsoft.com/en-us/azure/ai-foundry/what-is-azure-ai-foundry] - Who you would be signing with: Large cloud vendor product [https://learn.microsoft.com/en-us/azure/foundry/whats-new-foundry] - Last shipped an update: 2026-08-13 [https://learn.microsoft.com/en-us/azure/foundry/whats-new-foundry] ### Cost - Monthly cost per person: None [https://learn.microsoft.com/en-us/azure/ai-foundry/what-is-azure-ai-foundry] - Free tier: Azure free account gives $200 credit valid 30 days; the Foundry platform itself is free to use and you pay only for the models and compute you deploy. [https://azure.microsoft.com/en-us/pricing/details/phi-3/] - Cost to run it yourself: Mostly managed, with Foundry Local for on-device execution. Microsoft sets its own per-token prices rather than marking up a third party; Global Batch and Data Zone Batch deployments are 50% below the standard rate with a 24-hour completion target, and DeveloperTier deployments are cheap but capped at 24 hours with no SLA and no data-residency guarantee. [https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/deployment-types] - How the vendor makes money: Bundled — inference is the product - How pricing works, briefly: Pay-for-own-inference with an explicitly free platform layer. 'The Foundry platform is free to use and explore'; individual features billed at their normal rates. The gateway/routing layer is $0; models are billed separately. - Minimum commitment: None required. Microsoft Agent pre-purchase plan is a 1-year metered plan buying Agent Commit Units at tiered discounts — 20,000 ACUs = 5%, 100,000 = 10%, 500,000 = 15% (dollar prices shown as '$-'). - Charges that fire after you go over an allowance: None stated on either pricing page. - Prompt cache offered: Prefix cache - Premium on cache writes: None - Who captures the cache saving: Exact-prefix prompt caching, enabled by default, no opt-out. The first 1,024 tokens must be identical; a single character difference in the first 1,024 tokens = miss. Hits extend every 128 identical tokens. Caches clear within 5–10 minutes of inactivity and always within an hour, and are not shared across subscriptions. Discount: Standard cached-token discount is not published as a number; Provisioned deployments get up to 100% discount on input tokens. No cache-write charge documented. On Provisioned capacity, cached input can be effectively free. - What you can split spend by: Not stated on the pricing pages. - How you get cost data out: Not stated on the pricing pages. - Who pays the model bill: Not applicable ### Catalog - Models available: 11,000 [https://azure.microsoft.com/en-us/products/ai-foundry] - Works with standard OpenAI code: Yes [https://learn.microsoft.com/en-us/azure/ai-foundry/openai/supported-languages] - OpenAI chat endpoint: Yes - Embeddings endpoint: Yes - Image generation endpoint: Yes - Audio endpoints: Yes - Batch jobs endpoint: Yes - Needs the vendor’s own code library: No [https://learn.microsoft.com/en-us/azure/foundry/foundry-models/concepts/endpoints] - You can export your request history: Yes [https://learn.microsoft.com/en-us/azure/foundry/how-to/diagnostic-logging] - Settings can live in version control: Yes [https://learn.microsoft.com/en-us/azure/foundry/how-to/create-resource-terraform] - Embeddings: Yes [https://learn.microsoft.com/en-us/azure/ai-foundry/openai/azure-government] - Image generation: Yes [https://azure.microsoft.com/en-us/pricing/details/phi-3/] - Batch processing: Yes [https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/deployment-types] ### Routing & reliability - Uptime it promises in writing: 99.9% [https://azure.microsoft.com/en-us/products/ai-foundry/models/openai] - Load balancing: Yes [https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/deployment-types] - Rule-based routing: Yes [https://azure.microsoft.com/en-us/pricing/details/phi-3/] - Where you set fallbacks: Per request - Shape of the fallback chain: Single alternate - Upstream health tracking: Fixed, cannot change - Cross-region failover you control: Fixed, cannot change - Where you set load balancing: Fixed, cannot change ### Operations - Usage dashboards and logs: Yes [https://learn.microsoft.com/en-us/azure/ai-foundry/whats-new-azure-ai-foundry] - Quality testing: Yes [https://learn.microsoft.com/en-us/azure/ai-foundry/what-is-azure-ai-foundry] - What gets logged: Metadata only - You can turn logging off: Yes - Traces you can take elsewhere: OpenTelemetry - Where telemetry can go: Azure Monitor,Log Analytics,Event Hubs,Azure Storage - Can record user feedback: No - Scores live traffic: No ### Performance - What the request path runs on: Cloud provider API - You can run the request path yourself: No - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Yes [https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/openai/data-privacy] - What they keep if you change nothing: Nothing stored by default [https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/abuse-monitoring] - Could they train on your prompts: No [https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/openai/data-privacy] - Where it runs, and what you can pin: Processing stays in your specified geography — with a significant exception. Any deployment type labelled Global may process prompts and responses in any geography, and DataZone widens the boundary too. [https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/openai/data-privacy] - Where safety filters run: In the vendor’s cloud [https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/content-safety/data-privacy] - Who else touches the data: https://www.microsoft.com/en-us/trust-center/privacy/data-access [https://www.microsoft.com/en-us/trust-center/privacy/data-access] - SOC 2 audited: Yes [https://learn.microsoft.com/en-us/azure/compliance/offerings/offering-soc-2] - Will sign a HIPAA agreement: Yes [https://learn.microsoft.com/en-us/azure/compliance/offerings/offering-hipaa-us] - Can keep data in the EU: Yes [https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/deployment-types] - Does not retain your data: Yes [https://learn.microsoft.com/en-us/azure/ai-foundry/responsible-ai/openai/data-privacy] - Strips personal data: Yes [https://learn.microsoft.com/en-us/azure/ai-foundry/whats-new-azure-ai-foundry] - Content guardrails: Yes [https://learn.microsoft.com/en-us/azure/ai-foundry/responsible-ai/openai/data-privacy] - Blocks personal data in prompts: Inspects but lets it through - Blocks prompt injection: Can block the request - Blocks harmful content: Can block the request - Your own policy rules: Can block the request - Where guardrails run: On the vendor's servers ### Fit & integration - Work to try it: Set it up in a cloud console - Work to run it: Set it up in a cloud console - Steps on the quickstart: 54 - Can you self-host it today: No self-hosting - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Community provider package - Python framework integrations: LangChain,LangGraph,LlamaIndex - Kubernetes install: Helm supported, chart not named - Terraform support: Resources in the cloud’s provider - Reuses your cloud identity: Reuses your cloud identity - Fits behind your API gateway: The gateway vendor documents it - MCP support: MCP tools in the API - Needs your own provider key: Not needed - Gate before models work: Deployment and quota first - Official client languages: Python,JavaScript,.NET,Java ### Not published by this vendor Azure AI Foundry has not published a figure for: GitHub stars; Markup on model prices; Fee to add funds; Can use your own provider accounts; Cost of using your own accounts; Enterprise plan from; Discount on cached input; Model providers reachable; Anthropic messages endpoint; OpenAI Responses endpoint; Speech and audio; Video generation; Automatic failover; Response caching; Similar-question caching; Spending limits; Rate limits; Separate keys per team or app; Prompt versioning; MCP support; Delay it adds; Requests per second ceiling; How long they keep it; GDPR commitments; Runs fully disconnected; Where you set the timeout; Where you set retries; If the guardrail itself fails; Callable from Cloudflare Workers. --- ## Bifrost - Kind: Open source - Who runs it: Self-host only - Page: https://gatewayscore.com/providers/bifrost - Most recent check: 2026-08-31 - Evidence basis: used first-hand in production, plus documentation review Go-based open-source AI gateway focused on low proxy overhead, with an enterprise tier for clustering and SSO. ### Overview - What kind of product: Open source - Who runs it: Self-host only - Licence: Apache-2.0 [https://github.com/maximhq/bifrost] - Company: Maxim AI [https://github.com/maximhq/bifrost] - Who you would be signing with: Independent company [https://www.getmaxim.ai/blog/announcing-maxim-ais-general-availability-and-the-3m-funding-round-led-by-elevation-capital/] - Last shipped an update: 2026-08-27 [https://github.com/maximhq/bifrost/releases/tag/core%2Fv1.8.4] - GitHub stars: 7,600 [https://github.com/maximhq/bifrost] ### Cost - Markup on model prices: None [https://www.getmaxim.ai/pricing] - Fee to add funds: None [https://www.getmaxim.ai/pricing] - Monthly cost per person: None [https://www.getmaxim.ai/pricing] - Can use your own provider accounts: Yes [https://docs.getbifrost.ai/] - Cost of using your own accounts: Self-hosted with your own provider keys and vault integration; no gateway token fee. - Free tier: OSS tier is "Free Forever": drop-in gateway, OpenTelemetry observability, budgets, rate limits, virtual keys, custom routing, fallbacks, simple and semantic caching, MCP gateway, prompt repository. [https://www.getmaxim.ai/pricing] - Cost to run it yourself: OSS is free (Apache-2.0); infra cost only. Enterprise adds guardrails, cluster mode, adaptive load balancing, SAML/OIDC SSO, vault integration, log exports, audit logs, RBAC and SLAs at custom pricing after a 14-day trial; VPC, on-prem and air-gapped installs are enterprise options. [https://www.getmaxim.ai/pricing] - How the vendor makes money: Open source with a managed tier - How pricing works, briefly: OSS 'Free Forever' (Docker/K8s/Go binary) for developers, small teams and self-managed deployments. Enterprise is 'Custom Pricing' for VPC/on-prem/air-gapped, quote-only. - Minimum commitment: None for OSS. Enterprise not stated. - Charges that fire after you go over an allowance: None stated. Self-hosting infrastructure cost is yours. - Prompt cache offered: Both exact and semantic - Who captures the cache saving: Simple (exact) and semantic caching both in OSS. Semantic caching 'reduces costs and latency for similar queries with semantic similarity caching'. Bifrost prices no tokens, so caching saving is fully the customer's. MCP Code Mode claims to cut tokens by 50%+ separately. - What you can split spend by: Per virtual key, consumer, team and customer. Per-user and per-tag not stated. - How you get cost data out: OTel-compatible metrics and traces (any OTel backend can receive data). CSV, cost API, webhook-of-spend, S3 and warehouse export not stated. - Who pays the model bill: Your keys only ### Catalog - Model providers reachable: 23 [https://github.com/maximhq/bifrost] - Works with standard OpenAI code: Yes [https://github.com/maximhq/bifrost] - OpenAI chat endpoint: Yes - Anthropic messages endpoint: Yes - Embeddings endpoint: Yes - Image generation endpoint: Yes - Audio endpoints: Yes - Needs the vendor’s own code library: No [https://www.getmaxim.ai/docs/bifrost/usage/http-transport/integrations/openai-compatible] - You can export your request history: Yes [https://docs.getbifrost.ai/api-reference/logging/get-logs] - Settings can live in version control: Yes [https://docs.getbifrost.ai/deployment-guides/config-json] - Embeddings: Yes [https://docs.getbifrost.ai/] - Image generation: Yes [https://docs.getbifrost.ai/] - Speech and audio: Yes [https://docs.getbifrost.ai/] ### Routing & reliability - Uptime it promises in writing: 99.999% [https://www.getmaxim.ai/bifrost/resources/enterprise-deployment] - Automatic failover: Yes [https://www.getmaxim.ai/pricing] - Load balancing: Yes [https://www.getmaxim.ai/pricing] - Rule-based routing: Yes [https://www.getmaxim.ai/pricing] - Response caching: Yes [https://www.getmaxim.ai/pricing] - Similar-question caching: Yes [https://www.getmaxim.ai/pricing] - Where you set the timeout: In config - Where you set retries: In config - Where you set fallbacks: In config - Shape of the fallback chain: Weighted split - Where you set load balancing: In config ### Operations - Usage dashboards and logs: Yes [https://www.getmaxim.ai/pricing] - Spending limits: Yes [https://www.getmaxim.ai/pricing] - Rate limits: Yes [https://www.getmaxim.ai/pricing] - Separate keys per team or app: Yes [https://www.getmaxim.ai/pricing] - Prompt versioning: Yes [https://www.getmaxim.ai/pricing] - MCP support: Yes [https://www.getmaxim.ai/pricing] - What gets logged: Your choice - You can turn logging off: Yes - Traces you can take elsewhere: OpenTelemetry - Where telemetry can go: OpenTelemetry,Prometheus,Datadog,Langfuse,Maxim,S3,Webhook - Can record user feedback: No - Scores live traffic: No ### Performance - Delay it adds: 0.99 ms [https://www.getmaxim.ai/bifrost/resources/benchmarks] - Requests per second ceiling: 5,000 rps [https://www.getmaxim.ai/bifrost] - What the request path runs on: Compiled binary - You can run the request path yourself: Yes - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Depends how you deploy it [https://www.getmaxim.ai/docs/self-hosting/overview] - What they keep if you change nothing: Yes — prompts and replies [https://docs.getbifrost.ai/enterprise/log-exports] - How long they keep it: 365 [https://docs.getbifrost.ai/enterprise/log-exports] - Where it runs, and what you can pin: Your own infrastructure. Log offload targets S3 or GCS in a region you name. Maxim offers full VPC isolation or a VPC-peered hybrid. [https://docs.getbifrost.ai/enterprise/log-exports] - Where safety filters run: In your own infrastructure [https://docs.getbifrost.ai/enterprise/log-exports] - Content guardrails: Yes [https://www.getmaxim.ai/pricing] - Runs fully disconnected: Yes [https://www.getmaxim.ai/pricing] - Blocks personal data in prompts: Can block the request - Blocks prompt injection: Can block the request - Blocks harmful content: Can block the request - Your own policy rules: Can block the request - Where guardrails run: Either, your choice - Third-party guardrail vendors: Microsoft Presidio,Azure AI Language,AWS Bedrock Guardrails,Google Model Armor,Gitleaks ### Fit & integration - Work to try it: Run something locally first - Work to run it: Deploy it on your infrastructure - Steps on the quickstart: 4 - Can you self-host it today: Install command published - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Claimed compatible, no package named - Python framework integrations: LangChain - Kubernetes install: Official Helm chart - Terraform support: Official modules - Reuses your cloud identity: Reuses your cloud identity - MCP support: MCP gateway or registry - Needs your own provider key: Optional - Official client languages: Go ### Not published by this vendor Bifrost has not published a figure for: Enterprise plan from; Discount on cached input; Premium on cache writes; Models available; OpenAI Responses endpoint; Batch jobs endpoint; Video generation; Batch processing; Quality testing; Could they train on your prompts; Who else touches the data; SOC 2 audited; Will sign a HIPAA agreement; GDPR commitments; Can keep data in the EU; Does not retain your data; Strips personal data; Upstream health tracking; Cross-region failover you control; If the guardrail itself fails; Callable from Cloudflare Workers; Fits behind your API gateway; Gate before models work. --- ## Braintrust Gateway - Kind: Managed gateway - Who runs it: Managed or self-host - Page: https://gatewayscore.com/providers/braintrust-gateway - Most recent check: 2026-08-29 - Evidence basis: documentation review only Multi-provider gateway inside Braintrust's eval and observability platform, with caching and span-level tracing. ### Overview - What kind of product: Managed gateway - Who runs it: Managed or self-host [https://www.braintrust.dev/docs/deploy/gateway] - Licence: Open core [https://www.braintrust.dev/docs/guides/proxy] - Company: Braintrust - Who you would be signing with: Independent company [https://www.braintrust.dev/blog/announcing-series-b] - GitHub stars: 409 [https://api.github.com/repos/braintrustdata/braintrust-proxy] ### Cost - Monthly cost per person: None [https://www.braintrust.dev/pricing] - Can use your own provider accounts: Yes [https://www.braintrust.dev/docs/deploy/gateway] - Cost of using your own accounts: Provider keys are configured at organization or project level (project keys override org keys); the proxy can even be used with only a provider key and no Braintrust account. [https://www.braintrust.dev/docs/guides/proxy] - Free tier: Starter $0/month: unlimited users, $10 model credits, 1 GB processed data, 10k scores, 14-day retention. The Braintrust-hosted Gateway is free during public preview, with pricing to be announced before GA. [https://www.braintrust.dev/docs/deploy/gateway] - Cost to run it yourself: Self-hosted Braintrust deployments run the Gateway inside the customer's own data plane so LLM traffic stays in their account; the standalone proxy is MIT-licensed on GitHub. On-prem/hybrid deployment is an Enterprise-plan feature, and independent reviews note there is no free self-hosted tier below Enterprise. [https://www.braintrust.dev/docs/deploy/gateway] - How the vendor makes money: Platform fee plus usage meters - How pricing works, briefly: Flat monthly platform fee + included model credits then per-token rates + usage overages (processed data, scores, retention). Starter $0 with $10/mo credits; Pro $249/mo with $249/mo credits (drops to $100/mo credits after 2026-09-01); Enterprise annual invoice. AI proxy blog says the gateway itself is free 'as a beta', but Braintrust's docs index the proxy page as 'deprecated' — treat the free-proxy framing as unverified. - Minimum commitment: None for Starter/Pro. Enterprise is annual-invoiced. - Charges that fire after you go over an allowance: Processed data: 1 GB then +$4/GB (Starter) / 5 GB then +$3/GB (Pro). Scores: 10k then $2.50/1k (Starter) / 50k then $1.50/1k (Pro). Retention: 14 days (Starter) / 30 days then +$0.50/GB/month up to 180 days (Pro). Model credits do not roll over. Retention bills what you ingest into each month you keep beyond the base window, not what's currently stored. Deleting data does not reduce processed-data charges. - Prompt cache offered: Exact-match cache - Who captures the cache saving: Exact-request response caching in the proxy: requests with temperature=0 or `seed` are cached, encrypted per API key, controllable via x-bt-use-cache. No gateway-layer cached-token pricing; Braintrust's own model rates do publish cache-hit prices (GLM-5.2 81% off, Kimi K3 90% off). Cache hits avoid the provider call entirely, but cached traffic still counts as processed data at ingestion. - What you can split spend by: Not stated as cost splits. Environments tag object versions but are not described as cost attribution. - How you get cost data out: S3 / Google Cloud Storage scheduled export automations on Enterprise only; price not stated. CSV, webhook and warehouse export not stated. - Who pays the model bill: Your keys or their credits ### Catalog - Models available: 100 [https://www.braintrust.dev/docs/guides/proxy] - Model providers reachable: 18 [https://www.braintrust.dev/docs/deploy/supported-models] - Works with standard OpenAI code: Yes [https://www.braintrust.dev/docs/deploy/gateway] - OpenAI chat endpoint: Yes - Anthropic messages endpoint: Yes - OpenAI Responses endpoint: Yes - Embeddings endpoint: Yes - Audio endpoints: Partly - Needs the vendor’s own code library: No [https://www.braintrust.dev/docs/deploy/gateway] - You can export your request history: Yes [https://www.braintrust.dev/docs/admin/data-management/export] - Embeddings: Yes [https://www.braintrust.dev/docs/deploy/gateway] - Speech and audio: Yes [https://www.braintrust.dev/docs/guides/proxy] ### Routing & reliability - Automatic failover: Yes [https://www.braintrust.dev/docs/deploy/gateway] - Load balancing: Yes [https://www.braintrust.dev/docs/guides/proxy] - Response caching: Yes [https://www.braintrust.dev/docs/guides/proxy] - Where you set fallbacks: Per request - Shape of the fallback chain: Ordered list - Upstream health tracking: Fixed, cannot change - Cross-region failover you control: Fixed, cannot change - Where you set load balancing: Fixed, cannot change ### Operations - Usage dashboards and logs: Yes [https://www.braintrust.dev/docs/deploy/gateway] - Separate keys per team or app: Yes [https://www.braintrust.dev/docs/guides/proxy] - Prompt versioning: Yes [https://www.braintrust.dev/docs/guides/proxy] - Quality testing: Yes [https://www.braintrust.dev/pricing] - What gets logged: Your choice - You can turn logging off: Yes - Traces you can take elsewhere: OpenTelemetry - Where telemetry can go: Braintrust - Can record user feedback: Yes - Scores live traffic: Yes ### Performance - What the request path runs on: Edge worker - You can run the request path yourself: Yes - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Depends how you deploy it [https://www.braintrust.dev/docs/security] - What they keep if you change nothing: Yes — prompts and replies [https://www.braintrust.dev/docs/observe] - How long they keep it: 14 [https://www.braintrust.dev/docs/admin/data-management/retention] - Could they train on your prompts: No [https://www.braintrust.dev/blog/security-data-control] - Where it runs, and what you can pin: Hosted data plane in the US or the EU on Pro. BYOC and self-hosted keep data in your own cloud account and region. [https://www.braintrust.dev/legal/dpa] - Where safety filters run: Either, depending on deployment [https://www.braintrust.dev/blog/security-data-control] - Who else touches the data: https://www.braintrust.dev/legal/dpa [https://www.braintrust.dev/legal/dpa] - SOC 2 audited: Yes [https://www.braintrust.dev/pricing] - Will sign a HIPAA agreement: Yes [https://www.braintrust.dev/pricing] - Can keep data in the EU: Yes [https://www.braintrust.dev/docs/deploy/gateway] - Content guardrails: No [https://vantaige.io/ai-tool/braintrust] ### Fit & integration - Work to try it: Change one base URL - Work to run it: Change one base URL - Steps on the quickstart: 0 - Can you self-host it today: Offered, but no command published - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Documented integration, not a provider - Python framework integrations: LangChain,LlamaIndex - Callable from Cloudflare Workers: Documented from your Worker - Kubernetes install: Official Helm chart - Terraform support: Official modules - Reuses your cloud identity: Reuses your cloud identity - Fits behind your API gateway: Service mesh routing documented - MCP support: Hosted MCP server - Needs your own provider key: Optional - Gate before models work: No gate - Official client languages: TypeScript,Python,Go,Java,Ruby,C# ### Not published by this vendor Braintrust Gateway has not published a figure for: Last shipped an update; Markup on model prices; Fee to add funds; Enterprise plan from; Discount on cached input; Premium on cache writes; Image generation endpoint; Batch jobs endpoint; Settings can live in version control; Image generation; Video generation; Batch processing; Uptime it promises in writing; Rule-based routing; Similar-question caching; Spending limits; Rate limits; MCP support; Delay it adds; Requests per second ceiling; GDPR commitments; Does not retain your data; Strips personal data; Runs fully disconnected; Where you set the timeout; Where you set retries; Blocks personal data in prompts; Blocks prompt injection; Blocks harmful content; Your own policy rules; Where guardrails run; If the guardrail itself fails. --- ## Cloudflare AI Gateway - Kind: Cloud platform - Who runs it: Managed only - Page: https://gatewayscore.com/providers/cloudflare-ai-gateway - Most recent check: 2026-08-29 - Evidence basis: used first-hand in production, plus documentation review Edge proxy in front of a curated set of AI providers, with caching, rate limiting, DLP and analytics. ### Overview - What kind of product: Cloud platform - Who runs it: Managed only [https://developers.cloudflare.com/ai-gateway/reference/pricing/] - Licence: Proprietary [https://developers.cloudflare.com/ai-gateway/] - Company: Cloudflare - Who you would be signing with: Large cloud vendor product [https://developers.cloudflare.com/ai-gateway/changelog/] - Last shipped an update: 2026-08-19 [https://developers.cloudflare.com/ai-gateway/changelog/] ### Cost - Markup on model prices: None [https://developers.cloudflare.com/ai-gateway/reference/pricing/] - Fee to add funds: 5% [https://developers.cloudflare.com/ai-gateway/reference/pricing/] - Monthly cost per person: None [https://developers.cloudflare.com/ai-gateway/reference/pricing/] - Can use your own provider accounts: Yes [https://developers.cloudflare.com/ai-gateway/features/] - Cost of using your own accounts: BYOK is a built-in feature at no stated extra cost: provider keys are stored encrypted in Cloudflare and rotated without code changes across 20+ providers. [https://developers.cloudflare.com/ai-gateway/features/] - Free tier: AI Gateway core features (dashboard analytics, caching, rate limiting) are free on all plans; DLP scanning is free on all plans; Workers Free stores 100,000 logs total across gateways, Workers Paid 10,000,000 logs per gateway. [https://developers.cloudflare.com/ai-gateway/reference/pricing/] - How the vendor makes money: Percentage on tokens or top-ups - How pricing works, briefly: Free gateway core + credit top-up fee (Unified Billing, 5% on credit purchases) + plan-tied log/Logpush limits. Inference passed through at cost — 0% markup. - Minimum commitment: None stated. Enterprise customers reach out to their account team. - Charges that fire after you go over an allowance: The bite is on logs, not requests. Persistent log storage capped at 100,000 logs total across all gateways on Workers Free and 10,000,000 logs per gateway on Workers Paid, with automatic log deletion. Logpush is Workers Paid only, includes 10M requests/month then charges +$0.05/million. Guardrails usage is billed as Workers AI token-based inference. - Prompt cache offered: Exact-match cache - Who captures the cache saving: Exact-match caching: based on exact match of the entire request. Disabled by default. Serves cached provider responses from Cloudflare's cache. No differential cached-token pricing published. A cache hit skips the provider call entirely — customer captures the full saving and Cloudflare charges nothing for the cache itself. The only related cost is log storage of the cached requests. - What you can split spend by: Not stated. - How you get cost data out: Logpush on Workers Paid (10M requests/month included, then $0.05/million). CSV, cost API, webhook and warehouse export not stated. - Who pays the model bill: Your keys or their credits ### Catalog - Model providers reachable: 23 [https://developers.cloudflare.com/ai-gateway/usage/providers/] - OpenAI chat endpoint: Yes - Anthropic messages endpoint: Yes - Needs the vendor’s own code library: No [https://developers.cloudflare.com/ai-gateway/usage/chat-completion/] - You can export your request history: Yes [https://developers.cloudflare.com/ai-gateway/observability/logging/logpush/] - Settings can live in version control: Yes [https://developers.cloudflare.com/api/terraform/resources/ai_gateway/] - Image generation: Yes [https://developers.cloudflare.com/ai-gateway/usage/providers/] - Speech and audio: Yes [https://developers.cloudflare.com/ai-gateway/usage/providers/] ### Routing & reliability - Automatic failover: Yes [https://developers.cloudflare.com/ai-gateway/] - Rule-based routing: Yes [https://developers.cloudflare.com/ai-gateway/features/] - Response caching: Yes [https://developers.cloudflare.com/ai-gateway/reference/pricing/] - Where you set the timeout: Per request - Where you set retries: Per request - Where you set fallbacks: Per request - Shape of the fallback chain: Weighted split - Cross-region failover you control: Fixed, cannot change - Where you set load balancing: Dashboard only ### Operations - Usage dashboards and logs: Yes [https://developers.cloudflare.com/ai-gateway/features/] - Spending limits: Yes [https://developers.cloudflare.com/ai-gateway/features/] - Rate limits: Yes [https://developers.cloudflare.com/ai-gateway/features/] - What gets logged: Full prompts and responses - You can turn logging off: Yes - Where telemetry can go: Logpush,R2,Webhook - Can record user feedback: Yes - Scores live traffic: Yes ### Performance - What the request path runs on: Edge worker - You can run the request path yourself: No - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Yes [https://developers.cloudflare.com/ai-gateway/features/guardrails/] - What they keep if you change nothing: Yes — prompts and replies [https://developers.cloudflare.com/ai-gateway/observability/logging/] - Could they train on your prompts: No [https://www.cloudflare.com/trust-hub/responsible-ai/] - Where it runs, and what you can pin: A global edge with platform-level data-residency support, but no AI-Gateway-specific region pinning and no customer-VPC deployment. [https://www.cloudflare.com/trust-hub/responsible-ai/] - Where safety filters run: In the vendor’s cloud [https://developers.cloudflare.com/ai-gateway/features/guardrails/] - Who else touches the data: https://www.cloudflare.com/gdpr/subprocessors/ [https://www.cloudflare.com/gdpr/subprocessors/] - Strips personal data: Yes [https://developers.cloudflare.com/ai-gateway/features/] - Content guardrails: Yes [https://developers.cloudflare.com/ai-gateway/features/] - Runs fully disconnected: No [https://developers.cloudflare.com/ai-gateway/reference/pricing/] - Blocks personal data in prompts: Can block the request - Blocks prompt injection: Can block the request - Blocks harmful content: Can block the request - Your own policy rules: Can block the request - Where guardrails run: On the vendor's servers - If the guardrail itself fails: You choose ### Fit & integration - Work to try it: Set it up in a cloud console - Work to run it: Set it up in a cloud console - Steps on the quickstart: 13 - Can you self-host it today: No self-hosting - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Official provider package - Python framework integrations: LlamaIndex - Callable from Cloudflare Workers: Documented from your Worker - Terraform support: Official Terraform provider - Reuses your cloud identity: Static provider credentials only - MCP support: MCP tools in the API - Needs your own provider key: Not needed - Gate before models work: No gate - Official client languages: JavaScript,Python ### Not published by this vendor Cloudflare AI Gateway has not published a figure for: GitHub stars; Enterprise plan from; Cost to run it yourself; Discount on cached input; Premium on cache writes; Models available; Works with standard OpenAI code; OpenAI Responses endpoint; Embeddings endpoint; Image generation endpoint; Audio endpoints; Batch jobs endpoint; Embeddings; Video generation; Batch processing; Uptime it promises in writing; Load balancing; Similar-question caching; Separate keys per team or app; Prompt versioning; Quality testing; MCP support; Delay it adds; Requests per second ceiling; How long they keep it; SOC 2 audited; Will sign a HIPAA agreement; GDPR commitments; Can keep data in the EU; Does not retain your data; Upstream health tracking; Traces you can take elsewhere; Kubernetes install; Fits behind your API gateway. --- ## Fireworks AI - Kind: Inference provider - Who runs it: Managed only - Page: https://gatewayscore.com/providers/fireworks-ai - Most recent check: 2026-08-29 - Evidence basis: used first-hand in production, plus documentation review Inference provider serving open-weight models on its own stack, with fine-tuning and dedicated deployments. ### Overview - What kind of product: Inference provider - Who runs it: Managed only - Licence: Proprietary - Company: Fireworks AI [https://en.wikipedia.org/wiki/Fireworks_AI] - Who you would be signing with: Independent company [https://www.reuters.com/technology/nvidia-backed-startup-fireworks-valued-175-billion-latest-funding-2026-07-16/] - Last shipped an update: 2026-08-27 [https://docs.fireworks.ai/updates/changelog] ### Cost - Free tier: $1 in free credits on signup, then postpaid billing. [https://fireworks.ai/pricing] - Cost to run it yourself: No self-host option; Fireworks sets its own per-token serverless prices rather than marking up another vendor's list price, with DEFAULT, PRIORITY and FAST service tiers resolved live per model. Dedicated capacity is billed per GPU-hour, and batch inference plus supervised and reinforcement fine-tuning are priced separately. [https://docs.fireworks.ai/api-reference/list-models] - How the vendor makes money: Bundled — inference is the product - How pricing works, briefly: Pay-for-own-inference (Fireworks is the provider). Per-token serverless, per-GPU-second on-demand, per-token/GPU-hour training, enterprise quote. No separate gateway or platform fee — routing layer is $0. - Minimum commitment: None stated for inference. Fine-tuning has no stated minimum on the pricing page. - Charges that fire after you go over an allowance: None stated. Serverless is postpaid, so cost scales with tokens. Note: 'Checkpoint storage for serverless models is included during private preview' implies a future charge. - Prompt cache offered: Prefix cache - Discount on cached input: 50% - Premium on cache writes: None - Who captures the cache saving: Prompt caching with discounted cached tokens (prefix-style caching for serverless; mechanism not labelled exact-match or semantic). Default 50% discount, but exact discount varies by model. Pricing page separately lists 'Cached Prefill / 1M' rates for training that are 80% below prefill (e.g. Qwen 3.5 9B $0.132 vs $0.66; Kimi K3 $2.17 vs $10.87). No cache-write charge documented — close to a free win, but only serverless models are covered. - What you can split spend by: Not stated. - How you get cost data out: Not stated. - Who pays the model bill: Their credits only ### Catalog - Models available: 100 [https://docs.fireworks.ai/getting-started/introduction] - Works with standard OpenAI code: Yes [https://docs.fireworks.ai/tools-sdks/openai-compatibility] - OpenAI chat endpoint: Yes - Embeddings endpoint: Yes - Image generation endpoint: Yes - Audio endpoints: Yes - Batch jobs endpoint: Yes - Needs the vendor’s own code library: No [https://docs.fireworks.ai/tools-sdks/openai-compatibility] - You can export your request history: Yes [https://docs.fireworks.ai/accounts/exporting-usage-costs] - Embeddings: Yes [https://fireworks.ai/pricing] - Image generation: Yes [https://docs.fireworks.ai/getting-started/introduction] - Speech and audio: Yes [https://docs.fireworks.ai/getting-started/introduction] - Batch processing: Yes [https://docs.fireworks.ai/guides/batch-inference] ### Routing & reliability - Cross-region failover you control: In config ### Operations - What gets logged: Metadata only - You can turn logging off: Yes - Traces you can take elsewhere: Vendor format only - Can record user feedback: No - Scores live traffic: No ### Performance - What the request path runs on: Undisclosed vendor service - You can run the request path yourself: No - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Yes [https://docs.fireworks.ai/deployments/regions] - What they keep if you change nothing: Nothing stored by default [https://docs.fireworks.ai/guides/security_compliance/data_handling] - How long they keep it: 0 [https://docs.fireworks.ai/guides/security_compliance/data_handling] - Could they train on your prompts: Only if you opt in [https://docs.fireworks.ai/guides/security_compliance/data_handling] - Where it runs, and what you can pin: Region groupings for global (the default), US, Europe and Asia-Pacific, plus around 19 pinnable single regions including Iowa, Frankfurt, Iceland and Tokyo. [https://docs.fireworks.ai/deployments/regions] - Where safety filters run: No guardrails offered [https://docs.fireworks.ai/faq/security/infrastructure/model-security] - SOC 2 audited: Yes [https://docs.fireworks.ai/faq/enterprise/compliance/certifications] - Will sign a HIPAA agreement: Yes [https://docs.fireworks.ai/faq/enterprise/compliance/certifications] ### Fit & integration - Work to try it: Install a package - Work to run it: Install a package - Steps on the quickstart: 2 - Can you self-host it today: No self-hosting - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Official provider package - Python framework integrations: LangChain,LlamaIndex,CrewAI,PydanticAI,Strands - Kubernetes install: Helm supported, chart not named - Terraform support: Official modules - Reuses your cloud identity: Reuses your cloud identity - Fits behind your API gateway: The gateway vendor documents it - MCP support: MCP tools in the API - Needs your own provider key: Not needed - Gate before models work: No gate - Official client languages: Python,TypeScript ### Not published by this vendor Fireworks AI has not published a figure for: GitHub stars; Markup on model prices; Fee to add funds; Monthly cost per person; Can use your own provider accounts; Cost of using your own accounts; Enterprise plan from; Model providers reachable; Anthropic messages endpoint; OpenAI Responses endpoint; Settings can live in version control; Video generation; Uptime it promises in writing; Automatic failover; Load balancing; Rule-based routing; Response caching; Similar-question caching; Usage dashboards and logs; Spending limits; Rate limits; Separate keys per team or app; Prompt versioning; Quality testing; MCP support; Delay it adds; Requests per second ceiling; Who else touches the data; GDPR commitments; Can keep data in the EU; Does not retain your data; Strips personal data; Content guardrails; Runs fully disconnected; Where you set the timeout; Where you set retries; Where you set fallbacks; Shape of the fallback chain; Upstream health tracking; Where you set load balancing; Blocks personal data in prompts; Blocks prompt injection; Blocks harmful content; Your own policy rules; Where guardrails run; If the guardrail itself fails; Callable from Cloudflare Workers. --- ## Google Vertex AI - Kind: Cloud platform - Who runs it: Managed only - Page: https://gatewayscore.com/providers/google-vertex-ai - Most recent check: 2026-08-29 - Evidence basis: documentation review only Google Cloud's model platform for Gemini plus 200+ Model Garden models, now branded Gemini Enterprise Agent Platform. ### Overview - What kind of product: Cloud platform - Who runs it: Managed only - Licence: Proprietary - Company: Google Cloud [https://cloud.google.com/vertex-ai] - Who you would be signing with: Large cloud vendor product [https://cloud.google.com/vertex-ai/sla] - Last shipped an update: 2026-08-23 [https://docs.cloud.google.com/vertex-ai/docs/release-notes] ### Cost - Free tier: $300 in free credits for new Google Cloud customers; Grounding with Google Search includes 5,000 free queries per month. [https://cloud.google.com/vertex-ai/generative-ai/pricing] - Cost to run it yourself: Managed only. Google sets its own per-token and per-modality prices rather than marking up a third party; Batch mode is 50% below interactive pricing, and Provisioned Throughput is sold in Generative AI Scale Units (GSUs) on 1-week to 1-year terms. Gemini 3.7/3.6 Flash carries introductory pricing of $0.75/1M input and $3.75/1M output through Dec 31 2026, rising to $1.50/$7.50 afterwards. [https://cloud.google.com/vertex-ai/generative-ai/pricing] - How the vendor makes money: Bundled — inference is the product - How pricing works, briefly: Pay-for-own-inference, bundled routing; per-token model pricing plus separately metered grounding, caching storage and provisioned throughput. No platform fee, subscription or per-seat charge — routing layer effectively $0. - Minimum commitment: None for pay-as-you-go. Provisioned Throughput is sold in GSUs with 1-week, 1-month, 3-month and 1-year commitments. Flexible Savings Plans are spend-based committed use discounts. - Charges that fire after you go over an allowance: No log/trace retention or request-volume overage. Feature meters: grounding with Google Search $35 per 1,000 requests above 1,500/day (Gemini 2.0 Flash), Web Grounding for enterprise $45 per 1,000 requests, grounding with your data $2.50 per 1,000 prompts. Explicit context cache storage billed per token-hour ($0.0000045/token-hour for Gemini 3.1 Pro / 3 Pro / 2.5 Pro; $0.000001 for Flash/Flash-Lite). - Prompt cache offered: Both exact and semantic - Who captures the cache saving: Implicit cached input plus explicit Context Cache (prefix-based, not semantic). Prices cached input tokens, cache hits, explicit context-cache storage, and Anthropic-style cache writes. Examples: Gemini 3.1 Pro cached input $0.20/1M vs full input; Gemini 2.5 Pro $0.125 (≤200K) / $0.25 (>200K); Claude Opus 5 cache hit $0.50/1M vs 5-min cache write $6.25/1M; Claude Sonnet 5 cache hit $0.20 promotional / $0.30 from Sep 1 2026; DeepSeek-V3.1 cache hit $0.06; GLM-5 cache hit $0.10. Gemini implicit caching has no cache-write charge beyond storage; Anthropic models on Vertex charge explicit 5-min and 1-hour cache write rates above the input rate. A single universal cached-token discount or premium percentage is not stated. Explicit caching adds a storage meter (per token-hour) and Anthropic-style writes cost more than plain input — short-lived or low-reuse caches can be a net loss. - What you can split spend by: Billing tags are the only stated mechanism — 'to split out Computer Use costs, apply billing tags'. Per key/user/team/customer not stated. Published cost levers: Flexible Savings Plans, Provisioned Throughput GSUs, Model Optimizer configured for cost/quality/balance. - How you get cost data out: Not stated on the pricing page. - Who pays the model bill: Not applicable ### Catalog - Models available: 200 [https://cloud.google.com/vertex-ai] - Works with standard OpenAI code: Yes [https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/call-vertex-using-openai-library] - OpenAI chat endpoint: Yes - Embeddings endpoint: Yes - Image generation endpoint: Yes - Audio endpoints: Yes - Needs the vendor’s own code library: No [https://cloud.google.com/vertex-ai/generative-ai/docs/migrate/openai/overview] - You can export your request history: Yes [https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/request-response-logging] - Settings can live in version control: Yes [https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/vertex_ai_endpoint] - Embeddings: Yes [https://cloud.google.com/vertex-ai/generative-ai/pricing] - Image generation: Yes [https://cloud.google.com/vertex-ai/generative-ai/pricing] - Speech and audio: Yes [https://cloud.google.com/vertex-ai/generative-ai/pricing] - Video generation: Yes [https://cloud.google.com/vertex-ai/generative-ai/pricing] - Batch processing: Yes [https://cloud.google.com/vertex-ai/generative-ai/pricing] ### Routing & reliability - Uptime it promises in writing: 99.9% [https://cloud.google.com/vertex-ai/sla] - Response caching: Yes [https://cloud.google.com/vertex-ai/generative-ai/pricing] - Upstream health tracking: Fixed, cannot change - Cross-region failover you control: Fixed, cannot change - Where you set load balancing: Fixed, cannot change ### Operations - Prompt versioning: Yes [https://cloud.google.com/vertex-ai] - Quality testing: Yes [https://cloud.google.com/vertex-ai] - What gets logged: Nothing stored - You can turn logging off: Yes - Traces you can take elsewhere: OpenTelemetry - Where telemetry can go: Cloud Logging,Cloud Monitoring,BigQuery - Can record user feedback: No - Scores live traffic: No ### Performance - What the request path runs on: Cloud provider API - You can run the request path yourself: No - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Yes [https://cloud.google.com/vertex-ai/generative-ai/docs/learn/locations] - What they keep if you change nothing: Nothing stored by default [https://cloud.google.com/vertex-ai/generative-ai/docs/data-governance] - How long they keep it: 0 [https://cloud.google.com/vertex-ai/generative-ai/docs/data-governance] - Could they train on your prompts: No [https://docs.cloud.google.com/vertex-ai/generative-ai/docs/vertex-ai-zero-data-retention] - Where it runs, and what you can pin: Around 29 regional endpoints across the Americas, Europe, Asia-Pacific and the Middle East, plus jurisdictional multi-region endpoints that keep processing inside a boundary. Google explicitly warns against the global endpoint if you have processing requirements, because you cannot know which region handles the request. [https://cloud.google.com/vertex-ai/generative-ai/docs/learn/locations] - Where safety filters run: In the vendor’s cloud [https://cloud.google.com/vertex-ai/generative-ai/docs/data-governance] - Who else touches the data: https://cloud.google.com/terms/subprocessors [https://cloud.google.com/terms/subprocessors] - SOC 2 audited: Yes [https://cloud.google.com/security/compliance/soc-2] - Will sign a HIPAA agreement: Yes [https://cloud.google.com/security/compliance/hipaa-compliance] - Can keep data in the EU: Yes [https://cloud.google.com/vertex-ai/generative-ai/docs/learn/locations] - Does not retain your data: Yes [https://cloud.google.com/vertex-ai/generative-ai/docs/data-governance] - Blocks personal data in prompts: Can block the request - Blocks prompt injection: Can block the request - Blocks harmful content: Can block the request - Your own policy rules: Can block the request - Where guardrails run: On the vendor's servers - Third-party guardrail vendors: Google Model Armor,Sensitive Data Protection ### Fit & integration - Work to try it: Set it up in a cloud console - Work to run it: Set it up in a cloud console - Steps on the quickstart: 15 - Can you self-host it today: No self-hosting - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Official provider package - Python framework integrations: LangChain,LlamaIndex - Callable from Cloudflare Workers: Documented from your Worker - Kubernetes install: Only generic cluster docs - Terraform support: Resources in the cloud’s provider - Reuses your cloud identity: Reuses your cloud identity - Fits behind your API gateway: The cloud ships its own AI gateway - MCP support: MCP tools in the API - Needs your own provider key: Not needed - Gate before models work: You enable it first - Official client languages: Python,Go,Java,Node.js ### Not published by this vendor Google Vertex AI has not published a figure for: GitHub stars; Markup on model prices; Fee to add funds; Monthly cost per person; Can use your own provider accounts; Cost of using your own accounts; Enterprise plan from; Discount on cached input; Premium on cache writes; Model providers reachable; Anthropic messages endpoint; OpenAI Responses endpoint; Batch jobs endpoint; Automatic failover; Load balancing; Rule-based routing; Similar-question caching; Usage dashboards and logs; Spending limits; Rate limits; Separate keys per team or app; MCP support; Delay it adds; Requests per second ceiling; GDPR commitments; Strips personal data; Content guardrails; Runs fully disconnected; Where you set the timeout; Where you set retries; Where you set fallbacks; Shape of the fallback chain; If the guardrail itself fails. --- ## Groq - Kind: Inference provider - Who runs it: Managed only - Page: https://gatewayscore.com/providers/groq - Most recent check: 2026-08-29 - Evidence basis: used first-hand in production, plus documentation review Inference provider running open-weight models on its own LPU hardware for very high output speed. ### Overview - What kind of product: Inference provider - Who runs it: Managed only - Licence: Proprietary - Company: Groq [https://en.wikipedia.org/wiki/Groq] - Who you would be signing with: Independent company [https://groq.com/newsroom/groq-closes-usd350-million-series-a-building-the-world-s-leading-ai-inference-cloud] - Last shipped an update: 2026-04-18 [https://console.groq.com/docs/changelog] ### Cost - Free tier: Free tier with per-model rate limits (for example gpt-oss-120b at 30 requests/min, 1,000 requests/day, 8,000 tokens/min); the paid Developer plan raises this to about 1,000 RPM and 250,000 TPM and adds Batch and Flex processing. [https://console.groq.com/docs/rate-limits] - Cost to run it yourself: No self-host option; Groq sets its own per-token prices on its own LPU/LPX hardware rather than marking up another vendor's list price. Representative rates: gpt-oss-120b $0.15 input / $0.60 output per 1M tokens, gpt-oss-20b $0.075/$0.30, Qwen3.6-27B $0.60/$3.00. [https://console.groq.com/docs/models] - How the vendor makes money: Bundled — inference is the product - How pricing works, briefly: Pay-for-own-inference on a free/Developer/Enterprise ladder; pay-as-you-go per token with progressive billing. No separate gateway or platform fee — routing/API layer is $0. - Minimum commitment: None stated. Bills in arrears; progressive billing thresholds at $1, $10, $100, $500 and $1,000 (India: $1, $10, then recurring $100). $0.50 minimum before billing. - Charges that fire after you go over an allowance: None stated. Downgrading stops usage charges immediately and returns the account to Free tier rate limits and restrictions. - Prompt cache offered: Prefix cache - Discount on cached input: 50% - Premium on cache writes: None - Who captures the cache saving: Prefix prompt caching, free to use. 50% discount on cached input tokens; pricing discount applies only on successful cache hits. No cache-write charge. Clean win for the customer — no feature fee, no write premium — with one caveat: batch requests already receive a 50% discount on all tokens, no additional discount applied to cached tokens in batch requests. The two discounts do not stack. - What you can split spend by: Not stated. - How you get cost data out: Not stated. - Who pays the model bill: Their credits only ### Catalog - Works with standard OpenAI code: Yes [https://console.groq.com/docs/openai] - OpenAI chat endpoint: Yes - OpenAI Responses endpoint: Yes - Audio endpoints: Yes - Batch jobs endpoint: Yes - Needs the vendor’s own code library: No [https://console.groq.com/docs/openai] - Settings can live in version control: No [https://console.groq.com/docs/spend-limits] - Speech and audio: Yes [https://console.groq.com/docs/models] - Batch processing: Yes [https://console.groq.com/docs/rate-limits] ### Operations - Rate limits: Yes [https://console.groq.com/docs/rate-limits] - What gets logged: Metadata only - You can turn logging off: Yes - Traces you can take elsewhere: OpenTelemetry - Can record user feedback: No - Scores live traffic: No ### Performance - Requests per second ceiling: 1,000 rps [https://console.groq.com/docs/models] - What the request path runs on: Undisclosed vendor service - You can run the request path yourself: No - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Yes [https://console.groq.com/docs/your-data] - What they keep if you change nothing: Nothing stored by default [https://console.groq.com/docs/your-data] - How long they keep it: 0 [https://console.groq.com/docs/your-data] - Where it runs, and what you can pin: Retained data is US-only, in Google Cloud buckets. Inference itself spans 13 data centres on four continents including Finland, the UK, Saudi Arabia and Australia, but no customer region pinning is documented. [https://groq.com/platform] - Where safety filters run: No guardrails offered [https://console.groq.com/docs/your-data] - Who else touches the data: https://trust.groq.com/subprocessors [https://console.groq.com/docs/legal/customer-data-processing-addendum] - SOC 2 audited: Yes [https://trust.groq.com/] - Can keep data in the EU: No [https://console.groq.com/docs/your-data] - Does not retain your data: Yes [https://console.groq.com/docs/your-data] ### Fit & integration - Work to try it: Install a package - Work to run it: Install a package - Steps on the quickstart: 0 - Can you self-host it today: No self-hosting - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Official provider package - Python framework integrations: LangChain,LlamaIndex - Callable from Cloudflare Workers: Documented by Cloudflare, not the vendor - MCP support: MCP tools in the API - Needs your own provider key: Not needed - Official client languages: Python,TypeScript ### Not published by this vendor Groq has not published a figure for: GitHub stars; Markup on model prices; Fee to add funds; Monthly cost per person; Can use your own provider accounts; Cost of using your own accounts; Enterprise plan from; Models available; Model providers reachable; Anthropic messages endpoint; Embeddings endpoint; Image generation endpoint; You can export your request history; Embeddings; Image generation; Video generation; Uptime it promises in writing; Automatic failover; Load balancing; Rule-based routing; Response caching; Similar-question caching; Usage dashboards and logs; Spending limits; Separate keys per team or app; Prompt versioning; Quality testing; MCP support; Delay it adds; Could they train on your prompts; Will sign a HIPAA agreement; GDPR commitments; Strips personal data; Content guardrails; Runs fully disconnected; Where you set the timeout; Where you set retries; Where you set fallbacks; Shape of the fallback chain; Upstream health tracking; Cross-region failover you control; Where you set load balancing; Blocks personal data in prompts; Blocks prompt injection; Blocks harmful content; Your own policy rules; Where guardrails run; If the guardrail itself fails; Kubernetes install; Terraform support; Reuses your cloud identity; Fits behind your API gateway; Gate before models work. --- ## Helicone - Kind: Managed gateway - Who runs it: Managed or self-host - Page: https://gatewayscore.com/providers/helicone - Most recent check: 2026-08-29 - Evidence basis: used first-hand in production, plus documentation review Open-source LLM observability platform with an OpenAI-compatible AI gateway attached. ### Overview - What kind of product: Managed gateway - Who runs it: Managed or self-host [https://docs.helicone.ai/faq/compliance] - Licence: Apache-2.0 [https://docs.helicone.ai/references/open-source] - Company: Helicone (Mintlify) [https://chatforest.com/reviews/helicone-llm-observability-gateway/] - Who you would be signing with: Maintenance mode — no new features [https://www.mintlify.com/blog/mintlify-acquires-helicone] - Last shipped an update: 2025-08-21 [https://github.com/Helicone/helicone/releases/tag/v2025.08.21-1] - GitHub stars: 6,109 [https://api.github.com/repos/Helicone/helicone] ### Cost - Markup on model prices: None [https://docs.helicone.ai/gateway/overview] - Monthly cost per person: None [https://www.helicone.ai/pricing] - Can use your own provider accounts: Yes [https://docs.helicone.ai/getting-started/quick-start] - Cost of using your own accounts: Bring your own provider keys, or add Helicone credits and pay provider rates with 0% markup while Helicone manages provider keys. [https://docs.helicone.ai/gateway/overview] - Free tier: Hobby: free, 1 seat, 10,000 requests/month, 1 GB storage, 7-day retention, 10 logs/min ingestion. [https://www.helicone.ai/pricing] - Cost to run it yourself: Main platform repo is Apache-2.0 and self-hostable (infrastructure cost only); the standalone Rust AI gateway repo is GPL-3.0 and was last pushed 2025-11-21, which an independent review flags as effectively stale. [https://continuumcode.ai/guides/helicone-review/] - How the vendor makes money: Platform fee plus usage meters - How pricing works, briefly: Flat per-tier platform fee (Hobby free, Pro $79/mo, Team $799/mo, Enterprise custom) plus usage-based requests/storage. Gateway credits at 0% markup. 50% off first year for startups; $100 credit for OSS; free for students. - Minimum commitment: None stated for Pro/Team. Enterprise not stated. - Charges that fire after you go over an allowance: Pro/Team/Enterprise all include 10K requests + 1 GB storage with 'usage-based ↑' after — the per-unit overage rates are not published. Calculator implies storage pricing (0.30 GB → $0.97) without a stated rate. - Prompt cache offered: Exact-match cache - Who captures the cache saving: Exact-match gateway cache on Cloudflare's edge; cache key hashes seed, URL, full body, relevant headers and bucket index. Semantic cache not stated. Can combine with provider prompt caching. Helicone charges no cache surcharge, but cached responses count against your storage allowance. - What you can split spend by: Per user via Helicone-User-Id header; per tag/property via Custom Properties (project, feature, workflow stage, ticket). Per key/team not stated. Costs are estimates based on the model returned in the response body, not billed pass-through. - How you get cost data out: Query API for filtered retrieval; property-filtered webhooks. CSV, S3 and warehouse export not stated; pricing page 'Data export' line has no format, tier or price. - Who pays the model bill: Your keys or their credits ### Catalog - Models available: 111 [https://api.helicone.ai/v1/public/model-registry/models] - Model providers reachable: 21 [https://api.helicone.ai/v1/public/model-registry/models] - Works with standard OpenAI code: Yes [https://docs.helicone.ai/gateway/overview] - OpenAI chat endpoint: Yes - Anthropic messages endpoint: Yes - Needs the vendor’s own code library: No [https://docs.helicone.ai/getting-started/how-to-integrate] - You can export your request history: Yes [https://docs.helicone.ai/rest/request/post-v1requestquery-clickhouse] - Settings can live in version control: Yes [https://github.com/Helicone/ai-gateway] ### Routing & reliability - Automatic failover: Yes [https://docs.helicone.ai/gateway/overview] - Response caching: Yes [https://docs.helicone.ai/gateway/overview] - Where you set retries: Per request - Where you set fallbacks: Per request - Shape of the fallback chain: Ordered list - Upstream health tracking: Fixed, cannot change - Where you set load balancing: In config ### Operations - Usage dashboards and logs: Yes [https://docs.helicone.ai/gateway/overview] - Rate limits: Yes [https://docs.helicone.ai/gateway/overview] - Prompt versioning: Yes [https://docs.helicone.ai/gateway/overview] - Quality testing: Yes [https://www.helicone.ai/pricing] - What gets logged: Full prompts and responses - You can turn logging off: Yes - Traces you can take elsewhere: OpenTelemetry - Where telemetry can go: OpenTelemetry,Datadog,Webhook,PostHog - Can record user feedback: Yes - Scores live traffic: Partly ### Performance - What the request path runs on: Compiled binary - You can run the request path yourself: Yes - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Depends how you deploy it [https://docs.helicone.ai/faq/compliance] - What they keep if you change nothing: Yes — prompts and replies [https://docs.helicone.ai/features/advanced-usage/omit-logs] - How long they keep it: 7 [https://www.helicone.ai/pricing] - Where it runs, and what you can pin: US or EU for the hosted service, or self-host on your own Docker, Kubernetes or cloud. [https://docs.helicone.ai/faq/compliance] - Where safety filters run: No guardrails offered [https://docs.helicone.ai/faq/compliance] - SOC 2 audited: Yes [https://www.helicone.ai/pricing] - Will sign a HIPAA agreement: Yes [https://www.helicone.ai/pricing] - GDPR commitments: Yes [https://docs.helicone.ai/faq/compliance] - Can keep data in the EU: Yes [https://docs.helicone.ai/faq/compliance] - Does not retain your data: Yes [https://docs.helicone.ai/faq/compliance] - Strips personal data: No [https://continuumcode.ai/guides/helicone-review/] - Content guardrails: No [https://continuumcode.ai/guides/helicone-review/] - Blocks harmful content: Can block the request - Where guardrails run: Either, your choice - Third-party guardrail vendors: OpenAI Moderation ### Fit & integration - Work to try it: Change one base URL - Work to run it: Change one base URL - Steps on the quickstart: 2 - Can you self-host it today: Install command published - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Official provider package - Python framework integrations: LlamaIndex - Callable from Cloudflare Workers: Their gateway runs on Workers, not yours - Kubernetes install: Official Helm chart - Terraform support: Terraform code in the repo only - MCP support: MCP client tooling - Needs your own provider key: Not hosted, yes self-hosted - Gate before models work: No gate - Official client languages: Python,TypeScript ### Not published by this vendor Helicone has not published a figure for: Fee to add funds; Enterprise plan from; Discount on cached input; Premium on cache writes; OpenAI Responses endpoint; Embeddings endpoint; Image generation endpoint; Audio endpoints; Batch jobs endpoint; Embeddings; Image generation; Speech and audio; Video generation; Batch processing; Uptime it promises in writing; Load balancing; Rule-based routing; Similar-question caching; Spending limits; Separate keys per team or app; MCP support; Delay it adds; Requests per second ceiling; Could they train on your prompts; Who else touches the data; Runs fully disconnected; Where you set the timeout; Cross-region failover you control; Blocks personal data in prompts; Blocks prompt injection; Your own policy rules; If the guardrail itself fails; Reuses your cloud identity; Fits behind your API gateway. --- ## Kong AI Gateway - Kind: Managed gateway - Who runs it: Managed or self-host - Page: https://gatewayscore.com/providers/kong-ai-gateway - Most recent check: 2026-08-29 - Evidence basis: documentation review only AI plugins on the Kong API gateway, adding LLM routing, guardrails and token limits to existing API infrastructure. ### Overview - What kind of product: Managed gateway - Who runs it: Managed or self-host - Licence: Open core [https://github.com/Kong/kong] - Company: Kong Inc. [https://konghq.com/about] - Who you would be signing with: Independent company [https://konghq.com/company/press-room/press-release/kong-to-unveil-ai-connectivity-vision-and-roadmap] - Last shipped an update: 2026-08-26 [https://developer.konghq.com/gateway/changelog/] - GitHub stars: 44,100 [https://github.com/Kong/kong] ### Cost - Can use your own provider accounts: Yes [https://developer.konghq.com/ai-gateway/] - Cost of using your own accounts: You supply your own upstream LLM credentials, stored centrally in Kong's config store; Kong charges for gateways, not tokens. - Free tier: 30-day Konnect free trial with enterprise functionality; separately, Kong Gateway core is Apache-2.0 and free to self-host. [https://konghq.com/pricing] - Cost to run it yourself: Kong Gateway core is Apache-2.0 and self-hostable at infra cost only. The Plus plan is billed per gateway per month with the amount not published; fully self-hosted gateways plus audit logs and SSO require the custom-priced, annually billed Enterprise plan. Third-party reviews note the OSS build lacks semantic caching, analytics and compliance features found in Enterprise. [https://konghq.com/pricing] - How the vendor makes money: Flat monthly platform fee - How pricing works, briefly: Flat per-gateway platform fee (Konnect Plus, no dollar figure published) + enterprise-only annual quote. Kong does not resell inference. Plans billed on a calendar month basis in arrears. - Minimum commitment: Enterprise 'billed annually' with duration set out in the individual Order Form; no minimum amount stated. Plus is monthly. - Charges that fire after you go over an allowance: No metered overage billing published. Plus enforces soft limits via email + in-app notifications prompting you to reduce usage or upgrade. - Prompt cache offered: Semantic cache - Who captures the cache saving: Semantic caching included in Plus under 'Cost Control'. Exact-match and provider passthrough not stated. Kong prices no tokens, so semantic cache hits save on the customer's provider bill; cached requests may still count toward Plus request/analytics limits. - What you can split spend by: Plus includes 'AI Observability: token-level tracking and real-time cost analytics'. Per key/user/team/tag/customer splits not stated. - How you get cost data out: None stated on the pricing page. - Who pays the model bill: Your keys only ### Catalog - Works with standard OpenAI code: Yes [https://developer.konghq.com/ai-gateway/] - OpenAI chat endpoint: Yes - Anthropic messages endpoint: Yes - OpenAI Responses endpoint: Yes - Embeddings endpoint: Yes - Image generation endpoint: Yes - Audio endpoints: Yes - Batch jobs endpoint: Yes - Needs the vendor’s own code library: No [https://developer.konghq.com/plugins/ai-proxy/] - You can export your request history: Yes [https://developer.konghq.com/plugins/http-log/] - Settings can live in version control: Yes [https://developer.konghq.com/deck/file/format/] - Embeddings: Yes [https://developer.konghq.com/ai-gateway/] ### Routing & reliability - Uptime it promises in writing: 99.9% [https://konghq.com/pricing] - Automatic failover: Yes [https://developer.konghq.com/ai-gateway/] - Load balancing: Yes [https://developer.konghq.com/ai-gateway/] - Rule-based routing: Yes [https://developer.konghq.com/ai-gateway/] - Similar-question caching: Yes [https://developer.konghq.com/ai-gateway/] - Where you set the timeout: In config - Where you set retries: In config - Where you set fallbacks: In config - Shape of the fallback chain: Weighted split - Upstream health tracking: In config - Where you set load balancing: In config ### Operations - Usage dashboards and logs: Yes [https://developer.konghq.com/ai-gateway/] - Rate limits: Yes [https://konghq.com/pricing] - Prompt versioning: Yes [https://developer.konghq.com/ai-gateway/] - MCP support: Yes [https://konghq.com/pricing] - What gets logged: Your choice - You can turn logging off: Yes - Traces you can take elsewhere: OpenTelemetry - Where telemetry can go: OpenTelemetry,Prometheus,Datadog,StatsD,HTTP log,Kafka,Loki - Can record user feedback: No - Scores live traffic: No ### Performance - Delay it adds: 12 ms [https://apiscout.dev/guides/portkey-vs-kong-ai-gateway-llm-routing-2026] - Requests per second ceiling: 8,200 rps [https://apiscout.dev/guides/portkey-vs-kong-ai-gateway-llm-routing-2026] - What the request path runs on: Interpreted proxy - You can run the request path yourself: Yes - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Depends how you deploy it [https://developer.konghq.com/gateway/deployment-topologies/] - What they keep if you change nothing: Nothing stored by default [https://docs.jp.konghq.com/hub/kong-inc/ai-proxy/configuration/] - How long they keep it: 30 [https://konghq.com/pricing] - Where it runs, and what you can pin: Konnect control-plane geographies in Australia, the EU, the Middle East, the US, India and Singapore, with dedicated data planes across roughly 17 AWS, 13 Azure and 11 Google regions. Self-hosted and on-premises data planes are supported. [https://developer.konghq.com/konnect-platform/geos/] - Where safety filters run: In your own infrastructure [https://developer.konghq.com/plugins/ai-sanitizer/] - Who else touches the data: https://konghq.com/legal/subprocessors [https://konghq.com/legal/subprocessors] - SOC 2 audited: Yes [https://trust.konghq.com/] - GDPR commitments: Yes [https://trust.konghq.com/] - Strips personal data: Yes [https://developer.konghq.com/ai-gateway/] - Content guardrails: Yes [https://developer.konghq.com/ai-gateway/] - Blocks personal data in prompts: Can block the request - Blocks prompt injection: Can block the request - Blocks harmful content: Can block the request - Your own policy rules: Can block the request - Where guardrails run: Either, your choice - Third-party guardrail vendors: AWS Bedrock Guardrails,Azure Content Safety,Google Model Armor,Lakera ### Fit & integration - Work to try it: Run something locally first - Work to run it: Deploy it on your infrastructure - Steps on the quickstart: 3 - Can you self-host it today: Install command published - Works with the OpenAI SDK: Partly - Python framework integrations: LangChain - Kubernetes install: Official Helm chart - Terraform support: Official Terraform provider - Reuses your cloud identity: Reuses your cloud identity - Fits behind your API gateway: It is the API gateway - MCP support: MCP gateway or registry - Needs your own provider key: Required ### Not published by this vendor Kong AI Gateway has not published a figure for: Markup on model prices; Fee to add funds; Monthly cost per person; Enterprise plan from; Discount on cached input; Premium on cache writes; Models available; Model providers reachable; Image generation; Speech and audio; Video generation; Batch processing; Response caching; Spending limits; Separate keys per team or app; Quality testing; Could they train on your prompts; Will sign a HIPAA agreement; Can keep data in the EU; Does not retain your data; Runs fully disconnected; Cross-region failover you control; If the guardrail itself fails; Vercel AI SDK support; Callable from Cloudflare Workers; Gate before models work. --- ## LiteLLM - Kind: Open source - Who runs it: Self-host only - Page: https://gatewayscore.com/providers/litellm - Most recent check: 2026-08-31 - Evidence basis: used first-hand in production, plus documentation review Open-source AI gateway and Python SDK that puts one OpenAI-compatible API in front of many LLM providers. ### Overview - What kind of product: Open source - Who runs it: Self-host only - Licence: MIT [https://github.com/BerriAI/litellm] - Company: BerriAI [https://www.ycombinator.com/companies/litellm] - Who you would be signing with: Independent company [https://www.ycombinator.com/companies/litellm] - Last shipped an update: 2026-08-23 [https://github.com/BerriAI/litellm/releases/tag/v1.98.0] - GitHub stars: 57,500 [https://github.com/BerriAI/litellm] ### Cost - Markup on model prices: None [https://www.litellm.ai/#pricing] - Fee to add funds: None [https://www.litellm.ai/#pricing] - Monthly cost per person: None [https://www.litellm.ai/#pricing] - Can use your own provider accounts: Yes [https://www.litellm.ai/#pricing] - Cost of using your own accounts: Self-hosted only: you configure your own upstream provider keys and pay providers directly; no gateway token fee. - Free tier: Open-source gateway is $0 forever: 140+ providers, virtual keys, teams, budgets, load balancing, RPM/TPM limits, guardrails. [https://www.litellm.ai/#pricing] - Cost to run it yourself: OSS is free (MIT); you pay only for your own containers plus Postgres and Redis. Enterprise features (SSO, RBAC, JWT auth, SCIM, audit logs, support SLAs) require a paid LiteLLM commercial license whose price is not published. [https://www.litellm.ai/#pricing] - How the vendor makes money: Open source with a managed tier - How pricing works, briefly: OSS $0 free forever, self-hosted. Enterprise is talk-to-sales, sized to annual gateway request capacity, deployment architecture and support needs — 'never per token'. Volume discount tiers exist but no minimum amount is published. - Minimum commitment: Enterprise is annual term; no stated minimum amount. - Charges that fire after you go over an allowance: None stated for OSS or Enterprise. Practical exposure is self-hosted infrastructure and database cost. - Prompt cache offered: Both exact and semantic - Who captures the cache saving: Exact-match and semantic self-hosted caches: In-Memory, Disk, Redis, Qdrant Semantic, Redis Semantic, Valkey Semantic, S3 bucket, GCS bucket. LiteLLM prices no tokens; provider cached-token rates flow through in its cost calculations. Saving is 100% the customer's; the cost you add is running Redis/Qdrant yourself. - What you can split spend by: The strongest of any product here: spend logged per hashed api_key, internal user, end-user/customer, team_id, tags, model_group, api_base, User-Agent (tracked as a tag) and arbitrary custom headers/metadata. - How you get cost data out: API + database: /spend/logs, /user/info, /user/daily/activity, /gateway/daily/activity, /global/spend/report?group_by=customer|team|api_key|internal_user, x-litellm-response-cost header, and the LiteLLM_SpendLogs table you own. CSV, webhook, S3 and warehouse export not stated as first-class features. - Who pays the model bill: Your keys only ### Catalog - Model providers reachable: 140 [https://www.litellm.ai/] - Works with standard OpenAI code: Yes [https://docs.litellm.ai/docs/simple_proxy] - OpenAI chat endpoint: Yes - Anthropic messages endpoint: Yes - OpenAI Responses endpoint: Yes - Embeddings endpoint: Yes - Image generation endpoint: Yes - Audio endpoints: Yes - Batch jobs endpoint: Yes - Needs the vendor’s own code library: No [https://docs.litellm.ai/docs/proxy/user_keys] - You can export your request history: Yes [https://docs.litellm.ai/docs/proxy/cost_tracking] - Settings can live in version control: Yes [https://docs.litellm.ai/docs/proxy/configs] - Embeddings: Yes [https://github.com/BerriAI/litellm] - Image generation: Yes [https://github.com/BerriAI/litellm] - Speech and audio: Yes [https://github.com/BerriAI/litellm] - Batch processing: Yes [https://github.com/BerriAI/litellm] ### Routing & reliability - Automatic failover: Yes [https://www.litellm.ai/#pricing] - Load balancing: Yes [https://www.litellm.ai/#pricing] - Rule-based routing: Yes [https://www.litellm.ai/] - Response caching: Yes [https://docs.litellm.ai/docs/proxy/prod] - Similar-question caching: Yes [https://docs.litellm.ai/docs/proxy/prod] - Where you set the timeout: Per request - Where you set retries: Per request - Where you set fallbacks: In config - Shape of the fallback chain: Weighted split - Upstream health tracking: In config - Cross-region failover you control: In config - Where you set load balancing: In config ### Operations - Usage dashboards and logs: Yes [https://github.com/BerriAI/litellm] - Spending limits: Yes [https://www.litellm.ai/#pricing] - Rate limits: Yes [https://www.litellm.ai/#pricing] - Separate keys per team or app: Yes [https://www.litellm.ai/#pricing] - MCP support: Yes [https://github.com/BerriAI/litellm] - What gets logged: Your choice - You can turn logging off: Yes - Traces you can take elsewhere: OpenTelemetry - Where telemetry can go: OpenTelemetry,Datadog,Langfuse,Langsmith,Arize Phoenix,Prometheus,S3,GCS,Athina,Braintrust,Helicone,Lago,Logfire,Sentry,Slack,Webhook,Galileo,OpenMeter,DynamoDB,Azure Blob Storage - Can record user feedback: No - Scores live traffic: Partly ### Performance - Delay it adds: 0.66 ms [https://www.litellm.ai/] - Requests per second ceiling: 2,800 rps [https://www.litellm.ai/] - What the request path runs on: Interpreted proxy - You can run the request path yourself: Yes - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: No [https://docs.litellm.ai/docs/data_security] - What they keep if you change nothing: Metadata only, not content [https://docs.litellm.ai/docs/proxy/ui_logs] - Could they train on your prompts: No [https://docs.litellm.ai/docs/data_security] - Where it runs, and what you can pin: Your own infrastructure, including air-gapped. Enterprise offers a multi-region control plane. [https://www.litellm.ai/enterprise] - Where safety filters run: In your own infrastructure [https://docs.litellm.ai/docs/proxy/guardrails/pii_masking_v2] - Strips personal data: Yes [https://www.litellm.ai/#pricing] - Content guardrails: Yes [https://www.litellm.ai/#pricing] - Runs fully disconnected: Yes [https://www.litellm.ai/#pricing] - Blocks personal data in prompts: Can block the request - Blocks prompt injection: Can block the request - Blocks harmful content: Can block the request - Your own policy rules: Can block the request - Where guardrails run: Either, your choice - If the guardrail itself fails: You choose - Third-party guardrail vendors: Microsoft Presidio,AWS Bedrock Guardrails,Azure Content Safety,OpenAI Moderation,Lakera,Aporia,Pangea,Guardrails AI,Pillar Security ### Fit & integration - Work to try it: Run something locally first - Work to run it: Deploy it on your infrastructure - Steps on the quickstart: 0 - Can you self-host it today: Install command published - Works with the OpenAI SDK: Yes - Python framework integrations: LangChain,LlamaIndex - Callable from Cloudflare Workers: Workers AI as an upstream model - Kubernetes install: Official Helm chart - Terraform support: Official modules - Reuses your cloud identity: Reuses your cloud identity - MCP support: MCP gateway or registry - Needs your own provider key: Not needed - Official client languages: Python ### Not published by this vendor LiteLLM has not published a figure for: Enterprise plan from; Discount on cached input; Premium on cache writes; Models available; Video generation; Uptime it promises in writing; Prompt versioning; Quality testing; How long they keep it; Who else touches the data; SOC 2 audited; Will sign a HIPAA agreement; GDPR commitments; Can keep data in the EU; Does not retain your data; Vercel AI SDK support; Fits behind your API gateway; Gate before models work. --- ## LLM Gateway - Kind: Open source - Who runs it: Managed or self-host - Page: https://gatewayscore.com/providers/llmgateway - Most recent check: 2026-08-29 - Evidence basis: used first-hand in production, plus documentation review AGPL-licensed OpenAI-compatible gateway available as one self-hosted Docker image or a hosted service with credit fees. ### Overview - What kind of product: Open source - Who runs it: Managed or self-host - Licence: AGPL-3.0 [https://github.com/theopenco/llmgateway] - Company: Polar Lights LLC [https://llmgateway.io/about] - Who you would be signing with: Independent company [https://llmgateway.io/about] - Last shipped an update: 2026-08-24 [https://github.com/theopenco/llmgateway/releases/tag/v1.14.0] - GitHub stars: 601 [https://github.com/theopenco/llmgateway] ### Cost - Markup on model prices: None [https://llmgateway.io/pricing] - Fee to add funds: 5% [https://llmgateway.io/pricing] - Monthly cost per person: None [https://llmgateway.io/pricing] - Can use your own provider accounts: Yes [https://llmgateway.io/pricing] - Cost of using your own accounts: BYOK included on the free plan at no fee; self-hosted requests carry no credit fee and no BYOK metering. [https://llmgateway.io/blog/open-source-openrouter-alternatives] - Free tier: $0 forever: no seats or minimums, BYOK free, 3 free models limited to 20 req/min, 30-day data retention; hosted credits carry a 5% platform fee. [https://llmgateway.io/pricing] - Cost to run it yourself: The whole platform (gateway, API, dashboard, worker) ships as one Docker image, ghcr.io/theopenco/llmgateway-unified:latest, free forever under AGPLv3 with no credit fee or BYOK metering; the same codebase powers the managed cloud. Code in the repo's ee/ directory requires a separate commercial enterprise license (contact@llmgateway.io). [https://llmgateway.io/open-source] - How the vendor makes money: Platform fee plus usage meters - How pricing works, briefly: Free OSS self-host (AGPLv3) + credit top-up fee on the hosted platform (5% flat, +1.5% international card fee) + optional storage charge + Enterprise quote. No seats, no minimums, no subscription. Provider rates passed through 0% markup. BYOK 0% fee. - Minimum commitment: None — 'no minimums, no subscription'. Enterprise is custom with volume discounts and a 30-day trial. - Charges that fire after you go over an allowance: The only after-the-fact charge is storage: full data retention bills stored requests at $0.01 per 1M tokens; metadata retention is free. Free-plan retention is 30 days; Enterprise is unlimited. No request-volume overage or export fee stated. - Prompt cache offered: No gateway-owned cache - Who captures the cache saving: Prompt caching is offered ('cache prompts for faster responses'); BYOK users 'still get unified analytics, caching, and failover'. Mechanism (exact-match / semantic / provider passthrough) not stated. LLM Gateway's only fee is 5% on credit purchases (0% on BYOK), so cache savings flow to the customer. - What you can split spend by: Per project (cost ceilings and per-project routing overrides) with 'Advanced analytics'. Per key/user/team/tag/customer not stated. - How you get cost data out: No CSV/API/webhook/S3/warehouse cost export stated; only alerting webhooks and enterprise audit logs. - Who pays the model bill: Your keys or their credits ### Catalog - Models available: 200 [https://llmgateway.io/pricing] - Model providers reachable: 40 [https://llmgateway.io/pricing] - Works with standard OpenAI code: Yes [https://llmgateway.io/] - OpenAI chat endpoint: Yes - Anthropic messages endpoint: Yes - Image generation endpoint: Partly - Needs the vendor’s own code library: No [https://docs.llmgateway.io/] - Settings can live in version control: No [https://docs.llmgateway.io/custom-providers] ### Routing & reliability - Automatic failover: Yes [https://llmgateway.io/about] - Load balancing: Yes [https://llmgateway.io/about] - Rule-based routing: Yes [https://llmgateway.io/pricing] - Response caching: Yes [https://llmgateway.io/about] - Where you set load balancing: In config ### Operations - Usage dashboards and logs: Yes [https://llmgateway.io/about] - Spending limits: Yes [https://llmgateway.io/pricing] - Rate limits: Yes [https://llmgateway.io/pricing] - What gets logged: Metadata only - You can turn logging off: Yes - Can record user feedback: No - Scores live traffic: No ### Performance - What the request path runs on: Interpreted proxy - You can run the request path yourself: Yes ### Security & compliance - Does your prompt reach their servers: Depends how you deploy it [https://llmgateway.io/legal/privacy] - What they keep if you change nothing: Metadata only, not content [https://docs.llmgateway.io/features/data-retention] - How long they keep it: 30 [https://docs.llmgateway.io/features/data-retention] - Could they train on your prompts: No [https://llmgateway.io/legal/privacy] - Where it runs, and what you can pin: Processed and stored in the EU or the US, with per-project routing overrides to pin regulated workloads to specific regions and providers. Self-hosting supported. [https://llmgateway.io/legal/privacy] - Where safety filters run: Either, depending on deployment [https://llmgateway.io/blog/soc2-type-ii] - Who else touches the data: https://security.llmgateway.io/ [https://llmgateway.io/blog/soc2-type-ii] - SOC 2 audited: Yes [https://llmgateway.io/] - Strips personal data: Yes [https://llmgateway.io/pricing] - Content guardrails: Yes [https://llmgateway.io/pricing] - Where guardrails run: Either, your choice ### Fit & integration - Work to try it: Change one base URL - Work to run it: Change one base URL - Steps on the quickstart: 6 - Can you self-host it today: Offered, but no command published - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Official provider package - Kubernetes install: Official Helm chart - Terraform support: Official modules - MCP support: Hosted MCP server - Needs your own provider key: Not hosted, yes self-hosted - Gate before models work: No gate - Official client languages: TypeScript ### Not published by this vendor LLM Gateway has not published a figure for: Enterprise plan from; Discount on cached input; Premium on cache writes; OpenAI Responses endpoint; Embeddings endpoint; Audio endpoints; Batch jobs endpoint; You can export your request history; Embeddings; Image generation; Speech and audio; Video generation; Batch processing; Uptime it promises in writing; Similar-question caching; Separate keys per team or app; Prompt versioning; Quality testing; MCP support; Delay it adds; Requests per second ceiling; Will sign a HIPAA agreement; GDPR commitments; Can keep data in the EU; Does not retain your data; Runs fully disconnected; Where you set the timeout; Where you set retries; Where you set fallbacks; Shape of the fallback chain; Upstream health tracking; Cross-region failover you control; Streaming responses; Blocks personal data in prompts; Blocks prompt injection; Blocks harmful content; Your own policy rules; If the guardrail itself fails; Traces you can take elsewhere; Callable from Cloudflare Workers; Reuses your cloud identity; Fits behind your API gateway. --- ## OpenRouter - Kind: Managed marketplace - Who runs it: Managed only - Page: https://gatewayscore.com/providers/openrouter - Most recent check: 2026-08-29 - Evidence basis: used first-hand in production, plus documentation review Hosted marketplace that routes one OpenAI-compatible API to models from many inference providers. ### Overview - What kind of product: Managed marketplace - Who runs it: Managed only [https://api7.ai/openrouter-vs-vercel-ai-gateway] - Licence: Proprietary [https://api7.ai/openrouter-vs-vercel-ai-gateway] - Company: OpenRouter, Inc. [https://en.wikipedia.org/wiki/OpenRouter] - Who you would be signing with: Being acquired — deal not yet closed [https://openrouter.ai/blog/announcements/openrouter-is-joining-stripe/] - Last shipped an update: 2026-08-19 [https://openrouter.ai/docs/changelog] ### Cost - Markup on model prices: None [https://openrouter.ai/docs/faq] - Fee to add funds: 5.5% [https://openrouter.ai/docs/faq] - Can use your own provider accounts: Yes [https://openrouter.ai/docs/use-cases/byok] - Cost of using your own accounts: Free up to $25,000/mo of list-price inference (pay-as-you-go) or $200,000/mo (Enterprise), then 5% of the equivalent OpenRouter cost; measured by list-price cost, not request count. [https://openrouter.ai/pricing] - Free tier: Free tier with 25+ free models and 50 free-model requests/day; 1,000/day after purchasing at least $10 in credits. [https://openrouter.ai/pricing] - How the vendor makes money: Percentage on tokens or top-ups - How pricing works, briefly: Credit top-up fee (5.5% Stripe / 5% crypto) with a $0.80 minimum; 0% token markup; BYOK charged 5% above a monthly list-price allowance ($25k Pay-as-you-go / $200k Enterprise). - Minimum commitment: None stated. Unused credits may expire one year after purchase. - Charges that fire after you go over an allowance: No log/trace-retention or request-volume overage. Only after-the-fact charge is the BYOK 5% once the monthly list-price allowance is exceeded. - Prompt cache offered: Passes provider caching through - Who captures the cache saving: Provider caching flows through unchanged. Read discounts vary by provider: 75%/50% for OpenAI, 90% for Anthropic/Alibaba/DeepSeek, 50% for Groq, ~80% for Z.AI, 75% for Gemini implicit. Cache-write premium: 0% for pre-GPT-5.6 OpenAI/Grok/Moonshot/Groq/Gemini; +25% for GPT-5.6+, Alibaba explicit and Anthropic 5-min; +100% for Anthropic 1-hour. OpenRouter itself adds 0% on cached traffic. - What you can split spend by: Per model, provider and API key on the Activity page; session_id grouping across turns. Per-user/team/customer not stated. - How you get cost data out: API only (/api/v1/key, credits API, /api/v1/generation). CSV, webhook, S3 and warehouse export not stated. - Who pays the model bill: Your keys or their credits ### Catalog - Models available: 500 [https://openrouter.ai/pricing] - Model providers reachable: 80 [https://openrouter.ai/pricing] - Works with standard OpenAI code: Yes [https://openrouter.ai/docs/faq] - OpenAI chat endpoint: Yes - Anthropic messages endpoint: Yes - OpenAI Responses endpoint: Yes - Embeddings endpoint: Yes - Image generation endpoint: Yes - Audio endpoints: Yes - Batch jobs endpoint: Yes - Needs the vendor’s own code library: No [https://openrouter.ai/docs/quickstart] - You can export your request history: Yes [https://openrouter.ai/docs/cookbook/administration/activity-export] - Settings can live in version control: No [https://openrouter.ai/docs/guides/features/presets] - Image generation: Yes [https://openrouter.ai/models] - Video generation: Yes [https://openrouter.ai/models] ### Routing & reliability - Automatic failover: Yes [https://openrouter.ai/enterprise] - Load balancing: Yes [https://zackproser.com/blog/openrouter-vs-vercel-ai-gateway] - Rule-based routing: Yes [https://openrouter.ai/pricing] - Response caching: Yes [https://openrouter.ai/pricing] - Similar-question caching: No [https://api7.ai/openrouter-vs-vercel-ai-gateway] - Where you set fallbacks: Per request - Shape of the fallback chain: Ordered list - Upstream health tracking: Fixed, cannot change - Where you set load balancing: Per request ### Operations - Usage dashboards and logs: Yes [https://openrouter.ai/enterprise] - Spending limits: Yes [https://openrouter.ai/pricing] - Rate limits: Yes [https://openrouter.ai/pricing] - Separate keys per team or app: Yes [https://openrouter.ai/docs/features/provisioning-api-keys] - What gets logged: Metadata only - You can turn logging off: Yes - Where telemetry can go: CSV export - Can record user feedback: No - Scores live traffic: No ### Performance - What the request path runs on: Edge worker - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Yes [https://openrouter.ai/privacy] - What they keep if you change nothing: Metadata only, not content [https://openrouter.ai/docs/guides/privacy/data-collection] - How long they keep it: 0 [https://openrouter.ai/docs/guides/privacy/data-collection] - Could they train on your prompts: No [https://openrouter.ai/privacy] - Where it runs, and what you can pin: US servers by default. Enterprise adds EU region locking and in-region routing. No self-hosted or customer-VPC option. [https://openrouter.ai/enterprise] - Where safety filters run: In the vendor’s cloud [https://openrouter.ai/docs/guides/features/guardrails] - Who else touches the data: https://openrouter.ai/authorized-sub-processors [https://openrouter.ai/authorized-sub-processors] - SOC 2 audited: Yes [https://openrouter.ai/enterprise] - GDPR commitments: Yes [https://openrouter.ai/enterprise] - Can keep data in the EU: Yes [https://openrouter.ai/docs/features/privacy-and-logging] - Does not retain your data: Yes [https://openrouter.ai/enterprise] - Strips personal data: No [https://api7.ai/openrouter-vs-vercel-ai-gateway] - Content guardrails: No [https://api7.ai/openrouter-vs-vercel-ai-gateway] - Blocks personal data in prompts: Can block the request - Blocks prompt injection: Can block the request - Your own policy rules: Can block the request - Where guardrails run: On the vendor's servers - Third-party guardrail vendors: Microsoft Presidio ### Fit & integration - Work to try it: Change one base URL - Work to run it: Change one base URL - Steps on the quickstart: 0 - Can you self-host it today: No self-hosting - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Official provider package - Python framework integrations: LangChain,LlamaIndex - Callable from Cloudflare Workers: Their gateway runs on Workers, not yours - Terraform support: Community provider - Reuses your cloud identity: Static provider credentials only - Fits behind your API gateway: The gateway vendor documents it - MCP support: Hosted MCP server - Needs your own provider key: Not needed - Gate before models work: No gate - Official client languages: TypeScript,Python,Go ### Not published by this vendor OpenRouter has not published a figure for: GitHub stars; Monthly cost per person; Enterprise plan from; Cost to run it yourself; Discount on cached input; Premium on cache writes; Embeddings; Speech and audio; Batch processing; Uptime it promises in writing; Prompt versioning; Quality testing; MCP support; Delay it adds; Requests per second ceiling; Will sign a HIPAA agreement; Runs fully disconnected; Where you set the timeout; Where you set retries; Cross-region failover you control; You can run the request path yourself; Blocks harmful content; If the guardrail itself fails; Traces you can take elsewhere; Kubernetes install. --- ## Orq.ai Router - Kind: Managed gateway - Who runs it: Managed or self-host - Page: https://gatewayscore.com/providers/orq-ai-router - Most recent check: 2026-08-29 - Evidence basis: documentation review only Managed EU-hosted AI gateway and router bundled with evaluation, observability and governance tooling. ### Overview - What kind of product: Managed gateway - Who runs it: Managed or self-host - Licence: Proprietary - Company: Orq.ai - Who you would be signing with: Independent company [https://orq.ai/press-releases] - Last shipped an update: 2026-08-26 [https://docs.orq.ai/changelog] ### Cost - Markup on model prices: None [https://orq.ai/pricing] - Fee to add funds: 4.5% [https://orq.ai/pricing] - Can use your own provider accounts: Yes [https://orq.ai/pricing] - Cost of using your own accounts: BYOK free for the first 1M requests/month, then 4%; no markup on model traffic itself. [https://orq.ai/pricing] - Free tier: Pay-as-you-go starts free: 100k spans, 1 GB processed data, 500 agent runs, unlimited AI Gateway seats, 100 req/min limit, 30-day retention. [https://orq.ai/pricing] - Cost to run it yourself: No open-source or free self-host option. Enterprise-only sovereign deployments: customer-owned VPC on AWS or Azure via marketplace, or fully on-prem/air-gapped via a single Helm chart, priced custom and billed annually. [https://docs.orq.ai/docs/enterprise/sovereign-ai] - How the vendor makes money: Platform fee plus usage meters - How pricing works, briefly: Models billed at provider list prices with 0% Orq markup. Routing on Orq credits adds a 4.5% top-up fee; BYOK is free up to 1M req/mo then 4%. Seat, module and data overages layer on top. Enterprise annual. - Minimum commitment: None stated for Pay-as-you-go. Enterprise agreements run on annual terms. - Charges that fire after you go over an allowance: Spans 100k/mo included then €7 per 100k. Processed data 1 GB/mo then €3/GB. Agent runs 500/mo then €0.01/run. Knowledge Bases include 2.5 GB document processing then €0.10/MB. Retention 30 days, no published price for extending. Usage beyond allowance bills as separate line items — nothing blocks. - Prompt cache offered: No gateway-owned cache - Who captures the cache saving: Pricing page lists caching in the platform comparison but does not name a mechanism; AI Gateway docs page timed out on fetch. Because model traffic is billed at provider rates with no Orq markup, any provider-side cache discount flows to the customer, but Orq publishes no cached-token pricing of its own. - What you can split spend by: Attribution by workspace, project, identity, API key, provider and model. Tag/customer splits not stated. - How you get cost data out: CSV/JSON of traces, experiments and datasets; Dashboards & Reporting API; SIEM export to Splunk/Datadog — all with no stated additional cost. S3/warehouse export not stated. - Who pays the model bill: Your keys only ### Catalog - Models available: 500 [https://orq.ai/pricing] - Model providers reachable: 30 [https://orq.ai/pricing] - Works with standard OpenAI code: Yes [https://docs.orq.ai/docs/ai-gateway] - OpenAI chat endpoint: Yes - OpenAI Responses endpoint: Yes - Embeddings endpoint: Yes - Image generation endpoint: Yes - Audio endpoints: Yes - Needs the vendor’s own code library: No [https://docs.orq.ai/docs/ai-gateway/features/openai-compatible-api] - You can export your request history: Yes [https://docs.orq.ai/docs/ai-studio/observability/telemetry-query-api] - Settings can live in version control: No [https://docs.orq.ai/docs/deployments/routing] - Embeddings: Yes [https://docs.orq.ai/docs/ai-gateway] - Image generation: Yes [https://docs.orq.ai/docs/ai-gateway] - Speech and audio: Yes [https://docs.orq.ai/docs/ai-gateway] - Video generation: Yes [https://docs.orq.ai/docs/ai-gateway] ### Routing & reliability - Automatic failover: Yes [https://docs.orq.ai/docs/ai-gateway] - Load balancing: Yes [https://docs.orq.ai/docs/ai-gateway] - Rule-based routing: Yes [https://docs.orq.ai/docs/ai-gateway] - Response caching: Yes [https://docs.orq.ai/docs/ai-gateway] - Where you set the timeout: Per request - Where you set retries: Per request - Where you set fallbacks: Per request - Shape of the fallback chain: Ordered list - Where you set load balancing: Per request ### Operations - Usage dashboards and logs: Yes [https://orq.ai/] - Spending limits: Yes [https://orq.ai/pricing] - Rate limits: Yes [https://orq.ai/pricing] - Prompt versioning: Yes [https://orq.ai/] - Quality testing: Yes [https://orq.ai/] - MCP support: Yes [https://docs.orq.ai/docs/ai-gateway] - What gets logged: Your choice - You can turn logging off: Yes - Traces you can take elsewhere: OpenTelemetry - Where telemetry can go: Webhook - Can record user feedback: Yes - Scores live traffic: Yes ### Performance - What the request path runs on: Undisclosed vendor service - You can run the request path yourself: Yes - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Depends how you deploy it [https://docs.orq.ai/docs/enterprise/deployment-options] - What they keep if you change nothing: Yes — prompts and replies [https://docs.orq.ai/docs/administer/data-compliance] - Could they train on your prompts: No [https://docs.orq.ai/docs/administer/data-compliance] - Where it runs, and what you can pin: EU only for the hosted platform — there is no US region, and EU residency is included on every plan. Customer AWS or Azure VPC and on-premises or air-gapped installs are offered. [https://docs.orq.ai/docs/enterprise/eu-regions-faq] - Where safety filters run: Either, depending on deployment [https://docs.orq.ai/docs/enterprise/deployment-options] - SOC 2 audited: Yes [https://orq.ai/security] - Will sign a HIPAA agreement: Yes [https://docs.orq.ai/docs/enterprise/sovereign-ai] - GDPR commitments: Yes [https://orq.ai/security] - Can keep data in the EU: Yes [https://orq.ai/security] - Does not retain your data: Yes [https://orq.ai/security] - Strips personal data: Yes [https://orq.ai/security] - Content guardrails: Yes [https://orq.ai/] - Runs fully disconnected: Yes [https://docs.orq.ai/docs/enterprise/sovereign-ai] - Blocks personal data in prompts: Can block the request - Blocks prompt injection: Can block the request - Blocks harmful content: Can block the request - Your own policy rules: Can block the request - Where guardrails run: On the vendor's servers - If the guardrail itself fails: Request proceeds ### Fit & integration - Work to try it: Set it up in a cloud console - Work to run it: Set it up in a cloud console - Steps on the quickstart: 7 - Can you self-host it today: Offered, but no command published - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Official provider package - Python framework integrations: LlamaIndex - Kubernetes install: Official Helm chart - Terraform support: Official Terraform provider - Reuses your cloud identity: Static provider credentials only - MCP support: Hosted MCP server - Needs your own provider key: Required - Gate before models work: You enable it first - Official client languages: Python,Node.js ### Not published by this vendor Orq.ai Router has not published a figure for: GitHub stars; Monthly cost per person; Enterprise plan from; Discount on cached input; Premium on cache writes; Anthropic messages endpoint; Batch jobs endpoint; Batch processing; Uptime it promises in writing; Similar-question caching; Separate keys per team or app; Delay it adds; Requests per second ceiling; How long they keep it; Who else touches the data; Upstream health tracking; Cross-region failover you control; Callable from Cloudflare Workers; Fits behind your API gateway. --- ## Portkey - Kind: Managed gateway - Who runs it: Managed or self-host - Page: https://gatewayscore.com/providers/portkey - Most recent check: 2026-08-29 - Evidence basis: used first-hand in production, plus documentation review Open-core AI gateway with a hosted control plane for observability, prompt management and governance. ### Overview - What kind of product: Managed gateway - Who runs it: Managed or self-host [https://portkey.ai/docs] - Licence: Open core [https://github.com/Portkey-AI/gateway] - Company: Portkey, Inc. (Palo Alto Networks) - Who you would be signing with: Acquired [https://www.paloaltonetworks.com/company/press/2026/palo-alto-networks-completes-acquisition-of-portkey-to-secure-ai-agents] - Last shipped an update: 2026-08-21 [https://portkey.ai/docs/changelog/enterprise] - GitHub stars: 12,848 [https://api.github.com/repos/Portkey-AI/gateway] ### Cost - Can use your own provider accounts: Yes [https://portkey.ai/docs] - Cost of using your own accounts: Requests use your own provider credentials via provider integrations; enterprise deployments can also bring their own KMS encryption keys. [https://portkey.ai/blog/gateway-2-0] - Free tier: Developer plan free: 10,000 recorded logs/month, 3-day log and 30-day metric retention, universal API, fallbacks, load balancing, retries, 3 prompt templates, simple caching, deterministic guardrails. [https://portkey.ai/pricing] - Cost to run it yourself: The gateway core is MIT-licensed and free to self-host (you pay only your own infrastructure and model providers); the Gateway 2.0 release moved circuit breakers, semantic cache, budget limits, model catalog, metadata governance and config management into open source. Self-hosting drops the hosted dashboards/control plane, and an On-Prem Enterprise Gateway (gRPC, SSO, SCIM, AWS KMS, RBAC, JWT, audit logs, multi-workspace) is sold separately. [https://portkey.ai/blog/gateway-2-0] - How the vendor makes money: Platform fee plus usage meters - How pricing works, briefly: Free OSS self-host + free Dev tier + flat monthly platform fee ($49/mo Pro per docs) + request overage + Enterprise quote. Public pricing page shows tiers without dollar amounts and describes the free allowance as '10k recorded logs' per month; docs comparison lists 'Pro $49/Month' with '10k requests/month'. These two framings contradict. - Minimum commitment: Not stated. - Charges that fire after you go over an allowance: Dev 10K requests/mo with 'No Overage Allowed' (docs) — but pricing page says exceeding the log limit only stops recording, not requests. Pro 100K then $9/mo per additional 100K up to 3M requests. Enterprise custom. Retention-overage price and export cost not stated. - Prompt cache offered: Both exact and semantic - Who captures the cache saving: Exact/'simple' caching on all tiers; semantic caching from the paid tier (Production/Pro). No cached-token pricing published. Semantic cache savings are paid-tier-only. - What you can split spend by: Custom metadata and filters on all tiers. Per key/user/team/tag/customer cost splits not explicitly stated. - How you get cost data out: 'Data Export to Data Lakes' on Enterprise only. CSV/API/webhook/S3 not stated; price not stated. - Who pays the model bill: Your keys only ### Catalog - Models available: 250 [https://portkey.ai/docs] - Works with standard OpenAI code: Yes [https://portkey.ai/docs] - OpenAI chat endpoint: Yes - Anthropic messages endpoint: Yes - Embeddings endpoint: Yes - Image generation endpoint: Yes - Audio endpoints: Yes - Needs the vendor’s own code library: No [https://docs.portkey.ai/docs/integrations/llms/openai] - You can export your request history: Yes [https://docs.portkey.ai/docs/product/observability/logs-export] - Settings can live in version control: Yes [https://docs.portkey.ai/docs/api-reference/config-object] ### Routing & reliability - Uptime it promises in writing: 99.9% [https://portkey.ai/docs/product/product-feature-comparison] - Automatic failover: Yes [https://portkey.ai/pricing] - Load balancing: Yes [https://portkey.ai/pricing] - Rule-based routing: Yes [https://portkey.ai/blog/gateway-2-0] - Response caching: Yes [https://portkey.ai/pricing] - Similar-question caching: Yes [https://portkey.ai/pricing] - Where you set the timeout: Per request - Where you set retries: In config - Where you set fallbacks: In config - Shape of the fallback chain: Ordered list - Where you set load balancing: In config ### Operations - Usage dashboards and logs: Yes [https://portkey.ai/pricing] - Spending limits: Yes [https://docs.portkey.ai/docs/product/enterprise-offering] - Rate limits: Yes [https://docs.portkey.ai/docs/product/enterprise-offering] - Prompt versioning: Yes [https://portkey.ai/pricing] - Quality testing: Yes [https://portkey.ai/pricing] - MCP support: Yes [https://portkey.ai/blog/gateway-2-0] - What gets logged: Your choice - You can turn logging off: Yes - Traces you can take elsewhere: OpenTelemetry - Where telemetry can go: OpenTelemetry,Datadog,Grafana,LangSmith - Can record user feedback: Yes - Scores live traffic: Partly ### Performance - What the request path runs on: Interpreted proxy - You can run the request path yourself: Yes - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Depends how you deploy it [https://portkey.ai/docs/enterprise/security] - What they keep if you change nothing: Yes — prompts and replies [https://docs.portkey.ai/docs/introduction/what-is-portkey] - How long they keep it: 90 [https://portkey.ai/docs/enterprise/security] - Where it runs, and what you can pin: Region pinning with data stored in your selected region and requests served in-region across a network of about 310 data centres, plus customer-VPC and fully disconnected air-gapped deployment. [https://portkey.ai/docs/enterprise/security] - Where safety filters run: Either, depending on deployment [https://portkey.ai/docs/enterprise/security] - SOC 2 audited: Yes [https://portkey.ai/docs] - Will sign a HIPAA agreement: Yes [https://portkey.ai/docs] - GDPR commitments: Yes [https://portkey.ai/docs] - Does not retain your data: Yes [https://portkey.ai/docs] - Strips personal data: Yes [https://docs.portkey.ai/docs/product/enterprise-offering] - Content guardrails: Yes [https://portkey.ai/pricing] - Blocks personal data in prompts: Can block the request - Blocks prompt injection: Can block the request - Blocks harmful content: Can block the request - Your own policy rules: Can block the request - Where guardrails run: Either, your choice - Third-party guardrail vendors: Acuvity,Aporia,AWS Bedrock Guardrails,Azure Content Safety,Javelin,Lasso,Mistral,Pangea,Palo Alto Prisma AIRS,Patronus AI,Pillar Security,Qualifire ### Fit & integration - Work to try it: Install a package - Work to run it: Install a package - Steps on the quickstart: 3 - Can you self-host it today: Install command published - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Official provider package - Python framework integrations: LangChain,LlamaIndex - Callable from Cloudflare Workers: Their gateway runs on Workers, not yours - Kubernetes install: Official Helm chart - Terraform support: Official Terraform provider - MCP support: MCP gateway or registry - Needs your own provider key: Required - Official client languages: Python,JavaScript ### Not published by this vendor Portkey has not published a figure for: Markup on model prices; Fee to add funds; Monthly cost per person; Enterprise plan from; Discount on cached input; Premium on cache writes; Model providers reachable; OpenAI Responses endpoint; Batch jobs endpoint; Embeddings; Image generation; Speech and audio; Video generation; Batch processing; Separate keys per team or app; Delay it adds; Requests per second ceiling; Could they train on your prompts; Who else touches the data; Can keep data in the EU; Runs fully disconnected; Upstream health tracking; Cross-region failover you control; If the guardrail itself fails; Reuses your cloud identity; Fits behind your API gateway; Gate before models work. --- ## Requesty - Kind: Managed marketplace - Who runs it: Managed only - Page: https://gatewayscore.com/providers/requesty - Most recent check: 2026-08-29 - Evidence basis: used first-hand in production, plus documentation review Hosted router with a flat 5% fee on inference, EU data residency and enterprise governance controls. ### Overview - What kind of product: Managed marketplace - Who runs it: Managed only [https://www.truefoundry.com/blog/requesty-vs-openrouter] - Company: Requesty - Who you would be signing with: Independent company [https://www.requesty.ai/blog/requesty-raises-3m] - Last shipped an update: 2026-08-26 [https://docs.requesty.ai/changelog] ### Cost - Markup on model prices: 5% [https://www.requesty.ai/pricing] - Monthly cost per person: None [https://requesty.ai/] - Can use your own provider accounts: Yes [https://requesty.ai/] - Cost of using your own accounts: 0% markup on your own provider contracts when using BYOK; keeps negotiated provider and committed-use pricing. [https://requesty.ai/] - Free tier: Free plan: all free models, 200 requests/day, routing, caching, fallbacks, EU residency, no credit card. [https://www.requesty.ai/pricing] - How the vendor makes money: Percentage on tokens or top-ups - How pricing works, briefly: Flat 5% per-token markup, no seats, no minimum spend. Free tier caps free-model use at 200 req/day; Enterprise is custom. - Minimum commitment: None — 'no minimum spend'. - Charges that fire after you go over an allowance: No retention, request-volume or export overage stated on the pricing page. - Prompt cache offered: Passes provider caching through - Discount on cached input: 90% - Who captures the cache saving: Passthrough of provider caching with automatic breakpoint insertion; Requesty explicitly does not store responses server-side. Vendor says cache hits are billed 'at a fraction of the normal input token cost (up to 90% savings)'; cache writes 'have extra costs' for some providers, no number published. A per-request `false` toggle lets you avoid write premiums. - What you can split spend by: Per API key and per service account (each carries its own cap and usage). User/team/tag/customer splits listed as feature names without confirmation. - How you get cost data out: 'Data exports' is listed in the comparison table; a Management API can be queried programmatically. Specific formats (CSV/S3/warehouse) not stated. - Who pays the model bill: Your keys or their credits ### Catalog - Models available: 600 [https://www.requesty.ai/pricing] - Model providers reachable: 20 [https://www.requesty.ai/pricing] - Works with standard OpenAI code: Yes [https://docs.requesty.ai/] - OpenAI chat endpoint: Yes - Anthropic messages endpoint: Yes - OpenAI Responses endpoint: Yes - Embeddings endpoint: Yes - Image generation endpoint: Yes - Audio endpoints: Yes - Needs the vendor’s own code library: No [https://docs.requesty.ai/frameworks/openai] - You can export your request history: Yes [https://docs.requesty.ai/features/analytics-exports] - Settings can live in version control: No [https://docs.requesty.ai/features/latency-routing] - Embeddings: Yes [https://docs.requesty.ai/] - Image generation: Yes [https://docs.requesty.ai/] - Speech and audio: Yes [https://docs.requesty.ai/] ### Routing & reliability - Uptime it promises in writing: 99.99% [https://www.requesty.ai/enterprise] - Automatic failover: Yes [https://requesty.ai/] - Load balancing: Yes [https://docs.requesty.ai/] - Rule-based routing: Yes [https://www.requesty.ai/pricing] - Response caching: Yes [https://docs.requesty.ai/] - Similar-question caching: Yes [https://requesty.ai/] - Where you set retries: Dashboard only - Where you set fallbacks: Dashboard only - Shape of the fallback chain: Ordered list - Where you set load balancing: Dashboard only ### Operations - Usage dashboards and logs: Yes [https://requesty.ai/] - Spending limits: Yes [https://requesty.ai/] - Rate limits: Yes [https://requesty.ai/] - Prompt versioning: Yes [https://requesty.ai/] - MCP support: Yes [https://www.requesty.ai/pricing] - What gets logged: Full prompts and responses - Traces you can take elsewhere: OpenTelemetry - Can record user feedback: Partly - Scores live traffic: No ### Performance - Delay it adds: 16 ms [https://www.requesty.ai/blog/agentic-routing-benchmarked] - What the request path runs on: Undisclosed vendor service - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Yes [https://www.requesty.ai/security] - What they keep if you change nothing: Yes — prompts and replies [https://www.requesty.ai/security] - How long they keep it: 30 [https://www.requesty.ai/security] - Could they train on your prompts: No [https://www.requesty.ai/privacy] - Where it runs, and what you can pin: EU only — all infrastructure in Frankfurt, with an EU endpoint. No US region and no customer-VPC option. [https://www.requesty.ai/security] - Where safety filters run: In the vendor’s cloud [https://www.requesty.ai/security] - Who else touches the data: https://www.requesty.ai/privacy/subprocessors [https://www.requesty.ai/privacy/subprocessors] - SOC 2 audited: No [https://requesty.ai/] - GDPR commitments: Yes [https://requesty.ai/] - Can keep data in the EU: Yes [https://requesty.ai/] - Does not retain your data: Yes [https://requesty.ai/] - Strips personal data: Yes [https://requesty.ai/] - Content guardrails: Yes [https://requesty.ai/] - Runs fully disconnected: No [https://www.truefoundry.com/blog/requesty-vs-openrouter] - Blocks personal data in prompts: Inspects but lets it through - Where guardrails run: On the vendor's servers ### Fit & integration - Work to try it: Change one base URL - Work to run it: Change one base URL - Steps on the quickstart: 4 - Can you self-host it today: No self-hosting - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Official provider package - Python framework integrations: LangChain,LlamaIndex - Callable from Cloudflare Workers: Documented from your Worker - Kubernetes install: Official Helm chart - MCP support: MCP gateway or registry - Needs your own provider key: Not needed - Gate before models work: No gate - Official client languages: Python,TypeScript ### Not published by this vendor Requesty has not published a figure for: Licence; GitHub stars; Fee to add funds; Enterprise plan from; Cost to run it yourself; Premium on cache writes; Batch jobs endpoint; Video generation; Batch processing; Separate keys per team or app; Quality testing; Requests per second ceiling; Will sign a HIPAA agreement; Where you set the timeout; Upstream health tracking; Cross-region failover you control; You can run the request path yourself; Blocks prompt injection; Blocks harmful content; Your own policy rules; If the guardrail itself fails; You can turn logging off; Terraform support; Reuses your cloud identity; Fits behind your API gateway. --- ## Together AI - Kind: Inference provider - Who runs it: Managed only - Page: https://gatewayscore.com/providers/together-ai - Most recent check: 2026-08-31 - Evidence basis: used first-hand in production, plus documentation review Inference provider running open-weight models on its own GPUs, with fine-tuning and dedicated endpoints. ### Overview - What kind of product: Inference provider - Who runs it: Managed only - Licence: Proprietary - Company: Together AI [https://www.together.ai/about-us] - Who you would be signing with: Independent company [https://www.reuters.com/legal/transactional/together-ai-raises-800-million-83-billion-valuation-2026-07-01/] - Last shipped an update: 2026-08-28 [https://docs.together.ai/docs/changelog] ### Cost - Cost to run it yourself: No self-host option; Together sets its own per-token prices rather than marking up another vendor's list price. Representative serverless rates include MiniMax M3 at $0.30 per 1M input, gpt-oss-120B at $0.15, DeepSeek V4 Flash at $0.14, and Kimi K3 at $3.00 input / $15.00 output. Discounted batch rates and provisioned throughput sold in PTUs are also published, plus embeddings from $0.02 per 1M tokens. [https://www.together.ai/pricing] - How the vendor makes money: Bundled — inference is the product - How pricing works, briefly: Pay-for-own-inference across serverless per-token, Batch API, Provisioned Throughput (PTUs), dedicated single-tenant endpoints, GPU clusters, sandbox and storage. Enterprise quote for some SKUs. No separate gateway/platform fee — routing layer is $0. - Minimum commitment: None for on-demand serverless. Fine-tuning has a $4.00 minimum charge per job. PTU savings estimates assume continuous 24/7 provisioning (~43,800 min/mo) but no commitment is required. - Charges that fire after you go over an allowance: None stated. Cost risks are structural — dedicated endpoints and PTUs bill for provisioned capacity whether or not you use it; storage is billed per GiB/month. - Prompt cache offered: Prefix cache - Premium on cache writes: None - Who captures the cache saving: Automatic prefix caching on select serverless models, best-effort, plus per-replica caching on dedicated endpoints. Prefix-based: only the longest matching prefix counts. Serverless cache is shared across the fleet and entries are evicted as traffic shifts — cache hits aren't guaranteed and there's no configurable retention window. Docs say 'a steep discount', pricing table publishes numbers: MiniMax M3 $0.06 vs $0.30 (80% off), Kimi K3 $0.30 vs $3.00 (90% off), DeepSeek V4 Flash $0.03 vs $0.14 (~79% off), GLM-5.3 $0.26 vs $1.40 (~81% off), Qwen3.8-2.4T-A95B $0.25 vs $2.00 (87.5% off). No cache-write charge. Only dedicated endpoints make the saving reliable. - What you can split spend by: Not stated. - How you get cost data out: Not stated. - Who pays the model bill: Their credits only ### Catalog - Models available: 200 [https://www.together.ai/models] - Model providers reachable: 40 [https://www.together.ai/models] - Works with standard OpenAI code: Yes [https://docs.together.ai/docs/introduction] - OpenAI chat endpoint: Yes - Embeddings endpoint: Yes - Image generation endpoint: Yes - Audio endpoints: Yes - Batch jobs endpoint: Yes - Needs the vendor’s own code library: No [https://docs.together.ai/docs/inference/openai-compatibility] - Settings can live in version control: No [https://docs.together.ai/docs/billing-credits] - Embeddings: Yes [https://www.together.ai/pricing] - Image generation: Yes [https://docs.together.ai/docs/serverless-models] - Speech and audio: Yes [https://docs.together.ai/docs/serverless-models] - Video generation: Yes [https://docs.together.ai/docs/serverless-models] - Batch processing: Yes [https://www.together.ai/pricing] ### Operations - What gets logged: Metadata only - You can turn logging off: Yes - Traces you can take elsewhere: OpenTelemetry - Can record user feedback: No - Scores live traffic: No ### Performance - What the request path runs on: Undisclosed vendor service - You can run the request path yourself: No - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Yes [https://docs.together.ai/docs/privacy-and-security] - What they keep if you change nothing: Nothing stored by default [https://docs.together.ai/docs/privacy-and-security] - How long they keep it: 0 [https://docs.together.ai/docs/privacy-and-security] - Could they train on your prompts: Only if you opt in [https://docs.together.ai/docs/privacy-and-security] - Where it runs, and what you can pin: US by default. EU data centres exist for inference and storage inside the EU but only on Scale and Enterprise plans, serverless endpoints offer no region selection, and private networking and VPC-based deployments are supported. [https://support.together.ai/articles/8079447813-eu-data-centers-and-dedicated-model-deployment] - Where safety filters run: No guardrails offered [https://docs.together.ai/docs/privacy-and-security] - SOC 2 audited: Yes [https://www.together.ai/blog/soc-2-compliance] - Will sign a HIPAA agreement: Yes [https://www.together.ai/blog/soc-2-compliance] - Content guardrails: Yes [https://docs.together.ai/docs/serverless-models] - Blocks harmful content: Can block the request - Where guardrails run: On the vendor's servers ### Fit & integration - Work to try it: Install a package - Work to run it: Install a package - Steps on the quickstart: 3 - Can you self-host it today: No self-hosting - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Official provider package - Python framework integrations: LangChain,LlamaIndex - Kubernetes install: Only generic cluster docs - MCP support: Mentioned in docs only - Needs your own provider key: Not needed - Official client languages: Python,TypeScript ### Not published by this vendor Together AI has not published a figure for: GitHub stars; Markup on model prices; Fee to add funds; Monthly cost per person; Can use your own provider accounts; Cost of using your own accounts; Free tier; Enterprise plan from; Discount on cached input; Anthropic messages endpoint; OpenAI Responses endpoint; You can export your request history; Uptime it promises in writing; Automatic failover; Load balancing; Rule-based routing; Response caching; Similar-question caching; Usage dashboards and logs; Spending limits; Rate limits; Separate keys per team or app; Prompt versioning; Quality testing; MCP support; Delay it adds; Requests per second ceiling; Who else touches the data; GDPR commitments; Can keep data in the EU; Does not retain your data; Strips personal data; Runs fully disconnected; Where you set the timeout; Where you set retries; Where you set fallbacks; Shape of the fallback chain; Upstream health tracking; Cross-region failover you control; Where you set load balancing; Blocks personal data in prompts; Blocks prompt injection; Your own policy rules; If the guardrail itself fails; Callable from Cloudflare Workers; Terraform support; Reuses your cloud identity; Fits behind your API gateway; Gate before models work. --- ## TrueFoundry AI Gateway - Kind: Cloud platform - Who runs it: Managed or self-host - Page: https://gatewayscore.com/providers/truefoundry-ai-gateway - Most recent check: 2026-08-29 - Evidence basis: used first-hand in production, plus documentation review Closed-source enterprise AI gateway sold on request tiers, deployable as SaaS or inside the customer's own cloud. ### Overview - What kind of product: Cloud platform - Who runs it: Managed or self-host - Licence: Proprietary [https://llmtools.cc/tool/truefoundry/] - Company: TrueFoundry - Who you would be signing with: Independent company [https://www.truefoundry.com/blog/announcing-our-19m-series-a-scaling-ai-deployment-with-autonomous-agents-on-autopilot] - Last shipped an update: 2026-08-21 [https://www.truefoundry.com/docs/changelog] ### Cost - Can use your own provider accounts: Yes [https://www.truefoundry.com/ai-gateway] - Cost of using your own accounts: You connect your own provider accounts and keys; TrueFoundry bills by request tier, not tokens. - Free tier: Developer plan $0/month: 50,000 requests/month, 3 users, 50,000 MCP tool calls, up to 10 saved prompts. [https://www.truefoundry.com/pricing] - Cost to run it yourself: No open-source edition; the data plane is proprietary. Vendor estimates roughly $600-$1,000/month of your own hosting cost to run the gateway (or control plus gateway) plane in your cloud, on top of the plan fee; fully managed SaaS carries no hosting cost. VPC, on-prem, air-gapped and multi-cloud installs are enterprise deployments, and a third-party review notes Helm-based ops effort is required. [https://www.truefoundry.com/pricing] - How the vendor makes money: Flat monthly platform fee - How pricing works, briefly: Free developer tier + flat monthly platform fee + per-unit usage overage + enterprise quote + optional self-host infra cost. Developer $0/mo, Pro $499/mo, Pro Plus $2,999/mo, Enterprise custom. TrueFoundry does not resell inference. - Minimum commitment: None stated. Enterprise implied custom. - Charges that fire after you go over an allowance: Request overage priced in blocks: 2M requests and 5 API keys for additional $499/month; Pro Plus 'contact sales'. Self-hosting adds ~$600–$1,000/month of infrastructure cost when you self-host the Gateway or both the Control and Gateway planes. Log/trace retention overage and export cost not stated. - Prompt cache offered: Both exact and semantic - Who captures the cache saving: Exact ('Simple caching') and semantic caching both listed as features; provider passthrough not stated. No cached-token pricing published. TrueFoundry charges per request/platform, not per token, so caching cuts your provider bill and you keep the saving — but cached requests presumably still consume the tier's request allowance, and the feature list does not mark which tier includes semantic caching. - What you can split spend by: 'Cost per team/user/model/application' plus custom metadata and metadata filtering. Per-key, per-tag and per-customer not stated. - How you get cost data out: 'Export to custom storage buckets', 'Data Lake Export', 'Connect multiple storage bucket' and 'Export to other monitoring platforms'. CSV, cost API and webhook export not stated; export pricing not stated. - Who pays the model bill: Your keys only ### Catalog - Models available: 1,600 [https://www.truefoundry.com/ai-gateway] - Works with standard OpenAI code: Yes [https://www.truefoundry.com/ai-gateway] - OpenAI chat endpoint: Yes - Anthropic messages endpoint: Yes - Embeddings endpoint: Yes - Image generation endpoint: Yes - Audio endpoints: Yes - Batch jobs endpoint: Yes - Needs the vendor’s own code library: No [https://www.truefoundry.com/docs/ai-gateway/quick-start] - You can export your request history: Yes [https://docs.truefoundry.com/ai-gateway/export-logstraces] - Settings can live in version control: Yes [https://www.truefoundry.com/docs/ai-gateway/virtual-model] - Embeddings: Yes [https://www.truefoundry.com/ai-gateway] ### Routing & reliability - Uptime it promises in writing: 99.9% [https://www.truefoundry.com/service-level-agreement] - Automatic failover: Yes [https://www.truefoundry.com/ai-gateway] - Load balancing: Yes [https://www.truefoundry.com/ai-gateway] - Rule-based routing: Yes [https://www.truefoundry.com/ai-gateway] - Response caching: Yes [https://www.truefoundry.com/ai-gateway] - Similar-question caching: Yes [https://www.truefoundry.com/ai-gateway] - Where you set retries: In config - Where you set fallbacks: In config - Shape of the fallback chain: Ordered list - Upstream health tracking: In config - Cross-region failover you control: In config - Where you set load balancing: In config ### Operations - Usage dashboards and logs: Yes [https://www.truefoundry.com/ai-gateway] - Spending limits: Yes [https://www.truefoundry.com/ai-gateway] - Rate limits: Yes [https://www.truefoundry.com/ai-gateway] - Prompt versioning: Yes [https://www.truefoundry.com/ai-gateway] - MCP support: Yes [https://www.truefoundry.com/ai-gateway] - What gets logged: Your choice - You can turn logging off: Yes - Traces you can take elsewhere: OpenTelemetry - Where telemetry can go: OpenTelemetry,Datadog,Prometheus,Grafana,S3,Webhook - Can record user feedback: No - Scores live traffic: No ### Performance - Delay it adds: 3 ms [https://www.truefoundry.com/ai-gateway] - What the request path runs on: Interpreted proxy - You can run the request path yourself: Yes - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Depends how you deploy it [https://www.truefoundry.com/docs/ai-gateway/modes-of-deployment] - What they keep if you change nothing: Yes — prompts and replies [https://www.truefoundry.com/docs/ai-gateway/request-logging] - How long they keep it: 90 [https://www.truefoundry.com/docs/platform/security-and-compliance] - Where it runs, and what you can pin: You choose the region where the gateway is deployed and where logs and traces are stored, and you can bring your own S3, Azure Blob or GCS bucket. Gateway planes run active-active across multiple regions. [https://www.truefoundry.com/docs/platform/security-and-compliance] - Where safety filters run: Either, depending on deployment [https://www.truefoundry.com/docs/platform/security-and-compliance] - Who else touches the data: https://www.truefoundry.com/docs/platform/security-and-compliance [https://www.truefoundry.com/docs/platform/security-and-compliance] - SOC 2 audited: Yes [https://www.truefoundry.com/ai-gateway] - GDPR commitments: Yes [https://www.truefoundry.com/ai-gateway] - Strips personal data: Yes [https://www.truefoundry.com/ai-gateway] - Content guardrails: Yes [https://www.truefoundry.com/ai-gateway] - Runs fully disconnected: Yes [https://www.truefoundry.com/ai-gateway] - Blocks personal data in prompts: Can block the request - Blocks prompt injection: Can block the request - Blocks harmful content: Can block the request - Your own policy rules: Can block the request - Where guardrails run: Either, your choice - Third-party guardrail vendors: Azure AI Language,Azure Content Safety,OpenAI Moderation,Microsoft Presidio ### Fit & integration - Work to try it: Set it up in a cloud console - Work to run it: Deploy it on your infrastructure - Steps on the quickstart: 4 - Can you self-host it today: Offered, but no command published - Works with the OpenAI SDK: Yes - Python framework integrations: LangChain,LangGraph,LlamaIndex - Kubernetes install: Official Helm chart - Terraform support: Terraform code in the repo only - Reuses your cloud identity: Reuses your cloud identity - MCP support: MCP gateway or registry - Needs your own provider key: Required - Official client languages: Python,Node.js,Go,Rust,Java ### Not published by this vendor TrueFoundry AI Gateway has not published a figure for: GitHub stars; Markup on model prices; Fee to add funds; Monthly cost per person; Enterprise plan from; Discount on cached input; Premium on cache writes; Model providers reachable; OpenAI Responses endpoint; Image generation; Speech and audio; Video generation; Batch processing; Separate keys per team or app; Quality testing; Requests per second ceiling; Could they train on your prompts; Will sign a HIPAA agreement; Can keep data in the EU; Does not retain your data; Where you set the timeout; If the guardrail itself fails; Vercel AI SDK support; Callable from Cloudflare Workers; Fits behind your API gateway; Gate before models work. --- ## Vercel AI Gateway - Kind: Managed gateway - Who runs it: Managed only - Page: https://gatewayscore.com/providers/vercel-ai-gateway - Most recent check: 2026-08-29 - Evidence basis: used first-hand in production, plus documentation review Vercel-operated gateway that routes AI SDK and OpenAI-format requests to many providers with zero token markup. ### Overview - What kind of product: Managed gateway - Who runs it: Managed only [https://api7.ai/openrouter-vs-vercel-ai-gateway] - Licence: Proprietary [https://mcp.directory/blog/vercel-ai-gateway-vs-portkey-vs-openrouter-vs-litellm-2026] - Company: Vercel - Who you would be signing with: Independent company [https://vercel.com/blog/series-f] - Last shipped an update: 2026-08-28 [https://vercel.com/changelog/hy4-preview-now-available-on-ai-gateway] ### Cost - Markup on model prices: None [https://vercel.com/docs/ai-gateway/pricing] - Fee to add funds: None [https://vercel.com/docs/ai-gateway/pricing] - Monthly cost per person: $20/user/mo [https://vercel.com/pricing] - Can use your own provider accounts: Yes [https://vercel.com/docs/ai-gateway/byok] - Cost of using your own accounts: No markup or fee on BYOK, but BYOK requires purchased AI Gateway Credits (paid tier); failed BYOK requests fall back to Vercel system credentials and are billed against credits, and BYOK spend is not covered by budgets. [https://vercel.com/docs/ai-gateway/byok] - Free tier: Every team gets free AI Gateway credits on a subset of models with lower per-model rate limits; the amount is not published. Pro plans include $20 of credit across resources. [https://vercel.com/docs/ai-gateway/pricing] - How the vendor makes money: Percentage on tokens or top-ups - How pricing works, briefly: Credit-based pay-as-you-go with 0% markup and no platform fee on tokens. Monetizes gateway features à la carte: provider allowlists, ZDR, custom reporting writes/queries, Trace Drains. Enterprise can pay by invoice with no processing fees. Pricing page last updated 2026-08-23. - Minimum commitment: None. Credits purchasable at any time with no obligation to renew; custom volume discounts on Enterprise. - Charges that fire after you go over an allowance: Trace Drains meter trace events delivered and trace data transferred; Pro plans include no allowance for either. Team-wide provider allowlist: $0.10 per 1,000 successful requests. Team-wide zero data retention: $0.10 per 1,000 requests. Custom Reporting: $0.075 per 1,000 tag/user/quota-entity writes and $5 per 1,000 reporting-endpoint queries. All billed outside credits. - Prompt cache offered: No gateway-owned cache - Who captures the cache saving: Pricing page does not state whether the gateway caches or passes through provider caching, and no cached-token pricing is published. Because Vercel charges provider list price with 0% markup, any provider cache discount would reach the customer unchanged. - What you can split spend by: By tags, user IDs and quota entity IDs via Custom Reporting (queried through the reporting endpoint at $5/1k queries). Per team/key/customer not explicitly stated. - How you get cost data out: Reporting API plus Trace Drains (Pro/Enterprise, metered). CSV/S3/warehouse not stated. - Who pays the model bill: Your keys or their credits ### Catalog - Models available: 360 [https://ai-gateway.vercel.sh/v1/models] - Works with standard OpenAI code: Yes [https://vercel.com/docs/ai-gateway/byok] - OpenAI chat endpoint: Yes - Anthropic messages endpoint: Yes - OpenAI Responses endpoint: Yes - Embeddings endpoint: Yes - Image generation endpoint: Yes - Audio endpoints: Yes - Needs the vendor’s own code library: No [https://vercel.com/docs/ai-gateway/sdks-and-apis/openai-chat-completions] - You can export your request history: Yes [https://vercel.com/docs/ai-gateway/observability-and-spend/logs] - Embeddings: Yes [https://vercel.com/pricing] - Image generation: Yes [https://vercel.com/pricing] - Speech and audio: Yes [https://zackproser.com/blog/openrouter-vs-vercel-ai-gateway] - Video generation: Yes [https://zackproser.com/blog/openrouter-vs-vercel-ai-gateway] ### Routing & reliability - Uptime it promises in writing: 99.99% [https://vercel.com/legal/sla] - Automatic failover: Yes [https://vercel.com/docs/ai-gateway] - Load balancing: Yes [https://vercel.com/pricing] - Rule-based routing: Yes [https://vercel.com/docs/ai-gateway/provider-options] - Response caching: Yes [https://vercel.com/docs/ai-gateway/provider-options] - Similar-question caching: No [https://api7.ai/openrouter-vs-vercel-ai-gateway] - Where you set the timeout: Per request - Where you set fallbacks: Per request - Shape of the fallback chain: Ordered list - Upstream health tracking: Fixed, cannot change - Where you set load balancing: Per request ### Operations - Usage dashboards and logs: Yes [https://vercel.com/pricing] - Spending limits: Yes [https://vercel.com/docs/ai-gateway/byok] - Rate limits: Yes [https://vercel.com/docs/ai-gateway/pricing] - Separate keys per team or app: Yes [https://vercel.com/docs/ai-gateway/byok] - What gets logged: Your choice - You can turn logging off: Yes - Where telemetry can go: CSV export - Can record user feedback: No - Scores live traffic: No ### Performance - What the request path runs on: Undisclosed vendor service - You can run the request path yourself: No - Streaming responses: Yes ### Security & compliance - Does your prompt reach their servers: Yes [https://vercel.com/legal/dpa] - What they keep if you change nothing: Yes — prompts and replies [https://vercel.com/docs/ai-gateway/observability-and-spend/logs] - How long they keep it: 30 [https://vercel.com/docs/ai-gateway/observability-and-spend/logs] - Could they train on your prompts: No [https://vercel.com/docs/ai-gateway/security-and-compliance/disallow-prompt-training] - Where it runs, and what you can pin: US primary, and the data-processing agreement permits transfer anywhere in the world. No customer-VPC option and no documented gateway region pinning. [https://vercel.com/legal/dpa] - Where safety filters run: No guardrails offered [https://vercel.com/docs/ai-gateway/security-and-compliance] - Who else touches the data: https://security.vercel.com/ [https://security.vercel.com/] - SOC 2 audited: Yes [https://vercel.com/security] - Will sign a HIPAA agreement: Yes [https://vercel.com/security] - GDPR commitments: Yes [https://vercel.com/security] - Does not retain your data: Yes [https://vercel.com/docs/ai-gateway/pricing] - Content guardrails: No [https://api7.ai/openrouter-vs-vercel-ai-gateway] - Runs fully disconnected: No [https://api7.ai/openrouter-vs-vercel-ai-gateway] - Where guardrails run: On the vendor's servers ### Fit & integration - Work to try it: Install a package - Work to run it: Install a package - Steps on the quickstart: 5 - Can you self-host it today: No self-hosting - Works with the OpenAI SDK: Yes - Vercel AI SDK support: Official provider package - Python framework integrations: LangChain,LlamaIndex - Reuses your cloud identity: Its own OIDC token - MCP support: MCP client tooling - Needs your own provider key: Not needed - Gate before models work: Free tier sees fewer models - Official client languages: TypeScript,Python ### Not published by this vendor Vercel AI Gateway has not published a figure for: GitHub stars; Enterprise plan from; Cost to run it yourself; Discount on cached input; Premium on cache writes; Model providers reachable; Batch jobs endpoint; Settings can live in version control; Batch processing; Prompt versioning; Quality testing; MCP support; Delay it adds; Requests per second ceiling; Can keep data in the EU; Strips personal data; Where you set retries; Cross-region failover you control; Blocks personal data in prompts; Blocks prompt injection; Blocks harmful content; Your own policy rules; If the guardrail itself fails; Traces you can take elsewhere; Callable from Cloudflare Workers; Kubernetes install; Terraform support; Fits behind your API gateway. --- # Head-to-head comparisons ## OpenRouter vs Vercel AI Gateway Page: https://gatewayscore.com/compare/openrouter-vs-vercel-ai-gateway The question that decides it: Do you need the long tail of the model catalog, or do you already pay Vercel for seats? Verdict: Already on Vercel with the AI SDK: Vercel AI Gateway, because your seats are a sunk cost and there is no fee on top of tokens. Anywhere else, or if you need the long tail of open-weight models across many upstream providers: OpenRouter, and accept the 5.5% credit fee. These two look like direct competitors and price almost nowhere near each other. Both publish zero token markup, so neither adds a percentage to the per-token rate you would pay the model vendor. The difference is what surrounds that. OpenRouter takes 5.5% when you buy credits, with a minimum of $0.80 — which means a $5 top-up costs you $5.80, an effective 16% on small amounts. Vercel takes nothing on credits and nothing on tokens, but Pro seats are $20 per user per month. That makes the arithmetic almost entirely about your existing bill. If you are already paying Vercel for three Pro seats to host your app, the gateway is genuinely free at the margin and OpenRouter cannot beat it on cost. If you are not on Vercel, those seats are a new $60 a month line item to access a gateway, and OpenRouter is cheaper until your credit purchases are large enough that 5.5% exceeds the seat cost. Catalog is the other axis, and it is not close. OpenRouter publishes 500+ models across 80+ upstream providers; Vercel publishes 360 models and does not publish a provider count. More importantly, OpenRouter typically offers several upstream deployments of the same open-weight model, which is what makes its price sorting and automatic failover useful. If you are running Llama or Qwen and want the cheapest healthy endpoint at any moment, that pool of providers is the product. Vercel gives you provider ordering and fallbacks too, but from a shallower pool. Neither offers a semantic cache or native content guardrails, so if you need either, both of these are the wrong shelf — look at Portkey or Kong instead. Both are managed-only, so neither is an option if you need to self-host. --- ## OpenRouter vs Requesty Page: https://gatewayscore.com/compare/openrouter-vs-requesty The question that decides it: Do you bring your own provider keys, and do you need a completed SOC 2 attestation today? Verdict: Bringing your own provider keys: Requesty, because its markup drops to 0% and you keep the routing, caching and EU residency for free. Buying credits through the gateway, or needing an attested SOC 2 report right now: OpenRouter. The fee shapes here are different in a way that changes who should pick what. OpenRouter charges 0% markup on tokens and 5.5% when you buy credits. Requesty charges 5% on every token routed but has no credit top-up fee, no seat fee and no minimum spend. If all your traffic flows through gateway-purchased credits, OpenRouter is materially cheaper — 5.5% once on the way in beats 5% on everything forever. Bring your own provider keys and it inverts. Requesty charges 0% markup on BYOK traffic, which makes it free at the margin while still giving you routing, fallbacks, semantic caching, PII redaction and an EU-only endpoint. OpenRouter supports BYOK too, but its own documentation describes two different meters for when BYOK charges apply, which makes it harder to plan around. Compliance is the sharper split. OpenRouter is SOC 2 attested. Requesty states SOC 2 Type II is in progress, not complete. If you are selling into an enterprise that will ask for the report during procurement, that difference decides the question regardless of price. Requesty also gates its governance features — RBAC, PII detection, guardrails, SSO — to its Enterprise tier, so the controls that make it attractive to a regulated buyer are not on the plan a small team would start on. Requesty does have real advantages beyond BYOK pricing: a semantic cache, PII redaction and guardrails that OpenRouter does not offer at all, plus an EU-only routing endpoint with in-region logging, caching and analytics. If EU residency is a hard requirement and you can live with the SOC 2 status, it is the more capable product on paper. --- ## LiteLLM vs Portkey Page: https://gatewayscore.com/compare/litellm-vs-portkey The question that decides it: Do you want to build and run the governance layer yourself, or pay for one that already exists? Verdict: Want maximum provider breadth for free and have platform engineers to run it: LiteLLM. Want RBAC, SSO, audit logs, guardrails and prompt management working this week without building them: Portkey, at $49/month for the Production tier. Both have an open-source core and both can be self-hosted, so the licence is not the dividing line people expect. LiteLLM is MIT with the whole proxy free forever, including virtual keys, team budgets, RPM/TPM limits and guardrails. Portkey is open core: an MIT-licensed gateway covering routing, fallbacks, conditional routing, semantic cache, budgets and guardrails, with a hosted control plane on top. Coverage favours LiteLLM decisively. 140+ providers against Portkey's 40 to 48. Neither publishes a model total worth quoting - LiteLLM states none at all, and Portkey's own pages range from 250 to 2,300 - so providers are the honest measure, and on that measure nothing else in this catalog is close to LiteLLM. It also has by far the largest community here at 57,500 GitHub stars, which matters when you are debugging a proxy at 2am and need someone to have hit your problem first. The counterweight is scale. Third-party operators report the Python proxy hitting a GIL bottleneck past roughly 300 requests per second per instance, with P99 latency blowing out to seconds and memory growth ending in OOM. An independent 2026 comparison recommends staying under 250-300 RPS per instance. That is not fatal — you shard instances — but it is real operational work, and it means LiteLLM's free licence is not the same as a free deployment. Portkey's pitch is that the governance layer is already built. RBAC, SSO/SCIM, audit logs, budget and rate limits, PII-redaction guardrails, plus a compliance posture spanning SOC 2, HIPAA, GDPR and ISO 27001 with custom BAAs available. For LiteLLM, SSO, RBAC, audit logs and SCIM all sit behind a commercial licence whose price is not published. If you need those controls, the honest comparison is not "free versus $49" — it is "$49 versus a quote". The catch on Portkey is its meter. Pricing is on recorded logs rather than tokens, at $9 per additional 100,000 requests past the plan allowance. High-request, small-payload workloads — agent loops especially — inflate that number fast, and it is exactly the shape of traffic that has been growing. Model it against your actual request count, not your token count. --- ## LiteLLM vs Bifrost Page: https://gatewayscore.com/compare/litellm-vs-bifrost The question that decides it: Do you value provider breadth and a large operator community, or a Go runtime without a documented throughput ceiling? Verdict: Almost always LiteLLM, for coverage and community. Choose Bifrost if your traffic profile is high, steady RPS through a small set of providers and you would rather not engineer around a Python proxy — but do your own load test, because neither vendor's numbers are independent. This is the closest genuine like-for-like pair in the catalog. Both are free to self-host, both are permissively licensed (MIT for LiteLLM, Apache-2.0 for Bifrost), both support air-gapped deployment, and both ship virtual keys, budgets, semantic caching, guardrails, MCP and config-as-code at no cost. Neither has any compliance certification of its own, because neither is a service — you deploy it, so the attestations are yours to earn. Coverage is where they separate. LiteLLM lists 140 providers; Bifrost states 20+. Neither publishes a model total - both have been quoted with one, and neither vendor's own pages support it - so the provider gap is the number to judge on, and for most teams it decides the question. The community gap points the same way: 57,500 GitHub stars against 7,600. When you need a Terraform module, a Helm values example, or someone who has already debugged your exact upstream, LiteLLM has an order of magnitude more people to have written it down. Bifrost's argument is the runtime. It is Go rather than Python, which sidesteps the GIL contention that third-party operators report in LiteLLM past roughly 300 RPS per instance. Bifrost claims 0.99 ms of added latency at 5,000 RPS; LiteLLM claims 0.66 ms at 2,800 RPS. Both of those are vendor-run against mock upstreams, and no independent benchmark exists for either. The latency figures in particular are not comparable in any meaningful way — they were produced under different conditions by parties with an interest in the result. What is more credible than either benchmark is the structural point: a Go proxy does not have a GIL, and LiteLLM operators have publicly documented hitting a Python concurrency wall. If your workload is sustained high RPS and you would rather not run a fleet of sharded Python instances, that is a reasonable engineering preference. Just note the price: Bifrost gates guardrails, cluster mode, adaptive load balancing, SSO, RBAC and audit logs to an enterprise tier at unpublished prices, and its own provider and model counts are inconsistent across its pages. --- ## Portkey vs Helicone Page: https://gatewayscore.com/compare/portkey-vs-helicone The question that decides it: Are you starting something new, or do you already run Helicone and are deciding whether to stay? Verdict: For a new build, Portkey. Helicone was acquired by Mintlify in March 2026 and is reported to be in maintenance mode with no new features planned, which is disqualifying for a component on your critical request path. On paper these overlap heavily: both are open-core-ish gateways with observability, both self-hostable, both claiming SOC 2 and HIPAA, both with zero token markup. Helicone is Apache-2.0 with 6,109 GitHub stars and genuinely strong request analytics — sessions, per-user breakdowns, custom properties, an HQL query language and alerting. If the question were purely "which has better observability", Helicone would have a real case. But the question is not that. Helicone was acquired by Mintlify on 3 March 2026, and is reported to be in maintenance mode with no new features planned. Its standalone Rust gateway repository has not had a functional commit since July 2025. Putting a component that is not being developed on the path every model request takes is a risk that no feature advantage offsets — when a provider changes an API, a maintenance-mode proxy is the thing that does not get updated. Portkey is the opposite posture: an MIT-licensed gateway core with a full governance stack on top — RBAC, SSO/SCIM, audit logs, budget and rate limits, PII-redaction guardrails — plus an MCP registry with OAuth 2.1. It has 12,848 stars, roughly double Helicone's, and is actively developed. Its compliance claims cover SOC 2, HIPAA, GDPR and ISO 27001 with custom BAAs. There is one honest caveat in Portkey's favour that is not really about Helicone: third-party reviews note Portkey has a slower development cadence and a smaller self-hosting community than LiteLLM. If active development is your deciding criterion, LiteLLM is the more active project than either. But between these two specifically, Portkey is being built and Helicone is not. If you already run Helicone and it works, there is no emergency. Apache-2.0 means you can self-host it indefinitely and nobody can take it away. The advice is to stop building new dependencies on it and to have a migration target picked before you need one. --- ## Helicone vs Braintrust Gateway Page: https://gatewayscore.com/compare/helicone-vs-braintrust-gateway The question that decides it: Do you want to measure production traffic after the fact, or gate quality before you ship? Verdict: Neither, if the gateway is the point — both are platform features with real uncertainty attached. If you must pick: Braintrust when your team already runs evals there and wants CI quality gates; Helicone only if you self-host it and want request analytics. Both of these are observability products with a gateway attached, not gateways with observability attached, and that framing matters. You would pick either because you already want the platform. The two have different unresolved risks, and you should weigh both rather than treating one as safe. Helicone was acquired by Mintlify in March 2026 and is reported to be in maintenance mode with no new features planned; its standalone Rust gateway has not had a functional commit since July 2025. Braintrust's hosted Gateway is free during public preview with pricing to be announced before general availability — which means the cost of a component on your critical path is currently unknown. On capability they are genuinely different products. Braintrust is eval-centric: spans, scorers, datasets and CI quality gates, so you can block a deploy when output quality regresses. Helicone is production-analytics-centric: sessions, per-user breakdowns, custom properties, an HQL query language, alerting. One tells you whether a change is safe to ship; the other tells you what is happening to real users now. Teams often want both, and neither replaces the other. Two practical differentiators. Helicone is Apache-2.0 and genuinely self-hostable, which is a real hedge against the maintenance-mode risk — you can keep running it whatever Mintlify decides. Braintrust self-hosting is effectively Enterprise-only, so its preview-pricing risk has no equivalent escape hatch. Conversely, Braintrust supports unlimited users on all plans and offers time-limited temporary credentials for frontend and mobile clients, which is a genuinely useful feature neither of the others here has. Neither offers real-time guardrails. Braintrust evaluates after the fact by design; Helicone's guardrails and PII redaction are weaker than Portkey's or OpenRouter's. If you need to block a bad prompt or response inline, look elsewhere. --- ## Amazon Bedrock vs Google Vertex AI Page: https://gatewayscore.com/compare/amazon-bedrock-vs-google-vertex-ai The question that decides it: Which cloud already holds your data, and do you need FedRAMP High or explicit per-feature ZDR documentation? Verdict: Whichever cloud your data is already in — the egress and integration cost dominates any feature difference. Where that is genuinely open: Bedrock for FedRAMP High and the widest third-party model vendor list; Vertex AI for the clearest documented data-governance guarantees. Neither of these is a gateway in the sense the rest of this catalog means it. Both only route to models their own cloud hosts, so neither can fail over to the other, or to OpenAI directly. If you want cross-cloud failover you need a real gateway in front of one or both, and that changes the shape of the whole decision. Given that, the honest answer is usually "the one you are already on". Vertex AI's value drops sharply if your data is not already in Google Cloud, once you count egress and integration. The same logic applies in reverse to Bedrock. Model quality differences between the two are far smaller than the cost of moving your data across a cloud boundary to reach them. Where the decision is genuinely open, the differentiators are concrete. Bedrock brings 19 model providers plus custom model import under one AWS API and one bill, and a compliance posture including FedRAMP High — which is decisive for US public sector work and has no equivalent on the Vertex side of this comparison. It also has built-in Guardrails, evaluations, Knowledge Bases and AgentCore rather than bolt-on tooling, and cost levers in Batch and Flex at 50% off standard. Vertex AI's edge is documentation quality on data governance, which sounds unglamorous and matters enormously in regulated procurement. It publishes per-feature zero-retention conditions and a dedicated EU multi-region endpoint, and Generative AI on Vertex AI is explicitly named in scope for SOC 2 Type II with quarterly audits. When a security reviewer asks "prove this specific feature does not retain our prompts", Vertex is the easier document set to answer from. It also has the broadest first-party modality coverage — Gemini text, Imagen, Veo video, Chirp audio, Lyria music, embeddings — in one platform. Both have the same two structural problems. Pricing is fragmented in ways that make forecasting genuinely hard: Bedrock across tiers, Provisioned Throughput units and per-feature surcharges; Vertex across tokens, modality units, GSUs, grounding calls and cache storage. And both are deep lock-in — IAM, regions and fine-tuned artifacts do not port from one to the other. --- ## Amazon Bedrock vs Azure AI Foundry Page: https://gatewayscore.com/compare/amazon-bedrock-vs-azure-ai-foundry The question that decides it: Do you need first-party OpenAI models and catalog breadth, or AWS-native governance and FedRAMP High? Verdict: Already on AWS, or need FedRAMP High: Bedrock. Already on Microsoft, or you want first-party OpenAI models with a migration that is mostly a base-URL change: Azure AI Foundry. Catalog size is the most eye-catching difference and the most misleading. Azure AI Foundry publishes 11,000+ models against Bedrock's 100. But Bedrock's number counts models from 19 curated model providers, while Azure's counts an enormous partner and open-weight catalog of wildly varying production readiness. The useful question is not how many models exist but whether the specific ones you need are there, deployable in your region, at a price you can forecast. The genuine Azure advantage is first-party OpenAI. Azure OpenAI clients are thin wrappers over the official OpenAI SDKs, so migrating an existing OpenAI integration is mostly a base-URL and credential change. If your code already calls OpenAI and you need enterprise governance around it, that is the shortest path in this entire catalog. Azure also ships a built-in Model Router, and gives fine-grained residency control through Global, Data Zone, single-region and Azure Government deployment types. Bedrock's advantage is depth of compliance and integrated tooling. ISO, SOC, CSA STAR Level 2, GDPR, FedRAMP High, and HIPAA-eligible with an AWS BAA. FedRAMP High in particular has no counterpart in this comparison and settles US public sector questions on its own. Bedrock also builds in Guardrails, evaluations, Knowledge Bases and AgentCore rather than treating them as separate products, and offers Batch and Flex at 50% off standard. Both share the same limits as any hyperscaler platform: no cross-cloud routing, so multi-cloud failover needs an external gateway; deep coupling to their own cloud's IAM, regions and quotas; and pricing that is hard to forecast — Bedrock across tiers and Provisioned Throughput, Azure around PTUs and deployment types. Azure carries one extra irritation worth naming: constant renaming, from Azure AI Studio to Azure AI Foundry to Microsoft Foundry, which leaves docs and portal labels inconsistent and makes it harder to confirm which attestations cover which brand. --- ## Cloudflare AI Gateway vs Vercel AI Gateway Page: https://gatewayscore.com/compare/cloudflare-ai-gateway-vs-vercel-ai-gateway The question that decides it: Which platform do you already deploy on, and do you need documented gateway-level compliance? Verdict: On Cloudflare Workers: Cloudflare AI Gateway, because the core features are free on every plan. On Vercel: Vercel AI Gateway. If you need documented SOC 2 or HIPAA for the gateway itself, Vercel — Cloudflare publishes no AI Gateway-specific posture. Both of these are features of a hosting platform rather than standalone products, and both are priced to make you use the rest of the platform. That makes the decision mostly pre-made: you use the one belonging to the cloud your app already runs on, because the latency and integration story only works from inside. Cloudflare's offer is unusually generous on features. Core gateway functionality — dashboard analytics, caching, rate limiting — is free on every Cloudflare plan including the free one, with no markup on provider tokens. DLP scanning for PII, financial and healthcare data is free on all plans, along with llama-guard-based guardrails. Dynamic Routing gives visual, code-free routing by user segment, geography, content or A/B split. Spend limits can be scoped by model, provider or custom metadata with automatic blocking. That is a lot of governance to get for nothing, and materially more than Vercel offers — Vercel has no semantic cache and no native content guardrails at all. Vercel's offer is cleaner economics and better paperwork. Zero markup on tokens including BYOK, and no credit-purchase fee, against Cloudflare's 5% on credit purchases through Unified Billing. Vercel publishes 360 models and a public unauthenticated models endpoint with pricing; Cloudflare publishes roughly 23 upstream providers, no model count, and no public catalog endpoint. On compliance, Vercel documents SOC 2 Type 2, ISO 27001, HIPAA on enterprise and GDPR, and offers per-request ZDR and provider allowlists at no cost. Cloudflare publishes no SOC 2, HIPAA or ZDR posture specific to AI Gateway — the wider Cloudflare attestations may cover it, but you would need to confirm scope rather than cite a document. That compliance gap is the one thing that should override platform convenience. If a security reviewer needs a document naming the gateway, Vercel has one and Cloudflare does not. Everything else here is a preference; that is a procurement blocker. --- ## Groq vs Together AI Page: https://gatewayscore.com/compare/groq-vs-together-ai The question that decides it: Is perceived output speed your product differentiator, or do you need catalog breadth, fine-tuning and a HIPAA BAA? Verdict: Groq when output speed is the feature your users notice — streaming chat, voice, anything interactive. Together AI when you need catalog breadth, cheap fine-tuning, a dedicated-GPU growth path, or a HIPAA BAA. These are both single-source inference providers, not gateways. Neither offers cross-provider routing, load balancing or failover, which means an outage at either has no in-platform fallback. If uptime matters, you want one of these behind a gateway that can fail over to the other, rather than either one alone. That is the most important architectural point about both. Groq's case is speed, and it is not marginal. Up to 1,000 tokens per second on GPT-OSS-20B on its own LPU hardware is the highest published output speed in this catalog, and per-token prices are very low — gpt-oss-120b at $0.15 input and $0.60 output per million tokens. For streaming chat, voice agents or anything where the user watches text appear, that is a difference they can feel. Groq also makes zero data retention self-serve for every customer, globally or per feature, with no retention of inference data by default, which is a stronger default than most. Together AI's case is breadth and growth path. 200+ open-weight models across text, vision, image, video, audio and embeddings under one OpenAI-compatible API, against Groq's open-weight-only catalog with no published model count. Fine-tuning is inexpensive from $0.48 per million tokens with a $4 job minimum, and there is a documented path from serverless to dedicated endpoints, GPU clusters and provisioned throughput as volume grows. Together also has HIPAA BAAs in place with partners, where Groq has no documented BAA. The compliance gap decides this for regulated buyers. Groq retains data in US Google Cloud buckets with no EU residency option and no documented HIPAA BAA. Together has SOC 2 Type 2 completed, a public trust center and partner BAAs — though its GDPR posture, EU residency and zero-retention specifics are not documented on any vendor page found, so it is better rather than complete. Neither is a good answer if contractual EU residency is a hard requirement. --- ## Fireworks AI vs Together AI Page: https://gatewayscore.com/compare/fireworks-ai-vs-together-ai The question that decides it: Do you need selectable latency tiers and reinforcement fine-tuning, or broader modality coverage and cheaper supervised fine-tuning? Verdict: Together AI for most teams: broader catalog, cheaper fine-tuning, more modalities. Fireworks when you need selectable latency tiers on the same model, reinforcement fine-tuning, or documented CMEK encryption states. These two are genuinely hard to separate, which is itself the useful finding. Both are single-source inference providers serving open-weight models on their own stacks with OpenAI-compatible APIs. Both state SOC 2 Type II and HIPAA compliance with public trust centers. Both offer batch inference and fine-tuning. Neither offers closed frontier models, cross-provider routing or failover. Neither documents EU residency or zero-data-retention on any vendor page found. If you are choosing between them on compliance, you will not find a winner — you will find two similar gaps. Fireworks differentiates on latency control and training sophistication. It offers selectable DEFAULT, PRIORITY and FAST service tiers, so you can pay for predictable latency on the same model rather than accepting whatever multi-tenant serverless gives you. It supports both supervised and reinforcement fine-tuning, with managed LoRA from $0.50 per million tokens — reinforcement fine-tuning in particular is a capability Together does not list. Fireworks also documents CMEK encryption states and offers region-restricted deployments when a workload must stay in a specific geography. Together differentiates on breadth and price. 200+ models against Fireworks' 100, spanning text, vision, image, video, audio and embeddings, where Fireworks is open-weight text-centric. Fine-tuning is cheaper at $0.48 per million tokens with a $4 job minimum. And Together publishes a model list endpoint with per-model pricing, licence and context length, which makes programmatic catalog checks possible — useful if you want to detect a model deprecation before it breaks your product. One practical difference in getting started: Fireworks gives a one-time $1 credit, which is too small for realistic load testing. Together publishes no free tier at all and third-party reporting says new accounts need a prepaid minimum balance. Neither lets you evaluate seriously without spending money, so budget for a paid proof-of-concept either way. --- ## Kong AI Gateway vs Apache APISIX Page: https://gatewayscore.com/compare/kong-ai-gateway-vs-apache-apisix-ai-gateway The question that decides it: Do you need LLM-native features and compliance paperwork, or is free and fast with basic proxying enough? Verdict: Already running Kong and need semantic caching, PII sanitization and guardrails on AI traffic: Kong, and accept that you will negotiate the price. Already running APISIX and need basic multi-provider proxying with retries and token rate limits: APISIX, free and permanently so. Neither of these is an LLM-first product. Both are mature API gateways — Kong on NGINX/OpenResty with 44,100 GitHub stars, APISIX also OpenResty-based with 16,800 — that grew AI plugins. You would choose either because you already run it for your regular API traffic and want AI requests governed by the same gateway, plugins and ops tooling. Adopting either from scratch purely for LLM routing is usually the wrong call; an LLM-native gateway will be a better fit. Kong's AI plugin set is genuinely deep: semantic caching and semantic routing, a RAG injector, prompt templates, MCP and A2A gateways, plus a PII sanitizer covering 20 categories across 9 languages and integrations with Azure, AWS, GCP and Lakera guardrails. Its compliance posture is documented and broad — SOC 2 Type 2, GDPR, CCPA, PCI DSS, CSA STAR Level 1, NIST 800-218. For a regulated enterprise that already standardised on Kong, that combination is hard to match. The price of that is opacity, and it is significant. Kong publishes no pricing for the AI Gateway: Plus is quoted per gateway per month with no published amount, and Enterprise is custom. Worse for evaluation purposes, Plus caps the AI Gateway at five unique LLM models, which pushes any real multi-model deployment to an Enterprise conversation. Third-party reviews also report the free OSS build lacks semantic caching, analytics and compliance features — so the open-source Kong you can try is not the Kong being described. APISIX is the opposite trade. The entire project is Apache-2.0 with no paid tier, no gated features and no vendor account required, and it publishes the fastest data-plane numbers here at roughly 18,000 QPS per core at under 0.2 ms added latency. But the AI plugins are thin: no response or semantic caching, no budgets, no virtual keys, no prompt management, and only about ten documented provider integrations against 20-140 for LLM-native gateways. It also has no compliance certifications, support SLA or hosted control plane from the project itself, because it is a foundation project rather than a vendor. One caution that applies to both, and to this whole category: Kong's headline performance advantage comes from a Kong-run benchmark against mock LLMs, and a third-party guide reports that independent tests show far smaller gaps. APISIX's published numbers are for generic proxying, not LLM paths. Neither figure tells you what your traffic will do. --- ## OpenRouter vs LiteLLM Page: https://gatewayscore.com/compare/openrouter-vs-litellm The question that decides it: Do you want one commercial relationship and one bill, or do you want to hold every provider account and key yourself? Verdict: If you want to call a model this afternoon with one account, one key and one invoice, OpenRouter, and the 5.5% credit fee is the price of not doing procurement. If you already have accounts with the providers you use, or anything about your data path has to stay inside infrastructure you control, LiteLLM: it is MIT, it costs nothing per request, and it is the only one of the two you can run air-gapped. These are not two versions of the same product. OpenRouter is managed only and LiteLLM is self-host only, so the deployment question is settled before any feature comparison starts. If you cannot run a container with Postgres and Redis behind it, LiteLLM is not available to you. If your data cannot leave infrastructure you control, OpenRouter is not available to you. Most teams find one of those two sentences decides the page. The money works in opposite directions. OpenRouter publishes zero token markup and takes 5.5% when you buy credits, with a minimum of $0.80 per card purchase, so small top-ups are punitive and large ones are close to the headline rate. LiteLLM charges nothing at all: no markup, no credit fee, no seats. What it charges instead is your time and your infrastructure, and the worked example on our cost page assumes $40 a month of compute and database to run it, which is a guess about your setup rather than a published price. The real difference is who holds the provider relationship. Through OpenRouter you have one account and one key, and OpenRouter has the accounts with the upstream providers. That is the entire convenience, and it is substantial: no negotiating, no per-vendor onboarding, no separate invoices, and a pool of 83 providers behind 400 to 500 models with automatic failover between them. With LiteLLM you bring your own keys to 140+ providers, which means you keep whatever rates, credits and contractual terms you have already negotiated, and you keep custody of the secrets. On the compliance columns they are not comparable so much as differently shaped. OpenRouter publishes SOC 2, zero data retention, a zero-day retention window and metadata-only logging, which is a real audited posture from a vendor you are trusting. LiteLLM publishes no SOC 2 report, because there is no service to audit: the posture is whatever your own deployment has, which is either much better or much worse than OpenRouter depending entirely on you. LiteLLM is also the only side that can run air-gapped. --- ## OpenRouter vs Portkey Page: https://gatewayscore.com/compare/openrouter-vs-portkey The question that decides it: Do you need to reach the most models, or to control how your team uses the ones you already reach? Verdict: Reaching the long tail of open-weight models across many upstream providers is the job: OpenRouter. Stopping prompts leaking data, caching semantically similar calls, versioning prompts and running evals is the job: Portkey, at $49 a month for the Production plan, or free while you stay under 10,000 logged requests. These two are usually shortlisted together and they answer different questions. OpenRouter is a marketplace: its product is breadth and price discovery across 83 upstream providers, and it deliberately does very little to what passes through it. Portkey is a control plane: its product is what happens to the request on the way past, and its catalogue is a means to that end. The capability columns are unusually one-sided, and it is worth saying which absences are documented rather than merely unverified. OpenRouter explicitly does not offer a semantic cache or content guardrails — those are recorded as no, not as unknown. Portkey offers both, plus prompt management and evals, neither of which OpenRouter publishes at all. If any of those four is on your requirements list, this comparison is already over. Cost inverts depending on volume and on what you are counting. OpenRouter charges 5.5% on credit purchases and nothing else, so its cost scales with spend forever. Portkey charges nothing on tokens and $49 a month for Production, with 100,000 recorded logs included and $9 per additional 100,000 requests, so its cost scales with how much you log rather than what you spend on inference. At the workload on our cost page, that is $8.25 for OpenRouter against $49 for Portkey. Invert the token spend and the answer inverts with it. Portkey also has a deployment option OpenRouter cannot match: its gateway core is MIT and self-hostable, so the data path can stay in your infrastructure while the control plane stays hosted. It is the only side of this pair with a HIPAA BAA. Against that, its catalogue figure is the least trustworthy number on this page — its own pages give both 250 and 2,300 models depending on where you look, and its models endpoint requires an API key, so there is nothing public to count. --- ## LiteLLM vs Cloudflare AI Gateway Page: https://gatewayscore.com/compare/litellm-vs-cloudflare-ai-gateway The question that decides it: Do you want the gateway inside your infrastructure, or at an edge you never have to run? Verdict: If nobody on the team wants to operate a proxy, Cloudflare AI Gateway is close to free and effectively zero maintenance, and its 5% fee only applies if you opt into paying providers through Cloudflare. If you need provider breadth, air-gapped operation, or control over what gets logged, LiteLLM. The operational asymmetry is the whole comparison. Cloudflare AI Gateway is a hosted endpoint you point your existing SDK at; there is nothing to deploy, patch or scale, and the core features — analytics, caching, rate limiting and DLP scanning — are free on every plan including the free one. LiteLLM is a container you run with Postgres and Redis behind it, and everything about its availability is your problem. Provider coverage is not close. LiteLLM reaches 140+ providers; Cloudflare enumerates 24. Cloudflare's list covers the names most teams actually use, and its documentation is explicit that features work across the supported set, but if your requirement is "we must be able to call anything", it is the wrong shelf. Neither publishes a model total, and neither should be judged on one. The logging default deserves more attention than it usually gets. Cloudflare AI Gateway logs the full request and response by default. That is genuinely useful — it is what makes the analytics and the replay worth having — and it can be turned off. But it means prompt and completion content sits in Cloudflare's logs unless you go and change it, which is a different starting posture from LiteLLM, where logging is configurable and the logs land wherever you put them. If you handle regulated data, this is the sentence to take to your reviewer. Money is nearly a non-issue on both, in different ways. LiteLLM charges nothing and costs you infrastructure. Cloudflare charges nothing for the gateway itself and takes 5% only if you use Unified Billing to pay upstream providers through Cloudflare — bring your own provider keys instead and that fee does not arise. The honest cost comparison is therefore your engineers' time against a 5% opt-in. --- ## Vercel AI Gateway vs LiteLLM Page: https://gatewayscore.com/compare/vercel-ai-gateway-vs-litellm The question that decides it: Are the seats already on your invoice, and can you tolerate a gateway you cannot self-host? Verdict: Already shipping on Vercel with seats you are paying for anyway: the gateway is free at the margin and nothing here beats it on convenience. Not on Vercel, or unwilling to send prompts through a vendor you cannot self-host: LiteLLM, which costs nothing per request and runs wherever you put it. Both publish zero token markup and zero credit fee, which makes the fee comparison unusually clean: the question is seats against infrastructure. Vercel Pro is $20 per user per month, so a three-person team is $60 whether or not the gateway is used at all. LiteLLM is free, and our cost page assumes $40 a month of compute and database to run it — an assumption about your setup, not a published price. If you are already paying for the Vercel seats, the marginal cost of the gateway is zero and LiteLLM cannot beat that. Vercel's add-on pricing is where the clean story gets complicated, and one line deserves naming: zero data retention is $0.10 per 1,000 requests on Pro and Enterprise. On most platforms in this catalogue ZDR is a setting; here it is metered. At the 250,000 requests on our cost page that is $25 a month on top of the seats, and a team-wide provider allowlist is another $0.10 per 1,000 successful requests. Neither is exotic governance — they are the kind of thing a security review asks for by default. Deployment settles a lot of shortlists. Vercel AI Gateway is managed only and explicitly not air-gappable. LiteLLM is self-host only. There is no overlap, so if either constraint is hard for you, the comparison resolves before you reach the features. On provider breadth LiteLLM leads clearly with 140+ providers; Vercel publishes 200 to 350 models and no provider count. The compliance columns favour Vercel on paper: SOC 2, a HIPAA BAA, zero data retention available and a 30-day retention window, all audited and published. LiteLLM publishes none of those, because there is no hosted service to audit — the posture is whatever your deployment has. That is not automatically worse, but it means you are the one producing evidence for the reviewer instead of forwarding a report. --- ## Portkey vs Kong AI Gateway Page: https://gatewayscore.com/compare/portkey-vs-kong-ai-gateway The question that decides it: Is this a new AI product to adopt, or one more route on the API gateway your platform team already operates? Verdict: If Kong already sits in front of your services, the AI Gateway is a plugin on infrastructure your platform team already knows, and that institutional advantage usually outweighs everything else on this page. If it does not, Portkey is the AI-native choice, with published pricing, deeper AI-specific features and far fewer surprises about what tier you need. Both are open core, both can be self-hosted or bought as a service, and both do the routing, guardrails, semantic caching and prompt management you would expect. The difference is lineage. Kong AI Gateway is an AI capability added to a mature API gateway with 44,100 GitHub stars, and it inherits that platform's operational model, its config-as-code, and its performance: 8,200 requests per second and 12ms of measured overhead, the fastest figures in this catalogue. Portkey was built for LLM traffic from the start and is deeper on the things only LLM traffic needs. Pricing transparency separates them sharply, and not in Kong's favour. Portkey publishes its numbers: free Developer plan with 10,000 recorded logs a month, $49 a month for Production with 100,000 logs and 30-day retention, $9 per additional 100,000 requests. Kong publishes a 30-day Konnect trial and a Plus plan billed per gateway per month with the amount not published, and pushes fully self-hosted gateways, audit logs and SSO into a custom-priced annual Enterprise contract. If you need a number before you can get approval, only one of these gives you one. There is a specific trap in Kong's Plus tier worth knowing before you plan around it: it caps the AI Gateway at five unique LLM models, with more available only on Enterprise. That is a small number for anything doing model comparison, gradual migration or per-task routing, and it is the kind of limit that is discovered after the architecture is committed rather than before. On the AI-specific columns Portkey is ahead where it matters for most teams: it publishes evals, a HIPAA BAA and zero data retention, none of which Kong publishes. Kong leads on raw throughput and on provider-independent operational maturity, and its guardrails and prompt management are real. Coverage is thin on both sides and neither publishes a model total — Portkey lists 40 to 48 providers, Kong 17. --- ## Amazon Bedrock vs OpenRouter Page: https://gatewayscore.com/compare/amazon-bedrock-vs-openrouter The question that decides it: Does the model call have to stay inside your AWS account and its compliance boundary? Verdict: If you are an AWS shop and the answer to that question is yes, Bedrock, and the catalogue limits are the price of the boundary. If not, OpenRouter reaches 83 providers against Bedrock's 19 and costs 5.5% on credits with no per-feature metering. This is a compliance-boundary question wearing a catalogue-size costume. Bedrock runs inside your AWS account, under IAM, in a region you choose, with the data path governed by the agreement you already have with AWS. For a regulated team that already passed an AWS review, that is most of the work already done, and no gateway in this catalogue can offer it. OpenRouter is a separate vendor, a separate contract and a separate review. Neither charges a conventional gateway fee, but they recover it differently and that is the part people miss. Bedrock sets its own per-token prices rather than marking up a third party, so the routing layer is genuinely $0 — then it meters the governance. Guardrails content filters are $0.15 per 1,000 text units, sensitive-information filtering $0.10, Automated Reasoning checks $0.17, and Intelligent Prompt Routing $1.00 per 1,000 requests. OpenRouter takes 5.5% on credits and does not meter features, largely because it does not offer these ones. Catalogue is where OpenRouter wins and the margin is real: 83 providers and 400 to 500 models, against Bedrock's 19 providers and a documented "100+ foundation models" with no exact count published. More usefully, OpenRouter typically carries several upstream deployments of the same open-weight model, which is what makes price sorting and failover between them worth anything. Bedrock gives you one path to each model, which is the trade you accept for the boundary. On data handling both are strong, and Bedrock is the strictest thing in this catalogue: it logs nothing by default, retains for zero days, does not train on customer data and will sign a HIPAA BAA. OpenRouter also publishes zero-day retention, zero data retention and metadata-only logging, plus SOC 2 and EU residency, which is more than most managed gateways here. The difference is not the policy so much as who you are trusting to keep it. --- # Glossary Page: https://gatewayscore.com/glossary **Gateway.** A single endpoint you send all your AI requests to, which then forwards them to whichever model you asked for. One integration instead of one per vendor. **BYOK — bring your own key.** You keep your own accounts and contracts with OpenAI, Anthropic and the rest, and the gateway routes through your keys. You keep your negotiated rates and any committed-spend discounts; the gateway charges you for the plumbing, not the tokens. **Markup.** A percentage the gateway adds on top of what the model actually costs. Some charge none at all and make money elsewhere. **Credit or top-up fee.** A cut taken when you add money to a prepaid balance, typically around 5%. Easy to miss because it is not a markup on tokens — but you pay it on every dollar you load. **Token.** Roughly three quarters of a word. Billing is per million tokens, and output tokens usually cost several times more than input. **Failover.** When the provider you asked for is down or rate-limiting you, the gateway automatically retries somewhere else. The single most valuable reliability feature these products offer. **Load balancing.** Spreading traffic across several keys or providers so no single one hits its rate limit. Different from failover, which only reacts to failure. **Conditional routing.** Rules that pick the model per request — cheap model for simple work, expensive one for hard work, a specific model for one customer. **Semantic caching.** Returning a stored answer when a new question means the same thing as an old one, even if worded differently. Saves more than exact-match caching but can return a subtly wrong answer, so it needs care. **Virtual keys.** Separate scoped keys you issue per team, app, or customer, each with its own budget and limits, without handing out your real provider credentials. **Observability.** Logs, traces, and dashboards showing what was sent, what came back, what it cost, and how long it took. The thing you desperately want the first time a bill surprises you. **Guardrails.** Automated checks on requests and responses — blocking prohibited content, enforcing output formats, catching prompt injection. **ZDR — zero data retention.** A commitment that your prompts and responses are not stored after the request completes. Frequently a paid add-on or an enterprise-only option rather than the default. **SOC 2.** An audit of a company’s security controls. Type 1 checks the design at a point in time; Type 2 checks they actually operated over months. Procurement teams ask for this constantly. **HIPAA BAA.** A signed contract required before you can put US health data through a vendor. Without one you cannot legally use the service for protected health information, regardless of how secure it is. **Data residency.** A guarantee about which countries your data is processed in. Usually matters for EU obligations or public-sector contracts. **OpenAI-compatible.** Speaks the same API shape as OpenAI’s, so existing client libraries work by changing a base URL. This is what makes switching gateways cheap — and worth insisting on. **Self-hosted.** You run the software on your own infrastructure. No third party sees your traffic, and there is no vendor fee — but you own the uptime, the patching, and the upgrades. **Air-gapped.** Runs with no outbound internet access at all. A hard requirement in some defence and regulated environments, and a genuine constraint on which products are even eligible. **Latency overhead.** The delay the gateway itself adds, separate from the model’s own thinking time. Usually single-digit milliseconds and irrelevant next to a model taking several seconds — treat large claimed differences with suspicion. # Field definitions ## Overview **What kind of product.** Marketplaces resell many providers behind one key. Gateways add governance on top. Open-source projects you run yourself. Cloud platforms are hyperscaler surfaces. Inference providers host models on their own hardware. **Who runs it.** Managed means the vendor operates it. Self-host means you run it on your own infrastructure. Both means you can choose. **Licence.** Proprietary products cannot be inspected or forked. Open licences such as MIT and Apache-2.0 let you audit, modify, and run the code without permission. **Company.** The organisation that maintains the product. **Who you would be signing with.** Whether the product is still an independent company, has been acquired, is a large cloud vendor’s product line, is run by a software foundation, or has been put into maintenance mode. Maintenance mode means bug fixes and security patches only — no new features. **Last shipped an update.** The date of the most recent release or version tag. A product that has not shipped in a year is a different risk from one that shipped last week, regardless of what its marketing site says. **GitHub stars.** A rough proxy for community size on open-source projects. Not a quality measure. ## Cost **Markup on model prices.** How much the product adds on top of what the underlying model provider charges. Zero means you pay the same per-token price you would pay the model provider directly. **Fee to add funds.** A percentage charged when you top up your balance, separate from token prices. It is easy to miss because it does not appear on the per-token price list. **Monthly cost per person.** A recurring per-user platform charge that applies regardless of how much you use the models. **Can use your own provider accounts.** Bring Your Own Key: you keep direct contracts with OpenAI, Anthropic and others, and the gateway only routes traffic. This preserves negotiated rates and committed-spend discounts. **Cost of using your own accounts.** What the product charges to route traffic through your own provider keys. **Free tier.** What you can do without paying, useful for evaluation. **Enterprise plan from.** Annual entry price for the enterprise tier, where one is published or credibly reported. **Cost to run it yourself.** What self-hosting actually costs once you account for infrastructure and any paid tier. **How the vendor makes money.** The shape of the vendor’s bill: does the routing layer charge a percentage on top of tokens, a flat monthly fee, both, neither (because inference is the product), or nothing at all (open source with no paid tier). **How pricing works, briefly.** A one-paragraph description that covers the caveats a pricing category cannot: introductory rates, per-feature meters, tier gating, and pricing that resets on a specific date. **Minimum commitment.** Whether the vendor requires a minimum contract term, a minimum spend, or a provisioned-capacity purchase to get its published rate. **Charges that fire after you go over an allowance.** The line items that scale with usage after an included allowance is exhausted — log storage, extra requests, per-feature meters, data export — which is where cost estimates usually go wrong. **Prompt cache offered.** Whether the gateway offers its own response cache, what kind of match it does (exact request, prefix, semantic), or simply passes provider caching through unchanged. **Discount on cached input.** How much cheaper cached tokens are than fresh input, when the vendor publishes a single figure. Bundled-inference clouds usually price this per model instead of as one number. **Premium on cache writes.** How much more the first write of a cached prefix costs versus a plain input token. A high write premium and a low hit rate can leave you paying more than you save, so this matters as much as the read discount. **Who captures the cache saving.** Whether the customer keeps the full saving from caching or the vendor captures part of it — and any conditions attached (write premium, storage fees, best-effort hits). **What you can split spend by.** The dimensions the vendor documents for splitting spend — per key, per user, per team, per tag, per customer. Matters if you need to chargeback internally or bill an end customer. **How you get cost data out.** The mechanisms the vendor publishes for exporting cost and usage data: CSV, an API, webhooks, S3, a data warehouse, or nothing at all. Any per-unit price is included. **Who pays the model bill.** Whether you bring your own provider accounts (BYOK), buy inference from this vendor, or can do either. This is the single biggest commercial difference between these products: it decides who holds the contract with the model provider and who carries the spend. ## Catalog **Models available.** How many models you can call, shown as a range because several vendors publish different totals on different pages. Vendor-reported either way, so counts are not directly comparable — some count every provider variant of the same model separately. **Model providers reachable.** How many distinct model providers or labs you can reach, shown as a range where the vendor’s own pages disagree. More providers usually means better redundancy when one has an outage. **Works with standard OpenAI code.** If yes, you can usually switch to it by changing one base URL, and switch away just as easily. This is the main defence against lock-in. **OpenAI chat endpoint.** The endpoint almost every application ports first. "Not documented" means the vendor never states it, which is different from a documented no. **Anthropic messages endpoint.** Whether Anthropic-shaped calls work without rewriting them. Several products support this only as SDK compatibility or provider passthrough rather than a native endpoint — the detail page says which. **OpenAI Responses endpoint.** The newer stateful OpenAI surface. Support is much thinner across this market than chat completions. **Embeddings endpoint.** Whether you can generate vectors through the same gateway, or need a second integration for retrieval workloads. **Image generation endpoint.** Whether image models are reachable through the same surface as text. **Audio endpoints.** Speech-to-text and text-to-speech. Frequently the first gap in an otherwise complete gateway. **Batch jobs endpoint.** Asynchronous bulk processing, usually at a discount. Commonly undocumented, and commonly the reason a migration stalls late. **Needs the vendor’s own code library.** A proprietary client library spreads through your codebase and has to be torn out again if you leave. “No” is the better answer here, and it means the standard OpenAI client works. **You can export your request history.** Whether you can get your own request logs, traces, or usage records back out — through an API, a bulk export, or a download. Decides whether you leave with your history or abandon it. **Settings can live in version control.** Whether routing, fallback, and budget rules can be declared in a file you keep in Git, rather than existing only as settings clicked into a hosted dashboard. **Embeddings.** Text-to-vector models, needed for search and retrieval features. **Image generation.** Whether image models are reachable through the same interface. **Speech and audio.** Text-to-speech or transcription models through the same interface. **Video generation.** Whether video models are reachable through the same interface. **Batch processing.** Submitting large jobs for cheaper, slower processing. Often 50% off for work that is not time-sensitive. ## Routing & reliability **Uptime it promises in writing.** The uptime percentage in a published, contractual service level agreement. A public status page is not an SLA — it reports what happened, it does not promise anything or pay you back when it breaks. **Automatic failover.** When a model provider goes down or rate-limits you, traffic moves to a backup automatically instead of returning errors to your users. **Load balancing.** Spreads requests across several providers or keys to raise your effective rate limit. **Rule-based routing.** Send different requests to different models based on rules — for example a cheap model for free users and a strong model for paying ones. **Response caching.** Reuses the answer when the exact same request comes in again, which cuts both cost and latency. **Similar-question caching.** Reuses an answer when a new question means roughly the same thing as an earlier one. Saves far more than exact-match caching but can return subtly wrong answers if tuned loosely. **Where you set the timeout.** Where a request timeout can be set: per request, in a config file, in the vendor dashboard, or nowhere. Nine of the twenty products documented here do not describe a request timeout at all, so the worst case of a hung upstream call is unknowable from the docs. **Where you set retries.** Where retry count and backoff are configured. Worth knowing alongside billing: a retried streaming call can be charged more than once. **Where you set fallbacks.** Where the fallback chain is defined. Almost every product claims fallback; the useful question is whether you can change it from code or only by hand in a dashboard. **Shape of the fallback chain.** An ordered list tries targets in sequence; a weighted split sends a percentage of traffic to each, which is what you need to trial a new model on 5% of requests. Weighted splits are much rarer than the marketing implies. **Upstream health tracking.** Whether the product notices a failing upstream and stops sending traffic to it, and whether you can tune the thresholds. This is what turns a provider outage into a blip rather than a sustained error rate, and only four of the twenty expose it. **Cross-region failover you control.** Whether you can define what happens when a region degrades. A vendor running many regions is not the same as a vendor letting you configure failover between them; only three document a user-controlled mechanism. **Where you set load balancing.** Where traffic distribution across upstreams or keys is configured. ## Operations **Usage dashboards and logs.** Built-in visibility into what was sent, what came back, what it cost, and how long it took. **Spending limits.** Hard caps that stop spend before it becomes a surprise invoice. The single most valuable control for a small team. **Rate limits.** Caps on request volume per key or per user, useful for protecting against abuse and runaway loops. **Separate keys per team or app.** Issue scoped keys with their own budgets and permissions so you can attribute cost and revoke access without rotating everything. **Prompt versioning.** Store and version prompts outside your code so they can be changed without a deploy. **Quality testing.** Built-in tooling to score model output against test cases, so you can tell whether a model swap made things better or worse. **MCP support.** Native support for the Model Context Protocol, the emerging standard for connecting models to external tools. **What gets logged.** Whether full prompts and responses are stored, only metadata, or nothing. Full-body logging is the most useful debugging feature here and the one most likely to need a conversation with your compliance team. **You can turn logging off.** Whether prompt and response bodies can be suppressed while still keeping usage metrics. Nineteen of the twenty document a way to do this; the mechanisms range from a per-request header to an organisation-wide setting. **Traces you can take elsewhere.** Whether the product emits OpenTelemetry, a proprietary format, or nothing. OpenTelemetry means the traces land in the tooling you already run instead of only in the vendor’s dashboard. **Where telemetry can go.** Documented sinks for logs and metrics. This is a good proxy for how replaceable the vendor’s own dashboard is: around twenty destinations means you never have to depend on it, while a CSV download means you do. **Can record user feedback.** Whether there is an API to attach a rating or score to a logged request, which is what lets production traffic feed quality work later. **Scores live traffic.** Whether automated scorers can run against real production requests, rather than only against a test set you assemble yourself. ## Performance **Delay it adds.** Extra time the product itself adds to each request, on top of however long the model takes. Usually irrelevant next to multi-second model latency, but it matters for high-volume or streaming-sensitive workloads. **Requests per second ceiling.** Published sustained request rate before the product becomes the bottleneck. Only relevant at genuinely high volume. **What the request path runs on.** The comparable way to talk about latency here. An edge worker, a compiled Go or Rust binary, and a Python proxy have different overhead floors no matter which figures each vendor publishes. Products that never disclose their runtime are recorded as undisclosed rather than assumed. **You can run the request path yourself.** Whether the component that actually carries your prompts can run on your own infrastructure. Distinct from a vendor offering a self-hosted control plane while still proxying traffic through their network. **Streaming responses.** Whether token-by-token streaming is documented. The caveats matter more than the yes: some products cannot cancel a stream without still being billed, and several timeout and fallback mechanisms stop applying once the first token has been sent. ## Security & compliance **Does your prompt reach their servers.** Whether the text you send passes through this company’s own infrastructure. If it does, every other promise on this page is a policy commitment rather than a physical impossibility. Self-hosted products can answer no outright. **What they keep if you change nothing.** Defaults matter more than options. A product that stores full prompts and replies unless you find the right header will have stored them by the time you read the docs. **How long they keep it.** Default retention for request content, in days. Zero means nothing is kept. Read the note: several products keep nothing as a rule but make timed exceptions for abuse review or specific models. **Could they train on your prompts.** Whether the vendor may use your prompts and outputs to train models. “Not published” means we could not find any position, which is not the same as a no — ask for it in writing. **Where it runs, and what you can pin.** Which regions are offered and whether you can force processing to stay in one. A global endpoint that silently picks a region is a different compliance story from an endpoint you pin yourself. **Where safety filters run.** A filter that strips personal data only helps if it runs before the data leaves your boundary. If guardrails execute in the vendor’s cloud, the vendor has already received whatever you wanted redacted. **Who else touches the data.** The published list of third parties the vendor passes your data to. No list means you cannot know the full chain, which most data-protection agreements require you to. **SOC 2 audited.** An independent audit of security controls. Enterprise buyers and their procurement teams routinely require it. **Will sign a HIPAA agreement.** Required before you may send protected health information through the service. Without a signed BAA, healthcare data is off limits. **GDPR commitments.** Published data processing terms for handling personal data of people in the EU and UK. **Can keep data in the EU.** Requests can be processed inside the EU rather than routed to US infrastructure. Often the deciding constraint for European customers. **Does not retain your data.** Prompts and responses are not stored after the request completes. Sometimes a paid add-on rather than the default. **Strips personal data.** Detects and removes identifiers such as names, emails, and card numbers before the request reaches the model provider. **Content guardrails.** Policy checks on inputs and outputs — blocking unsafe content, enforcing formats, or catching prompt-injection attempts. **Runs fully disconnected.** Can be deployed in a network with no internet access, which some regulated and defence environments require. **Blocks personal data in prompts.** Whether personal data detection sits on the request path and can stop the call, merely inspects and forwards it, or is not documented. A control that only reports is a logging feature, not a policy control. **Blocks prompt injection.** Whether injection and jailbreak detection can stop a request. Most products offering this call a partner classifier rather than shipping their own. **Blocks harmful content.** Whether hate, violence, sexual and self-harm categories are checked inline and can stop a request, in either direction. **Your own policy rules.** Whether you can add your own rule — a regex, a webhook, or your own classifier — rather than choosing from the vendor library. **Where guardrails run.** Whether guardrail evaluation happens inside your infrastructure or on the vendor’s servers. This decides whether the prompt you are trying to protect leaves your network in order to be checked. **If the guardrail itself fails.** What happens when the guardrail service times out or errors: does the request proceed unchecked, or is it blocked? This is the worst-documented field in the entire catalogue — only two vendors state it plainly, which means most teams are running a control whose failure behaviour they cannot know. **Third-party guardrail vendors.** Named external guardrail services the product can call. A long list means the product is a router for policy engines rather than a policy engine itself — which also means another vendor bill and another hop. ## Fit & integration **Work to try it.** The shape of the work on the vendor’s own quickstart, from swapping one base URL through to deploying infrastructure. An ordinal class rather than a duration, because elapsed time depends on accounts and quota we cannot see. **Work to run it.** The same scale applied to the vendor’s recommended production path. For several products this is much heavier than the quickstart, which is exactly why both are recorded. **Steps on the quickstart.** A literal count of numbered steps on the vendor’s quickstart, recorded as evidence beside the work shape. Zero means the page publishes no numbered procedure at all. Large counts usually mean interleaved language tracks rather than more work. **Can you self-host it today.** Whether an install command is actually published. Several products advertise self-hosting while publishing no command to start from, which a plain yes/no would hide. **Works with the OpenAI SDK.** Whether an existing OpenAI-compatible client can be pointed at it by changing the base URL and key. **Vercel AI SDK support.** Whether a first-party AI SDK provider package exists, or only a community package, a documented workaround, or the generic OpenAI provider pointed at a custom base URL. **Python framework integrations.** LangChain, LangGraph, LlamaIndex and similar orchestration frameworks with a documented integration. **Callable from Cloudflare Workers.** Whether the docs show calling this product from your own Worker. Deliberately separated from the several products whose own gateway runs on Workers, which is a fact about their infrastructure and not about your edge compatibility. **Kubernetes install.** Whether a named, published Helm chart exists, versus Helm being referenced with no chart named, versus generic cluster documentation that has nothing to do with this product. **Terraform support.** Whether you can declare this in code: an official provider, official modules, resources inside a hyperscaler’s provider, a community provider, or only Terraform code shipped in a repo. **Reuses your cloud identity.** Whether you can authenticate with IAM roles, workload identity or managed identities instead of another long-lived API key. Distinguished from products that only accept static upstream provider credentials. **Fits behind your API gateway.** Whether AI traffic can go through a gateway you already run, and who documented that — the product’s vendor or the gateway’s. **MCP support.** Which kind of MCP support this is: a gateway that governs many MCP servers, a hosted MCP server you connect to, MCP tools accepted inside the completion API, or client tooling. These are different products behind one acronym. **Needs your own provider key.** Whether an upstream provider account and key must exist before your first call works. A prerequisite rather than a step, and it can differ between the hosted and self-hosted forms of the same product. **Gate before models work.** Whether an enablement click, a quota grant, a paid tier or an approval form stands between a valid key and a working model call. **Official client languages.** Languages with a first-party client library. An empty list can still mean the product is usable from any language via an OpenAI-compatible SDK.