Envoy AI Gateway Open source

Overview

Interpret these fields: Self-hosted vs managed LLM gateways · Who still owns your LLM gateway?

What kind of product Category ?

Open source Verified Sep 2, 2026

Marketplaces resell many providers behind one key. Gateways add governance on top. Open-source projects you run yourself. Cloud platforms are hyperscaler surfaces. Inference providers host models on their own hardware.

Who runs it Deployment model ?

Self-host only Verified Sep 2, 2026

Managed means the vendor operates it. Self-host means you run it on your own infrastructure. Both means you can choose.

Licence Licence ?

Apache-2.0 Verified Sep 2, 2026

Proprietary products cannot be inspected or forked. Open licences such as MIT and Apache-2.0 let you audit, modify, and run the code without permission.

Company Company ?

Envoy AI Gateway open-source project in the `envoyproxy` GitHub organization; maintainers are drawn from Tetrate, Bloomberg, Tencent, Netflix and Nutanix plus the KServe, Kubeflow, Envoy Proxy and Envoy Gateway projects Not verified

The organisation that maintains the product.

Who you would be signing with Vendor status ?

Run by a software foundation Verified Sep 2, 2026

Whether the product is still an independent company, has been acquired, is a large cloud vendor’s product line, is run by a software foundation, or has been put into maintenance mode. Maintenance mode means bug fixes and security patches only — no new features.

Last shipped an update Latest release ?

2026-08-21 Verified Sep 2, 2026

v1.1.0, published 2026-08-21T18:23:16Z, not a prerelease ([releases API](https://api.github.com/repos/envoyproxy/ai-gateway/releases/latest)). It is the first minor release on the stable 1.x API and added `/tokenize` across providers, `credentialOverride` for per-request credentials, `GatewayConfig.spec.forwardProxy` for HTTP CONNECT egress, `streamIdleTimeout`, MCP `hostnames` and CEL `backendSelector`, the `gen_ai` tracing convention plus a Grafana dashboard, JSON controller logs, and Helm PDB/topology-spread support ([v1.1 release notes](https://aigateway.envoyproxy.io/release-notes/v1.1), 2026-09-02).

The date of the most recent release or version tag. A product that has not shipped in a year is a different risk from one that shipped last week, regardless of what its marketing site says.

GitHub stars GitHub stars ?

2,112 Verified Sep 17, 2026

A rough proxy for community size on open-source projects. Not a quality measure.

Cost

Interpret these fields: How LLM gateway pricing works · LLM gateway spending limits: stop a runaway agent bill?

Markup on model prices Token markup ?

None Verified Sep 2, 2026

How much the product adds on top of what the underlying model provider charges. Zero means you pay the same per-token price you would pay the model provider directly.

Fee to add funds Credit purchase fee ?

Not published Not verified

A percentage charged when you top up your balance, separate from token prices. It is easy to miss because it does not appear on the per-token price list.

Monthly cost per person Seat fee ?

Not published Not verified

A recurring per-user platform charge that applies regardless of how much you use the models.

Can use your own provider accounts BYOK supported ?

Yes Verified Sep 2, 2026

Bring Your Own Key: you keep direct contracts with OpenAI, Anthropic and others, and the gateway only routes traffic. This preserves negotiated rates and committed-spend discounts.

Cost of using your own accounts BYOK terms ?

No fee of any kind: the project takes no payment and no tokens pass through vendor infrastructure, so BYOK carries no surcharge ([LICENSE](https://raw.githubusercontent.com/envoyproxy/ai-gateway/main/LICENSE); [upstream auth](https://aigateway.envoyproxy.io/docs/capabilities/security/upstream-auth), 2026-09-02). Verified Sep 2, 2026

What the product charges to route traffic through your own provider keys.

Free tier Free tier ?

The whole project is free: Apache-2.0 licence, Helm charts on Docker Hub (`oci://docker.io/envoyproxy/ai-gateway-helm`), a CLI image (`envoyproxy/ai-gateway-cli`) and per-release CLI binaries, with no account, plan or quota. Your costs are the Kubernetes cluster and the upstream provider tokens. Not verified

What you can do without paying, useful for evaluation.

Enterprise plan from Enterprise plan from ?

Not published Not verified

Annual entry price for the enterprise tier, where one is published or credibly reported.

Cost to run it yourself Self-host cost ?

Self-hosting is the only mode and carries no licence fee: `helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm` followed by `helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm` ([installation](https://aigateway.envoyproxy.io/docs/getting-started/installation)). Real cost drivers are the Envoy Gateway control plane and Envoy data plane it requires, the AI Gateway controller (documented starting requests 100m CPU / 128Mi, limits 500m / 256Mi, HPA example min 2 max 10 replicas at 70% CPU) and a Redis instance if you enable token rate limiting or quotas ([scaling](https://aigateway.envoyproxy.io/docs/capabilities/scaling); [rate limiting](https://aigateway.envoyproxy.io/docs/capabilities/traffic/usage-based-ratelimiting), 2026-09-02). Verified Sep 2, 2026

What self-hosting actually costs once you account for infrastructure and any paid tier.

How the vendor makes money Pricing model ?

Open source, no paid tier Verified Sep 2, 2026

The shape of the vendor’s bill: does the routing layer charge a percentage on top of tokens, a flat monthly fee, both, neither (because inference is the product), or nothing at all (open source with no paid tier).

How pricing works, briefly Pricing model detail ?

Apache-2.0 open source with no vendor-priced tier at all: the site has no pricing page and the only artifacts are Helm charts, container images and CLI binaries ([LICENSE](https://raw.githubusercontent.com/envoyproxy/ai-gateway/main/LICENSE); [installation](https://aigateway.envoyproxy.io/docs/getting-started/installation); [CLI install](https://aigateway.envoyproxy.io/docs/cli/aigwinstall), 2026-09-02). A managed product built on the same data plane is sold separately by Tetrate as Tetrate Agent Router Service, which appears in this project only as one more upstream provider you can route to ([supported providers](https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/supported-providers)) - it is a different product and is not scored here. Not verified

A one-paragraph description that covers the caveats a pricing category cannot: introductory rates, per-feature meters, tier gating, and pricing that resets on a specific date.

Minimum commitment Minimum commitment ?

None. Apache-2.0 licence, no contract, no seat count and no vendor relationship ([LICENSE](https://raw.githubusercontent.com/envoyproxy/ai-gateway/main/LICENSE)). Not verified

Whether the vendor requires a minimum contract term, a minimum spend, or a provisioned-capacity purchase to get its published rate.

Charges that fire after you go over an allowance Overage terms ?

n.a. - there is no vendor meter to overrun. Nothing on the [installation](https://aigateway.envoyproxy.io/docs/getting-started/installation) or [release notes](https://aigateway.envoyproxy.io/release-notes/) pages describes paid units; quota enforcement is something you configure for your own tenants with `QuotaPolicy` ([quota policy](https://aigateway.envoyproxy.io/docs/capabilities/traffic/quota-policy)). Not verified

The line items that scale with usage after an included allowance is exhausted — log storage, extra requests, per-feature meters, data export — which is where cost estimates usually go wrong.

Prompt cache offered Cache mechanism ?

Passes provider caching through Verified Sep 2, 2026

Whether the gateway offers its own response cache, what kind of match it does (exact request, prefix, semantic), or simply passes provider caching through unchanged.

Discount on cached input Cache-read discount ?

Not published Not verified

How much cheaper cached tokens are than fresh input, when the vendor publishes a single figure. Bundled-inference clouds usually price this per model instead of as one number.

Premium on cache writes Cache-write premium ?

Not published Not verified

How much more the first write of a cached prefix costs versus a plain input token. A high write premium and a low hit rate can leave you paying more than you save, so this matters as much as the read discount.

Who captures the cache saving Cache economics ?

No gateway-side cache and therefore no cache pricing. What exists is provider-agnostic passthrough of Anthropic-style `cache_control: {"type": "ephemeral"}` breakpoints - native on Anthropic, translated for Claude on GCP Vertex AI and AWS Bedrock, minimum 1,024 cacheable tokens and at most 4 breakpoints, with `prompt_tokens_details.cached_tokens` returned in usage. Any discount is the provider's, not the gateway's ([prompt caching](https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/prompt-caching), 2026-09-02). Not verified

Whether the customer keeps the full saving from caching or the vendor captures part of it — and any conditions attached (write premium, storage fees, best-effort hits).

What you can split spend by Cost attribution ?

Per-request token and cost metadata rather than a billing console: `llmRequestCosts` on `AIGatewayRoute` exposes InputToken, CachedInputToken, OutputToken, TotalToken and CEL-computed costs as Envoy dynamic metadata under `io.envoy.ai_gateway`, surfaced in access logs as `gen_ai.*` fields; OpenTelemetry GenAI metrics carry provider and model attributes and can be labelled with arbitrary request headers (for example a tenant header) via `controller.metricsRequestHeaderAttributes`. Per-team or per-key rollups are yours to build in Prometheus or your log store. Not verified

The dimensions the vendor documents for splitting spend — per key, per user, per team, per tag, per customer. Matters if you need to chargeback internally or bill an end customer.

How you get cost data out Cost export ?

No cost CSV, invoice or billing API - the project has no billing. Cost signals leave through Prometheus metrics, OTLP traces and Envoy access logs that you own ([metrics](https://aigateway.envoyproxy.io/docs/capabilities/observability/metrics); [tracing](https://aigateway.envoyproxy.io/docs/capabilities/observability/tracing); [access logs](https://aigateway.envoyproxy.io/docs/capabilities/observability/accesslogs)). Not verified

The mechanisms the vendor publishes for exporting cost and usage data: CSV, an API, webhooks, S3, a data warehouse, or nothing at all. Any per-unit price is included.

Who pays the model bill BYOK mode ?

Your keys only Verified Sep 2, 2026

Whether you bring your own provider accounts (BYOK), buy inference from this vendor, or can do either. This is the single biggest commercial difference between these products: it decides who holds the contract with the model provider and who carries the spend.

Catalog

Interpret these fields: LLM gateway model counts: what “500+” means

Models available Models available ?

Not published Not verified

How many models you can call, shown as a range because several vendors publish different totals on different pages. Vendor-reported either way, so counts are not directly comparable — some count every provider variant of the same model separately.

Model providers reachable Upstream providers ?

16–19 Verified Sep 2, 2026

How many distinct model providers or labs you can reach, shown as a range where the vendor’s own pages disagree. More providers usually means better redundancy when one has an outage.

Works with standard OpenAI code OpenAI-compatible API ?

Yes Verified Sep 2, 2026

If yes, you can usually switch to it by changing one base URL, and switch away just as easily. This is the main defence against lock-in.

OpenAI chat endpoint POST /v1/chat/completions ?

Yes Not verified

The endpoint almost every application ports first. "Not documented" means the vendor never states it, which is different from a documented no.

Anthropic messages endpoint POST /v1/messages ?

Yes Not verified

Whether Anthropic-shaped calls work without rewriting them. Several products support this only as SDK compatibility or provider passthrough rather than a native endpoint — the detail page says which.

OpenAI Responses endpoint POST /v1/responses ?

Yes Not verified

The newer stateful OpenAI surface. Support is much thinner across this market than chat completions.

Embeddings endpoint POST /v1/embeddings ?

Yes Not verified

Whether you can generate vectors through the same gateway, or need a second integration for retrieval workloads.

Image generation endpoint POST /v1/images/generations ?

Yes Not verified

Whether image models are reachable through the same surface as text.

Audio endpoints POST /v1/audio/* ?

Yes Not verified

Speech-to-text and text-to-speech. Frequently the first gap in an otherwise complete gateway.

Batch jobs endpoint POST /v1/batches ?

Not documented Not verified

Asynchronous bulk processing, usually at a discount. Commonly undocumented, and commonly the reason a migration stalls late.

Needs the vendor’s own code library Requires a vendor-specific SDK ?

No Verified Sep 2, 2026

A proprietary client library spreads through your codebase and has to be torn out again if you leave. “No” is the better answer here, and it means the standard OpenAI client works.

You can export your request history Logs / usage data export ?

Yes Not verified

Whether you can get your own request logs, traces, or usage records back out — through an API, a bulk export, or a download. Decides whether you leave with your history or abandon it.

Settings can live in version control Declarative config-as-code ?

Yes Not verified

Whether routing, fallback, and budget rules can be declared in a file you keep in Git, rather than existing only as settings clicked into a hosted dashboard.

Embeddings Embeddings ?

Yes Not verified

Text-to-vector models, needed for search and retrieval features.

Image generation Image generation ?

Yes Not verified

Whether image models are reachable through the same interface.

Speech and audio Speech and audio ?

Yes Not verified

Text-to-speech or transcription models through the same interface.

Video generation Video generation ?

Yes Not verified

Whether video models are reachable through the same interface.

Batch processing Batch processing ?

Not published Not verified

Submitting large jobs for cheaper, slower processing. Often 50% off for work that is not time-sensitive.

Routing & reliability

Interpret these fields: How LLM gateway failover actually works · Does your LLM gateway promise any uptime? · Is routing destroying your prompt cache? · Changing models without breaking production

Uptime it promises in writing Contractual SLA uptime ?

Not published Not verified

The uptime percentage in a published, contractual service level agreement. A public status page is not an SLA — it reports what happened, it does not promise anything or pay you back when it breaks.

Automatic failover Automatic failover ?

Yes Verified Sep 2, 2026

When a model provider goes down or rate-limits you, traffic moves to a backup automatically instead of returning errors to your users.

Load balancing Load balancing ?

Yes Verified Sep 2, 2026

Spreads requests across several providers or keys to raise your effective rate limit.

Rule-based routing Conditional routing ?

Yes Verified Sep 2, 2026

Send different requests to different models based on rules — for example a cheap model for free users and a strong model for paying ones.

Response caching Response caching ?

Not published Not verified

Reuses the answer when the exact same request comes in again, which cuts both cost and latency.

Similar-question caching Semantic cache ?

Not published Not verified

Reuses an answer when a new question means roughly the same thing as an earlier one. Saves far more than exact-match caching but can return subtly wrong answers if tuned loosely.

Where you set the timeout Request timeout surface ?

In config Not verified

Config-file only, and split across two APIs. Request timeouts come from Envoy Gateway's `BackendTrafficPolicy` (the fallback example sets `timeout: 30s` with `perRetry.backOff.baseInterval 100ms` and `maxInterval 10s`) ([provider fallback](https://aigateway.envoyproxy.io/docs/capabilities/traffic/provider-fallback)); v1.1 added an AI-specific `AIGatewayRouteRule.streamIdleTimeout` that triggers failover if it fires before the first token and returns 504 if it fires mid-stream ([v1.1 release notes](https://aigateway.envoyproxy.io/release-notes/v1.1), 2026-09-02). No default value for either is stated on the pages fetched.

Where a request timeout can be set: per request, in a config file, in the vendor dashboard, or nowhere. Nine of the twenty products documented here do not describe a request timeout at all, so the worst case of a hung upstream call is unknowable from the docs.

Where you set retries Retry policy surface ?

In config Not verified

Retries are Envoy Gateway's, configured in YAML next to the AI routes: the documented example sets `numRetries: 5`, `numAttemptsPerPriority: 1`, exponential backoff `100ms`/`10s`, `retryOn.httpStatusCodes: [500]` and triggers `connect-failure` and `retriable-status-codes`, with `numAttemptsPerPriority` controlling how many tries each priority group gets before failing over ([provider fallback](https://aigateway.envoyproxy.io/docs/capabilities/traffic/provider-fallback), 2026-09-02).

Where retry count and backoff are configured. Worth knowing alongside billing: a retried streaming call can be charged more than once.

Where you set fallbacks Fallback surface ?

In config Not verified

Ordered priority groups: `backendRefs` in an `AIGatewayRoute` rule carry `priority: 0`, `priority: 1` and so on, and traffic moves to the next priority when the current one exhausts its attempts. Because the ext_proc filter chain is split into router-level and upstream-level filters, a failover to a different provider re-runs request translation and upstream auth for the new backend - which is what makes cross-provider fallback work rather than just cross-endpoint retry ([provider fallback](https://aigateway.envoyproxy.io/docs/capabilities/traffic/provider-fallback); [data plane](https://aigateway.envoyproxy.io/docs/concepts/architecture/data-plane), 2026-09-02). Weighted fallback is not documented.

Where the fallback chain is defined. Almost every product claims fallback; the useful question is whether you can change it from code or only by hand in a dashboard.

Shape of the fallback chain Fallback shape ?

Ordered list Not verified

An ordered list tries targets in sequence; a weighted split sends a percentage of traffic to each, which is what you need to trial a new model on 5% of requests. Weighted splits are much rarer than the marketing implies.

Upstream health tracking Health checks / circuit breaking ?

Not documented Not verified

n.a. - no active health check or circuit breaker for AI backends on any page fetched ([provider fallback](https://aigateway.envoyproxy.io/docs/capabilities/traffic/provider-fallback), [system architecture](https://aigateway.envoyproxy.io/docs/concepts/architecture/system-architecture), [data plane](https://aigateway.envoyproxy.io/docs/concepts/architecture/data-plane), [capabilities index](https://aigateway.envoyproxy.io/docs/)). Failure detection is reactive: retries and priority failover on connect failures and retriable status codes. The one exception is InferencePool, where an endpoint picker selects self-hosted endpoints from live metrics such as KV-cache usage and queue depth ([InferencePool support](https://aigateway.envoyproxy.io/docs/capabilities/inference/inferencepool-support), 2026-09-02).

Whether the product notices a failing upstream and stops sending traffic to it, and whether you can tune the thresholds. This is what turns a provider outage into a blip rather than a sustained error rate, and only four of the twenty expose it.

Cross-region failover you control Multi-region failover surface ?

Not documented Not verified

n.a. - no cross-region or multi-cluster failover construct is documented; the two-tier gateway pattern in the README is about tier-one ingress versus tier-two self-hosted model access, not geography ([README](https://raw.githubusercontent.com/envoyproxy/ai-gateway/main/README.md); [provider fallback](https://aigateway.envoyproxy.io/docs/capabilities/traffic/provider-fallback), 2026-09-02).

Whether you can define what happens when a region degrades. A vendor running many regions is not the same as a vendor letting you configure failover between them; only three document a user-controlled mechanism.

Where you set load balancing Load balancing surface ?

In config Not verified

Config-file. Multiple `backendRefs` at the same priority are load-balanced by Envoy, and Envoy Gateway's `BackendTrafficPolicy` governs the algorithm ([provider fallback](https://aigateway.envoyproxy.io/docs/capabilities/traffic/provider-fallback)); for self-hosted fleets an InferencePool endpoint picker does inference-aware selection on real-time KV-cache usage, queued requests and LoRA adapter state ([InferencePool support](https://aigateway.envoyproxy.io/docs/capabilities/inference/inferencepool-support), 2026-09-02).

Where traffic distribution across upstreams or keys is configured.

Operations

Interpret these fields: LLM gateway observability: traces, logs and export · Running coding agents through an LLM gateway · Changing models without breaking production

Usage dashboards and logs Observability ?

Yes Verified Sep 2, 2026

Built-in visibility into what was sent, what came back, what it cost, and how long it took.

Spending limits Budget controls ?

Yes Verified Sep 2, 2026

Hard caps that stop spend before it becomes a surprise invoice. The single most valuable control for a small team.

Rate limits Rate limits ?

Yes Verified Sep 2, 2026

Caps on request volume per key or per user, useful for protecting against abuse and runaway loops.

Separate keys per team or app Virtual keys ?

Not published Not verified

Issue scoped keys with their own budgets and permissions so you can attribute cost and revoke access without rotating everything.

Prompt versioning Prompt management ?

Not published Not verified

Store and version prompts outside your code so they can be changed without a deploy.

Quality testing Evals ?

Not published Not verified

Built-in tooling to score model output against test cases, so you can tell whether a model swap made things better or worse.

MCP support MCP support ?

Yes Verified Sep 2, 2026

Native support for the Model Context Protocol, the emerging standard for connecting models to external tools.

What gets logged Logged content ?

Your choice Verified Sep 2, 2026

Content capture is a switch, and the default depends on which semantic convention you pick. OpenInference (the default) records inputs and outputs unless you set `OPENINFERENCE_HIDE_INPUTS`, `OPENINFERENCE_HIDE_OUTPUTS`, `OPENINFERENCE_HIDE_EMBEDDINGS_TEXT` or `OPENINFERENCE_HIDE_EMBEDDINGS_VECTORS`; switching to `AI_GATEWAY_TRACING_SEMCONV=gen_ai` makes content opt-in via `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true` ([tracing](https://aigateway.envoyproxy.io/docs/capabilities/observability/tracing), 2026-09-02).

Whether full prompts and responses are stored, only metadata, or nothing. Full-body logging is the most useful debugging feature here and the one most likely to need a conversation with your compliance team.

You can turn logging off Body-logging opt-out ?

Yes Verified Sep 2, 2026

Yes, and it is total: with no OTLP endpoint configured nothing is exported at all, and per-field redaction flags plus the `gen_ai` semantic-convention mode let you keep spans without message content ([tracing](https://aigateway.envoyproxy.io/docs/capabilities/observability/tracing), 2026-09-02).

Whether prompt and response bodies can be suppressed while still keeping usage metrics. Nineteen of the twenty document a way to do this; the mechanisms range from a per-request header to an organisation-wide setting.

Traces you can take elsewhere Distributed tracing ?

OpenTelemetry Verified Sep 2, 2026

Full OpenTelemetry: set `OTEL_EXPORTER_OTLP_ENDPOINT` (globally or per-Gateway through `GatewayConfig.spec.extProc.kubernetes.env`) and the ext_proc emits spans, defaulting to **OpenInference** conventions with `AI_GATEWAY_TRACING_SEMCONV=gen_ai` as the alternative; MCP `CallTool` and `ListTools` operations get spans too, and a `session.id` header mapping groups multi-turn conversations. Arize Phoenix is the documented consumer, giving LLM-as-judge evaluation over production spans ([tracing](https://aigateway.envoyproxy.io/docs/capabilities/observability/tracing); [gateway config](https://aigateway.envoyproxy.io/docs/capabilities/gateway-config), 2026-09-02).

Whether the product emits OpenTelemetry, a proprietary format, or nothing. OpenTelemetry means the traces land in the tooling you already run instead of only in the vendor’s dashboard.

Where telemetry can go Export destinations ?

OpenTelemetry (OTLP), Prometheus, Envoy access logs, Arize Phoenix, Grafana Not verified

OTLP for traces to any collector ([tracing](https://aigateway.envoyproxy.io/docs/capabilities/observability/tracing)); Prometheus scrape for GenAI token and latency metrics, with custom labels lifted from request headers ([metrics](https://aigateway.envoyproxy.io/docs/capabilities/observability/metrics)); Envoy access logs carrying `gen_ai.*` token costs and MCP fields such as `mcp_tool_name` and `mcp_backend`, configured on `EnvoyProxy.spec.telemetry.accessLog` ([access logs](https://aigateway.envoyproxy.io/docs/capabilities/observability/accesslogs)). Named downstreams in the docs are Arize Phoenix via its Helm chart and a bundled Grafana dashboard at `examples/monitoring/grafana-dashboard.json` ([tracing](https://aigateway.envoyproxy.io/docs/capabilities/observability/tracing); [v1.1 release notes](https://aigateway.envoyproxy.io/release-notes/v1.1), 2026-09-02).

Documented sinks for logs and metrics. This is a good proxy for how replaceable the vendor’s own dashboard is: around twenty destinations means you never have to depend on it, while a CSV download means you do.

Can record user feedback Feedback capture API ?

No Not verified

n.a. - no thumbs-up/score ingestion API or feedback header on any page fetched ([tracing](https://aigateway.envoyproxy.io/docs/capabilities/observability/tracing); [metrics](https://aigateway.envoyproxy.io/docs/capabilities/observability/metrics); [API reference](https://aigateway.envoyproxy.io/docs/api/), 2026-09-02).

Whether there is an API to attach a rating or score to a logged request, which is what lets production traffic feed quality work later.

Scores live traffic Online eval hooks ?

Partly Verified Sep 2, 2026

Partial and second-hand: the gateway emits OpenInference-compatible spans with full message content, and the docs walk you through deploying Arize Phoenix so "LLM-as-judge" evaluations can run against production spans - the evaluation itself belongs to Phoenix, not to the gateway, which ships no dataset, scorer or experiment surface ([tracing](https://aigateway.envoyproxy.io/docs/capabilities/observability/tracing), 2026-09-02).

Whether automated scorers can run against real production requests, rather than only against a test set you assemble yourself.

Performance

Delay it adds Proxy overhead ?

2 ms Verified Sep 2, 2026

Extra time the product itself adds to each request, on top of however long the model takes. Usually irrelevant next to multi-second model latency, but it matters for high-volume or streaming-sensitive workloads.

Requests per second ceiling Throughput ?

Not published Not verified

Published sustained request rate before the product becomes the bottleneck. Only relevant at genuinely high volume.

What the request path runs on Architecture class ?

Compiled binary Verified Sep 2, 2026

Go control plane and Go ext_proc filter alongside the Envoy C++ data plane: repo language bytes are Go 7,634,320, MDX 1,427,966, CSS 44,361 and TypeScript 43,527, built against Go 1.26.4 ([GitHub API](https://api.github.com/repos/envoyproxy/ai-gateway); [v1.1 release notes](https://aigateway.envoyproxy.io/release-notes/v1.1), 2026-09-02). Requests traverse Envoy Proxy plus an External Processor container that does model extraction, routing, upstream auth, request/response translation and token accounting, with a Rate Limit Service and Redis for token limits; the filter chain is deliberately split into router-level and upstream-level stages so a retry to another provider re-runs translation and auth. Dynamic Modules are noted as a possible future alternative to ext_proc ([system architecture](https://aigateway.envoyproxy.io/docs/concepts/architecture/system-architecture); [data plane](https://aigateway.envoyproxy.io/docs/concepts/architecture/data-plane)).

The comparable way to talk about latency here. An edge worker, a compiled Go or Rust binary, and a Python proxy have different overhead floors no matter which figures each vendor publishes. Products that never disclose their runtime are recorded as undisclosed rather than assumed.

You can run the request path yourself Self-hostable data plane ?

Yes Verified Sep 2, 2026

Everything is published: Helm charts as OCI artifacts (`oci://docker.io/envoyproxy/ai-gateway-crds-helm` and `oci://docker.io/envoyproxy/ai-gateway-helm`, pinned in the docs to `--version v1.0.0`) ([installation](https://aigateway.envoyproxy.io/docs/getting-started/installation)); a CLI container image at `envoyproxy/ai-gateway-cli` and per-release `aigw` binaries for Linux and macOS on the GitHub releases page, or `go install ./cmd/aigw` from source ([CLI installation](https://aigateway.envoyproxy.io/docs/cli/aigwinstall), 2026-09-02); the v1.1.0 release itself is on GitHub ([releases API](https://api.github.com/repos/envoyproxy/ai-gateway/releases/latest)).

Whether the component that actually carries your prompts can run on your own infrastructure. Distinct from a vendor offering a self-hosted control plane while still proxying traffic through their network.

Streaming responses Streaming support ?

Yes Verified Sep 2, 2026

Yes - SSE streaming is listed as supported on chat completions and Anthropic messages, with translation preserved across providers including tool use and reasoning blocks ([supported endpoints](https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/supported-endpoints); [v1.0 release notes](https://aigateway.envoyproxy.io/release-notes/v1.0)). Two operational caveats the docs raise: Envoy Gateway's default 32 KB buffer limit "is not enough for most AI model responses", so the example manifest raises it to 50 MB via `ClientTrafficPolicy` ([basic usage](https://aigateway.envoyproxy.io/docs/getting-started/basic-usage)); and `streamIdleTimeout` returns 504 if it fires mid-stream, while firing before the first token triggers failover instead ([v1.1 release notes](https://aigateway.envoyproxy.io/release-notes/v1.1), 2026-09-02).

Whether token-by-token streaming is documented. The caveats matter more than the yes: some products cannot cancel a stream without still being billed, and several timeout and fallback mechanisms stop applying once the first token has been sent.

Security & compliance

Interpret these fields: LLM gateway compliance: SOC 2, HIPAA and evidence · How LLM gateway guardrails fail · Which LLM gateways store your prompts? · Do you need an MCP gateway as well?

Does your prompt reach their servers Prompt transits vendor ?

No Not verified

Prompts never reach a project-operated service. Traffic terminates in Envoy proxies and an ext_proc container inside your own cluster, and the control plane only reconciles CRDs and pushes xDS config ([system architecture](https://aigateway.envoyproxy.io/docs/concepts/architecture/system-architecture); [data plane](https://aigateway.envoyproxy.io/docs/concepts/architecture/data-plane), 2026-09-02). The only third parties on the path are the upstream providers you configure.

Whether the text you send passes through this company’s own infrastructure. If it does, every other promise on this page is a policy commitment rather than a physical impossibility. Self-hosted products can answer no outright.

What they keep if you change nothing Logging default ?

Metadata only, not content Verified Sep 2, 2026

Out of the box you get Prometheus metrics following OpenTelemetry GenAI semantic conventions - `gen_ai.client.token.usage`, `gen_ai.server.request.duration`, `gen_ai.server.time_to_first_token`, `gen_ai.server.time_per_output_token` - and Envoy access logs carrying token-cost dynamic metadata; that is metadata, not content ([metrics](https://aigateway.envoyproxy.io/docs/capabilities/observability/metrics); [access logs](https://aigateway.envoyproxy.io/docs/capabilities/observability/accesslogs), 2026-09-02). The load-bearing caveat is one level up: as soon as you set `OTEL_EXPORTER_OTLP_ENDPOINT`, tracing defaults to OpenInference conventions **with full request and response content included by default**, and every `OPENINFERENCE_HIDE_*` redaction flag defaults to false ([tracing](https://aigateway.envoyproxy.io/docs/capabilities/observability/tracing)).

Defaults matter more than options. A product that stores full prompts and replies unless you find the right header will have stored them by the time you read the docs.

How long they keep it Default content retention (days) ?

Not published Not verified

Retention is entirely yours - the project stores nothing and ships no datastore for request data. Metrics, traces and access logs land in the Prometheus, OTLP collector and log sinks you operate ([metrics](https://aigateway.envoyproxy.io/docs/capabilities/observability/metrics); [tracing](https://aigateway.envoyproxy.io/docs/capabilities/observability/tracing); [access logs](https://aigateway.envoyproxy.io/docs/capabilities/observability/accesslogs)).

Default retention for request content, in days. Zero means nothing is kept. Read the note: several products keep nothing as a rule but make timed exceptions for abuse review or specific models.

Could they train on your prompts Training on customer data ?

Not applicable Not verified

No vendor ever receives prompts, so there is nothing to train on; no training policy is published, and none would be meaningful for self-hosted Apache-2.0 software ([system architecture](https://aigateway.envoyproxy.io/docs/concepts/architecture/system-architecture)).

Whether the vendor may use your prompts and outputs to train models. “Not published” means we could not find any position, which is not the same as a no — ask for it in writing.

Where it runs, and what you can pin Region and residency control ?

Region is whatever your Kubernetes cluster and chosen upstreams are - there is no vendor region list. Kubernetes v1.32+ with Envoy Gateway v1.7.0+ is the only placement requirement stated ([prerequisites](https://aigateway.envoyproxy.io/docs/getting-started/prerequisites); [compatibility](https://aigateway.envoyproxy.io/docs/compatibility), 2026-09-02). Verified Sep 2, 2026

Which regions are offered and whether you can force processing to stay in one. A global endpoint that silently picks a region is a different compliance story from an endpoint you pin yourself.

Where safety filters run Guardrail execution location ?

No guardrails offered Verified Sep 2, 2026

There is no guardrail feature. The security documentation covers client-to-gateway access control by delegating to Envoy Gateway SecurityPolicy - JWT validation, JWT claim-based authorization, mTLS, external authorization, OIDC, basic auth, API key, IP allow/deny - and gateway-to-provider credential handling; content inspection is absent ([security](https://aigateway.envoyproxy.io/docs/capabilities/security/); [upstream auth](https://aigateway.envoyproxy.io/docs/capabilities/security/upstream-auth), 2026-09-02). No page in the site map carries the word guardrail, and neither the [v1.0](https://aigateway.envoyproxy.io/release-notes/v1.0) nor [v1.1](https://aigateway.envoyproxy.io/release-notes/v1.1) release notes introduce one.

A filter that strips personal data only helps if it runs before the data leaves your boundary. If guardrails execute in the vendor’s cloud, the vendor has already received whatever you wanted redacted.

Who else touches the data Subprocessor list ?

Not published Not verified

The published list of third parties the vendor passes your data to. No list means you cannot know the full chain, which most data-protection agreements require you to.

SOC 2 audited SOC 2 audited ?

Not published Not verified

An independent audit of security controls. Enterprise buyers and their procurement teams routinely require it.

Will sign a HIPAA agreement HIPAA BAA ?

Not published Not verified

Required before you may send protected health information through the service. Without a signed BAA, healthcare data is off limits.

GDPR commitments GDPR commitments ?

Not published Not verified

Published data processing terms for handling personal data of people in the EU and UK.

Can keep data in the EU EU data residency ?

Not published Not verified

Requests can be processed inside the EU rather than routed to US infrastructure. Often the deciding constraint for European customers.

Does not retain your data Zero data retention ?

Not applicable Not verified

Prompts and responses are not stored after the request completes. Sometimes a paid add-on rather than the default.

Strips personal data PII redaction ?

Not published Not verified

Detects and removes identifiers such as names, emails, and card numbers before the request reaches the model provider.

Content guardrails Content guardrails ?

Not published Not verified

Policy checks on inputs and outputs — blocking unsafe content, enforcing formats, or catching prompt-injection attempts.

Runs fully disconnected Air-gapped deployment ?

Not published Not verified

Can be deployed in a network with no internet access, which some regulated and defence environments require.

Blocks personal data in prompts PII / DLP enforcement ?

Not documented Not verified

n.a. - no PII detection, masking or redaction of request content on any page fetched ([security](https://aigateway.envoyproxy.io/docs/capabilities/security/); [header and body mutations](https://aigateway.envoyproxy.io/docs/capabilities/traffic/header-body-mutations); [v1.1 release notes](https://aigateway.envoyproxy.io/release-notes/v1.1)). The nearest thing is trace-level redaction: `OPENINFERENCE_HIDE_INPUTS` / `HIDE_OUTPUTS` / `HIDE_EMBEDDINGS_TEXT` / `HIDE_EMBEDDINGS_VECTORS` suppress prompt content in spans, all defaulting to false ([tracing](https://aigateway.envoyproxy.io/docs/capabilities/observability/tracing), 2026-09-02) - that protects your telemetry, not the model call.

Whether personal data detection sits on the request path and can stop the call, merely inspects and forwards it, or is not documented. A control that only reports is a logging feature, not a policy control.

Blocks prompt injection Injection / jailbreak enforcement ?

Not documented Not verified

n.a. - no prompt-injection or jailbreak detection documented ([security](https://aigateway.envoyproxy.io/docs/capabilities/security/); [capabilities index](https://aigateway.envoyproxy.io/docs/); [v1.1 release notes](https://aigateway.envoyproxy.io/release-notes/v1.1), 2026-09-02).

Whether injection and jailbreak detection can stop a request. Most products offering this call a partner classifier rather than shipping their own.

Blocks harmful content Toxicity / moderation enforcement ?

Not documented Not verified

n.a. - no moderation or toxicity filtering; nothing on the [security](https://aigateway.envoyproxy.io/docs/capabilities/security/) page or in the [v1.0](https://aigateway.envoyproxy.io/release-notes/v1.0) and [v1.1](https://aigateway.envoyproxy.io/release-notes/v1.1) release notes inspects prompt or completion content.

Whether hate, violence, sexual and self-harm categories are checked inline and can stop a request, in either direction.

Your own policy rules Custom policy hooks ?

Not documented Not verified

n.a. as a policy-driven content check. You can rewrite headers and bodies (`headerMutation`, `bodyMutation` on `AIServiceBackend`) and enforce arbitrary CEL expressions for MCP tool authorization ([header and body mutations](https://aigateway.envoyproxy.io/docs/capabilities/traffic/header-body-mutations); [MCP](https://aigateway.envoyproxy.io/docs/capabilities/mcp/)), and Envoy Gateway's external-authorization hook can front the gateway ([security](https://aigateway.envoyproxy.io/docs/capabilities/security/)) - but no custom guardrail evaluator is documented.

Whether you can add your own rule — a regex, a webhook, or your own classifier — rather than choosing from the vendor library.

Where guardrails run Guardrail execution location ?

Not documented Not verified

Whether guardrail evaluation happens inside your infrastructure or on the vendor’s servers. This decides whether the prompt you are trying to protect leaves your network in order to be checked.

If the guardrail itself fails Guardrail failure mode ?

Not documented Not verified

n.a. - no guardrail exists, so no fail-open/fail-closed behaviour is stated ([security](https://aigateway.envoyproxy.io/docs/capabilities/security/)). For the request path proper, the documented failure behaviour is Envoy retry and priority failover plus 429 on quota exhaustion ([provider fallback](https://aigateway.envoyproxy.io/docs/capabilities/traffic/provider-fallback); [quota policy](https://aigateway.envoyproxy.io/docs/capabilities/traffic/quota-policy)).

What happens when the guardrail service times out or errors: does the request proceed unchecked, or is it blocked? This is the worst-documented field in the entire catalogue — only two vendors state it plainly, which means most teams are running a control whose failure behaviour they cannot know.

Third-party guardrail vendors Guardrail integrations ?

Not published Not verified

Named external guardrail services the product can call. A long list means the product is a router for policy engines rather than a policy engine itself — which also means another vendor bill and another hop.

Fit & integration

Interpret these fields: How much does an LLM gateway lock you in? · Do you need an MCP gateway as well? · Running coding agents through an LLM gateway · Changing models without breaking production

Work to try it Evaluation work shape ?

Run something locally first Verified Sep 2, 2026

The shape of the work on the vendor’s own quickstart, from swapping one base URL through to deploying infrastructure. An ordinal class rather than a duration, because elapsed time depends on accounts and quota we cannot see.

Work to run it Production work shape ?

Deploy it on your infrastructure Verified Sep 2, 2026

The same scale applied to the vendor’s recommended production path. For several products this is much heavier than the quickstart, which is exactly why both are recorded.

Steps on the quickstart Numbered quickstart steps ?

4 Verified Sep 2, 2026

The getting-started guide numbers four sections (Prerequisites, Installation, Basic Usage, Connect Providers) rather than four commands; the Installation page itself has two numbered Helm steps and Prerequisites adds the Envoy Gateway install, so there is no single steps-to-first-call figure ([getting started](https://aigateway.envoyproxy.io/docs/getting-started/); [installation](https://aigateway.envoyproxy.io/docs/getting-started/installation), 2026-09-02).

A literal count of numbered steps on the vendor’s quickstart, recorded as evidence beside the work shape. Zero means the page publishes no numbered procedure at all. Large counts usually mean interleaved language tracks rather than more work.

Can you self-host it today Self-host install documentation ?

Install command published Not verified

`helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm --version v1.0.0 --namespace envoy-ai-gateway-system --create-namespace` then `helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm --version v1.0.0 --namespace envoy-ai-gateway-system --create-namespace`, after Envoy Gateway v1.7.0+; locally `docker run --rm -p 1975:1975 -e OPENAI_API_KEY=... envoyproxy/ai-gateway-cli run` or `go install ./cmd/aigw`

Whether an install command is actually published. Several products advertise self-hosting while publishing no command to start from, which a plain yes/no would hide.

Works with the OpenAI SDK OpenAI SDK drop-in ?

Yes Verified Sep 2, 2026

Yes: OpenAI paths are served at the gateway root, the quickstart is a plain curl to `$GATEWAY_URL/v1/chat/completions`, and `aigw run` intentionally consumes OpenAI SDK environment variables so an existing client only needs its base URL changed ([supported endpoints](https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/supported-endpoints); [basic usage](https://aigateway.envoyproxy.io/docs/getting-started/basic-usage); [aigw run](https://aigateway.envoyproxy.io/docs/cli/aigwrun), 2026-09-02).

Whether an existing OpenAI-compatible client can be pointed at it by changing the base URL and key.

Vercel AI SDK support AI SDK provider package ?

Via the OpenAI provider Verified Sep 2, 2026

No Vercel AI SDK provider package or integration page exists - the term appears nowhere on the pages fetched or in the site map. What is documented is generic OpenAI compatibility at the gateway root, which is what an `@ai-sdk/openai` `baseURL` override would target ([supported endpoints](https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/supported-endpoints); [aigw run](https://aigateway.envoyproxy.io/docs/cli/aigwrun), 2026-09-02).

Whether a first-party AI SDK provider package exists, or only a community package, a documented workaround, or the generic OpenAI provider pointed at a custom base URL.

Python framework integrations Documented Python frameworks ?

Not published Not verified

n.a. - no LangChain, LlamaIndex, DSPy or other framework integration is documented; the 469-URL site map contains no framework page and none of the docs pages fetched name one ([capabilities index](https://aigateway.envoyproxy.io/docs/); [supported endpoints](https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/supported-endpoints), 2026-09-02). In practice you would use each framework's OpenAI-compatible client with the gateway's base URL.

LangChain, LangGraph, LlamaIndex and similar orchestration frameworks with a documented integration.

Callable from Cloudflare Workers Cloudflare Workers support ?

Not documented Not verified

n.a. - no Cloudflare Workers or edge-runtime story; the data plane is Envoy in your cluster or the `aigw` binary on Linux/macOS ([data plane](https://aigateway.envoyproxy.io/docs/concepts/architecture/data-plane); [aigw run](https://aigateway.envoyproxy.io/docs/cli/aigwrun), 2026-09-02). Cloudflare is not even listed as an upstream provider ([supported providers](https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/supported-providers)).

Whether the docs show calling this product from your own Worker. Deliberately separated from the several products whose own gateway runs on Workers, which is a fact about their infrastructure and not about your edge compatibility.

Kubernetes install Helm chart availability ?

Official Helm chart Verified Sep 2, 2026

Kubernetes is the primary target, not a deployment option: two official OCI Helm charts (CRDs and controller), a documented CRD-ownership migration with `--take-ownership`, controller leader election with a horizontally scalable read-only extension server, an HPA example (70% CPU, min 2 / max 10), and v1.1 Helm additions for PodDisruptionBudget, topology spread constraints, pod labels and a restricted controller security context ([installation](https://aigateway.envoyproxy.io/docs/getting-started/installation); [scaling](https://aigateway.envoyproxy.io/docs/capabilities/scaling); [v1.1 release notes](https://aigateway.envoyproxy.io/release-notes/v1.1), 2026-09-02).

Whether a named, published Helm chart exists, versus Helm being referenced with no chart named, versus generic cluster documentation that has nothing to do with this product.

Terraform support Terraform provider or modules ?

Not documented Not verified

n.a. - no Terraform provider, module or example is published: nothing in the repository root or the 469-URL site map mentions Terraform, and the documented install path is Helm ([installation](https://aigateway.envoyproxy.io/docs/getting-started/installation); [GitHub repo](https://github.com/envoyproxy/ai-gateway), 2026-09-02).

Whether you can declare this in code: an official provider, official modules, resources inside a hyperscaler’s provider, a community provider, or only Terraform code shipped in a repo.

Reuses your cloud identity Cloud IAM reuse ?

Reuses your cloud identity Verified Sep 2, 2026

Native cloud identity on all three hyperscalers: AWS Bedrock through the default credential chain including EKS Pod Identity and IRSA (region only, no key material) or OIDC-to-STS temporary credentials; Azure OpenAI through Entra ID short-lived tokens; GCP Vertex AI through Application Default Credentials, service-account keys or Workload Identity Federation with Google STS ([connect providers](https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/connect-providers); [upstream auth](https://aigateway.envoyproxy.io/docs/capabilities/security/upstream-auth); [supported providers](https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/supported-providers), 2026-09-02).

Whether you can authenticate with IAM roles, workload identity or managed identities instead of another long-lived API key. Distinguished from products that only accept static upstream provider credentials.

Fits behind your API gateway API gateway integration ?

It is the API gateway Verified Sep 2, 2026

It is the gateway - but uniquely among these entries it is not a standalone one: Envoy Gateway is a hard prerequisite (v1.7.0+ for install, v1.8.1+ in the v1.0.x compatibility matrix) and AI Gateway is an extension server plus ext_proc filter on top of it, described as "an additive layer" ([prerequisites](https://aigateway.envoyproxy.io/docs/getting-started/prerequisites); [compatibility](https://aigateway.envoyproxy.io/docs/compatibility); [GOALS.md](https://raw.githubusercontent.com/envoyproxy/ai-gateway/main/GOALS.md), 2026-09-02).

Whether AI traffic can go through a gateway you already run, and who documented that — the product’s vendor or the gateway’s.

MCP support MCP surface shape ?

MCP gateway or registry Verified Sep 2, 2026

A genuine MCP gateway, configured with an `MCPRoute` CRD: multiple MCP servers multiplexed behind one endpoint, tool names namespaced by backend (`github__issue_read`), tool filtering by name or regex, OAuth 2.0 authorization-code flow with PKCE plus JWT scope/claim and CEL-based authorization with a configurable `defaultAction`, header forwarding, upstream API-key injection, streamable HTTP per the June 2025 MCP spec, and OTel spans plus Prometheus metrics for MCP traffic ([MCP](https://aigateway.envoyproxy.io/docs/capabilities/mcp/), 2026-09-02). The same thing runs locally via `aigw run --mcp-config` on `http://localhost:1975/mcp` ([aigw run](https://aigateway.envoyproxy.io/docs/cli/aigwrun)). A dedicated `MCPBackend` CRD is on the post-1.0 roadmap ([1.0 announcement](https://aigateway.envoyproxy.io/blog/v1.0-release-announcement)).

Which kind of MCP support this is: a gateway that governs many MCP servers, a hosted MCP server you connect to, MCP tools accepted inside the completion API, or client tooling. These are different products behind one acronym.

Needs your own provider key Upstream provider key required ?

Required Not verified

Yes for any real traffic: each `AIServiceBackend` needs a `BackendSecurityPolicy` with your provider credential ([connect providers](https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/connect-providers)). The exception is evaluation - the quickstart's `basic.yaml` routes to a mock backend (`some-cool-self-hosted-model`) so you can make a first call with no provider account at all ([basic usage](https://aigateway.envoyproxy.io/docs/getting-started/basic-usage), 2026-09-02).

Whether an upstream provider account and key must exist before your first call works. A prerequisite rather than a step, and it can differ between the hosted and self-hosted forms of the same product.

Gate before models work Model access gate ?

No gate Verified Sep 2, 2026

None: no approval, waitlist, enablement or quota step exists on the project side - whatever your provider credential can reach is reachable once you declare it in a route ([connect providers](https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/connect-providers); [supported endpoints](https://aigateway.envoyproxy.io/docs/capabilities/llm-integrations/supported-endpoints), 2026-09-02). Any gating you experience is your upstream provider's.

Whether an enablement click, a quota grant, a paid tier or an approval form stands between a valid key and a working model call.

Official client languages First-party client SDK languages ?

Not published Not verified

No client library is published or required. The documented pattern is pointing an OpenAI-compatible client at the gateway; `aigw run` reads the OpenAI SDK's environment variables so existing code needs no change, and the InferencePool page cites "seamless integration with OpenAI SDKs" ([aigw run](https://aigateway.envoyproxy.io/docs/cli/aigwrun); [InferencePool support](https://aigateway.envoyproxy.io/docs/capabilities/inference/inferencepool-support), 2026-09-02). No per-language SDK list, snippet set or package name appears on the pages fetched, so `fit_client_sdk_langs` is left empty rather than inferred from the curl examples.

Languages with a first-party client library. An empty list can still mean the product is usable from any language via an OpenAI-compatible SDK.