Higress Open source

Overview

Interpret these fields: Self-hosted vs managed LLM gateways · Who still owns your LLM gateway?

What kind of product Category ?

Open source Not verified

Marketplaces resell many providers behind one key. Gateways add governance on top. Open-source projects you run yourself. Cloud platforms are hyperscaler surfaces. Inference providers host models on their own hardware.

Who runs it Deployment model ?

Managed or self-host Verified Sep 2, 2026

Managed means the vendor operates it. Self-host means you run it on your own infrastructure. Both means you can choose.

Licence Licence ?

Apache-2.0 Verified Sep 2, 2026

Proprietary products cannot be inspected or forked. Open licences such as MIT and Apache-2.0 let you audit, modify, and run the code without permission.

Company Company ?

Alibaba Group; governed as a CNCF Sandbox project ("Copyright Higress a Series of LF Projects, LLC") Verified Sep 2, 2026

The organisation that maintains the product.

Who you would be signing with Vendor status ?

Run by a software foundation Not verified

Whether the product is still an independent company, has been acquired, is a large cloud vendor’s product line, is run by a software foundation, or has been put into maintenance mode. Maintenance mode means bug fixes and security patches only — no new features.

Last shipped an update Latest release ?

2026-08-13 Verified Sep 2, 2026

Latest non-prerelease is v2.2.4, published 2026-08-13T23:49:46Z via the GitHub releases API; the default branch was last pushed 2026-09-02T12:30:09Z ([releases/latest](https://api.github.com/repos/higress-group/higress/releases/latest)).

The date of the most recent release or version tag. A product that has not shipped in a year is a different risk from one that shipped last week, regardless of what its marketing site says.

GitHub stars GitHub stars ?

9,458 Verified Sep 26, 2026

A rough proxy for community size on open-source projects. Not a quality measure.

Cost

Interpret these fields: How LLM gateway pricing works · LLM gateway spending limits: stop a runaway agent bill?

Markup on model prices Token markup ?

None Not verified

How much the product adds on top of what the underlying model provider charges. Zero means you pay the same per-token price you would pay the model provider directly.

Fee to add funds Credit purchase fee ?

Not published Not verified

A percentage charged when you top up your balance, separate from token prices. It is easy to miss because it does not appear on the per-token price list.

Monthly cost per person Seat fee ?

Not published Not verified

A recurring per-user platform charge that applies regardless of how much you use the models.

Can use your own provider accounts BYOK supported ?

Yes Not verified

Bring Your Own Key: you keep direct contracts with OpenAI, Anthropic and others, and the gateway only routes traffic. This preserves negotiated rates and committed-spend discounts.

Cost of using your own accounts BYOK terms ?

No fee of any kind: the software is Apache-2.0 and nothing meters your traffic, so BYOK is the only mode and it costs nothing beyond your provider bills and your own compute ([GitHub API](https://api.github.com/repos/higress-group/higress), [ai-proxy plugin](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-proxy/)). Not verified

What the product charges to route traffic through your own provider keys.

Free tier Free tier ?

The entire gateway is free: Apache-2.0 licensed ([GitHub API](https://api.github.com/repos/higress-group/higress)) with the Community edition described as "Free & Open Source, Community Support" and "Local deployment, verify core capabilities in 5 minutes" ([AI gateway editions](https://higress.ai/en/ai-gateway/)). There is no request, seat or token cap in the open-source edition, and no Higress account exists to create. Not verified

What you can do without paying, useful for evaluation.

Enterprise plan from Enterprise plan from ?

Not published Not verified

Annual entry price for the enterprise tier, where one is published or credibly reported.

Cost to run it yourself Self-host cost ?

Your own infrastructure only — no licence, no seat fee, no metered tier. Sizing signals from the docs: gateway defaults to 2 replicas, the console to 1, and the optional o11y stack adds Grafana, Prometheus, Loki and PromTail to the same cluster ([Helm values](https://higress.ai/en/docs/latest/ops/deploy-by-helm/)). Redis is a hard dependency for ai-cache, ai-token-ratelimit, ai-quota and MCP hosting, so budget for it ([ai-cache](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-cache/), [ai-token-ratelimit](https://higress.ai/en/docs/latest/plugins/ai/api-consumer/ai-token-ratelimit/), [MCP quick start](https://higress.ai/en/docs/ai/mcp-quick-start/)). Autoscaling is your job: "Higress is based on K8s HPA and supports elastic scaling. The gateway is stateless and is a deployment." ([FAQ](https://higress.ai/en/docs/latest/overview/faq/)) Not verified

What self-hosting actually costs once you account for infrastructure and any paid tier.

How the vendor makes money Pricing model ?

Open source with a managed tier Verified Sep 2, 2026

The shape of the vendor’s bill: does the routing layer charge a percentage on top of tokens, a flat monthly fee, both, neither (because inference is the product), or nothing at all (open source with no paid tier).

How pricing works, briefly Pricing model detail ?

Apache-2.0 software with no price of its own, alongside a separately-sold managed edition: the AI gateway page presents Community ("Free & Open Source, Community Support", local deployment), Serverless via Alibaba Cloud AI Gateway, and a Feitian Exclusive Edition with negotiable SLA and ticket/DingTalk support ([AI gateway editions](https://higress.ai/en/ai-gateway/)). The quick start states "Serverless Standard starts at ¥0 and charges only for actual usage" and recommends Alibaba Cloud AI Gateway Enterprise for production without Kubernetes ([quick start](https://higress.ai/en/docs/latest/user/quickstart/)). No dollar or yuan rate card is published on the Higress site itself, and the managed edition is an Alibaba Cloud product, not a Higress-branded SaaS. Not verified

A one-paragraph description that covers the caveats a pricing category cannot: introductory rates, per-feature meters, tier gating, and pricing that resets on a specific date.

Minimum commitment Minimum commitment ?

None for the open-source edition — no account, contract or licence key is involved ([GitHub API](https://api.github.com/repos/higress-group/higress)). The managed alternatives are Alibaba Cloud commitments: Serverless Standard "starts at ¥0 and charges only for actual usage", while the Feitian Exclusive Edition is a negotiated commercial agreement ([quick start](https://higress.ai/en/docs/latest/user/quickstart/), [AI gateway editions](https://higress.ai/en/ai-gateway/)). Not verified

Whether the vendor requires a minimum contract term, a minimum spend, or a provisioned-capacity purchase to get its published rate.

Charges that fire after you go over an allowance Overage terms ?

n.a. — there is no meter to overrun. Higress bills nothing; the costs you incur are your own infrastructure plus the upstream provider bills against the API keys you configure in ai-proxy ([AI gateway editions](https://higress.ai/en/ai-gateway/), [ai-proxy plugin](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-proxy/)). Not verified

The line items that scale with usage after an included allowance is exhausted — log storage, extra requests, per-feature meters, data export — which is where cost estimates usually go wrong.

Prompt cache offered Cache mechanism ?

Both exact and semantic Verified Sep 2, 2026

Whether the gateway offers its own response cache, what kind of match it does (exact request, prefix, semantic), or simply passes provider caching through unchanged.

Discount on cached input Cache-read discount ?

Not published Not verified

How much cheaper cached tokens are than fresh input, when the vendor publishes a single figure. Bundled-inference clouds usually price this per model instead of as one number.

Premium on cache writes Cache-write premium ?

Not published Not verified

How much more the first write of a cached prefix costs versus a plain input token. A high write premium and a low hit rate can leave you paying more than you save, so this matters as much as the read discount.

Who captures the cache saving Cache economics ?

Both shapes, self-hosted, and Redis is mandatory. The ai-cache plugin does exact-match caching keyed by default on `messages.@reverse.0.content` with prefix `higress-ai-cache:` and `cacheTTL` defaulting to 0, i.e. never expire; `redis.serviceName` is required and `x-higress-skip-ai-cache: on` bypasses the cache per request ([ai-cache plugin](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-cache/)). Semantic caching is the same plugin pointed at an embedding service plus a vector database — the worked example uses Alibaba Cloud BaiLian `text-embedding-v3` at 1024 dimensions with DashVector and Cosine distance ([semantic cache guide](https://higress.ai/en/docs/ai/scene-guide/semantic-cache/)). No cached-token pricing exists because no vendor meters tokens here; a hit simply skips the upstream call, so you save the provider's own bill. Not verified

Whether the customer keeps the full saving from caching or the vendor captures part of it — and any conditions attached (write premium, storage fees, best-effort hits).

What you can split spend by Cost attribution ?

Per gateway, route, service and model: ai-statistics exposes input tokens, output tokens, time to first token (streaming) and total request time "observable across four dimensions: gateway, route, service, model", with `input_token`, `output_token`, `model`, `llm_service_duration` and `llm_first_token_duration` in the log line ([ai-statistics](https://higress.ai/en/docs/latest/plugins/ai/api-o11y/ai-statistics/)). Per-consumer attribution comes from key-auth's `X-Mse-Consumer` header plus ai-quota's per-consumer Redis counters ([key-auth](https://higress.ai/en/docs/latest/user/plugins/authentication/key-auth/), [ai-quota](https://higress.ai/en/docs/latest/user/plugins/ai/api-consumer/ai-quota/)). Attribution is in tokens, not currency — no price table or cost field is computed anywhere. Not verified

The dimensions the vendor documents for splitting spend — per key, per user, per team, per tag, per customer. Matters if you need to chargeback internally or bill an end customer.

How you get cost data out Cost export ?

n.a. — no CSV, billing API, webhook, S3 or warehouse cost export on [ai-statistics](https://higress.ai/en/docs/latest/plugins/ai/api-o11y/ai-statistics/), [ai-quota](https://higress.ai/en/docs/latest/user/plugins/ai/api-consumer/ai-quota/) or [the token management guide](https://higress.ai/en/docs/ai/scene-guide/token-management/). What you can export is token telemetry, via Prometheus scraping and OTLP ([Prometheus guide](https://higress.ai/en/docs/latest/user/prometheus/)). Not verified

The mechanisms the vendor publishes for exporting cost and usage data: CSV, an API, webhooks, S3, a data warehouse, or nothing at all. Any per-unit price is included.

Who pays the model bill BYOK mode ?

Your keys only Verified Sep 2, 2026

Whether you bring your own provider accounts (BYOK), buy inference from this vendor, or can do either. This is the single biggest commercial difference between these products: it decides who holds the contract with the model provider and who carries the spend.

Catalog

Interpret these fields: LLM gateway model counts: what “500+” means

Models available Models available ?

~100 Verified Sep 2, 2026

How many models you can call, shown as a range because several vendors publish different totals on different pages. Vendor-reported either way, so counts are not directly comparable — some count every provider variant of the same model separately.

Model providers reachable Upstream providers ?

26–31 Verified Sep 2, 2026

How many distinct model providers or labs you can reach, shown as a range where the vendor’s own pages disagree. More providers usually means better redundancy when one has an outage.

Works with standard OpenAI code OpenAI-compatible API ?

Yes Verified Sep 2, 2026

If yes, you can usually switch to it by changing one base URL, and switch away just as easily. This is the main defence against lock-in.

OpenAI chat endpoint POST /v1/chat/completions ?

Yes Verified Sep 2, 2026

The endpoint almost every application ports first. "Not documented" means the vendor never states it, which is different from a documented no.

Anthropic messages endpoint POST /v1/messages ?

Yes Verified Sep 2, 2026

Whether Anthropic-shaped calls work without rewriting them. Several products support this only as SDK compatibility or provider passthrough rather than a native endpoint — the detail page says which.

OpenAI Responses endpoint POST /v1/responses ?

Not documented Verified Sep 2, 2026

The newer stateful OpenAI surface. Support is much thinner across this market than chat completions.

Embeddings endpoint POST /v1/embeddings ?

Yes Verified Sep 2, 2026

Whether you can generate vectors through the same gateway, or need a second integration for retrieval workloads.

Image generation endpoint POST /v1/images/generations ?

Not documented Verified Sep 2, 2026

Whether image models are reachable through the same surface as text.

Audio endpoints POST /v1/audio/* ?

Not documented Verified Sep 2, 2026

Speech-to-text and text-to-speech. Frequently the first gap in an otherwise complete gateway.

Batch jobs endpoint POST /v1/batches ?

Not documented Verified Sep 2, 2026

Asynchronous bulk processing, usually at a discount. Commonly undocumented, and commonly the reason a migration stalls late.

Needs the vendor’s own code library Requires a vendor-specific SDK ?

No Not verified

A proprietary client library spreads through your codebase and has to be torn out again if you leave. “No” is the better answer here, and it means the standard OpenAI client works.

You can export your request history Logs / usage data export ?

Yes Not verified

Whether you can get your own request logs, traces, or usage records back out — through an API, a bulk export, or a download. Decides whether you leave with your history or abandon it.

Settings can live in version control Declarative config-as-code ?

Yes Not verified

Whether routing, fallback, and budget rules can be declared in a file you keep in Git, rather than existing only as settings clicked into a hosted dashboard.

Embeddings Embeddings ?

Yes Not verified

Text-to-vector models, needed for search and retrieval features.

Image generation Image generation ?

Not published Not verified

Whether image models are reachable through the same interface.

Speech and audio Speech and audio ?

Not published Not verified

Text-to-speech or transcription models through the same interface.

Video generation Video generation ?

Not published Not verified

Whether video models are reachable through the same interface.

Batch processing Batch processing ?

Not published Not verified

Submitting large jobs for cheaper, slower processing. Often 50% off for work that is not time-sensitive.

Routing & reliability

Interpret these fields: How LLM gateway failover actually works · Does your LLM gateway promise any uptime? · Is routing destroying your prompt cache? · Changing models without breaking production

Uptime it promises in writing Contractual SLA uptime ?

Not published Not verified

The uptime percentage in a published, contractual service level agreement. A public status page is not an SLA — it reports what happened, it does not promise anything or pay you back when it breaks.

Automatic failover Automatic failover ?

Yes Not verified

When a model provider goes down or rate-limits you, traffic moves to a backup automatically instead of returning errors to your users.

Load balancing Load balancing ?

Yes Not verified

Spreads requests across several providers or keys to raise your effective rate limit.

Rule-based routing Conditional routing ?

Yes Not verified

Send different requests to different models based on rules — for example a cheap model for free users and a strong model for paying ones.

Response caching Response caching ?

Yes Not verified

Reuses the answer when the exact same request comes in again, which cuts both cost and latency.

Similar-question caching Semantic cache ?

Yes Not verified

Reuses an answer when a new question means roughly the same thing as an earlier one. Saves far more than exact-match caching but can return subtly wrong answers if tuned loosely.

Where you set the timeout Request timeout surface ?

In config Verified Sep 2, 2026

`config_file`, but the LLM request timeout is the weak spot: ai-proxy's `timeout` (default **120000** ms) is documented as applying only to the context-data retrieval call, not to the model request ([ai-proxy plugin](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-proxy/)). The request timeout you actually want is the Ingress annotation `higress.io/timeout`, in seconds, with **no timeout by default** ([annotations](https://higress.ai/en/docs/latest/user/annotation/)). Connection-level values live in the ConfigMap: downstream `idleTimeout` 180 s, upstream `idleTimeout` 10 s ([global configuration](https://higress.ai/en/docs/latest/user/configmap/)). MCP proxying has its own `server.timeout`, default 5000 ms ([mcp-server plugin](https://higress.ai/en/docs/ai/mcp-server/)).

Where a request timeout can be set: per request, in a config file, in the vendor dashboard, or nowhere. Nine of the twenty products documented here do not describe a request timeout at all, so the worst case of a hung upstream call is unknowable from the docs.

Where you set retries Retry policy surface ?

In config Verified Sep 2, 2026

Off by default at the AI layer and capped at one extra attempt when switched on; no backoff strategy or jitter is documented anywhere ([ai-proxy retryOnFailure](https://higress.cn/docs/latest/plugins/ai/api-provider/ai-proxy/)). The generic Ingress retry annotation is separate and retries to a different upstream host rather than to a different model ([annotations](https://higress.ai/en/docs/latest/user/annotation/)). No retry keys appear in the global ConfigMap ([global configuration](https://higress.ai/en/docs/latest/user/configmap/)).

Where retry count and backoff are configured. Worth knowing alongside billing: a retried streaming call can be charged more than once.

Where you set fallbacks Fallback surface ?

Dashboard only Verified Sep 2, 2026

Model-level fallback is a console concept, not a plugin field: the home page states Higress "supports model-level Fallback" ([home page](https://higress.ai/en/)) and the multi-model proxy guide configures it in `AI Route Config`, where a DeepSeek route falls back to Alibaba Cloud `qwen-turbo` when the primary model fails or is rate-limited ([multi-model proxy guide](https://higress.ai/en/docs/ai/scene-guide/multi-proxy/)). The guide shows a single fallback target, not an ordered chain, and no YAML/CRD field for it appears in [ai-proxy (EN)](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-proxy/) or [ai-proxy (CN)](https://higress.cn/docs/latest/plugins/ai/api-provider/ai-proxy/). Alibaba Cloud's own comparison confirms cross-model failover exists in the open-source edition ("多模型间 Failover: 支持") ([Alibaba Cloud comparison](https://help.aliyun.com/zh/api-gateway/ai-gateway/product-overview/product-comparison)).

Where the fallback chain is defined. Almost every product claims fallback; the useful question is whether you can change it from code or only by hand in a dashboard.

Shape of the fallback chain Fallback shape ?

Single alternate Not verified

An ordered list tries targets in sequence; a weighted split sends a percentage of traffic to each, which is what you need to trial a new model on 5% of requests. Weighted splits are much rarer than the marketing implies.

Upstream health tracking Health checks / circuit breaking ?

In config Verified Sep 2, 2026

Health checking exists only at the API-key level, not the upstream level. ai-proxy `failover` is `enabled: false` by default with `failureThreshold` **3**, `successThreshold` **1**, `healthCheckInterval` **5000** ms, `healthCheckTimeout` **5000** ms, a required `healthCheckModel` and `failoverOnStatus` `["4.*","5.*"]`; an unhealthy `apiToken` is removed from rotation and re-added once probes pass ([ai-proxy, Chinese](https://higress.cn/docs/latest/plugins/ai/api-provider/ai-proxy/)), which the AI quick-start describes in prose as pausing requests on a token until health checks recover it ([AI quick start](https://higress.ai/en/docs/ai/quick-start/)). No circuit-breaker or outlier-detection annotation appears in [the annotation reference](https://higress.ai/en/docs/latest/user/annotation/) and no health-check keys appear in [the global ConfigMap](https://higress.ai/en/docs/latest/user/configmap/) — and Alibaba Cloud's comparison page explicitly marks "故障自动检测及恢复" (automatic fault detection and recovery) as 不支持 for the open-source edition ([Alibaba Cloud comparison](https://help.aliyun.com/zh/api-gateway/ai-gateway/product-overview/product-comparison)).

Whether the product notices a failing upstream and stops sending traffic to it, and whether you can tune the thresholds. This is what turns a provider outage into a blip rather than a sustained error rate, and only four of the twenty expose it.

Cross-region failover you control Multi-region failover surface ?

Not documented Verified Sep 2, 2026

n.a. — no cross-region or multi-AZ failover configuration on [the Helm guide](https://higress.ai/en/docs/latest/ops/deploy-by-helm/), [the global ConfigMap](https://higress.ai/en/docs/latest/user/configmap/) or [the annotation reference](https://higress.ai/en/docs/latest/user/annotation/). Alibaba Cloud's comparison page lists 多可用区部署 (multi-availability-zone deployment) as 自行构建, i.e. build it yourself, for the open-source edition ([Alibaba Cloud comparison](https://help.aliyun.com/zh/api-gateway/ai-gateway/product-overview/product-comparison)). Container images are mirrored to cn-hangzhou, us-west-1 and ap-southeast-7 registries, but that is image distribution, not traffic failover ([README](https://github.com/higress-group/higress/blob/main/README.md)).

Whether you can define what happens when a region degrades. A vendor running many regions is not the same as a vendor letting you configure failover between them; only three document a user-controlled mechanism.

Where you set load balancing Load balancing surface ?

In config Verified Sep 2, 2026

Two unrelated layers. Classic HTTP balancing is an Ingress annotation: `nginx.ingress.kubernetes.io/load-balance` defaults to `round_robin` with `least_conn` and `random` also supported, `ewma` is explicitly **not** supported and silently falls back to round-robin, and `upstream-hash-by` gives consistent hashing on `$request_uri`, `$host`, `$remote_addr`, a header or a query arg ([annotation compatibility](https://higress.ai/en/docs/latest/user/annotation/)). Across LLM credentials, ai-proxy simply picks an `apiToken` at random per request ([ai-proxy plugin](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-proxy/)). LLM-aware algorithms — "global least connections", "prefix matching" and "GPU-aware" load balancing — are described in a vendor engineering post rather than in the plugin reference ([Higress engineering post](https://medium.com/@higress_ai/no-increase-in-gpu-the-first-token-latency-decreases-by-50-new-practices-in-llm-service-load-5583192f9442)), and no `ai-load-balancer` page exists in the documentation index ([higress.ai/llms.txt](https://higress.ai/llms.txt)).

Where traffic distribution across upstreams or keys is configured.

Operations

Interpret these fields: LLM gateway observability: traces, logs and export · Running coding agents through an LLM gateway · Changing models without breaking production

Usage dashboards and logs Observability ?

Yes Not verified

Built-in visibility into what was sent, what came back, what it cost, and how long it took.

Spending limits Budget controls ?

Yes Not verified

Hard caps that stop spend before it becomes a surprise invoice. The single most valuable control for a small team.

Rate limits Rate limits ?

Yes Not verified

Caps on request volume per key or per user, useful for protecting against abuse and runaway loops.

Separate keys per team or app Virtual keys ?

Yes Not verified

Issue scoped keys with their own budgets and permissions so you can attribute cost and revoke access without rotating everything.

Prompt versioning Prompt management ?

Yes Not verified

Store and version prompts outside your code so they can be changed without a deploy.

Quality testing Evals ?

Not published Not verified

Built-in tooling to score model output against test cases, so you can tell whether a model swap made things better or worse.

MCP support MCP support ?

Yes Not verified

Native support for the Model Context Protocol, the emerging standard for connecting models to external tools.

What gets logged Logged content ?

Your choice Verified Sep 2, 2026

Configurable in both directions: `accessLogFormat` under the `mesh` field controls which access-log fields are emitted ([log description](https://higress.ai/en/docs/latest/ops/log/)), and ai-statistics has an explicit "record questions and answers" mode where `attributes` entries with `value_source` of `request_body`, `response_body` or `response_streaming_body` and `apply_to_log: true` write prompt and completion text into the log — both flags default to **false**, so content logging is opt-in ([ai-statistics](https://higress.ai/en/docs/latest/plugins/ai/api-o11y/ai-statistics/)).

Whether full prompts and responses are stored, only metadata, or nothing. Full-body logging is the most useful debugging feature here and the one most likely to need a conversation with your compliance team.

You can turn logging off Body-logging opt-out ?

Not documented Verified Sep 2, 2026

n.a. as a documented switch: [the log page](https://higress.ai/en/docs/latest/ops/log/) explains how to change `accessLogFormat` and how to view logs but states no way to disable access logging, and [the global ConfigMap](https://higress.ai/en/docs/latest/user/configmap/) and [the annotation reference](https://higress.ai/en/docs/latest/user/annotation/) contain no logging on/off key. In practice a self-hosted operator controls this at the Envoy/ConfigMap level, but Higress does not document it.

Whether prompt and response bodies can be suppressed while still keeping usage metrics. Nineteen of the twenty document a way to do this; the mechanisms range from a per-request header to an organisation-wide setting.

Traces you can take elsewhere Distributed tracing ?

OpenTelemetry Verified Sep 2, 2026

OpenTelemetry, plus SkyWalking and Zipkin, configured in the global ConfigMap under `tracing`: `enable` defaults to **false**, `sampling` to **100.0**, `timeout` to **500** ms, and only one exporter can take effect at a time; the `opentelemetry` block takes a `service` and a gRPC `port` ([global configuration](https://higress.ai/en/docs/latest/user/configmap/)). Spans are enriched by ai-statistics via `apply_to_span` attributes, and `trace_id` is an access-log field ([ai-statistics](https://higress.ai/en/docs/latest/plugins/ai/api-o11y/ai-statistics/), [log description](https://higress.ai/en/docs/latest/ops/log/)).

Whether the product emits OpenTelemetry, a proprietary format, or nothing. OpenTelemetry means the traces land in the tooling you already run instead of only in the vendor’s dashboard.

Where telemetry can go Export destinations ?

OpenTelemetry, Prometheus, Grafana, Loki, SkyWalking, Zipkin Not verified

Traces go to OpenTelemetry, SkyWalking or Zipkin collectors you name in the ConfigMap ([global configuration](https://higress.ai/en/docs/latest/user/configmap/)); metrics are scraped from the gateway's metrics port (default **15020** in the Docker Compose install) by Prometheus and drawn in Grafana, with `--set global.o11y.enabled=true` installing Grafana, Prometheus, Loki and PromTail for you, or an external Grafana URL pasted into the console instead ([Prometheus guide](https://higress.ai/en/docs/latest/user/prometheus/), [Helm values](https://higress.ai/en/docs/latest/ops/deploy-by-helm/), [Docker Compose options](https://higress.ai/en/docs/latest/ops/deploy-by-docker-compose/)). There is no vendor SaaS sink and no Datadog/LangSmith/S3 integration on any fetched page.

Documented sinks for logs and metrics. This is a good proxy for how replaceable the vendor’s own dashboard is: around twenty destinations means you never have to depend on it, while a CSV download means you do.

Can record user feedback Feedback capture API ?

No Not verified

No feedback, rating or annotation endpoint. ai-statistics can attach arbitrary attributes to logs and spans from request or response fields, which is the closest available mechanism, but nothing captures a user verdict ([ai-statistics](https://higress.ai/en/docs/latest/plugins/ai/api-o11y/ai-statistics/)); nothing appears in [the plugin marketplace](https://higress.ai/en/plugins/) either.

Whether there is an API to attach a rating or score to a logged request, which is what lets production traffic feed quality work later.

Scores live traffic Online eval hooks ?

No Verified Sep 2, 2026

No evaluation, scoring or experiment feature exists: the 41-plugin marketplace has no eval plugin ([plugin marketplace](https://higress.ai/en/plugins/)) and the documentation index lists none across overview, user guide, ops, developer, AI gateway and scene-guide sections ([higress.ai/llms.txt](https://higress.ai/llms.txt)). Traffic-shaping primitives that could support an offline comparison do exist — canary annotations (`canary-by-header`, `canary-weight`) and 多模型灰度 model-level canary — but no scoring is attached ([annotation reference](https://higress.ai/en/docs/latest/user/annotation/), [Alibaba Cloud comparison](https://help.aliyun.com/zh/api-gateway/ai-gateway/product-overview/product-comparison)).

Whether automated scorers can run against real production requests, rather than only against a test set you assemble yourself.

Performance

Delay it adds Proxy overhead ?

Not published Not verified

Extra time the product itself adds to each request, on top of however long the model takes. Usually irrelevant next to multi-second model latency, but it matters for high-volume or streaming-sensitive workloads.

Requests per second ceiling Throughput ?

Not published Not verified

Published sustained request rate before the product becomes the bottleneck. Only relevant at genuinely high volume.

What the request path runs on Architecture class ?

Compiled binary Verified Sep 2, 2026

Envoy data plane plus a Go control plane: the project is "built on Istio and Envoy" and extended with Wasm plugins written in Go, Rust or JavaScript ([what-is-Higress](https://higress.ai/en/docs/latest/overview/what-is-higress/)). Repo language bytes are Go 9,245,785, C++ 1,269,908, Rust 193,762, Shell 122,023, Python 88,069 and TypeScript 53,579 ([GitHub languages API](https://api.github.com/repos/higress-group/higress)), i.e. a compiled control plane and a compiled proxy, with plugin code compiled to Wasm rather than interpreted per request.

The comparable way to talk about latency here. An edge worker, a compiled Go or Rust binary, and a Python proxy have different overhead floors no matter which figures each vendor publishes. Products that never disclose their runtime are recorded as undisclosed rather than assumed.

You can run the request path yourself Self-hostable data plane ?

Yes Verified Sep 2, 2026

Container images and a Helm chart, both published: `higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:latest` runs console, HTTP and HTTPS on 8001/8080/8443 from one `docker run` ([README](https://github.com/higress-group/higress/blob/main/README.md)), the chart is `higress.io/higress` installed into `higress-system` ([README](https://github.com/higress-group/higress/blob/main/README.md), [Helm guide](https://higress.ai/en/docs/latest/ops/deploy-by-helm/)), and a Docker Compose bundle of apiserver/controller/pilot/gateway/console is generated by an installer script for non-Kubernetes hosts ([Docker Compose guide](https://higress.ai/en/docs/latest/ops/deploy-by-docker-compose/)). Alibaba Cloud Compute Nest deployment is also documented ([documentation index](https://higress.ai/llms.txt)). The FAQ notes there is no pre-built package beyond the images: "There is no existing one, you need to build it yourself. Currently, all Docker images are provided and can be pulled and used by yourself." ([FAQ](https://higress.ai/en/docs/latest/overview/faq/))

Whether the component that actually carries your prompts can run on your own infrastructure. Distinct from a vendor offering a self-hosted control plane while still proxying traffic through their network.

Streaming responses Streaming support ?

Yes Verified Sep 2, 2026

Yes, and it is a headline feature: Higress claims "true streaming processing" for SSE and supports the AI streaming (SSE) scenario ([what-is-Higress](https://higress.ai/en/docs/latest/overview/what-is-higress/)), and ai-cache handles both streaming and non-streaming responses ([ai-cache](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-cache/)). Two documented streaming caveats matter: retries only apply to non-streaming requests ([ai-proxy retryOnFailure](https://higress.cn/docs/latest/plugins/ai/api-provider/ai-proxy/)), and on SSE the ai-data-masking plugin may fail to restore a masked word split across chunks and may leak part of a sensitive word to the client ([ai-data-masking](https://higress.ai/en/docs/latest/user/plugins/ai/api-consumer/ai-data-masking/)).

Whether token-by-token streaming is documented. The caveats matter more than the yes: some products cannot cancel a stream without still being billed, and several timeout and fallback mechanisms stop applying once the first token has been sent.

Security & compliance

Interpret these fields: LLM gateway compliance: SOC 2, HIPAA and evidence · How LLM gateway guardrails fail · Which LLM gateways store your prompts? · Do you need an MCP gateway as well?

Does your prompt reach their servers Prompt transits vendor ?

No Verified Sep 2, 2026

No Higress-operated hop exists: the gateway is software you deploy, and requests go from your gateway straight to the provider endpoints configured in ai-proxy ([README](https://github.com/higress-group/higress/blob/main/README.md), [ai-proxy plugin](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-proxy/)). Two things do leave your perimeter if you enable them — ai-security-guard sends prompt and completion text to Alibaba Cloud's content-moderation API ([content security guide](https://higress.ai/en/docs/ai/scene-guide/application-safety/)), and semantic caching sends prompt text to whichever embedding service and vector database you configure, the worked example being Alibaba Cloud BaiLian plus DashVector ([semantic cache guide](https://higress.ai/en/docs/ai/scene-guide/semantic-cache/)).

Whether the text you send passes through this company’s own infrastructure. If it does, every other promise on this page is a policy commitment rather than a physical impossibility. Self-hosted products can answer no outright.

What they keep if you change nothing Logging default ?

Metadata only, not content Verified Sep 2, 2026

Access logs are on and JSON-formatted by default, and they are metadata: the 22 documented fields are `authority`, `bytes_received`, `bytes_sent`, `duration`, `method`, `path`, `request_id`, `response_code`, `response_flags`, `route_name`, `trace_id`, `upstream_host`, `upstream_service_time`, `user_agent`, `x_forwarded_for` and similar — request and response bodies are represented only by their byte counts ([log description](https://higress.ai/en/docs/latest/ops/log/)). Token counts, model name, `llm_service_duration` and `llm_first_token_duration` are added by the ai-statistics plugin ([ai-statistics](https://higress.ai/en/docs/latest/plugins/ai/api-o11y/ai-statistics/)).

Defaults matter more than options. A product that stores full prompts and replies unless you find the right header will have stored them by the time you read the docs.

How long they keep it Default content retention (days) ?

Not published Not verified

n.a. as a vendor-set value — Higress is self-hosted software with no vendor-side store. The bundled `global.o11y` stack installs Grafana, Prometheus, Loki and PromTail into your cluster and you own the retention settings ([Helm values](https://higress.ai/en/docs/latest/ops/deploy-by-helm/)); ai-cache entries live in a Redis you run, with `cacheTTL` defaulting to 0 meaning never expire ([ai-cache plugin](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-cache/)).

Default retention for request content, in days. Zero means nothing is kept. Read the note: several products keep nothing as a rule but make timed exceptions for abuse review or specific models.

Could they train on your prompts Training on customer data ?

Not applicable Not verified

not_applicable by construction rather than by policy: Higress is self-hosted Apache-2.0 software with no vendor telemetry endpoint on any fetched page ([GitHub API](https://api.github.com/repos/higress-group/higress), [README](https://github.com/higress-group/higress/blob/main/README.md)), so there is no Higress-side corpus to train on. No separate training or data-use policy is published — [the documentation index](https://higress.ai/llms.txt), [the FAQ](https://higress.ai/en/docs/latest/overview/faq/) and [the home page](https://higress.ai/en/) contain none.

Whether the vendor may use your prompts and outputs to train models. “Not published” means we could not find any position, which is not the same as a no — ask for it in writing.

Where it runs, and what you can pin Region and residency control ?

You choose, because you run it. There is no vendor region list; placement is wherever your Kubernetes cluster or host sits ([Helm guide](https://higress.ai/en/docs/latest/ops/deploy-by-helm/), [Docker Compose guide](https://higress.ai/en/docs/latest/ops/deploy-by-docker-compose/)). The only region-shaped facts published are container-registry mirrors in cn-hangzhou, us-west-1 and ap-southeast-7 ([README](https://github.com/higress-group/higress/blob/main/README.md)) and Alibaba Cloud's note that multi-AZ deployment is "build it yourself" for the open-source edition ([Alibaba Cloud comparison](https://help.aliyun.com/zh/api-gateway/ai-gateway/product-overview/product-comparison)). Not verified

Which regions are offered and whether you can force processing to stay in one. A global endpoint that silently picks a region is a different compliance story from an endpoint you pin yourself.

Where safety filters run Guardrail execution location ?

Either, depending on deployment Verified Sep 2, 2026

Enforcement is always in your own gateway process, but the classifier can be remote: ai-security-guard delegates to Alibaba Cloud's content-moderation API, which requires activating that Alibaba Cloud service ([content security guide](https://higress.ai/en/docs/ai/scene-guide/application-safety/)), whereas qwen3guard calls a Qwen3Guard-Gen model you can host yourself behind an OpenAI-compatible endpoint via vLLM or SGLang ([Higress content-security post](https://higress.ai/blog/higress-mmse_awbbpb_yafyyrc3t5wh0u55/)), and ai-data-masking runs entirely locally in Wasm ([ai-data-masking](https://higress.ai/en/docs/latest/user/plugins/ai/api-consumer/ai-data-masking/)).

A filter that strips personal data only helps if it runs before the data leaves your boundary. If guardrails execute in the vendor’s cloud, the vendor has already received whatever you wanted redacted.

Who else touches the data Subprocessor list ?

Not published Not verified

The published list of third parties the vendor passes your data to. No list means you cannot know the full chain, which most data-protection agreements require you to.

SOC 2 audited SOC 2 audited ?

Not published Not verified

An independent audit of security controls. Enterprise buyers and their procurement teams routinely require it.

Will sign a HIPAA agreement HIPAA BAA ?

Not published Not verified

Required before you may send protected health information through the service. Without a signed BAA, healthcare data is off limits.

GDPR commitments GDPR commitments ?

Not published Not verified

Published data processing terms for handling personal data of people in the EU and UK.

Can keep data in the EU EU data residency ?

Not published Not verified

Requests can be processed inside the EU rather than routed to US infrastructure. Often the deciding constraint for European customers.

Does not retain your data Zero data retention ?

Not applicable Not verified

Prompts and responses are not stored after the request completes. Sometimes a paid add-on rather than the default.

Strips personal data PII redaction ?

Yes Not verified

Detects and removes identifiers such as names, emails, and card numbers before the request reaches the model provider.

Content guardrails Content guardrails ?

Yes Not verified

Policy checks on inputs and outputs — blocking unsafe content, enforcing formats, or catching prompt-injection attempts.

Runs fully disconnected Air-gapped deployment ?

Not published Not verified

Can be deployed in a network with no internet access, which some regulated and defence environments require.

Blocks personal data in prompts PII / DLP enforcement ?

Can block the request Verified Sep 2, 2026

`sync_block` by default when the plugin is attached: ai-data-masking's `system_deny` (built-in sensitive-word rules, sourced from `github.com/houbb/sensitive-word`) and `deny_openai` both default to **true**, and `deny_code` defaults to **200** with the message "Sensitive words found in the question or answer have been blocked". A softer mode exists — `replace_roles` with regex/GROK patterns and `type: replace` or `hash`, plus `restore: true` to put the original values back into the model's answer ([ai-data-masking](https://higress.ai/en/docs/latest/user/plugins/ai/api-consumer/ai-data-masking/)). Documented failure mode on SSE: a masked word split across chunks may not be restored, and part of a sensitive word can reach the user.

Whether personal data detection sits on the request path and can stop the call, merely inspects and forwards it, or is not documented. A control that only reports is a logging feature, not a policy control.

Blocks prompt injection Injection / jailbreak enforcement ?

Can block the request Verified Sep 2, 2026

`sync_block`, but only through the newer `qwen3guard` plugin, and only as a by-product: the vendor engineering post states Qwen3Guard's official safety policy covers violence, illegal acts, sexual content, PII, self-harm, unethical behaviour, politically sensitive topics and copyright infringement "并在输入审核中包含 Jailbreak 检测" (jailbreak detection is included in input moderation). The plugin enables `checkRequest` and `checkResponse` by default, calls `Qwen/Qwen3Guard-Gen-4B`, uses `riskLevelBar: Unsafe` by default, and when the bar is met returns a refusal without calling the model at all ([Higress content-security post](https://higress.ai/blog/higress-mmse_awbbpb_yafyyrc3t5wh0u55/)). Caveats stated by the vendor: only the `Safety` verdict drives the decision, per-category actions are not implemented, and streaming interception cannot honour `denyCode`. No prompt-injection or jailbreak wording appears in the older [ai-security-guard](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-security-guard/) reference.

Whether injection and jailbreak detection can stop a request. Most products offering this call a partner classifier rather than shipping their own.

Blocks harmful content Toxicity / moderation enforcement ?

Can block the request Verified Sep 2, 2026

`sync_block` via ai-security-guard, which calls Alibaba Cloud's content-moderation service inline for both directions and denies the request when a risk label is returned; `checkRequest` and `checkResponse` both default to **false**, `requestCheckService` defaults to `llm_query_moderation`, `responseCheckService` to `llm_response_moderation`, and `denyCode` defaults to **200** — so a blocked call looks like a success to a naive client ([ai-security-guard](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-security-guard/)). It emits `ai_sec_request_deny`/`ai_sec_response_deny` metrics and `ai_sec_risklabel`/`ai_sec_deny_phase` span attributes. The console walkthrough registers Alibaba Cloud Content Safety as a service source and attaches "AI Safety Guard" to a route ([content security guide](https://higress.ai/en/docs/ai/scene-guide/application-safety/)).

Whether hate, violence, sexual and self-harm categories are checked inline and can stop a request, in either direction.

Your own policy rules Custom policy hooks ?

Can block the request Not verified

Custom policy is regex/GROK-based: ai-data-masking takes user-supplied `deny_words` and `replace_roles` regex patterns with per-rule `replace`/`hash` actions ([ai-data-masking](https://higress.ai/en/docs/latest/user/plugins/ai/api-consumer/ai-data-masking/)), and the generic `request-block` and `request-validation` plugins add path/body blocking and JSON-schema request validation ([plugin marketplace](https://higress.ai/en/plugins/)). Custom rules you add yourself default to replacement rather than denial unless you put them in the deny list, hence `sync_observe` as the default posture for custom rules specifically.

Whether you can add your own rule — a regex, a webhook, or your own classifier — rather than choosing from the vendor library.

Where guardrails run Guardrail execution location ?

Either, your choice Not verified

Whether guardrail evaluation happens inside your infrastructure or on the vendor’s servers. This decides whether the prompt you are trying to protect leaves your network in order to be checked.

If the guardrail itself fails Guardrail failure mode ?

Request proceeds Verified Sep 2, 2026

`fail_open`, stated explicitly and unusually candidly for the qwen3guard plugin: "当前插件选择 fail-open：记录警告并放行" — on connection failure, `timeoutMs` (default 2000 ms) expiry, a non-200 from the guard model, unparseable JSON or a missing `Safety` field, the request is logged and allowed through. The same post says compliance scenarios needing forced fail-close "不能被描述为已经满足" (cannot be described as satisfied) in the current version ([Higress content-security post](https://higress.ai/blog/higress-mmse_awbbpb_yafyyrc3t5wh0u55/)). No fail-open/fail-closed statement appears in the older [ai-security-guard](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-security-guard/) reference.

What happens when the guardrail service times out or errors: does the request proceed unchecked, or is it blocked? This is the worst-documented field in the entire catalogue — only two vendors state it plainly, which means most teams are running a control whose failure behaviour they cannot know.

Third-party guardrail vendors Guardrail integrations ?

Alibaba Cloud Content Moderation, Qwen3Guard Not verified

Named external guardrail services the product can call. A long list means the product is a router for policy engines rather than a policy engine itself — which also means another vendor bill and another hop.

Fit & integration

Interpret these fields: How much does an LLM gateway lock you in? · Do you need an MCP gateway as well? · Running coding agents through an LLM gateway · Changing models without breaking production

Work to try it Evaluation work shape ?

Run something locally first Not verified

The shape of the work on the vendor’s own quickstart, from swapping one base URL through to deploying infrastructure. An ordinal class rather than a duration, because elapsed time depends on accounts and quota we cannot see.

Work to run it Production work shape ?

Deploy it on your infrastructure Not verified

The same scale applied to the vendor’s recommended production path. For several products this is much heavier than the quickstart, which is exactly why both are recorded.

Steps on the quickstart Numbered quickstart steps ?

0 Not verified

No numbered step list exists to count. The AI gateway quick start is organised as screenshot-driven sections (install by Docker, log in to the console, configure an LLM provider key, configure an AI route, observe the AI dashboard) rather than enumerated steps ([AI quick start](https://higress.ai/en/docs/ai/quick-start/)), and the Kubernetes quick start is structured as three stages — install, configure, validate — with branching environment options rather than a linear count ([quick start](https://higress.ai/en/docs/latest/user/quickstart/)). Recorded as 0 to mean "vendor publishes no step count", not "zero work".

A literal count of numbered steps on the vendor’s quickstart, recorded as evidence beside the work shape. Zero means the page publishes no numbered procedure at all. Large counts usually mean interleaved language tracks rather than more work.

Can you self-host it today Self-host install documentation ?

Install command published Not verified

One container: `docker run -d --rm --name higress-ai -v ${PWD}:/data -p 8001:8001 -p 8080:8080 -p 8443:8443 higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:latest` (console 8001, HTTP 8080, HTTPS 8443). Kubernetes: `helm install higress -n higress-system higress.io/higress --create-namespace`. Non-Kubernetes AI gateway installer: `wget https://higress.cn/ai-gateway/install.sh` then run with bash.

Whether an install command is actually published. Several products advertise self-hosting while publishing no command to start from, which a plain yes/no would hide.

Works with the OpenAI SDK OpenAI SDK drop-in ?

Yes Not verified

Yes: point an OpenAI-format client at the gateway route and ai-proxy converts to whichever provider the route is bound to, because the plugin "implements AI proxy functionality based on OpenAI API contract" and detects the protocol from `/v1/chat/completions` ([ai-proxy plugin](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-proxy/)). Requests are authenticated with your own consumer credential in `x-api-key` rather than a provider key ([token management guide](https://higress.ai/en/docs/ai/scene-guide/token-management/)). No SDK change is needed; `protocol: original` is available when you would rather not be converted at all.

Whether an existing OpenAI-compatible client can be pointed at it by changing the base URL and key.

Vercel AI SDK support AI SDK provider package ?

Not documented Not verified

n.a. — the Vercel AI SDK is not mentioned on [the documentation index](https://higress.ai/llms.txt), [the ai-proxy reference](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-proxy/) or [the AI gateway page](https://higress.ai/en/ai-gateway/). In practice `@ai-sdk/openai` with a custom `baseURL` should work against the OpenAI-compatible route ([ai-proxy plugin](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-proxy/)), but Higress documents no such integration, so this is recorded as not documented rather than as compatibility.

Whether a first-party AI SDK provider package exists, or only a community package, a documented workaround, or the generic OpenAI provider pointed at a custom base URL.

Python framework integrations Documented Python frameworks ?

Not published Not verified

n.a. — neither LangChain nor LlamaIndex is documented anywhere in the official docs index or the plugin marketplace ([higress.ai/llms.txt](https://higress.ai/llms.txt), [plugin marketplace](https://higress.ai/en/plugins/)); a LangChain + Higress + Elasticsearch RAG walkthrough exists only as a third-party blog post, not vendor documentation. RAG is instead offered as a gateway plugin (ai-rag) and via ai-search ([plugin marketplace](https://higress.ai/en/plugins/)).

LangChain, LangGraph, LlamaIndex and similar orchestration frameworks with a documented integration.

Callable from Cloudflare Workers Cloudflare Workers support ?

Workers AI as an upstream model Not verified

Cloudflare appears only as a model source: ai-proxy has a `cloudflare` provider type for Cloudflare Workers AI, configured with `cloudflareAccountId` ([ai-proxy plugin](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-proxy/)). Higress itself is a self-hosted Envoy process and does not run on Workers ([what-is-Higress](https://higress.ai/en/docs/latest/overview/what-is-higress/)).

Whether the docs show calling this product from your own Worker. Deliberately separated from the several products whose own gateway runs on Workers, which is a fact about their infrastructure and not about your edge compatibility.

Kubernetes install Helm chart availability ?

Official Helm chart Verified Sep 2, 2026

Kubernetes is the first-class target: `helm repo add higress.io https://higress.io/helm-charts` then `helm install higress -n higress-system higress.io/higress --create-namespace`, with `global.local=true` for a kind cluster and `hgctl` to open the console ([Helm guide](https://higress.ai/en/docs/latest/ops/deploy-by-helm/), [quick start](https://higress.ai/en/docs/latest/user/quickstart/)). The README states Higress is a conformant Gateway API implementation, a conformant Gateway API Inference Extension implementation, and is listed in the official Kubernetes Ingress Controllers documentation ([README](https://github.com/higress-group/higress/blob/main/README.md)). It is also nginx-Ingress-annotation compatible, which is its main migration story ([annotation compatibility](https://higress.ai/en/docs/latest/user/annotation/)).

Whether a named, published Helm chart exists, versus Helm being referenced with no chart named, versus generic cluster documentation that has nothing to do with this product.

Terraform support Terraform provider or modules ?

Not documented Verified Sep 2, 2026

n.a. — no Terraform provider, module or registry reference on [the documentation index](https://higress.ai/llms.txt), [the Helm guide](https://higress.ai/en/docs/latest/ops/deploy-by-helm/) or [the Docker Compose guide](https://higress.ai/en/docs/latest/ops/deploy-by-docker-compose/). Infrastructure-as-code is expressed instead as Helm values and Kubernetes Ingress/ConfigMap objects, plus an Alibaba Cloud Compute Nest deployment path ([documentation index](https://higress.ai/llms.txt)).

Whether you can declare this in code: an official provider, official modules, resources inside a hyperscaler’s provider, a community provider, or only Terraform code shipped in a repo.

Reuses your cloud identity Cloud IAM reuse ?

Static provider credentials only Not verified

Cloud credentials are passed as static plugin fields, not assumed roles: AWS Bedrock takes `awsAccessKey`, `awsSecretKey` and `awsRegion`, and Google Vertex AI takes `vertexAuthKey`, `vertexRegion`, `vertexProjectId` and `vertexAuthServiceName` ([ai-proxy plugin](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-proxy/)). No IRSA, workload identity, instance-profile or credential-chain support is documented. IDaaS/OIDC integration for the control plane is listed as a commercial-edition feature (深度集成阿里云 IDAAS 产品) versus "build it yourself" for open source ([Alibaba Cloud comparison](https://help.aliyun.com/zh/api-gateway/ai-gateway/product-overview/product-comparison)), although an OIDC plugin exists for data-plane auth ([plugin marketplace](https://higress.ai/en/plugins/)).

Whether you can authenticate with IAM roles, workload identity or managed identities instead of another long-lived API key. Distinguished from products that only accept static upstream provider credentials.

Fits behind your API gateway API gateway integration ?

It is the API gateway Not verified

It is the gateway: Higress is described as an AI-native API gateway built on Istio and Envoy that unifies AI gateway, Kubernetes Ingress gateway, microservice gateway and security gateway roles in one deployment ([what-is-Higress](https://higress.ai/en/docs/latest/overview/what-is-higress/)), and the home page splits the AI side into LLM Gateway, MCP Gateway and Model Gateway (the last implementing the Gateway API Inference Extension with InferencePool and Endpoint Picker) ([home page](https://higress.ai/en/)).

Whether AI traffic can go through a gateway you already run, and who documented that — the product’s vendor or the gateway’s.

MCP support MCP surface shape ?

MCP gateway or registry Verified Sep 2, 2026

A real MCP gateway, not just a client. The `mcp-server` plugin both converts REST APIs into MCP tools with no code (`server.type: rest`, `tools[].requestTemplate`) and proxies existing MCP servers (`server.type: mcp-proxy` with `mcpServerURL`, `server.timeout` default 5000 ms), with `allowTools` allowlisting, `securitySchemes` plus `defaultDownstreamSecurity`/`defaultUpstreamSecurity`, and `passthroughAuthHeader` defaulting to **false** so client credentials are not leaked to backends ([mcp-server plugin](https://higress.ai/en/docs/ai/mcp-server/)). Hosting requires Higress >= 2.1.0 and Redis; the Nacos MCP registry needs Nacos >= 3.0 and Higress >= 2.1.2; SSE endpoints are exposed at `/{mcp route prefix}/{nacos name}/sse` and the 2025-03-26 streamable-HTTP protocol needs no ConfigMap entry ([MCP quick start](https://higress.ai/en/docs/ai/mcp-quick-start/)). The home page announces support for the 2026-07-28 MCP revision "with compatibility for existing protocols" ([home page](https://higress.ai/en/)), an `openapi-to-mcp` conversion tool and a hosted demo at https://mcp.higress.ai/ ([README](https://github.com/higress-group/higress/blob/main/README.md)).

Which kind of MCP support this is: a gateway that governs many MCP servers, a hosted MCP server you connect to, MCP tools accepted inside the completion API, or client tooling. These are different products behind one acronym.

Needs your own provider key Upstream provider key required ?

Required Not verified

Yes — you cannot make a first call without an upstream key. The installer prompts "enter the Aliyun Dashscope or other API-KEY" (you may skip and add it in the console instead) and "LLM Provider Management" is where keys for Alibaba Cloud, DeepSeek, Azure OpenAI, OpenAI, DouBao and others are stored ([AI quick start](https://higress.ai/en/docs/ai/quick-start/)); every ai-proxy provider block takes `apiTokens` or cloud credentials ([ai-proxy plugin](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-proxy/)).

Whether an upstream provider account and key must exist before your first call works. A prerequisite rather than a step, and it can differ between the hosted and self-hosted forms of the same product.

Gate before models work Model access gate ?

No gate Not verified

No gate: you add a provider block with your own `apiTokens` and the model is immediately routable, with `modelMapping` deciding which names clients may use ([ai-proxy plugin](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-proxy/)); the console equivalent is adding a key under LLM Provider Management ([AI quick start](https://higress.ai/en/docs/ai/quick-start/)). No approval, enablement, quota or waitlist step appears on either page.

Whether an enablement click, a quota grant, a paid tier or an approval form stands between a valid key and a working model call.

Official client languages First-party client SDK languages ?

Not published Not verified

n.a. — no Higress client library exists on [the ai-proxy reference](https://higress.ai/en/docs/latest/plugins/ai/api-provider/ai-proxy/), [the documentation index](https://higress.ai/llms.txt) or [the AI quick start](https://higress.ai/en/docs/ai/quick-start/); the documented calling convention is `curl` or any OpenAI-compatible SDK against the gateway route. The Go, Rust and JavaScript SDKs Higress does publish are for authoring Wasm gateway plugins, not for calling the gateway ([what-is-Higress](https://higress.ai/en/docs/latest/overview/what-is-higress/)), so no client-language list is recorded here.

Languages with a first-party client library. An empty list can still mean the product is usable from any language via an OpenAI-compatible SDK.