Managed or self-host Proprietary

TrueFoundry AI Gateway Cloud platform

TrueFoundry AI Gateway is a cloud AI platform: an OpenAI-compatible API in front of ~1,000 models from 27 providers. Its fees are not published. It can be self-hosted or used as a managed service. SOC 2 is published; zero data retention and a HIPAA BAA are not. You can point it at your own provider accounts. Beyond chat it also serves embeddings. Air-gapped deployment is available.

· 37 of 113 fields dated · 49 sources

Built by TrueFoundry, founded 2021 · India company — outside US jurisdiction

Compare with others Estimate cost Visit site ↗

Access at a glance

Whether this product can work for you at all, before features matter: who pays the model bill, where it can run, and which of your existing API calls keep working. Every value is the vendor’s own claim, linked to the page it came from.

TrueFoundry calls it the "TrueFoundry AI Gateway", "the proxy layer that sits between your applications and the LLM providers and MCP Servers", and "an enterprise-grade platform that enables users to access 1000+ LLMs using a unified interface while taking care of observability and governance", offering "one endpoint with an OpenAI-compatible schema for every provider" (Introduction to AI Gateway). It is a routing proxy with an observability/governance plane.

Who pays the model bill

Your keys only

You contract with each model provider directly and hold those accounts. The gateway never resells inference.

Providers are added as Model Accounts — "A Model Account represents one account of a model provider (e.g. OpenAI, Anthropic, AWS Bedrock)… You can add multiple accounts per provider, each with their own API keys" — and models are added "after providing API key" (Quick start). No TrueFoundry-sold inference credits are documented; pricing is platform subscription plus usage (TrueFoundry pricing).

Merchant of record: n.a. as an explicit statement about model usage; TrueFoundry invoices its platform tiers (Developer $0, Pro $499/month, Pro Plus $2,999/month, Enterprise custom, plus "additional usage at transparent, per-unit rates") while model calls run on the customer's own provider accounts (TrueFoundry pricing; Quick start).

Key handling: Upstream provider keys are held per Model Account, with separate accounts used to hold multiple keys for the same provider; self-hosted model registrations store optional "Auth Data" for upstream auth (Quick start; Self-hosted models). Caller auth uses a Personal Access Token (for development) or Virtual Account Token (for applications) created in the platform's Access section, and the gateway offers "RBAC and scoped keys for users, teams, and applications" plus spend limits and per-user/per-model/per-application throttles (Quick start; Introduction to AI Gateway). Data handling varies by topology: on SaaS "data is encrypted at rest and in transit" on infrastructure that is "SOC2, ISO27001, GDPR, and HIPAA compliant"; with customer-managed storage, request-response data still flows through the TrueFoundry control plane, which "might cache some of the data for faster queries, but doesn't do any long term retention of the data" and requires bucket access ("you will not be able to use this feature if you don't give access to TrueFoundry control plane access to your bucket"); fully self-hosted keeps both planes on customer infrastructure (Deployment options; Introduction to AI Gateway).

Where it can run

4 of 5 shapes documented
  • Vendor-hosted
  • Self-host
  • Your VPC
  • On-premise
  • Air-gapped

Dimmed shapes are not documented by the vendor, which is not the same as unsupported.

hosted SaaS ("You can run the AI Gateway as fully managed SaaS… hosted in multiple regions across the world"), self-host, hybrid/customer-cloud VPC, and on-prem: "SaaS, hybrid, or fully self-hosted in your own VPC" and "Yes, the AI Gateway supports on-premise deployments on any infrastructure or cloud provider" (Introduction to AI Gateway; Deployment options). Air-gapped: n.a. (explicitly not mentioned on the deployment pages fetched).

Four documented topologies with tier and cost: (1) "AI Gateway SAAS only" — Starter tier, $0 hosting, "you don't need to deploy any infrastructure on your end"; (2) "SaaS AI Gateway with data storage on your own infrastructure" — Starter, cost = "storage cost in S3"; (3) "Gateway Plane only" with TrueFoundry-hosted control plane — Enterprise, "~$600/month" hosting; (4) "Control Plane + Gateway Plane" on customer infrastructure — Enterprise, "~$800-1000/month" (Deployment options). Docker/Helm/Terraform specifics are not mentioned on the pages fetched: n.a. (Self-hosted gateway). Platform tiers are Developer $0, Pro $499/month, Pro Plus $2,999/month, Enterprise custom (TrueFoundry pricing).

API surfaces your code can keep using

6 of 7 documented
  • OpenAI chatPOST /v1/chat/completionsYes

    OpenAI-schema chat completions through the gateway base URL (https://gateway.truefoundry.ai for SaaS; self-hosted URLs come from the Playground code snippet), used with OpenAI(api_key=..., base_url="{GATEWAY_BASE_URL}") (Quick start; Introduction to AI Gateway).

  • Anthropic messagesPOST /v1/messagesYes *

    Yes as SDK compatibility: "drop-in support for OpenAI, Anthropic, and other provider SDKs", plus a /proxy API that can "forward provider-native requests through the AI Gateway while keeping logging, rate limiting, and budget controls" (Introduction to AI Gateway). Explicit /v1/messages path: n.a.

  • OpenAI ResponsesPOST /v1/responsesNot documented

    n.a. (not documented on the pages fetched).

  • EmbeddingsPOST /v1/embeddingsYes

    "Embedding (/embeddings)", with a linked Embeddings API reference (Introduction to AI Gateway).

  • ImagesPOST /v1/images/generationsYes

    "Image Generation (/images/generations)", "Image Edit (/images/edits)", "Image Variation (/images/variations)" (Introduction to AI Gateway).

  • AudioPOST /v1/audio/*Yes *

    Yes, both directions: Text To Speech and Speech to Text are documented, plus Audio Translation at /audio/translations and a Live/Realtime API (Introduction to AI Gateway).

  • Batch jobsPOST /v1/batchesYes

    "Batch (/batches)" — "run large workloads asynchronously at batch pricing" (Introduction to AI Gateway).

An asterisk marks a qualified verdict: support that is indirect (SDK compatibility or provider passthrough rather than a native endpoint), or a gap that is narrower or wider than the label suggests. Read the note before porting.

Drop-in base-URL swap with the OpenAI SDK (base_url="{GATEWAY_BASE_URL}", or OPENAI_BASE_URL/OPENAI_API_KEY env vars), with ready snippets for OpenAI SDK, LangChain, LangGraph, LlamaIndex, Node.js, Google ADK and cURL; you need the Gateway Base URL, API key and Model ID (Quick start). The surface is broad: also /rerank, /moderations, /files, fine-tune, a Compaction API and /proxy for provider-native passthrough (Introduction to AI Gateway). Streaming is marked supported for all listed chat providers (OpenAI, Azure OpenAI, Anthropic, Bedrock, Vertex, Cohere, Gemini, Groq, AI21, Cerebras, Wafer, SambaNova, Perplexity AI, Together AI, xAI, DeepInfra) with no documented caveats (Introduction to AI Gateway).

How much it reaches

Models~1,000
Upstream providers27

Providers: Counted from the vendor’s own published list; no aggregate total is published.

"1000+ LLMs" (Introduction to AI Gateway, undated).

n.a. as a stated total; the intro page enumerates 27 provider integrations (Gemini & Vertex AI, Google Gemini, AWS Bedrock, AWS SageMaker, Azure OpenAI, Azure AI Foundry, OpenAI, Cohere, Databricks, AI21, Anthropic, Together AI, xAI, DeepInfra, Perplexity AI, Mistral AI, Cloudera, Groq, ElevenLabs, Deepgram, Cartesia, Smallest AI, Snowflake Cortex, Self Hosted, OpenRouter, SambaNova, Cerebras) and adds "if you don't see the provider you need, there is a high chance it will just work as self-hosted models or OpenAI provider" (Introduction to AI Gateway, undated).

Whose models: All third-party or customer-run: TrueFoundry proxies provider APIs and customer-registered self-hosted models; the intro page does not claim any TrueFoundry-owned model hardware (Introduction to AI Gateway; Self-hosted models).

Your own endpoints: Yes: "you can easily integrate any OpenAI-compatible self-hosted model" by supplying "the endpoint URL" and optional "Auth Data which will be used for Authentication of request to the model"; documented targets include vLLM, Ollama, SGLang and Text Generation Inference running on "your own infrastructure", "on-premises GPUs", "private cloud instances", or "any model server not deployed through TrueFoundry", and registered models appear in the catalog next to cloud providers (Self-hosted models).

How it behaves in production

What happens when an upstream model is slow, wrong, or down — and what you can see and stop while it happens. Reliability features are recorded as where you configure them, not whether the vendor lists them, because almost every product here lists all of them.

5 of 6 reachable from code4 of 4 can block6 documented destinations

When something goes wrong

Each row says where the knob is, not whether the feature is on the marketing page. A control you can only reach by hand in someone else’s dashboard cannot be reviewed or version-controlled.

  • Request timeoutNot documented

    The vendor does not document this, so any behaviour you observe today is unversioned and may change.

    not_documented as a request-timeout key. What exists are SLA thresholds used for health, not for aborting a request: sla_cutoff.time_to_first_token_ms and sla_cutoff.time_per_output_token_ms (TrueFoundry virtual models / fallback).

  • RetriesIn config

    Changing it means editing configuration and shipping it, so behaviour is uniform across traffic until you redeploy.

    config_file (also editable in the UI). retry_config with attempts (default 0), delay (default 100 ms), and on_status_codes (default ["429","500","502","503"]). Retries occur on the same target; backoff beyond the fixed delay is not documented (TrueFoundry routing config, TrueFoundry virtual models / fallback).

  • Fallback to another modelIn config

    ORDERED for priority-based routing (priority, lower number = higher priority, 0 highest); fallback targets come from load_balance_targets, gated by fallback_status_codes (default ["401","403","404","408","429","500","502","503"]) and fallback_candidate (default true). Ordering depends on strategy: latency-based puts lower-latency targets first; weight-based sticky routing tries remaining healthy targets in sequence and that order is explicitly "not weight-based" (TrueFoundry virtual models / fallback).

  • Load balancingIn config

    Weights supported: routing_config.type of weight-based-routing (integer weight 0–100, must sum to 100), priority-based-routing, latency-based-routing ("No weights needed", selection by recent TPOT), and complexity-based-routing (simple/medium/complex). Targets are catalog identifiers (azure/gpt-4o, openai/gpt-4o, bedrock/llama, onprem/llama), with metadata_match, headers_override and override_params per target; rules are evaluated in order and the first match wins (TrueFoundry routing config, TrueFoundry virtual models / fallback).

  • Upstream health trackingIn config

    config_file, and it is passive health tracking plus cooldown ejection. The gateway "continuously monitors every target", marks failing targets unhealthy, moves them to the end of the list as last resort, and recovers automatically. Failure-based cooldown: 5xx/429/401/403, default threshold 2+ failures in a rolling 2-minute window. SLA-based cooldown (priority routing only) via sla_cutoff.time_per_output_token_ms / time_to_first_token_ms, averaged over a 3-minute rolling window using up to 10 samples with a minimum of 3; TTFT samples only come from streaming requests and TPOT samples require at least 30 output tokens (TrueFoundry virtual models / fallback). TrueFoundry's blog additionally describes classic circuit-breaker states (closed/open/half_open) as a pattern, but states no configuration keys for it (TrueFoundry LLM failover blog).

  • Cross-region failoverIn config

    SaaS requests are auto-tagged with tfy_gateway_region and tfy_gateway_zone, and you route on them with metadata_match (documented examples US → azure-us/gpt-4o, EU → azure-eu/gpt-4o, IN → azure-in/gpt-4o, and metadata_match: {tfy_gateway_zone: SFO}), with a catch-all priority: 1 target as the cross-region fallback. Both are true here: TrueFoundry runs multiple SaaS gateway regions AND you configure the cross-region target chain yourself (TrueFoundry virtual models / fallback). Multiple gateways can also be deployed cross-region under one control plane (TrueFoundry gateway plane architecture).

Fallback chain: Ordered list — Try A, then B, then C. Simple and predictable, but every failover is all-or-nothing.

Defaults: retry_config with attempts (default 0), delay (default 100 ms), and on_status_codes (default ["429","500","502","503"]).

Latency-based routing uses Time Per Output Token (total response time ÷ output tokens) over a 20-minute window with a 10-minute sticky epoch, chosen because TPOT folds TTFT and inter-token latency into one output-length-independent number (TrueFoundry virtual models / fallback). Control-plane/data-plane separation means gateway pods only take traffic after config sync (readiness probe) (TrueFoundry gateway plane architecture).

How fast the hop is

Interpreted proxy

Runs on an interpreted or JIT runtime (Lua, Python, Node). Overhead is higher than a compiled binary and more sensitive to concurrency, though a Lua-on-nginx proxy and a Python one are far apart.

interpreted_proxy (JS/edge runtime). "The AI Gateway is written using the Hono framework", described as "ultra-fast, minimalistic, and is designed for the edge" (TrueFoundry gateway plane architecture).

You can run the request path yourselfYes
StreamingYes

yes, but the artifact is not named on the fetched pages. Deployment is documented as SaaS, your VPC, on-prem or air-gapped, running as Kubernetes pods with readiness probes and optional HTTP_PROXY/HTTPS_PROXY/NO_PROXY forward-proxy env vars; no Docker image name or Helm chart name is stated (TrueFoundry gateway plane architecture, TrueFoundry LLM failover blog).

Streaming caveats: Supported, with the most precise streaming-fallback semantics in this set: the gateway waits for the "first non-empty stream chunk", and if Anthropic returns overloaded_error before that chunk it falls back to the next eligible target. Separate caveat: prompt_version_fqn "does not work with agents (when using MCP/tools)". Note also that TTFT SLA samples are only recorded on streaming requests (TrueFoundry routing config, TrueFoundry virtual models / fallback).

Published figures, grouped by what each one measured. Figures in different groups are different quantities and cannot be compared with one another — nor, in most cases, with another vendor’s figure in the same group.

Overhead added by the gateway

The only figures that speak to whether this product slows your application down. Still not comparable between vendors: each was measured on different hardware, at a different load, with a different payload.

  • +7 msVendor-published

    200-220 RPS on 1 vCPU / 1 GB RAM, tracing disabled. Percentile, payload and cache state not stated.

    Source
  • +8 msVendor-published

    Same hardware at 200-220 RPS with complete tracing enabled.

    Source
  • +12 msVendor-published

    350-370 RPS with complete tracing — the only figure here showing overhead rising with load.

    Source

Sustained capacity

Requests or queries per second sustained on the stated hardware.

  • 250 RPSsustainedVendor-published

    Single pod, 1 CPU / 1 GB RAM, no degradation stated until about 350 RPS.

    Source

All vendor self-published. The docs table (with RPS, hardware and tracing conditions) is the credible artifact; the blog's ~10ms and ~3–4 ms figures carry no methodology and the ~3–4 ms figure appears in a comparative context, so treat it as marketing rather than a measured result (TrueFoundry gateway plane architecture, TrueFoundry LLM failover blog).

What it will stop

4 of 4 can block

Two separate questions per control: can it stop a request at all, and what does it do before you change any settings? A control that inspects and forwards is a logging feature, however it is named.

  • Personal data in promptsCan block the request

    Out of the box: You pick the action when configuring

    Built-in PII Detection and Azure AI Language PII/PHI guardrails (Guardrails overview). Custom plugins distinguish mutation from validation: a Presidio POST /pii-redaction plugin returns transformed: true with a full replacement requestBody to redact, whereas a Guardrails-AI POST /pii-detection plugin "only detects and reports PII without redacting it" (Custom guardrails)

  • Prompt injection and jailbreaksCan block the request

    Out of the box: You pick the action when configuring

    Azure Prompt Shield is a built-in guardrail integration; external options include Model Armor, Prisma AIRS, CrowdStrike, and NeMo (Guardrails overview)

  • Harmful contentCan block the request

    Out of the box: Blocks out of the box

    Azure Content Safety moderation and OpenAI Moderations are supported; blocked requests return a configurable custom_error_message (Guardrails overview, Guardrails configuration)

  • Your own policiesCan block the request

    Out of the box: You pick the action when configuring

    Policy forms: built-in Regex and Secrets Detection rules, plus custom HTTP plugins bound to lifecycle hooks llm_input, llm_output, mcp_tool_pre_invoke, mcp_tool_post_invoke, returning either a validate verdict or a mutated request body (Guardrails overview, Guardrails configuration, Custom guardrails)

Where checks runEither, your choice
If the guardrail itself failsNot documented

Almost no vendor in this catalogue documents what happens when the guardrail service itself times out. If the control matters to you, this is a question worth asking before you sign.

not_documented as a global fail-open/fail-closed rule. For custom plugins, HTTP 4xx/5xx from the plugin signals a "processing or dependency failure only; not used for 'PII found' policy outcomes", which separates infrastructure failure from a policy verdict but does not state the resulting request disposition (Custom guardrails)

Calls out to: Azure AI Language, Azure Content Safety, OpenAI Moderation, Microsoft Presidio. Each is a separate vendor relationship and a separate hop on the request path.

TrueFoundry's guardrail analytics are the most enforcement-aware of any product here: outcomes are split into allowed, blocked, mutated, and audit_mode_blocked, with per-guardrail block/mutate rates and P50–P99 guardrail latency, so you can measure whether a control actually fires (Metrics dashboard). One documentation inconsistency: the analytics page labels "Total Flagged Requests" as "requests that were blocked by a guardrail" (Metrics dashboard).

What you can see

Exports widely
What gets loggedYour choice

You decide whether bodies are captured, by setting or by header.

You can turn bodies offYes

Set the global mode to NEVER, or send X-TFY-LOGGING-CONFIG: {"enabled": false} (stringified JSON) per request; metrics dashboards remain populated (Request logging, Metrics dashboard)

TracesOpenTelemetry

OpenTelemetry natively for both traces and metrics, with MCP tool and agent spans; trace storage is managed by TrueFoundry (Export OpenTelemetry data)

Global logging mode is HEADER_CONTROLLED (default behaviour: log unless the header says otherwise), ALWAYS, or NEVER; logged requests, viewable under AI Gateway → Monitor → Requests, include request content, and sensitive values can be redacted per subject/model/metadata (Request logging)

Where telemetry can go

  • OpenTelemetry
  • Datadog
  • Prometheus
  • Grafana
  • S3
  • Webhook

OTLP exporters to Grafana Cloud, Datadog (OTLP), and AWS CloudWatch, plus any OTLP endpoint (Export OpenTelemetry data); aggregated metrics download as CSV and per-datasource metrics APIs for model, MCP, guardrail, cache, routing, and agent metrics (Metrics dashboard)

Records user feedbackNo
Scores live trafficNo

n.a. — no feedback/score endpoint documented (Metrics dashboard)

n.a. on the fetched pages; the closest is audit_mode_blocked, a guardrail outcome that records what *would* have been blocked without blocking it (Metrics dashboard)

Depends on the vendor’s SaaS: Partially — the gateway is customer-deployed and telemetry can be exported to the customer's own stack, but "Traces storage is managed by TrueFoundry" (Export OpenTelemetry data)

Retention: n.a. — no retention window stated (Request logging)

Whether it fits how you work

How much work stands between you and a first call, how different that is from running it in production, and whether it slots into the stack you already have. Recorded as the shape of the work rather than a number of minutes — how long it takes you depends on which accounts and quota you already hold, which no comparison can know.

to try: cloud console setupto run: infrastructure rolloutfits 5 of 10 common stacks

Getting to a first call

4 numbered steps
Shape of the workSet it up in a cloud console

You cannot start from an empty editor. An account, a project or a deployed resource has to exist first, and that step is done by hand.

Read off: the vendor’s own quickstart — 4 numbered steps.

Before step one

Everything you need first: A TrueFoundry account (created in step 1) with a verified email, and a model-provider account plus API key to add models. Credit card, cloud account and cluster are not mentioned (Quick Start Guide: Setup & Integration - TrueFoundry Docs)

Copyable snippet: incomplete. client = OpenAI(api_key="your_truefoundry_api_key", base_url="{GATEWAY_BASE_URL}") — but no completed request call is shown on the page; snippets live in the Playground's Code Snippets tab (Quick Start Guide: Setup & Integration - TrueFoundry Docs)

Running it in production

The same scale applied to the path the vendor recommends for production traffic. Kept separate from the quickstart because for several products here the two are barely related pieces of work.

Shape of the workDeploy it on your infrastructure

This is an infrastructure project, not an integration. Expect charts or Terraform, networking, secrets and someone who owns the deployment.

Getting to production is a step up in kind from the quickstart, not just more of the same.

What production needs: An organization registered on TrueFoundry; Kubernetes 1.27+ with nodes having at least 100 GB ephemeral disk; dynamic PVC provisioning; an ingress controller or Istio service mesh; a domain and TLS certificate; outbound HTTPS access to TrueFoundry endpoints; PostgreSQL >= 13 (managed recommended); S3/GCS/Azure Blob or other S3-compatible blob storage; and TrueFoundry-provided Tenant Name, License Key (TFY_API_KEY) and Image Pull Secret. Sizing: medium tier ≥3 AI Gateway replicas (~500 req/s), large tier ≥10 replicas (~2,000 req/s) (Deploy AI Gateway - TrueFoundry Docs)

Can you run it yourself

Install command publishedOffered, but no command published

Self-hosting is advertised and the requirements are described, but no page publishes a command to start from. Expect to talk to the vendor before you can run it.

No literal install command is published on the self-hosting pages; they document topologies and prerequisites only (Self-hosting of Gateway - TrueFoundry Docs, Deploy AI Gateway - TrueFoundry Docs)

How it fits your stack

5 of 10

Each row is a thing you might already run. “With a caveat” means it works but not the way the vendor’s marketing implies — read the reason, because that is usually where the surprise lives.

  • FitsThe OpenAI SDKDrop-in once set up — but first-call work is cloud console setup.
  • NoThe Vercel AI SDKNo AI SDK route documented.
  • NoCloudflare WorkersNo Workers guidance published.
  • FitsKubernetestfy-llm-gateway for the Gateway Plane, truefoundry for the full control plane plus gateway plane
  • With a caveatTerraform or OpenTofuTerraform code in the repo only
  • NoAn existing API gatewayNothing published about running behind your gateway.
  • FitsCloud IAM I already runReuses IAM roles, workload identity or managed identities.
  • FitsLangChain or LlamaIndexLangChain, LangGraph, LlamaIndex
  • FitsMCP servers to governActs as an MCP gateway or registry.
  • NoNothing — plain Node or PythonA cloud console or resource has to exist before your first call.

Reading this the other way round — pick what you already run and see every product scored against it.

The integration surfaces behind those answers

  • Vercel AI SDKNot documented

    Nothing published. Assume the OpenAI-compatible route and verify it yourself.

    n.a. (not documented)

  • Cloudflare WorkersNot documented

    No Workers guidance either way. If you are edge-first, verify fetch-only compatibility yourself.

    n.a. (not documented)

  • KubernetesOfficial Helm chart

    A named, published chart. You can read its values file before committing to anything.

    Named: tfy-llm-gateway for the Gateway Plane, truefoundry for the full control plane plus gateway plane

    The AI Gateway can be deployed on Kubernetes using the tfy-llm-gateway Helm chart at https://github.com/truefoundry/infra-charts/tree/main/charts/tfy-llm-gateway. A separate full TrueFoundry control-plane and gateway-plane installation uses the truefoundry chart from https://github.com/truefoundry/infra-charts/tree/main/charts/truefoundry. (TrueFoundry deployment overview)

  • TerraformTerraform code in the repo only

    Terraform exists inside the repo or product but nothing is published for you to consume. You will be adapting their code, not calling a module.

    Named: the software ships with OpenTofu/Terraform code; no provider, module or Registry name is documented

    TrueFoundry states that its software ships with OpenTofu/Terraform code, but no official Terraform provider, module, or Terraform Registry name is documented. (TrueFoundry deployment overview)

  • Existing API gatewayNot documented

    Nothing published about sitting behind an existing gateway. Treat it as a separate hop you route to yourself.

    n.a. (not documented)

  • Cloud identityReuses your cloud identity

    Authenticate with the identity you already run — IAM roles, workload identity or managed identities. No long-lived key to rotate.

    Yes for platform-level cloud access: TrueFoundry documents AWS IAM Roles for Service Accounts (IRSA, sts:AssumeRoleWithWebIdentity with the eks.amazonaws.com/role-arn annotation) and GCP GKE Workload Identity; Azure is listed as "Coming Soon". (TrueFoundry docs - access cloud services with IAM)

  • MCPMCP gateway or registry

    It sits in front of many MCP servers and governs access to them. This is the shape you want if MCP sprawl is the problem you are solving.

    TrueFoundry documents an MCP Gateway that centralizes access to MCP servers, with dynamic tool discovery and invocation, a curated registry, and support for remote, virtual, OpenAPI, and stdio MCP servers over streamable HTTP. It also documents centralized authentication and combining tools from multiple MCP servers through Virtual MCP Servers. (TrueFoundry MCP Gateway overview)

Python frameworks
  • LangChain
  • LangGraph
  • LlamaIndex

TrueFoundry documents ready-to-use LangChain and LlamaIndex snippets. The LangChain integration uses pip install langchain-openai and the ChatOpenAI class; LangGraph examples use StateGraph, MessagesState, and related workflow APIs. (TrueFoundry LangChain integration)

First-party client libraries
  • Python
  • Node.js
  • Go
  • Rust
  • Java

Documented client and integration formats include OpenAI SDK, LangChain, LangGraph, Google ADK, LlamaIndex, Go-OpenAI, Rust-OpenAI, LangChain4j, Node.js, cURL, REST, and streaming APIs. Official OpenAI Agents SDK packages are openai-agents for Python and @openai/agents for JavaScript/TypeScript; documented classes include Agent, Runner, OpenAIChatCompletionsModel, and AsyncOpenAI. (TrueFoundry OpenAI Agents SDK integration)

Agent features: Through the OpenAI Agents SDK integration, TrueFoundry documents multi-agent workflows, handoffs, function tools, human-in-the-loop workflows, automatic tracing, session management, and conversation history across agent runs. The documentation does not state A2A support. (TrueFoundry OpenAI Agents SDK integration)

No install commands are given in the quickstart at all; auth uses a PAT or VAT from the Access section, and multiple provider accounts/keys can be added. Self-hosted PostgreSQL 17+ requires disabling SSL (force_ssl=0 on AWS, require_secure_transport=false on Azure) (Quick Start Guide: Setup & Integration - TrueFoundry Docs, Deploy AI Gateway - TrueFoundry Docs)

TrueFoundry supports fully managed SaaS, hybrid, private-VPC, on-premises, and fully self-hosted deployments, including use on any infrastructure or cloud provider and as a standalone gateway. The self-hosted Gateway Plane is stateless and connects securely to the TrueFoundry control plane over WebSockets; the gateway supports providers including OpenAI, Anthropic, Azure, and 1000+ LLMs overall. (TrueFoundry introduction to the LLM Gateway)

Silence in the docs: 4 of the integration questions on this card have no published answer either way. That is recorded as undocumented, not as a no — but it does mean you would be verifying it yourself.

What it does well

  • Broad routing feature set: weight, latency and priority routing, fallback chains, geo-aware routing and virtual models
  • Both simple and semantic caching plus budget and rate-limit controls per team, model and key
  • Deployable in the customer's VPC, on-prem, air-gapped or multi-cloud with SOC 2 and HIPAA/GDPR-ready configurations
  • Detailed built-in observability (P50/P90/P99, TTFT, traces, alerts) and MCP gateway with RBAC

Where it falls short

  • Not open source: the data plane is proprietary, so there is no free self-host path or community audit
  • Steep jump from the $0 Developer tier to $499/month Pro, with overage billed in $499 blocks
  • Performance figures are vendor claims only; a third-party review explicitly flags them as unverified
  • Self-hosting adds an estimated $600-$1,000/month of infra plus Helm/Kubernetes operational burden

Choose it when

Enterprises that want a fully managed or in-VPC AI gateway with guardrails, MCP governance and SSO/RBAC, and are comfortable with closed source.

Look elsewhere when

You require an open-source, auditable gateway or want to avoid per-request pricing tiers as volume grows.

Cloud platform: A hyperscaler surface offering several vendors models under one contract, one bill, and one identity system. Strong compliance story, limited to that cloud.

How hard is it to leave?

Derived from six published facts, not from an opinion. The weights are fixed and the same for every product — see the arithmetic.

100/100Easy to leave
Portability score breakdown for TrueFoundry AI Gateway
What helps you leave Points Source
Works with standard OpenAI codeSwitching away is a base-URL change rather than a rewrite of every call site.22/22 vendor page
No vendor-specific SDK requiredA proprietary client library spreads through your codebase and has to be torn out again.10/10 vendor page
Can use your own provider accountsYour keys and billing relationship stay yours, so removing the gateway does not cut off model access.20/20 vendor page
Can be self-hostedYou can run it yourself instead of accepting a pricing or policy change.20/20—
Configuration lives in version controlRouting and budget rules are a file you keep, not dashboard state you would have to rebuild.16/16 vendor page
Your request history can be exportedYou leave with your own logs instead of abandoning them.12/12 vendor page

Read the fine print: Export is requested by email and returned as JSON, not self-serve.

All six inputs are published, so this is scored against the full 100. This measures technical switching cost only. It does not price the engineering time to re-test prompts against a different routing stack.

Full specification

Every field we track. Blank fields say "Not published" rather than "No" — we do not infer an absence from silence. Switch to Technical in the header for the precise field names and the low-level details.

Overview

What kind of product Category
Cloud platform Not verified
Marketplaces resell many providers behind one key. Gateways add governance on top. Open-source projects you run yourself. Cloud platforms are hyperscaler surfaces. Inference providers host models on their own hardware.
Who runs it Deployment model
Managed or self-host Not verified
Managed means the vendor operates it. Self-host means you run it on your own infrastructure. Both means you can choose.
Licence Licence
Proprietary Verified 3 days ago
Proprietary products cannot be inspected or forked. Open licences such as MIT and Apache-2.0 let you audit, modify, and run the code without permission.
Company Company
TrueFoundry Not verified
The organisation that maintains the product.
Who you would be signing with Vendor status
Independent company Not verified
Whether the product is still an independent company, has been acquired, is a large cloud vendor’s product line, is run by a software foundation, or has been put into maintenance mode. Maintenance mode means bug fixes and security patches only — no new features.
Last shipped an update Latest release
2026-08-21 Not verified
The date of the most recent release or version tag. A product that has not shipped in a year is a different risk from one that shipped last week, regardless of what its marketing site says.
GitHub stars GitHub stars
Not published Not verified
A rough proxy for community size on open-source projects. Not a quality measure.

Cost

Markup on model prices Token markup
Not published Not verified
How much the product adds on top of what the underlying model provider charges. Zero means you pay the same per-token price you would pay the model provider directly.
Fee to add funds Credit purchase fee
Not published Not verified
A percentage charged when you top up your balance, separate from token prices. It is easy to miss because it does not appear on the per-token price list.
Monthly cost per person Seat fee
Not published Not verified
A recurring per-user platform charge that applies regardless of how much you use the models.
Can use your own provider accounts BYOK supported
Yes Verified 3 days ago
Bring Your Own Key: you keep direct contracts with OpenAI, Anthropic and others, and the gateway only routes traffic. This preserves negotiated rates and committed-spend discounts.
Cost of using your own accounts BYOK terms
You connect your own provider accounts and keys; TrueFoundry bills by request tier, not tokens. Not verified
What the product charges to route traffic through your own provider keys.
Free tier Free tier
Developer plan $0/month: 50,000 requests/month, 3 users, 50,000 MCP tool calls, up to 10 saved prompts. Verified 3 days ago
What you can do without paying, useful for evaluation.
Enterprise plan from Enterprise plan from
Not published Not verified
Annual entry price for the enterprise tier, where one is published or credibly reported.
Cost to run it yourself Self-host cost
No open-source edition; the data plane is proprietary. Vendor estimates roughly $600-$1,000/month of your own hosting cost to run the gateway (or control plus gateway) plane in your cloud, on top of the plan fee; fully managed SaaS carries no hosting cost. VPC, on-prem, air-gapped and multi-cloud installs are enterprise deployments, and a third-party review notes Helm-based ops effort is required. Verified 3 days ago
What self-hosting actually costs once you account for infrastructure and any paid tier.
How the vendor makes money Pricing model
Flat monthly platform fee Not verified
The shape of the vendor’s bill: does the routing layer charge a percentage on top of tokens, a flat monthly fee, both, neither (because inference is the product), or nothing at all (open source with no paid tier).
How pricing works, briefly Pricing model detail
Free developer tier + flat monthly platform fee + per-unit usage overage + enterprise quote + optional self-host infra cost. Developer $0/mo, Pro $499/mo, Pro Plus $2,999/mo, Enterprise custom. TrueFoundry does not resell inference. Not verified
A one-paragraph description that covers the caveats a pricing category cannot: introductory rates, per-feature meters, tier gating, and pricing that resets on a specific date.
Minimum commitment Minimum commitment
None stated. Enterprise implied custom. Not verified
Whether the vendor requires a minimum contract term, a minimum spend, or a provisioned-capacity purchase to get its published rate.
Charges that fire after you go over an allowance Overage terms
Request overage priced in blocks: 2M requests and 5 API keys for additional $499/month; Pro Plus 'contact sales'. Self-hosting adds ~$600–$1,000/month of infrastructure cost when you self-host the Gateway or both the Control and Gateway planes. Log/trace retention overage and export cost not stated. Not verified
The line items that scale with usage after an included allowance is exhausted — log storage, extra requests, per-feature meters, data export — which is where cost estimates usually go wrong.
Prompt cache offered Cache mechanism
Both exact and semantic Not verified
Whether the gateway offers its own response cache, what kind of match it does (exact request, prefix, semantic), or simply passes provider caching through unchanged.
Discount on cached input Cache-read discount
Not published Not verified
How much cheaper cached tokens are than fresh input, when the vendor publishes a single figure. Bundled-inference clouds usually price this per model instead of as one number.
Premium on cache writes Cache-write premium
Not published Not verified
How much more the first write of a cached prefix costs versus a plain input token. A high write premium and a low hit rate can leave you paying more than you save, so this matters as much as the read discount.
Who captures the cache saving Cache economics
Exact ('Simple caching') and semantic caching both listed as features; provider passthrough not stated. No cached-token pricing published. TrueFoundry charges per request/platform, not per token, so caching cuts your provider bill and you keep the saving — but cached requests presumably still consume the tier's request allowance, and the feature list does not mark which tier includes semantic caching. Not verified
Whether the customer keeps the full saving from caching or the vendor captures part of it — and any conditions attached (write premium, storage fees, best-effort hits).
What you can split spend by Cost attribution
'Cost per team/user/model/application' plus custom metadata and metadata filtering. Per-key, per-tag and per-customer not stated. Not verified
The dimensions the vendor documents for splitting spend — per key, per user, per team, per tag, per customer. Matters if you need to chargeback internally or bill an end customer.
How you get cost data out Cost export
'Export to custom storage buckets', 'Data Lake Export', 'Connect multiple storage bucket' and 'Export to other monitoring platforms'. CSV, cost API and webhook export not stated; export pricing not stated. Not verified
The mechanisms the vendor publishes for exporting cost and usage data: CSV, an API, webhooks, S3, a data warehouse, or nothing at all. Any per-unit price is included.
Who pays the model bill BYOK mode
Your keys only Not verified
Whether you bring your own provider accounts (BYOK), buy inference from this vendor, or can do either. This is the single biggest commercial difference between these products: it decides who holds the contract with the model provider and who carries the spend.

Spend governance in detail

Seven signals matter when a bill starts to hurt: who can spend, how much, on what, and who gets paged when it goes wrong. Everything below is drawn from the vendor’s own pricing and docs pages — how we read these.

  • Virtual or scoped keysYes

    Keys that carry their own budget and rate-limit policy, so an intern experiment cannot spend against a production budget.

    'Virtual models' are listed; per-tier API key counts (5 extra keys bundled with the $499 overage pack). Not branded as virtual keys explicitly.

  • Budget caps per keyNot published

    A dollar or token ceiling attached to an individual key. Where enforcement is soft, one over-limit request still completes before the block kicks in.

    Not stated as per-key.

  • Budget caps per team or workspaceYes

    A ceiling applied at a higher scope than one key — a team, a workspace, a customer, or an entire environment.

    'Budget limiting' listed without scope or tier.

  • Rate limiting as a cost controlYes

    Configurable request-per-time-window caps. Platform-set rate limits do not count as spend controls; user-configurable ones do.

    Listed, tier unstated.

  • Model allowlistsYes

    A policy that constrains which models a key or team can call, keeping expensive frontier models out of the wrong hands.

    'RBAC on models' and 'RBAC on MCPs' are listed — allowlisting by role rather than a named allowlist.

  • Spend alertsYes

    Alerts fired as spend approaches a threshold. Alerts that only fire after the meter has rolled over are marked as such.

    'Alerts' listed, tier unstated.

  • Webhook notificationsNot published

    Programmatic notifications on spend events, so budget breaches can page an on-call or open a ticket.

    Not stated.

Enforcement:Enforcement point not stated

Catalog

Models available Models available
~1,000 Verified 3 days ago
How many models you can call, shown as a range because several vendors publish different totals on different pages. Vendor-reported either way, so counts are not directly comparable — some count every provider variant of the same model separately.
Model providers reachable Upstream providers
27 Not verified
How many distinct model providers or labs you can reach, shown as a range where the vendor’s own pages disagree. More providers usually means better redundancy when one has an outage.
Works with standard OpenAI code OpenAI-compatible API
Yes Verified 3 days ago
If yes, you can usually switch to it by changing one base URL, and switch away just as easily. This is the main defence against lock-in.
OpenAI chat endpoint POST /v1/chat/completions
Yes Not verified
The endpoint almost every application ports first. "Not documented" means the vendor never states it, which is different from a documented no.
Anthropic messages endpoint POST /v1/messages
Yes Not verified
Whether Anthropic-shaped calls work without rewriting them. Several products support this only as SDK compatibility or provider passthrough rather than a native endpoint — the detail page says which.
OpenAI Responses endpoint POST /v1/responses
Not documented Not verified
The newer stateful OpenAI surface. Support is much thinner across this market than chat completions.
Embeddings endpoint POST /v1/embeddings
Yes Not verified
Whether you can generate vectors through the same gateway, or need a second integration for retrieval workloads.
Image generation endpoint POST /v1/images/generations
Yes Not verified
Whether image models are reachable through the same surface as text.
Audio endpoints POST /v1/audio/*
Yes Not verified
Speech-to-text and text-to-speech. Frequently the first gap in an otherwise complete gateway.
Batch jobs endpoint POST /v1/batches
Yes Not verified
Asynchronous bulk processing, usually at a discount. Commonly undocumented, and commonly the reason a migration stalls late.
Needs the vendor’s own code library Requires a vendor-specific SDK
No Verified 3 days ago
A proprietary client library spreads through your codebase and has to be torn out again if you leave. “No” is the better answer here, and it means the standard OpenAI client works.
You can export your request history Logs / usage data export
Yes Verified 3 days ago
Whether you can get your own request logs, traces, or usage records back out — through an API, a bulk export, or a download. Decides whether you leave with your history or abandon it.
Settings can live in version control Declarative config-as-code
Yes Verified 3 days ago
Whether routing, fallback, and budget rules can be declared in a file you keep in Git, rather than existing only as settings clicked into a hosted dashboard.
Embeddings Embeddings
Yes Verified 3 days ago
Text-to-vector models, needed for search and retrieval features.
Image generation Image generation
Not published Not verified
Whether image models are reachable through the same interface.
Speech and audio Speech and audio
Not published Not verified
Text-to-speech or transcription models through the same interface.
Video generation Video generation
Not published Not verified
Whether video models are reachable through the same interface.
Batch processing Batch processing
Not published Not verified
Submitting large jobs for cheaper, slower processing. Often 50% off for work that is not time-sensitive.

Routing & reliability

Uptime it promises in writing Contractual SLA uptime
99.9% Not verified
The uptime percentage in a published, contractual service level agreement. A public status page is not an SLA — it reports what happened, it does not promise anything or pay you back when it breaks.
Automatic failover Automatic failover
Yes Verified 3 days ago
When a model provider goes down or rate-limits you, traffic moves to a backup automatically instead of returning errors to your users.
Load balancing Load balancing
Yes Verified 3 days ago
Spreads requests across several providers or keys to raise your effective rate limit.
Rule-based routing Conditional routing
Yes Verified 3 days ago
Send different requests to different models based on rules — for example a cheap model for free users and a strong model for paying ones.
Response caching Response caching
Yes Verified 3 days ago
Reuses the answer when the exact same request comes in again, which cuts both cost and latency.
Similar-question caching Semantic cache
Yes Verified 3 days ago
Reuses an answer when a new question means roughly the same thing as an earlier one. Saves far more than exact-match caching but can return subtly wrong answers if tuned loosely.
Where you set the timeout Request timeout surface
Not documented Not verified

`not_documented` as a request-timeout key. What exists are SLA thresholds used for health, not for aborting a request: `sla_cutoff.time_to_first_token_ms` and `sla_cutoff.time_per_output_token_ms` ([TrueFoundry virtual models / fallback](https://www.truefoundry.com/docs/ai-gateway/fallback)).

Where a request timeout can be set: per request, in a config file, in the vendor dashboard, or nowhere. Nine of the twenty products documented here do not describe a request timeout at all, so the worst case of a hung upstream call is unknowable from the docs.
Where you set retries Retry policy surface
In config Not verified

`config_file` (also editable in the UI). `retry_config` with `attempts` (**default `0`**), `delay` (**default `100` ms**), and `on_status_codes` (**default `["429","500","502","503"]`**). Retries occur on the same target; backoff beyond the fixed `delay` is not documented ([TrueFoundry routing config](https://www.truefoundry.com/docs/ai-gateway/load-balancing-overview), [TrueFoundry virtual models / fallback](https://www.truefoundry.com/docs/ai-gateway/fallback)).

Where retry count and backoff are configured. Worth knowing alongside billing: a retried streaming call can be charged more than once.
Where you set fallbacks Fallback surface
In config Not verified

ORDERED for priority-based routing (`priority`, lower number = higher priority, `0` highest); fallback targets come from `load_balance_targets`, gated by `fallback_status_codes` (default `["401","403","404","408","429","500","502","503"]`) and `fallback_candidate` (default `true`). Ordering depends on strategy: latency-based puts lower-latency targets first; weight-based sticky routing tries remaining healthy targets in sequence and that order is explicitly "not weight-based" ([TrueFoundry virtual models / fallback](https://www.truefoundry.com/docs/ai-gateway/fallback)).

Where the fallback chain is defined. Almost every product claims fallback; the useful question is whether you can change it from code or only by hand in a dashboard.
Shape of the fallback chain Fallback shape
Ordered list Not verified
An ordered list tries targets in sequence; a weighted split sends a percentage of traffic to each, which is what you need to trial a new model on 5% of requests. Weighted splits are much rarer than the marketing implies.
Upstream health tracking Health checks / circuit breaking
In config Not verified

`config_file`, and it is passive health tracking plus cooldown ejection. The gateway "continuously monitors every target", marks failing targets unhealthy, moves them to the end of the list as last resort, and recovers automatically. Failure-based cooldown: 5xx/429/401/403, default threshold 2+ failures in a rolling 2-minute window. SLA-based cooldown (priority routing only) via `sla_cutoff.time_per_output_token_ms` / `time_to_first_token_ms`, averaged over a 3-minute rolling window using up to 10 samples with a minimum of 3; TTFT samples only come from streaming requests and TPOT samples require at least 30 output tokens ([TrueFoundry virtual models / fallback](https://www.truefoundry.com/docs/ai-gateway/fallback)). TrueFoundry's blog additionally describes classic circuit-breaker states (`closed`/`open`/`half_open`) as a pattern, but states no configuration keys for it ([TrueFoundry LLM failover blog](https://www.truefoundry.com/blog/llm-failover-load-balancing-provider-outages)).

Whether the product notices a failing upstream and stops sending traffic to it, and whether you can tune the thresholds. This is what turns a provider outage into a blip rather than a sustained error rate, and only four of the twenty expose it.
Cross-region failover you control Multi-region failover surface
In config Not verified

SaaS requests are auto-tagged with `tfy_gateway_region` and `tfy_gateway_zone`, and you route on them with `metadata_match` (documented examples `US → azure-us/gpt-4o`, `EU → azure-eu/gpt-4o`, `IN → azure-in/gpt-4o`, and `metadata_match: {tfy_gateway_zone: SFO}`), with a catch-all `priority: 1` target as the cross-region fallback. Both are true here: TrueFoundry runs multiple SaaS gateway regions AND you configure the cross-region target chain yourself ([TrueFoundry virtual models / fallback](https://www.truefoundry.com/docs/ai-gateway/fallback)). Multiple gateways can also be deployed cross-region under one control plane ([TrueFoundry gateway plane architecture](https://www.truefoundry.com/docs/platform/gateway-plane-architecture)).

Whether you can define what happens when a region degrades. A vendor running many regions is not the same as a vendor letting you configure failover between them; only three document a user-controlled mechanism.
Where you set load balancing Load balancing surface
In config Not verified

Weights supported: `routing_config.type` of `weight-based-routing` (integer `weight` 0–100, must sum to 100), `priority-based-routing`, `latency-based-routing` ("No weights needed", selection by recent TPOT), and `complexity-based-routing` (`simple`/`medium`/`complex`). Targets are catalog identifiers (`azure/gpt-4o`, `openai/gpt-4o`, `bedrock/llama`, `onprem/llama`), with `metadata_match`, `headers_override` and `override_params` per target; rules are evaluated in order and the first match wins ([TrueFoundry routing config](https://www.truefoundry.com/docs/ai-gateway/load-balancing-overview), [TrueFoundry virtual models / fallback](https://www.truefoundry.com/docs/ai-gateway/fallback)).

Where traffic distribution across upstreams or keys is configured.

Operations

Usage dashboards and logs Observability
Yes Verified 3 days ago
Built-in visibility into what was sent, what came back, what it cost, and how long it took.
Spending limits Budget controls
Yes Verified 3 days ago
Hard caps that stop spend before it becomes a surprise invoice. The single most valuable control for a small team.
Rate limits Rate limits
Yes Verified 3 days ago
Caps on request volume per key or per user, useful for protecting against abuse and runaway loops.
Separate keys per team or app Virtual keys
Not published Not verified
Issue scoped keys with their own budgets and permissions so you can attribute cost and revoke access without rotating everything.
Prompt versioning Prompt management
Yes Verified 3 days ago
Store and version prompts outside your code so they can be changed without a deploy.
Quality testing Evals
Not published Not verified
Built-in tooling to score model output against test cases, so you can tell whether a model swap made things better or worse.
MCP support MCP support
Yes Verified 3 days ago
Native support for the Model Context Protocol, the emerging standard for connecting models to external tools.
What gets logged Logged content
Your choice Not verified

Global logging mode is `HEADER_CONTROLLED` (default behaviour: log unless the header says otherwise), `ALWAYS`, or `NEVER`; logged requests, viewable under AI Gateway → Monitor → Requests, include request content, and sensitive values can be redacted per subject/model/metadata ([Request logging](https://www.truefoundry.com/docs/ai-gateway/request-logging))

Whether full prompts and responses are stored, only metadata, or nothing. Full-body logging is the most useful debugging feature here and the one most likely to need a conversation with your compliance team.
You can turn logging off Body-logging opt-out
Yes Not verified

Set the global mode to `NEVER`, or send `X-TFY-LOGGING-CONFIG: {"enabled": false}` (stringified JSON) per request; metrics dashboards remain populated ([Request logging](https://www.truefoundry.com/docs/ai-gateway/request-logging), [Metrics dashboard](https://www.truefoundry.com/docs/ai-gateway/analytics))

Whether prompt and response bodies can be suppressed while still keeping usage metrics. Nineteen of the twenty document a way to do this; the mechanisms range from a per-request header to an organisation-wide setting.
Traces you can take elsewhere Distributed tracing
OpenTelemetry Not verified

OpenTelemetry natively for both traces and metrics, with MCP tool and agent spans; trace storage is managed by TrueFoundry ([Export OpenTelemetry data](https://www.truefoundry.com/docs/ai-gateway/export-opentelemetry-data))

Whether the product emits OpenTelemetry, a proprietary format, or nothing. OpenTelemetry means the traces land in the tooling you already run instead of only in the vendor’s dashboard.
Where telemetry can go Export destinations
OpenTelemetry, Datadog, Prometheus, Grafana, S3, Webhook Not verified

OTLP exporters to Grafana Cloud, Datadog (OTLP), and AWS CloudWatch, plus any OTLP endpoint ([Export OpenTelemetry data](https://www.truefoundry.com/docs/ai-gateway/export-opentelemetry-data)); aggregated metrics download as CSV and per-datasource metrics APIs for model, MCP, guardrail, cache, routing, and agent metrics ([Metrics dashboard](https://www.truefoundry.com/docs/ai-gateway/analytics))

Documented sinks for logs and metrics. This is a good proxy for how replaceable the vendor’s own dashboard is: around twenty destinations means you never have to depend on it, while a CSV download means you do.
Can record user feedback Feedback capture API
No Not verified

`n.a.` — no feedback/score endpoint documented ([Metrics dashboard](https://www.truefoundry.com/docs/ai-gateway/analytics))

Whether there is an API to attach a rating or score to a logged request, which is what lets production traffic feed quality work later.
Scores live traffic Online eval hooks
No Not verified

`n.a.` on the fetched pages; the closest is `audit_mode_blocked`, a guardrail outcome that records what *would* have been blocked without blocking it ([Metrics dashboard](https://www.truefoundry.com/docs/ai-gateway/analytics))

Whether automated scorers can run against real production requests, rather than only against a test set you assemble yourself.

Performance

Delay it adds Proxy overhead
3 ms Verified 3 days ago
Extra time the product itself adds to each request, on top of however long the model takes. Usually irrelevant next to multi-second model latency, but it matters for high-volume or streaming-sensitive workloads.
Requests per second ceiling Throughput
Not published Not verified
Published sustained request rate before the product becomes the bottleneck. Only relevant at genuinely high volume.
What the request path runs on Architecture class
Interpreted proxy Not verified

`interpreted_proxy` (JS/edge runtime). "The AI Gateway is written using the Hono framework", described as "ultra-fast, minimalistic, and is designed for the edge" ([TrueFoundry gateway plane architecture](https://www.truefoundry.com/docs/platform/gateway-plane-architecture)).

The comparable way to talk about latency here. An edge worker, a compiled Go or Rust binary, and a Python proxy have different overhead floors no matter which figures each vendor publishes. Products that never disclose their runtime are recorded as undisclosed rather than assumed.
You can run the request path yourself Self-hostable data plane
Yes Not verified

`yes`, but the artifact is not named on the fetched pages. Deployment is documented as SaaS, your VPC, on-prem or air-gapped, running as Kubernetes pods with readiness probes and optional `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY` forward-proxy env vars; no Docker image name or Helm chart name is stated ([TrueFoundry gateway plane architecture](https://www.truefoundry.com/docs/platform/gateway-plane-architecture), [TrueFoundry LLM failover blog](https://www.truefoundry.com/blog/llm-failover-load-balancing-provider-outages)).

Whether the component that actually carries your prompts can run on your own infrastructure. Distinct from a vendor offering a self-hosted control plane while still proxying traffic through their network.
Streaming responses Streaming support
Yes Not verified

Supported, with the most precise streaming-fallback semantics in this set: the gateway waits for the "first non-empty stream chunk", and if Anthropic returns `overloaded_error` before that chunk it falls back to the next eligible target. Separate caveat: `prompt_version_fqn` "does not work with agents (when using MCP/tools)". Note also that TTFT SLA samples are only recorded on streaming requests ([TrueFoundry routing config](https://www.truefoundry.com/docs/ai-gateway/load-balancing-overview), [TrueFoundry virtual models / fallback](https://www.truefoundry.com/docs/ai-gateway/fallback)).

Whether token-by-token streaming is documented. The caveats matter more than the yes: some products cannot cancel a stream without still being billed, and several timeout and fallback mechanisms stop applying once the first token has been sent.

Security & compliance

Does your prompt reach their servers Prompt transits vendor
Depends how you deploy it Not verified

Three genuinely different data paths: managed SaaS where request and response data flows through TrueFoundry's control plane; gateway-in-your-infrastructure where all LLM traffic stays with you and TrueFoundry is not in the live request path; and fully self-hosted, where everything except the authentication and analytics servers runs on your infrastructure.

Whether the text you send passes through this company’s own infrastructure. If it does, every other promise on this page is a policy commitment rather than a physical impossibility. Self-hosted products can answer no outright.
What they keep if you change nothing Logging default
Yes — prompts and replies Not verified

On hosted deployments logging happens unless you suppress it: absent or true in the logging-config header means logging occurs. You can control whether bodies are logged and redact patterns, and a deny rule always wins.

Defaults matter more than options. A product that stores full prompts and replies unless you find the right header will have stored them by the time you read the docs.
How long they keep it Default content retention (days)
90 days Not verified

Ninety days for SaaS blob storage, configurable. In self-hosted modes the control plane does no long-term retention.

Default retention for request content, in days. Zero means nothing is kept. Read the note: several products keep nothing as a rule but make timed exceptions for abuse review or specific models.
Could they train on your prompts Training on customer data
Not published — silence, not a no Not verified

Neither the security page nor the deployment docs address training.

Whether the vendor may use your prompts and outputs to train models. “Not published” means we could not find any position, which is not the same as a no — ask for it in writing.
Where it runs, and what you can pin Region and residency control
You choose the region where the gateway is deployed and where logs and traces are stored, and you can bring your own S3, Azure Blob or GCS bucket. Gateway planes run active-active across multiple regions. Not verified
Which regions are offered and whether you can force processing to stay in one. A global endpoint that silently picks a region is a different compliance story from an endpoint you pin yourself.
Where safety filters run Guardrail execution location
Either, depending on deployment Not verified

The gateway plane evaluates PII, prompt-injection and content-filtering policy in memory on the request path — and in self-hosted modes that plane is yours.

A filter that strips personal data only helps if it runs before the data leaves your boundary. If guardrails execute in the vendor’s cloud, the vendor has already received whatever you wanted redacted.
Who else touches the data Subprocessor list
https://www.truefoundry.com/docs/platform/security-and-compliance Not verified
The published list of third parties the vendor passes your data to. No list means you cannot know the full chain, which most data-protection agreements require you to.
SOC 2 audited SOC 2 audited
Yes Verified 3 days ago
An independent audit of security controls. Enterprise buyers and their procurement teams routinely require it.
Will sign a HIPAA agreement HIPAA BAA
Not published Not verified
Required before you may send protected health information through the service. Without a signed BAA, healthcare data is off limits.
GDPR commitments GDPR commitments
Yes Verified 3 days ago
Published data processing terms for handling personal data of people in the EU and UK.
Can keep data in the EU EU data residency
Not published Not verified
Requests can be processed inside the EU rather than routed to US infrastructure. Often the deciding constraint for European customers.
Does not retain your data Zero data retention
Not published Not verified
Prompts and responses are not stored after the request completes. Sometimes a paid add-on rather than the default.
Strips personal data PII redaction
Yes Verified 3 days ago
Detects and removes identifiers such as names, emails, and card numbers before the request reaches the model provider.
Content guardrails Content guardrails
Yes Verified 3 days ago
Policy checks on inputs and outputs — blocking unsafe content, enforcing formats, or catching prompt-injection attempts.
Runs fully disconnected Air-gapped deployment
Yes Verified 3 days ago
Can be deployed in a network with no internet access, which some regulated and defence environments require.
Blocks personal data in prompts PII / DLP enforcement
Can block the request Not verified

Built-in PII Detection and Azure AI Language PII/PHI guardrails ([Guardrails overview](https://www.truefoundry.com/docs/ai-gateway/guardrails-overview)). Custom plugins distinguish mutation from validation: a Presidio `POST /pii-redaction` plugin returns `transformed: true` with a full replacement `requestBody` to redact, whereas a Guardrails-AI `POST /pii-detection` plugin "only detects and reports PII without redacting it" ([Custom guardrails](https://www.truefoundry.com/docs/ai-gateway/custom-guardrails))

Whether personal data detection sits on the request path and can stop the call, merely inspects and forwards it, or is not documented. A control that only reports is a logging feature, not a policy control.
Blocks prompt injection Injection / jailbreak enforcement
Can block the request Not verified

Azure Prompt Shield is a built-in guardrail integration; external options include Model Armor, Prisma AIRS, CrowdStrike, and NeMo ([Guardrails overview](https://www.truefoundry.com/docs/ai-gateway/guardrails-overview))

Whether injection and jailbreak detection can stop a request. Most products offering this call a partner classifier rather than shipping their own.
Blocks harmful content Toxicity / moderation enforcement
Can block the request Not verified

Azure Content Safety moderation and OpenAI Moderations are supported; blocked requests return a configurable `custom_error_message` ([Guardrails overview](https://www.truefoundry.com/docs/ai-gateway/guardrails-overview), [Guardrails configuration](https://www.truefoundry.com/docs/ai-gateway/guardrails-configuration))

Whether hate, violence, sexual and self-harm categories are checked inline and can stop a request, in either direction.
Your own policy rules Custom policy hooks
Can block the request Not verified

Policy forms: built-in Regex and Secrets Detection rules, plus custom HTTP plugins bound to lifecycle hooks `llm_input`, `llm_output`, `mcp_tool_pre_invoke`, `mcp_tool_post_invoke`, returning either a validate verdict or a mutated request body ([Guardrails overview](https://www.truefoundry.com/docs/ai-gateway/guardrails-overview), [Guardrails configuration](https://www.truefoundry.com/docs/ai-gateway/guardrails-configuration), [Custom guardrails](https://www.truefoundry.com/docs/ai-gateway/custom-guardrails))

Whether you can add your own rule — a regex, a webhook, or your own classifier — rather than choosing from the vendor library.
Where guardrails run Guardrail execution location
Either, your choice Not verified
Whether guardrail evaluation happens inside your infrastructure or on the vendor’s servers. This decides whether the prompt you are trying to protect leaves your network in order to be checked.
If the guardrail itself fails Guardrail failure mode
Not documented Not verified

`not_documented` as a global fail-open/fail-closed rule. For custom plugins, HTTP 4xx/5xx from the plugin signals a "processing or dependency failure only; not used for 'PII found' policy outcomes", which separates infrastructure failure from a policy verdict but does not state the resulting request disposition ([Custom guardrails](https://www.truefoundry.com/docs/ai-gateway/custom-guardrails))

What happens when the guardrail service times out or errors: does the request proceed unchecked, or is it blocked? This is the worst-documented field in the entire catalogue — only two vendors state it plainly, which means most teams are running a control whose failure behaviour they cannot know.
Third-party guardrail vendors Guardrail integrations
Azure AI Language, Azure Content Safety, OpenAI Moderation, Microsoft Presidio Not verified
Named external guardrail services the product can call. A long list means the product is a router for policy engines rather than a policy engine itself — which also means another vendor bill and another hop.

Compliance evidence

Graded by how strong the evidence is, not whether the word appears on the vendor’s website. An audited report and a marketing claim are different things, and only one of them will satisfy your own auditor.

  • SOC 2 Certified Type II programme for TrueFoundry SaaS
  • ISO 27001 Claimed, no evidence published asserted in deployment docs but absent from the security page's programme list
  • GDPR DPA Available on request standard DPAs available
  • HIPAA BAA Certified HIPAA programme maintained
  • FedRAMP Not published
  • ITAR Not published

Vendor source

Fit & integration

Work to try it Evaluation work shape
Set it up in a cloud console Not verified
The shape of the work on the vendor’s own quickstart, from swapping one base URL through to deploying infrastructure. An ordinal class rather than a duration, because elapsed time depends on accounts and quota we cannot see.
Work to run it Production work shape
Deploy it on your infrastructure Not verified
The same scale applied to the vendor’s recommended production path. For several products this is much heavier than the quickstart, which is exactly why both are recorded.
Steps on the quickstart Numbered quickstart steps
4 Not verified
A literal count of numbered steps on the vendor’s quickstart, recorded as evidence beside the work shape. Zero means the page publishes no numbered procedure at all. Large counts usually mean interleaved language tracks rather than more work.
Can you self-host it today Self-host install documentation
Offered, but no command published Not verified

No literal install command is published on the self-hosting pages; they document topologies and prerequisites only ([Self-hosting of Gateway - TrueFoundry Docs](https://www.truefoundry.com/docs/ai-gateway/gateway-self-hosted), [Deploy AI Gateway - TrueFoundry Docs](https://www.truefoundry.com/docs/platform/deploy-control-plane-and-gateway-plane))

Whether an install command is actually published. Several products advertise self-hosting while publishing no command to start from, which a plain yes/no would hide.
Works with the OpenAI SDK OpenAI SDK drop-in
Yes Not verified

Yes. TrueFoundry documents OpenAI-compatible drop-in usage by changing the client `base_url` and API key; the documented SaaS base URL is `https://gateway.truefoundry.ai`, while self-hosted examples use `{GATEWAY_BASE_URL}`. ([TrueFoundry AI Gateway quick start](https://www.truefoundry.com/docs/ai-gateway/quick-start))

Whether an existing OpenAI-compatible client can be pointed at it by changing the base URL and key.
Vercel AI SDK support AI SDK provider package
Not documented Not verified

n.a. (not documented)

Whether a first-party AI SDK provider package exists, or only a community package, a documented workaround, or the generic OpenAI provider pointed at a custom base URL.
Python framework integrations Documented Python frameworks
LangChain, LangGraph, LlamaIndex Not verified

TrueFoundry documents ready-to-use LangChain and LlamaIndex snippets. The LangChain integration uses `pip install langchain-openai` and the `ChatOpenAI` class; LangGraph examples use `StateGraph`, `MessagesState`, and related workflow APIs. ([TrueFoundry LangChain integration](https://www.truefoundry.com/docs/ai-gateway/langchain))

LangChain, LangGraph, LlamaIndex and similar orchestration frameworks with a documented integration.
Callable from Cloudflare Workers Cloudflare Workers support
Not documented Not verified

n.a. (not documented)

Whether the docs show calling this product from your own Worker. Deliberately separated from the several products whose own gateway runs on Workers, which is a fact about their infrastructure and not about your edge compatibility.
Kubernetes install Helm chart availability
Official Helm chart Not verified

The AI Gateway can be deployed on Kubernetes using the `tfy-llm-gateway` Helm chart at `https://github.com/truefoundry/infra-charts/tree/main/charts/tfy-llm-gateway`. A separate full TrueFoundry control-plane and gateway-plane installation uses the `truefoundry` chart from `https://github.com/truefoundry/infra-charts/tree/main/charts/truefoundry`. ([TrueFoundry deployment overview](https://www.truefoundry.com/docs/platform/deployment-overview))

Whether a named, published Helm chart exists, versus Helm being referenced with no chart named, versus generic cluster documentation that has nothing to do with this product.
Terraform support Terraform provider or modules
Terraform code in the repo only Not verified

TrueFoundry states that its software ships with OpenTofu/Terraform code, but no official Terraform provider, module, or Terraform Registry name is documented. ([TrueFoundry deployment overview](https://www.truefoundry.com/docs/platform/deployment-overview))

Whether you can declare this in code: an official provider, official modules, resources inside a hyperscaler’s provider, a community provider, or only Terraform code shipped in a repo.
Reuses your cloud identity Cloud IAM reuse
Reuses your cloud identity Not verified

Yes for platform-level cloud access: TrueFoundry documents AWS IAM Roles for Service Accounts (IRSA, `sts:AssumeRoleWithWebIdentity` with the `eks.amazonaws.com/role-arn` annotation) and GCP GKE Workload Identity; Azure is listed as "Coming Soon". ([TrueFoundry docs - access cloud services with IAM](https://www.truefoundry.com/docs/access-data-from-s3-or-other-clouds-services))

Whether you can authenticate with IAM roles, workload identity or managed identities instead of another long-lived API key. Distinguished from products that only accept static upstream provider credentials.
Fits behind your API gateway API gateway integration
Not documented Not verified

n.a. (not documented)

Whether AI traffic can go through a gateway you already run, and who documented that — the product’s vendor or the gateway’s.
MCP support MCP surface shape
MCP gateway or registry Not verified

TrueFoundry documents an MCP Gateway that centralizes access to MCP servers, with dynamic tool discovery and invocation, a curated registry, and support for remote, virtual, OpenAPI, and stdio MCP servers over streamable HTTP. It also documents centralized authentication and combining tools from multiple MCP servers through Virtual MCP Servers. ([TrueFoundry MCP Gateway overview](https://www.truefoundry.com/docs/ai-gateway/mcp/mcp-overview))

Which kind of MCP support this is: a gateway that governs many MCP servers, a hosted MCP server you connect to, MCP tools accepted inside the completion API, or client tooling. These are different products behind one acronym.
Needs your own provider key Upstream provider key required
Required Not verified

Effectively yes to have any model: "Select the provider you want to add models from and add models after providing API Key," though the page does not state a first-call rule explicitly ([Quick Start Guide: Setup & Integration - TrueFoundry Docs](https://docs.truefoundry.com/gateway/quick-start))

Whether an upstream provider account and key must exist before your first call works. A prerequisite rather than a step, and it can differ between the hosted and self-hosted forms of the same product.
Gate before models work Model access gate
Not documented Not verified

n.a. (not documented) on the quickstart, self-hosting or pricing pages ([Quick Start Guide: Setup & Integration - TrueFoundry Docs](https://docs.truefoundry.com/gateway/quick-start), [Deploy AI Gateway - TrueFoundry Docs](https://www.truefoundry.com/docs/platform/deploy-control-plane-and-gateway-plane))

Whether an enablement click, a quota grant, a paid tier or an approval form stands between a valid key and a working model call.
Official client languages First-party client SDK languages
Python, Node.js, Go, Rust, Java Not verified

Documented client and integration formats include OpenAI SDK, LangChain, LangGraph, Google ADK, LlamaIndex, Go-OpenAI, Rust-OpenAI, LangChain4j, Node.js, cURL, REST, and streaming APIs. Official OpenAI Agents SDK packages are `openai-agents` for Python and `@openai/agents` for JavaScript/TypeScript; documented classes include `Agent`, `Runner`, `OpenAIChatCompletionsModel`, and `AsyncOpenAI`. ([TrueFoundry OpenAI Agents SDK integration](https://www.truefoundry.com/docs/ai-gateway/openai-agents-sdk))

Languages with a first-party client library. An empty list can still mean the product is usable from any language via an OpenAI-compatible SDK.

Additional charges

These are the fees that do not appear on a per-token price list, and they are where estimates usually go wrong.

  • Pro plan $499/month (1M requests, 10 users)
  • Pro Plus plan $2,999/month (1M requests, 25 users)
  • Request/key overage $499/month per additional 2M requests and 5 API keys

How pricing actually works

No open-source edition; the data plane is proprietary. Vendor estimates roughly $600-$1,000/month of your own hosting cost to run the gateway (or control plus gateway) plane in your cloud, on top of the plan fee; fully managed SaaS carries no hosting cost. VPC, on-prem, air-gapped and multi-cloud installs are enterprise deployments, and a third-party review notes Helm-based ops effort is required.

Back to top ↑

Common questions

Answered from the fields above, so these move when the catalog moves. Every figure quoted here appears in the specification with its source.

Does TrueFoundry AI Gateway charge a markup on model prices?

TrueFoundry AI Gateway does not publish a token markup figure. Other charges on the page: pro plan ($499/month (1M requests, 10 users)), pro plus plan ($2,999/month (1M requests, 25 users)) and request/key overage ($499/month per additional 2M requests and 5 API keys).

Can TrueFoundry AI Gateway be self-hosted?

Yes. TrueFoundry AI Gateway can be run on your own infrastructure or used as a managed service. The licence is Proprietary. Air-gapped installation is supported.

Is TrueFoundry AI Gateway SOC 2 audited, and will it sign a HIPAA BAA?

A SOC 2 report is available for TrueFoundry AI Gateway but it does not publish a HIPAA business associate agreement. It offers a GDPR data processing agreement. Each of these is linked to the vendor's own page in the compliance section below.

Does TrueFoundry AI Gateway retain your prompts?

TrueFoundry AI Gateway does not publish a zero-data-retention position. Whether prompt and response bodies are logged is configurable. Logging can be turned off. Stated retention is 90 days.

Can you use your own provider keys with TrueFoundry AI Gateway?

Yes. TrueFoundry AI Gateway can route through your own accounts with the underlying model providers, so inference is billed to you directly. You connect your own provider accounts and keys; TrueFoundry bills by request tier, not tokens.

How many models does TrueFoundry AI Gateway support?

TrueFoundry AI Gateway states ~1,000 models, drawn from 27 upstream providers. Vendor-stated 1,600+ models (chat, completion, embedding, reranking) on the AI Gateway page; other vendor pages cite 250+ LLMs. The figure on this page is dated and carries its source.

Back to top ↑

What has changed here

  1. catalog entry catalog entry Not published Added to the catalog source ↗
See this in the full changelog Back to top ↑

Usually weighed against

TrueFoundry AI Gateway vs Amazon Bedrock Compare → TrueFoundry AI Gateway vs Azure AI Foundry Compare → TrueFoundry AI Gateway vs Cloudflare AI Gateway Compare →