Open source Self-host only MIT

LiteLLM

Open-source AI gateway and Python SDK that puts one OpenAI-compatible API in front of many LLM providers.

Built by BerriAI, founded 2023 · US company

Access at a glance

Whether this product can work for you at all, before features matter: who pays the model bill, where it can run, and which of your existing API calls keep working. Every value is the vendor’s own claim, linked to the page it came from.

Titled "LiteLLM AI Gateway (LLM Proxy)" and elsewhere "LiteLLM Proxy (LLM Gateway)", it is a self-hosted proxy that maps many provider APIs to OpenAI-format input/output (LiteLLM AI Gateway, LiteLLM providers). Not a model platform: "input, output and exceptions are mapped to the OpenAI format for all supported models" (LiteLLM proxy client examples).

Who pays the model bill

Your keys only

You contract with each model provider directly and hold those accounts. The gateway never resells inference.

The proxy is configured with the customer's own upstream provider keys, and it can additionally "allow users to pass in their OpenAI API key"; callers authenticate with LiteLLM virtual keys (e.g. sk-test-proxy-key-123) (LiteLLM proxy client examples).

Merchant of record: n.a. as an explicit statement: the LiteLLM pages fetched do not state who invoices model usage; LiteLLM Enterprise sells the software licence while the deployment runs on the customer's infrastructure and keys (LiteLLM Enterprise, LiteLLM proxy client examples).

Key handling: Keys stay in the customer's deployment: Kubernetes manifests use a Secret for keys and a ConfigMap for config.yaml (LiteLLM production deployment), and Enterprise adds secret-manager integrations — AWS Secrets Manager-style vaults, Google KMS, Google Secret Manager, HashiCorp Vault, CyberArk and custom secret managers — plus key rotations and read/write to the secret manager (LiteLLM Enterprise). Access is organized as "Organizations → Teams → Projects → Keys"; OSS includes request/response logging, Enterprise adds per-key/per-team routing to Langfuse, Langsmith and Arize, log export to GCS/Azure Blob, and "disable logging per team — GDPR-friendly opt-out at the team level" (LiteLLM Enterprise). Since it is self-hosted, no vendor plane sees request content by default (LiteLLM production deployment).

Where it can run

3 of 5 shapes documented
  • Vendor-hosted
  • Self-host
  • Your VPC
  • On-premise
  • Air-gapped

Dimmed shapes are not documented by the vendor, which is not the same as unsupported.

self-host via Docker, Helm, Terraform, Kubernetes, or a local binary (LiteLLM production deployment, LiteLLM proxy client examples); hybrid/VPC in the customer's own cloud (the AWS Terraform module reuses an existing VPC, subnets, database and Redis) (LiteLLM production deployment); air-gapped on Enterprise — "self-hosted or air-gapped, and yours to run" (LiteLLM Enterprise). Hosted SaaS: n.a. (not documented on the pages fetched).

Official images publish to ghcr.io/berriai mirrored at docker.litellm.ai/berriai (e.g. docker.litellm.ai/berriai/litellm:v1.90.2); the litellm-helm chart covers monolithic deployments on EKS/GKE/AKS and a componentized litellm chart splits gateway (LLM traffic, port 4000), backend (management API/UI, 4001) and ui (3000); official Terraform modules exist for AWS ECS Fargate and Google Cloud Run, and plain Kubernetes manifests are documented (LiteLLM production deployment). Licensing is a flag, not a build: litellm_license is "optional, omit for open source" (LiteLLM production deployment); Enterprise adds SSO+SCIM and support, self-served as "an instant key, emailed to you" (LiteLLM Enterprise).

API surfaces your code can keep using

7 of 7 documented
  • OpenAI chatPOST /v1/chat/completionsYes

    /chat/completions (examples use curl http://0.0.0.0:4000/v1/chat/completions) (LiteLLM proxy client examples).

  • Anthropic messagesPOST /v1/messagesYes

    "LiteLLM Proxy is Anthropic-compatible" with /messages (LiteLLM proxy client examples).

  • OpenAI ResponsesPOST /v1/responsesYes *

    Yes, referenced as a supported surface: Manus agents are called "through LiteLLM's OpenAI-compatible Responses API" (LiteLLM providers). Explicit /responses path: n.a.

  • EmbeddingsPOST /v1/embeddingsYes

    /embeddings (LiteLLM proxy client examples).

  • ImagesPOST /v1/images/generationsYes

    /image/generations (LiteLLM proxy client examples); provider-level image generation and editing are documented for Black Forest Labs, Fal AI and RunwayML (LiteLLM providers).

  • AudioPOST /v1/audio/*Yes *

    Yes, both directions: /audio/transcriptions (STT) and /audio/speech (TTS) (LiteLLM proxy client examples); provider examples include Deepgram's /listen and Nvidia Riva transcription (LiteLLM providers).

  • Batch jobsPOST /v1/batchesYes

    "Batches API endpoints" are exposed, plus a separate beta "batch completions" that fans one request out to N models via a comma-separated model string ("model"="llama3,gpt-3.5-turbo") (LiteLLM proxy client examples).

An asterisk marks a qualified verdict: support that is indirect (SDK compatibility or provider passthrough rather than a native endpoint), or a gap that is narrower or wider than the label suggests. Read the note before porting.

Drop-in base-URL swap: "Just change the base_url, api_key and model… LiteLLM Proxy is OpenAI-Compatible, it works with any project that calls OpenAI", and it is also Azure OpenAI-, Anthropic- and Vertex AI-compatible (/chat/completion, /embedding for Vertex) (LiteLLM proxy client examples). No proprietary SDK required — documented clients are the OpenAI SDK, LangChain, LlamaIndex, Instructor and curl (LiteLLM proxy client examples, LiteLLM production deployment). Streaming works with "stream": true (LiteLLM proxy client examples); Assistants and Fine-Tuning endpoints are also passed through (LiteLLM proxy client examples).

How much it reaches

ModelsNot published
Upstream providersNot published

n.a.: no total is stated; coverage is expressed per provider ("LiteLLM supports all models on VLLM", "We support ALL Fireworks AI models", CometAPI "500+ AI models") (LiteLLM providers, undated).

n.a.: the providers index states no total, though it enumerates a long list of provider sections (OpenAI, Azure OpenAI, Azure AI, Vertex AI, Bedrock, HuggingFace, Nvidia NIM, Perplexity, RunwayML, WatsonX, xAI, Databricks, Snowflake Cortex, Together AI, vLLM, Ollama and many more) (LiteLLM providers, undated).

Whose models: All third-party or customer-run: LiteLLM ships no models and integrates provider APIs plus self-hosted servers (vLLM, Ollama, LM Studio, Llamafile, Lemonade, Nvidia Riva, custom torch-serve) (LiteLLM providers).

Your own endpoints: Yes: "Call your custom torch-serve / internal LLM APIs via LiteLLM", with documented support for vLLM ("all models on VLLM"), Ollama ("all models from Ollama"), AWS SageMaker ("All Sagemaker Huggingface Jumpstart Models"), LM Studio, Llamafile, and OpenAI-compatible endpoints selected by choosing openai as provider with an upstream URL (LiteLLM providers).

How it behaves in production

What happens when an upstream model is slow, wrong, or down — and what you can see and stop while it happens. Reliability features are recorded as where you configure them, not whether the vendor lists them, because almost every product here lists all of them.

6 of 6 reachable from code4 of 4 can block20 documented destinations

When something goes wrong

Each row says where the knob is, not whether the feature is on the marketing page. A control you can only reach by hand in someone else’s dashboard cannot be reviewed or version-controlled.

  • Request timeoutPer request

    Your application decides per call, so one noisy endpoint can have its own timeout without a redeploy.

    Also available:In config

    per_request and config_file. Config: litellm_settings.request_timeout: 10 ("raise Timeout error if call takes longer than 10s") (LiteLLM reliability); router_settings.timeout: 30, per-deployment litellm_params.timeout: 300, and per-request "timeout": 1 in the JSON body (or extra_body={"timeout": 1} from OpenAI SDKs). Streaming has its own key stream_timeout (first-chunk deadline). No default value is stated (LiteLLM timeouts).

  • RetriesPer request

    Also available:In config

    per_request and config_file. num_retries (e.g. litellm_settings.num_retries: 3), with precedence x-litellm-num-retries header → request body → deployment litellm_paramslitellm_settings. Default count is not stated: n.a. Backoff: exponential for RateLimitError, immediate retry for generic errors; minimum delay configurable via retry_after. Per-error counts via RetryPolicy (e.g. TimeoutErrorRetries: 2, RateLimitErrorRetries: 3). Note max_retries is the provider SDK's internal counter and is forced to 0 on router paths to avoid (1 + N) ** 2 upstream calls (LiteLLM routing).

  • Fallback to another modelIn config

    Changing it means editing configuration and shipping it, so behaviour is uniform across traffic until you redeploy.

    config_file (also per-request). ORDERED: fallbacks: [{"gpt-3.5-turbo": ["gpt-4"]}], and "Fallbacks are done in-order". Specialized variants: default_fallbacks, content_policy_fallbacks, context_window_fallbacks. Fallback fires only after num_retries is exhausted (LiteLLM reliability). WEIGHTED failover also exists inside a model group via enable_weighted_failover (requires routing_strategy="simple-shuffle", uses weight/rpm/tpm, capped by max_fallbacks, default 5; async entrypoints only) (LiteLLM routing).

  • Load balancingIn config

    Deployments sharing a model_name form a load-balanced model group; routing_strategy selects the algorithm (e.g. "least-busy", "simple-shuffle"), and weights are supported through weight/rpm/tpm in the weighted-failover path (LiteLLM routing, LiteLLM timeouts).

  • Upstream health trackingIn config

    config_file, and it is cooldown-based ejection (LiteLLM does not call it circuit breaking): allowed_fails (default 3), cooldown_time (default 5s), disable_cooldowns, plus AllowedFailsPolicy per error type. Cooldown triggers: 429, >50% failures in the current minute, and non-retryable 401/404/408. Cooled deployments are removed from the pool and automatically reintroduced (LiteLLM routing).

  • Cross-region failoverIn config

    "LiteLLM will handle retrying in different regions if a call fails", with region_name (e.g. "eu") auto-inferred for Vertex AI, Bedrock and WatsonX, and weighted failover documented across Azure regional copies (eastus2, swedencentral). LiteLLM does not run the regions itself — you register per-region deployments (LiteLLM routing).

Fallback chain: Weighted split — Traffic splits by percentage across targets, so you can shift 5% to a new model and watch it before committing.

Deepest reliability surface of any self-hostable option here (retry precedence chain, per-error-type retry and failure policies, cooldowns, ordered + weighted failover, region inference), and the docs explicitly warn about retry amplification if the provider SDK's own max_retries is left on (LiteLLM routing). Kubernetes probes are documented at /health/readiness and /health/liveliness (LiteLLM production deployment).

How fast the hop is

Interpreted proxy

Runs on an interpreted or JIT runtime (Lua, Python, Node). Overhead is higher than a compiled binary and more sensitive to concurrency, though a Lua-on-nginx proxy and a Python one are far apart.

Currently in transition and the repo states both: the repository description is "The fastest, litest AI Gateway. Rust core with Python SDK", while the language breakdown is Python 84%, TypeScript 13.2%, Rust 0.7%. Best single label from the evidence: interpreted_proxy with a stated Rust core (BerriAI/litellm).

You can run the request path yourselfYes
StreamingYes

Docker images ghcr.io/berriai/litellm:<release-tag> (cosign-signed; -stable tags after 12-hour load tests) plus componentized litellm-gateway/-backend/-ui images, Helm charts litellm-helm and litellm (oci://ghcr.io/berriai/litellm-helm), and Terraform modules for AWS ECS Fargate and GCP Cloud Run (BerriAI/litellm, LiteLLM production deployment).

Streaming caveats: Supported via "stream": true. Documented streaming machinery/caveats: stream_timeout bounds only the first chunk and can be used to abort hanging providers and retry another model; SSE keepalives via keepalive_seconds (clamped 1–300 s; 0 at deployment level is a hard disable) or proxy-wide sse_keepalive_ping_interval_seconds, sending : ping comment frames; client-side override requires allow_client_keepalive_override: true (header x-litellm-keepalive-seconds) (LiteLLM timeouts).

Published figures, grouped by what each one measured. Figures in different groups are different quantities and cannot be compared with one another — nor, in most cases, with another vendor’s figure in the same group.

Overhead added by the gateway

The only figures that speak to whether this product slows your application down. Still not comparable between vendors: each was measured on different hardware, at a different load, with a different payload.

  • 8 msp95Vendor-published

    At 1k RPS. The README does not say whether this is added overhead or end-to-end, and gives no payload or hardware.

    Source

Sustained capacity

Requests or queries per second sustained on the stated hardware.

  • 1,000 RPSload conditionVendor-published

    Stated only as the load the 8ms p95 figure was taken at; hardware not stated.

    Source

The 8 ms / 1k RPS figure is vendor self-published (BerriAI/litellm). Flag: the widely circulated LiteLLM latency numbers (P50 38.65 s, ~40 ms overhead, 44.84 req/s) are published by a RIVAL, Maxim AI's Bifrost, not by LiteLLM (Bifrost benchmarks) — they are not comparable to LiteLLM's own claim and should not be reported as LiteLLM data.

What it will stop

4 of 4 can block

Two separate questions per control: can it stop a request at all, and what does it do before you change any settings? A control that inspects and forwards is a logging feature, however it is named.

  • Personal data in promptsCan block the request

    Out of the box: You pick the action when configuring

    Presidio integration supports per-entity MASK or BLOCK, runs in pre_call mode, and can skip system/tool messages; masked data is forwarded downstream with pass_data: true (Guardrails quick start, Policy flow builder)

  • Prompt injection and jailbreaksCan block the request

    Out of the box: You pick the action when configuring

    sync_block via third parties — e.g. Lakera configured as guardrail: lakera, mode: pre_call, plus PromptGuard, Gray Swan, and others; detection is delegated to the vendor's classifier (Policy flow builder, Guardrail providers)

  • Harmful contentCan block the request

    Out of the box: You pick the action when configuring

    sync_block — moderation guardrails (e.g. OpenAI moderation, Azure Content Safety, Bedrock Guardrails) block on policy failure such as toxic content (Guardrail providers, Policy flow builder)

  • Your own policiesCan block the request

    Out of the box: You pick the action when configuring

    Policies are declared in YAML pipelines with on_pass, on_fail, on_error branches per step (allow, block, next, modify_response with a custom message), and custom guardrail classes can be registered; key-level permissions control which guardrails apply (Policy flow builder, Guardrails quick start)

Where checks runEither, your choice
If the guardrail itself failsYou choose

Configurable, defaults to the policy branch: technical errors (timeouts, network errors, guardrail not loaded, connection failure) are handled by on_error, and "If on_error is omitted, the pipeline uses on_fail for error outcomes". The docs' own example fails open: "If backup_scanner errors → allow the request (set on_error to block if you prefer fail-closed)" (Policy flow builder)

Calls out to: Microsoft Presidio, AWS Bedrock Guardrails, Azure Content Safety, OpenAI Moderation, Lakera, Aporia, Pangea, Guardrails AI, Pillar Security. Each is a separate vendor relationship and a separate hop on the request path.

LiteLLM is the only product in this set that gives an explicit, documented three-way distinction between pass, policy failure, and technical error, with independent branch actions for each — the cleanest fail-open/fail-closed control available (Policy flow builder).

What you can see

Exports widely
What gets loggedYour choice

You decide whether bodies are captured, by setting or by header.

You can turn bodies offYes

yes — leave "Store Prompts in Spend Logs" off or set general_settings.store_prompts_in_spend_logs: false; additionally turn_off_message_logging, per-request header x-litellm-enable-message-redaction, and the no-log flag suppress message content on callbacks (Spend log settings, Logging)

TracesOpenTelemetry

OpenTelemetry is a first-class callback, alongside Langfuse, Arize, Langtrace, MLflow, and Langsmith for chain/agent-level traces (Logging)

"Store Prompts in Spend Logs" controls whether request messages and response content are stored; when disabled only metadata (tokens, cost, model) is kept (Spend log settings)

Where telemetry can go

  • OpenTelemetry
  • Datadog
  • Langfuse
  • Langsmith
  • Arize Phoenix
  • Prometheus
  • S3
  • GCS
  • Athina
  • Braintrust
  • Helicone
  • Lago
  • Logfire
  • Sentry
  • Slack
  • Webhook
  • Galileo
  • OpenMeter
  • DynamoDB
  • Azure Blob Storage

Langfuse, OpenTelemetry, GCS buckets, Amazon S3, Azure Blob Storage, GCS Pub/Sub, AWS SQS, DynamoDB, Datadog, Azure Sentinel, Lunary, MLflow, Deepeval, Langsmith, Arize, Langtrace, Galileo, OpenMeter, Sentry, Athina (Logging)

Records user feedbackNo
Scores live trafficPartly

n.a. — no native feedback/score endpoint on the fetched pages; scoring is delegated to callback platforms (Logging)

Indirectly, and without LiteLLM SaaS: Deepeval, MLflow, Langsmith, Galileo, Athina and Arize callbacks receive traffic for offline/online evaluation in those tools (Logging)

Depends on the vendor’s SaaS: No — the proxy is self-hosted, spend logs live in the customer's database, and callbacks ship to destinations the customer chooses (Logging, Spend log settings)

Retention: Configurable via maximum_spend_logs_retention_period (e.g. 7d, 30d) settable from the Admin UI without a restart; if unset, logs follow config/default behaviour and no fixed default is documented (Spend log settings)

Whether it fits how you work

How much work stands between you and a first call, how different that is from running it in production, and whether it slots into the stack you already have. Recorded as the shape of the work rather than a number of minutes — how long it takes you depends on which accounts and quota you already hold, which no comparison can know.

to try: cli or containerto run: infrastructure rolloutfits 5 of 10 common stacks

Getting to a first call

No numbered procedure published
Shape of the workRun something locally first

Nothing works until you have a process running. Fine on a laptop, but it means there is no zero-install way to try it.

Read off: the vendor’s own quickstart — no numbered procedure published.

Why the count is not the work: The getting-started page is prose plus commands rather than a numbered procedure.

Before step one

  • Your own provider keyNot needed

    You can make a first call with only this product’s key. No upstream provider account needed.

    For the documented first call: it uses huggingface/bigcode/starcoder and states HUGGINGFACE_API_KEY is "[OPTIONAL]" (Quick Start - LiteLLM Proxy CLI). Production secrets do include a provider key, e.g. --from-literal=OPENAI_API_KEY="<provider-key>" (Production Deployment - LiteLLM)

  • Payment methodNot documented

    n.a. (not documented) on either page; no enterprise/licence requirement is stated and litellm_license is optional in the Terraform examples (Quick Start - LiteLLM Proxy CLI, Production Deployment - LiteLLM)

  • Gate before models answerNot documented

    The docs do not say, so budget for a surprise on the first model you actually want.

    n.a. (not documented) (Quick Start - LiteLLM Proxy CLI)

Everything you need first: No account, credit card, cloud account, cluster, or provider key is stated; the Hugging Face key in the example is explicitly "[OPTIONAL]". Python 3.10+ is required for LiteLLM 1.84.0+ (Quick Start - LiteLLM Proxy CLI)

Running it in production

The same scale applied to the path the vendor recommends for production traffic. Kept separate from the quickstart because for several products here the two are barely related pieces of work.

Shape of the workDeploy it on your infrastructure

This is an infrastructure project, not an integration. Expect charts or Terraform, networking, secrets and someone who owns the deployment.

Getting to production is a step up in kind from the quickstart, not just more of the same.

What production needs: A Kubernetes cluster (EKS, GKE, AKS or other), external PostgreSQL (required for authentication/tracking) and Redis (required for multiple instances), cloud ingress/load balancer, and secrets management; the Terraform path provisions networking, databases, Redis, object storage, secrets, load balancer and migrations (AWS: Aurora PostgreSQL, ElastiCache, S3, ALB; GCP: Cloud SQL, Memorystore, GCS, HTTPS load balancer) (Production Deployment - LiteLLM)

Can you run it yourself

Install command publishedInstall command published

There is a command you can copy and run, so you can evaluate the self-hosted path yourself today.

uv tool install 'litellm[proxy]' / pip install 'litellm[proxy]' for local (Quick Start - LiteLLM Proxy CLI); production images ghcr.io/berriai/litellm and docker.litellm.ai/berriai/litellm:v1.90.2, plus helm install litellm oci://ghcr.io/berriai/litellm-helm -f values.yaml and helm upgrade --install litellm oci://ghcr.io/berriai/litellm/chart/litellm --version 1.89.2 -f values.yaml, with kubectl create secret generic litellm-masterkey|litellm-db|litellm-env ... (Production Deployment - LiteLLM)

How it fits your stack

5 of 10

Each row is a thing you might already run. “With a caveat” means it works but not the way the vendor’s marketing implies — read the reason, because that is usually where the surprise lives.

  • FitsThe OpenAI SDKDrop-in once set up — but first-call work is cli or container.
  • NoThe Vercel AI SDKNo AI SDK route documented.
  • NoCloudflare WorkersWorkers AI is an upstream model here, which is the opposite direction.
  • FitsKuberneteslitellm-helm at oci://ghcr.io/berriai/litellm-helm — the GitHub page describes it as community maintained
  • With a caveatTerraform or OpenTofuOfficial modules
  • NoAn existing API gatewayNothing published about running behind your gateway.
  • FitsCloud IAM I already runReuses IAM roles, workload identity or managed identities.
  • FitsLangChain or LlamaIndexLangChain, LlamaIndex
  • FitsMCP servers to governActs as an MCP gateway or registry.
  • With a caveatNothing — plain Node or PythonYou have to run a process locally before any call works.

Reading this the other way round — pick what you already run and see every product scored against it.

The integration surfaces behind those answers

  • Vercel AI SDKNot documented

    Nothing published. Assume the OpenAI-compatible route and verify it yourself.

    No official Vercel AI SDK provider package is documented; instead LiteLLM documents Vercel AI Gateway as an upstream provider route (vercel_ai_gateway/), and Vercel documents LiteLLM as a framework integration using the Python litellm package. (Vercel AI Gateway LiteLLM framework integration)

  • Cloudflare WorkersWorkers AI as an upstream model

    Cloudflare Workers AI is reachable as a model behind this product. That is the opposite direction from running your code on Workers.

    LiteLLM documents Cloudflare Workers AI support using the cloudflare/ model prefix, including cloudflare/@cf/meta/llama-2-7b-chat-int8 and cloudflare/@hf/thebloke/codellama-7b-instruct-awq. (LiteLLM Cloudflare Workers AI provider)

  • KubernetesOfficial Helm chart

    A named, published chart. You can read its values file before committing to anything.

    Named: litellm-helm at oci://ghcr.io/berriai/litellm-helm — the GitHub page describes it as community maintained

    Documented by:community maintainers, per the chart's own GitHub page

    LiteLLM deployment documentation lists the Helm charts litellm-helm at oci://ghcr.io/berriai/litellm-helm and litellm at oci://ghcr.io/berriai/litellm/chart/litellm. (LiteLLM deployment documentation)

  • TerraformOfficial modules

    Vendor-published modules exist. Some are off-Registry Git references, so pin them explicitly.

    Named: BerriAI/litellm/aws and BerriAI/litellm/google, plus a terraform-provider-litellm

    LiteLLM deployment documentation lists official Terraform modules BerriAI/litellm/aws and BerriAI/litellm/google, along with the resource-management provider terraform-provider-litellm. (LiteLLM deployment documentation)

  • Existing API gatewayNot documented

    Nothing published about sitting behind an existing gateway. Treat it as a separate hop you route to yourself.

    n.a. (not documented)

  • Cloud identityReuses your cloud identity

    Authenticate with the identity you already run — IAM roles, workload identity or managed identities. No long-lived key to rotate.

    For AWS Bedrock AgentCore MCP servers, LiteLLM supports AWS SigV4 and can use AWS access keys or the boto3 credential chain, including IAM roles and environment variables. The documented configuration includes auth_type: "aws_sigv4", aws_region_name, and aws_service_name: bedrock-agentcore. (LiteLLM MCP documentation)

  • MCPMCP gateway or registry

    It sits in front of many MCP servers and governs access to them. This is the shape you want if MCP sprawl is the problem you are solving.

    LiteLLM Proxy provides an MCP Gateway with endpoints including /mcp-rest/tools/list and /mcp-rest/tools/call. It supports listing and calling tools, prompts, resources, Streamable HTTP, SSE, and stdio, with MCP permissions by Key, Team, Organization, and end user. (LiteLLM MCP documentation)

Python frameworks
  • LangChain
  • LlamaIndex

LiteLLM documents integrations with LangChain and LlamaIndex. Examples include LangChain classes such as ChatOpenAI and OpenAIEmbeddings, and LlamaIndex classes including AzureOpenAI, AzureOpenAIEmbedding, VectorStoreIndex, and SimpleDirectoryReader. (LiteLLM user keys and integrations)

First-party client libraries
  • Python

Documented clients and integrations include the OpenAI SDK with openai.OpenAI, the Anthropic SDK with anthropic.Anthropic, the LiteLLM Python SDK, Mistral SDK, LlamaIndex, and LangChain for JavaScript and Python. The quick-start examples also show langchain.chat_models.ChatOpenAI and langchain.embeddings.OpenAIEmbeddings. (LiteLLM proxy quick start)

Agent features: LiteLLM supports OpenAI-compatible tool/function calls for MCP tools and can automatically execute returned MCP tool calls when require_approval: "never", then feed the results back to the model. Multi-step tool-result-to-model flow is documented through litellm.experimental_mcp_client.call_openai_tool; sessions, threads, and A2A are not documented. (LiteLLM MCP documentation)

Version trap documented: on Python 3.9 pip silently resolves to 1.83.9 without an error, so you must check python --version, upgrade to 3.10+ and reinstall; uv tool install provisions a compatible Python automatically. Config via your_config.yaml and litellm --config your_config.yaml. Production adds LITELLM_SALT_KEY (set once, never change), a /health/readiness check, and an Admin UI at /ui; the proxy claims "1.5k+ requests/second during load tests" (Quick Start - LiteLLM Proxy CLI, Production Deployment - LiteLLM)

LiteLLM is documented as either a Proxy Server or a Python SDK; the Proxy Server can run via the LiteLLM CLI or Docker and listens at http://0.0.0.0:4000. The product documentation also describes self-hosted deployment on AWS, Google Cloud, Azure, and Kubernetes clusters. (LiteLLM documentation home)

Silence in the docs: 4 of the integration questions on this card have no published answer either way. That is recorded as undocumented, not as a no — but it does mean you would be verifying it yourself.

What it does well

  • Widest provider and model coverage of any gateway here (140+ providers)
  • MIT-licensed with no usage fees, plus Helm/Terraform/Docker install paths and air-gapped support
  • Virtual keys, team budgets, RPM/TPM limits and guardrails in the free tier
  • Response and semantic caching backed by Redis, S3 or GCS
  • Very broad API surface: chat, responses, embeddings, images, audio, batches, rerank, MCP

Where it falls short

  • Third-party operators report the Python proxy hitting a GIL bottleneck past ~300 RPS per instance with P99 blowing out to seconds and memory growth to OOM
  • Independent 2026 comparison recommends staying under 250-300 RPS per instance and disputes vendor cache/MCP feature claims
  • Vendor benchmark numbers are self-run and conflict sharply with rivals' self-run numbers (Kong measured LiteLLM 859% slower than Kong)
  • SSO, RBAC, audit logs and SCIM sit behind an unpublished-price commercial license

Choose it when

Platform teams that want a free, self-hosted, maximally broad provider abstraction with virtual keys and budgets, at moderate request volumes.

Look elsewhere when

You need a single instance to sustain thousands of RPS with tight tail latency, or you need certified compliance artifacts from a vendor rather than your own deployment.

Open source: You run the gateway yourself. Full control over the data path and no per-request vendor fee, in exchange for operating it.

How hard is it to leave?

Derived from six published facts, not from an opinion. The weights are fixed and the same for every product — see the arithmetic.

100/100Easy to leave
Portability score breakdown for LiteLLM
What helps you leave Points Source
Works with standard OpenAI codeSwitching away is a base-URL change rather than a rewrite of every call site.22/22 vendor page
No vendor-specific SDK requiredA proprietary client library spreads through your codebase and has to be torn out again.10/10 vendor page
Can use your own provider accountsYour keys and billing relationship stay yours, so removing the gateway does not cut off model access.20/20 vendor page
Can be self-hostedYou can run it yourself instead of accepting a pricing or policy change.20/20
Configuration lives in version controlRouting and budget rules are a file you keep, not dashboard state you would have to rebuild.16/16 vendor page
Your request history can be exportedYou leave with your own logs instead of abandoning them.12/12 vendor page

All six inputs are published, so this is scored against the full 100. This measures technical switching cost only. It does not price the engineering time to re-test prompts against a different routing stack.

Full specification

Every field we track. Blank fields say "Not published" rather than "No" — we do not infer an absence from silence. Switch to Technical in the header for the precise field names and the low-level details.

Overview

What kind of product Category
Open source Not verified
Marketplaces resell many providers behind one key. Gateways add governance on top. Open-source projects you run yourself. Cloud platforms are hyperscaler surfaces. Inference providers host models on their own hardware.
Who runs it Deployment model
Self-host only Not verified
Managed means the vendor operates it. Self-host means you run it on your own infrastructure. Both means you can choose.
Licence Licence
MIT Verified 2 days ago
Proprietary products cannot be inspected or forked. Open licences such as MIT and Apache-2.0 let you audit, modify, and run the code without permission.
Company Company
BerriAI Verified 2 days ago
The organisation that maintains the product.
Who you would be signing with Vendor status
Independent company Not verified
Whether the product is still an independent company, has been acquired, is a large cloud vendor’s product line, is run by a software foundation, or has been put into maintenance mode. Maintenance mode means bug fixes and security patches only — no new features.
Last shipped an update Latest release
2026-08-23 Not verified

Latest stable release v1.98.0. A v1.100.0 release candidate was tagged on 30 August 2026; release candidates are excluded here.

The date of the most recent release or version tag. A product that has not shipped in a year is a different risk from one that shipped last week, regardless of what its marketing site says.
GitHub stars GitHub stars
57,500 Verified 2 days ago
A rough proxy for community size on open-source projects. Not a quality measure.

Cost

Markup on model prices Token markup
None Verified 2 days ago
How much the product adds on top of what the underlying model provider charges. Zero means you pay the same per-token price you would pay the model provider directly.
Fee to add funds Credit purchase fee
None Verified 2 days ago
A percentage charged when you top up your balance, separate from token prices. It is easy to miss because it does not appear on the per-token price list.
Monthly cost per person Seat fee
None Verified 2 days ago
A recurring per-user platform charge that applies regardless of how much you use the models.
Can use your own provider accounts BYOK supported
Yes Verified 2 days ago
Bring Your Own Key: you keep direct contracts with OpenAI, Anthropic and others, and the gateway only routes traffic. This preserves negotiated rates and committed-spend discounts.
Cost of using your own accounts BYOK terms
Self-hosted only: you configure your own upstream provider keys and pay providers directly; no gateway token fee. Not verified
What the product charges to route traffic through your own provider keys.
Free tier Free tier
Open-source gateway is $0 forever: 140+ providers, virtual keys, teams, budgets, load balancing, RPM/TPM limits, guardrails. Verified 2 days ago
What you can do without paying, useful for evaluation.
Enterprise plan from Enterprise plan from
Not published Not verified
Annual entry price for the enterprise tier, where one is published or credibly reported.
How the vendor makes money Pricing model
Open source with a managed tier Not verified
The shape of the vendor’s bill: does the routing layer charge a percentage on top of tokens, a flat monthly fee, both, neither (because inference is the product), or nothing at all (open source with no paid tier).
How pricing works, briefly Pricing model detail
OSS $0 free forever, self-hosted. Enterprise is talk-to-sales, sized to annual gateway request capacity, deployment architecture and support needs — 'never per token'. Volume discount tiers exist but no minimum amount is published. Not verified
A one-paragraph description that covers the caveats a pricing category cannot: introductory rates, per-feature meters, tier gating, and pricing that resets on a specific date.
Minimum commitment Minimum commitment
Enterprise is annual term; no stated minimum amount. Not verified
Whether the vendor requires a minimum contract term, a minimum spend, or a provisioned-capacity purchase to get its published rate.
Charges that fire after you go over an allowance Overage terms
None stated for OSS or Enterprise. Practical exposure is self-hosted infrastructure and database cost. Not verified
The line items that scale with usage after an included allowance is exhausted — log storage, extra requests, per-feature meters, data export — which is where cost estimates usually go wrong.
Prompt cache offered Cache mechanism
Both exact and semantic Not verified
Whether the gateway offers its own response cache, what kind of match it does (exact request, prefix, semantic), or simply passes provider caching through unchanged.
Discount on cached input Cache-read discount
Not published Not verified
How much cheaper cached tokens are than fresh input, when the vendor publishes a single figure. Bundled-inference clouds usually price this per model instead of as one number.
Premium on cache writes Cache-write premium
Not published Not verified
How much more the first write of a cached prefix costs versus a plain input token. A high write premium and a low hit rate can leave you paying more than you save, so this matters as much as the read discount.
Who captures the cache saving Cache economics
Exact-match and semantic self-hosted caches: In-Memory, Disk, Redis, Qdrant Semantic, Redis Semantic, Valkey Semantic, S3 bucket, GCS bucket. LiteLLM prices no tokens; provider cached-token rates flow through in its cost calculations. Saving is 100% the customer's; the cost you add is running Redis/Qdrant yourself. Not verified
Whether the customer keeps the full saving from caching or the vendor captures part of it — and any conditions attached (write premium, storage fees, best-effort hits).
What you can split spend by Cost attribution
The strongest of any product here: spend logged per hashed api_key, internal user, end-user/customer, team_id, tags, model_group, api_base, User-Agent (tracked as a tag) and arbitrary custom headers/metadata. Not verified
The dimensions the vendor documents for splitting spend — per key, per user, per team, per tag, per customer. Matters if you need to chargeback internally or bill an end customer.
How you get cost data out Cost export
API + database: /spend/logs, /user/info, /user/daily/activity, /gateway/daily/activity, /global/spend/report?group_by=customer|team|api_key|internal_user, x-litellm-response-cost header, and the LiteLLM_SpendLogs table you own. CSV, webhook, S3 and warehouse export not stated as first-class features. Not verified
The mechanisms the vendor publishes for exporting cost and usage data: CSV, an API, webhooks, S3, a data warehouse, or nothing at all. Any per-unit price is included.
Who pays the model bill BYOK mode
Your keys only Not verified
Whether you bring your own provider accounts (BYOK), buy inference from this vendor, or can do either. This is the single biggest commercial difference between these products: it decides who holds the contract with the model provider and who carries the spend.

Spend governance in detail

Seven signals matter when a bill starts to hurt: who can spend, how much, on what, and who gets paged when it goes wrong. Everything below is drawn from the vendor’s own pricing and docs pages — how we read these.

  • Virtual or scoped keysYes — oss

    Keys that carry their own budget and rate-limit policy, so an intern experiment cannot spend against a production budget.

    Free. /key/generate accepts models, user_id, team_id, metadata, aliases, budgets and limits.

  • Budget caps per keyYes — oss

    A dollar or token ceiling attached to an individual key. Where enforcement is soft, one over-limit request still completes before the block kicks in.

    Free, pre-request. max_budget + budget_duration, with admin-side upperbound_key_generate_params.

  • Budget caps per team or workspaceYes — oss

    A ceiling applied at a higher scope than one key — a team, a workspace, a customer, or an entire environment.

    Teams carry max_budget; owner budgets apply to keys with no team.

  • Rate limiting as a cost controlYes — oss

    Configurable request-per-time-window caps. Platform-set rate limits do not count as spend controls; user-configurable ones do.

    Free, pre-request. tpm_limit, rpm_limit, max_parallel_requests, /key/block and /key/unblock.

  • Model allowlistsYes — oss

    A policy that constrains which models a key or team can call, keeping expensive frontier models out of the wrong hands.

    Free. Keys specify allowed models, evaluated against the key row itself.

  • Spend alertsNot published

    Alerts fired as spend approaches a threshold. Alerts that only fire after the meter has rolled over are marked as such.

    Not stated on the pages fetched.

  • Webhook notificationsNot published

    Programmatic notifications on spend events, so budget breaches can page an on-call or open a ticket.

    Not stated on the pages fetched.

Enforcement:Enforced before each request

Catalog

Models available Models available
Not published Verified 2 days ago
How many models you can call, shown as a range because several vendors publish different totals on different pages. Vendor-reported either way, so counts are not directly comparable — some count every provider variant of the same model separately.
Model providers reachable Upstream providers
Not published Verified 2 days ago
How many distinct model providers or labs you can reach, shown as a range where the vendor’s own pages disagree. More providers usually means better redundancy when one has an outage.
Works with standard OpenAI code OpenAI-compatible API
Yes Verified 2 days ago
If yes, you can usually switch to it by changing one base URL, and switch away just as easily. This is the main defence against lock-in.
OpenAI chat endpoint POST /v1/chat/completions
Yes Not verified
The endpoint almost every application ports first. "Not documented" means the vendor never states it, which is different from a documented no.
Anthropic messages endpoint POST /v1/messages
Yes Not verified
Whether Anthropic-shaped calls work without rewriting them. Several products support this only as SDK compatibility or provider passthrough rather than a native endpoint — the detail page says which.
OpenAI Responses endpoint POST /v1/responses
Yes Not verified
The newer stateful OpenAI surface. Support is much thinner across this market than chat completions.
Embeddings endpoint POST /v1/embeddings
Yes Not verified
Whether you can generate vectors through the same gateway, or need a second integration for retrieval workloads.
Image generation endpoint POST /v1/images/generations
Yes Not verified
Whether image models are reachable through the same surface as text.
Audio endpoints POST /v1/audio/*
Yes Not verified
Speech-to-text and text-to-speech. Frequently the first gap in an otherwise complete gateway.
Batch jobs endpoint POST /v1/batches
Yes Not verified
Asynchronous bulk processing, usually at a discount. Commonly undocumented, and commonly the reason a migration stalls late.
Needs the vendor’s own code library Requires a vendor-specific SDK
No Verified 2 days ago
A proprietary client library spreads through your codebase and has to be torn out again if you leave. “No” is the better answer here, and it means the standard OpenAI client works.
You can export your request history Logs / usage data export
Yes Verified 2 days ago
Whether you can get your own request logs, traces, or usage records back out — through an API, a bulk export, or a download. Decides whether you leave with your history or abandon it.
Settings can live in version control Declarative config-as-code
Yes Verified 2 days ago
Whether routing, fallback, and budget rules can be declared in a file you keep in Git, rather than existing only as settings clicked into a hosted dashboard.
Embeddings Embeddings
Yes Verified 2 days ago
Text-to-vector models, needed for search and retrieval features.
Image generation Image generation
Yes Verified 2 days ago
Whether image models are reachable through the same interface.
Speech and audio Speech and audio
Yes Verified 2 days ago
Text-to-speech or transcription models through the same interface.
Video generation Video generation
Not published Not verified
Whether video models are reachable through the same interface.
Batch processing Batch processing
Yes Verified 2 days ago
Submitting large jobs for cheaper, slower processing. Often 50% off for work that is not time-sensitive.

Routing & reliability

Uptime it promises in writing Contractual SLA uptime
Not published Not verified
The uptime percentage in a published, contractual service level agreement. A public status page is not an SLA — it reports what happened, it does not promise anything or pay you back when it breaks.
Automatic failover Automatic failover
Yes Verified 2 days ago
When a model provider goes down or rate-limits you, traffic moves to a backup automatically instead of returning errors to your users.
Load balancing Load balancing
Yes Verified 2 days ago
Spreads requests across several providers or keys to raise your effective rate limit.
Rule-based routing Conditional routing
Yes Verified 2 days ago
Send different requests to different models based on rules — for example a cheap model for free users and a strong model for paying ones.
Response caching Response caching
Yes Verified 2 days ago
Reuses the answer when the exact same request comes in again, which cuts both cost and latency.
Similar-question caching Semantic cache
Yes Verified 2 days ago
Reuses an answer when a new question means roughly the same thing as an earlier one. Saves far more than exact-match caching but can return subtly wrong answers if tuned loosely.
Where you set the timeout Request timeout surface
Per request Not verified

`per_request` and `config_file`. Config: `litellm_settings.request_timeout: 10` ("raise Timeout error if call takes longer than 10s") ([LiteLLM reliability](https://docs.litellm.ai/docs/proxy/reliability)); `router_settings.timeout: 30`, per-deployment `litellm_params.timeout: 300`, and per-request `"timeout": 1` in the JSON body (or `extra_body={"timeout": 1}` from OpenAI SDKs). Streaming has its own key `stream_timeout` (first-chunk deadline). No default value is stated ([LiteLLM timeouts](https://docs.litellm.ai/docs/proxy/timeout)).

Where a request timeout can be set: per request, in a config file, in the vendor dashboard, or nowhere. Nine of the twenty products documented here do not describe a request timeout at all, so the worst case of a hung upstream call is unknowable from the docs.
Where you set retries Retry policy surface
Per request Not verified

`per_request` and `config_file`. `num_retries` (e.g. `litellm_settings.num_retries: 3`), with precedence `x-litellm-num-retries` header → request body → deployment `litellm_params` → `litellm_settings`. Default count is not stated: `n.a.` Backoff: exponential for `RateLimitError`, immediate retry for generic errors; minimum delay configurable via `retry_after`. Per-error counts via `RetryPolicy` (e.g. `TimeoutErrorRetries: 2`, `RateLimitErrorRetries: 3`). Note `max_retries` is the provider SDK's internal counter and is forced to `0` on router paths to avoid `(1 + N) ** 2` upstream calls ([LiteLLM routing](https://docs.litellm.ai/docs/routing)).

Where retry count and backoff are configured. Worth knowing alongside billing: a retried streaming call can be charged more than once.
Where you set fallbacks Fallback surface
In config Not verified

`config_file` (also per-request). ORDERED: `fallbacks: [{"gpt-3.5-turbo": ["gpt-4"]}]`, and "Fallbacks are done in-order". Specialized variants: `default_fallbacks`, `content_policy_fallbacks`, `context_window_fallbacks`. Fallback fires only after `num_retries` is exhausted ([LiteLLM reliability](https://docs.litellm.ai/docs/proxy/reliability)). WEIGHTED failover also exists inside a model group via `enable_weighted_failover` (requires `routing_strategy="simple-shuffle"`, uses `weight`/`rpm`/`tpm`, capped by `max_fallbacks`, default `5`; async entrypoints only) ([LiteLLM routing](https://docs.litellm.ai/docs/routing)).

Where the fallback chain is defined. Almost every product claims fallback; the useful question is whether you can change it from code or only by hand in a dashboard.
Shape of the fallback chain Fallback shape
Weighted split Not verified
An ordered list tries targets in sequence; a weighted split sends a percentage of traffic to each, which is what you need to trial a new model on 5% of requests. Weighted splits are much rarer than the marketing implies.
Upstream health tracking Health checks / circuit breaking
In config Not verified

`config_file`, and it is cooldown-based ejection (LiteLLM does not call it circuit breaking): `allowed_fails` (default `3`), `cooldown_time` (default `5s`), `disable_cooldowns`, plus `AllowedFailsPolicy` per error type. Cooldown triggers: `429`, >50% failures in the current minute, and non-retryable `401`/`404`/`408`. Cooled deployments are removed from the pool and automatically reintroduced ([LiteLLM routing](https://docs.litellm.ai/docs/routing)).

Whether the product notices a failing upstream and stops sending traffic to it, and whether you can tune the thresholds. This is what turns a provider outage into a blip rather than a sustained error rate, and only four of the twenty expose it.
Cross-region failover you control Multi-region failover surface
In config Not verified

"LiteLLM will handle retrying in different regions if a call fails", with `region_name` (e.g. `"eu"`) auto-inferred for Vertex AI, Bedrock and WatsonX, and weighted failover documented across Azure regional copies (`eastus2`, `swedencentral`). LiteLLM does not run the regions itself — you register per-region deployments ([LiteLLM routing](https://docs.litellm.ai/docs/routing)).

Whether you can define what happens when a region degrades. A vendor running many regions is not the same as a vendor letting you configure failover between them; only three document a user-controlled mechanism.
Where you set load balancing Load balancing surface
In config Not verified

Deployments sharing a `model_name` form a load-balanced model group; `routing_strategy` selects the algorithm (e.g. `"least-busy"`, `"simple-shuffle"`), and weights are supported through `weight`/`rpm`/`tpm` in the weighted-failover path ([LiteLLM routing](https://docs.litellm.ai/docs/routing), [LiteLLM timeouts](https://docs.litellm.ai/docs/proxy/timeout)).

Where traffic distribution across upstreams or keys is configured.

Operations

Usage dashboards and logs Observability
Yes Verified 2 days ago
Built-in visibility into what was sent, what came back, what it cost, and how long it took.
Spending limits Budget controls
Yes Verified 2 days ago
Hard caps that stop spend before it becomes a surprise invoice. The single most valuable control for a small team.
Rate limits Rate limits
Yes Verified 2 days ago
Caps on request volume per key or per user, useful for protecting against abuse and runaway loops.
Separate keys per team or app Virtual keys
Yes Verified 2 days ago
Issue scoped keys with their own budgets and permissions so you can attribute cost and revoke access without rotating everything.
Prompt versioning Prompt management
Not published Not verified
Store and version prompts outside your code so they can be changed without a deploy.
Quality testing Evals
Not published Not verified
Built-in tooling to score model output against test cases, so you can tell whether a model swap made things better or worse.
MCP support MCP support
Yes Verified 2 days ago
Native support for the Model Context Protocol, the emerging standard for connecting models to external tools.
What gets logged Logged content
Your choice Not verified

"Store Prompts in Spend Logs" controls whether request messages and response content are stored; when disabled only metadata (tokens, cost, model) is kept ([Spend log settings](https://docs.litellm.ai/docs/proxy/ui_spend_log_settings))

Whether full prompts and responses are stored, only metadata, or nothing. Full-body logging is the most useful debugging feature here and the one most likely to need a conversation with your compliance team.
You can turn logging off Body-logging opt-out
Yes Not verified

`yes` — leave "Store Prompts in Spend Logs" off or set `general_settings.store_prompts_in_spend_logs: false`; additionally `turn_off_message_logging`, per-request header `x-litellm-enable-message-redaction`, and the `no-log` flag suppress message content on callbacks ([Spend log settings](https://docs.litellm.ai/docs/proxy/ui_spend_log_settings), [Logging](https://docs.litellm.ai/docs/proxy/logging))

Whether prompt and response bodies can be suppressed while still keeping usage metrics. Nineteen of the twenty document a way to do this; the mechanisms range from a per-request header to an organisation-wide setting.
Traces you can take elsewhere Distributed tracing
OpenTelemetry Not verified

OpenTelemetry is a first-class callback, alongside Langfuse, Arize, Langtrace, MLflow, and Langsmith for chain/agent-level traces ([Logging](https://docs.litellm.ai/docs/proxy/logging))

Whether the product emits OpenTelemetry, a proprietary format, or nothing. OpenTelemetry means the traces land in the tooling you already run instead of only in the vendor’s dashboard.
Where telemetry can go Export destinations
OpenTelemetry, Datadog, Langfuse, Langsmith, Arize Phoenix, Prometheus, S3, GCS, Athina, Braintrust, Helicone, Lago, Logfire, Sentry, Slack, Webhook, Galileo, OpenMeter, DynamoDB, Azure Blob Storage Not verified

Langfuse, OpenTelemetry, GCS buckets, Amazon S3, Azure Blob Storage, GCS Pub/Sub, AWS SQS, DynamoDB, Datadog, Azure Sentinel, Lunary, MLflow, Deepeval, Langsmith, Arize, Langtrace, Galileo, OpenMeter, Sentry, Athina ([Logging](https://docs.litellm.ai/docs/proxy/logging))

Documented sinks for logs and metrics. This is a good proxy for how replaceable the vendor’s own dashboard is: around twenty destinations means you never have to depend on it, while a CSV download means you do.
Can record user feedback Feedback capture API
No Not verified

`n.a.` — no native feedback/score endpoint on the fetched pages; scoring is delegated to callback platforms ([Logging](https://docs.litellm.ai/docs/proxy/logging))

Whether there is an API to attach a rating or score to a logged request, which is what lets production traffic feed quality work later.
Scores live traffic Online eval hooks
Partly Not verified

Indirectly, and without LiteLLM SaaS: Deepeval, MLflow, Langsmith, Galileo, Athina and Arize callbacks receive traffic for offline/online evaluation in those tools ([Logging](https://docs.litellm.ai/docs/proxy/logging))

Whether automated scorers can run against real production requests, rather than only against a test set you assemble yourself.

Performance

Delay it adds Proxy overhead
0.66 ms Verified 2 days ago
Extra time the product itself adds to each request, on top of however long the model takes. Usually irrelevant next to multi-second model latency, but it matters for high-volume or streaming-sensitive workloads.
Requests per second ceiling Throughput
2,800 rps Verified 2 days ago
Published sustained request rate before the product becomes the bottleneck. Only relevant at genuinely high volume.
What the request path runs on Architecture class
Interpreted proxy Not verified

Currently in transition and the repo states both: the repository description is "The fastest, litest AI Gateway. Rust core with Python SDK", while the language breakdown is Python 84%, TypeScript 13.2%, Rust 0.7%. Best single label from the evidence: `interpreted_proxy` with a stated Rust core ([BerriAI/litellm](https://github.com/BerriAI/litellm)).

The comparable way to talk about latency here. An edge worker, a compiled Go or Rust binary, and a Python proxy have different overhead floors no matter which figures each vendor publishes. Products that never disclose their runtime are recorded as undisclosed rather than assumed.
You can run the request path yourself Self-hostable data plane
Yes Not verified

Docker images `ghcr.io/berriai/litellm:<release-tag>` (cosign-signed; `-stable` tags after 12-hour load tests) plus componentized `litellm-gateway`/`-backend`/`-ui` images, Helm charts `litellm-helm` and `litellm` (`oci://ghcr.io/berriai/litellm-helm`), and Terraform modules for AWS ECS Fargate and GCP Cloud Run ([BerriAI/litellm](https://github.com/BerriAI/litellm), [LiteLLM production deployment](https://docs.litellm.ai/docs/proxy/deploy)).

Whether the component that actually carries your prompts can run on your own infrastructure. Distinct from a vendor offering a self-hosted control plane while still proxying traffic through their network.
Streaming responses Streaming support
Yes Not verified

Supported via `"stream": true`. Documented streaming machinery/caveats: `stream_timeout` bounds only the first chunk and can be used to abort hanging providers and retry another model; SSE keepalives via `keepalive_seconds` (clamped 1–300 s; `0` at deployment level is a hard disable) or proxy-wide `sse_keepalive_ping_interval_seconds`, sending `: ping` comment frames; client-side override requires `allow_client_keepalive_override: true` (header `x-litellm-keepalive-seconds`) ([LiteLLM timeouts](https://docs.litellm.ai/docs/proxy/timeout)).

Whether token-by-token streaming is documented. The caveats matter more than the yes: some products cannot cancel a stream without still being billed, and several timeout and fallback mechanisms stop applying once the first token has been sent.

Security & compliance

Does your prompt reach their servers Prompt transits vendor
No Not verified

Self-hosted: no personal data is collected or transmitted to LiteLLM's servers, and enterprise supports air-gapped deployment so neither your data nor your keys leave your infrastructure.

Whether the text you send passes through this company’s own infrastructure. If it does, every other promise on this page is a policy commitment rather than a physical impossibility. Self-hosted products can answer no outright.
What they keep if you change nothing Logging default
Metadata only, not content Not verified

Content logging is off by default — only metadata such as tokens, cost and model is stored. Storing request and response bodies requires opting in with store_prompts_in_spend_logs.

Defaults matter more than options. A product that stores full prompts and replies unless you find the right header will have stored them by the time you read the docs.
How long they keep it Default content retention (days)
Not published Not verified

No default deletion. Logs persist according to your configuration until you set something like maximum_spend_logs_retention_period.

Default retention for request content, in days. Zero means nothing is kept. Read the note: several products keep nothing as a rule but make timed exceptions for abuse review or specific models.
Could they train on your prompts Training on customer data
No Not verified

True by construction rather than by policy: no data or telemetry reaches LiteLLM's servers when self-hosted. No separate training policy is published.

Whether the vendor may use your prompts and outputs to train models. “Not published” means we could not find any position, which is not the same as a no — ask for it in writing.
Where it runs, and what you can pin Region and residency control
Your own infrastructure, including air-gapped. Enterprise offers a multi-region control plane. Not verified
Which regions are offered and whether you can force processing to stay in one. A global endpoint that silently picks a region is a different compliance story from an endpoint you pin yourself.
Where safety filters run Guardrail execution location
In your own infrastructure Not verified

PII masking requires Presidio analyzer and anonymizer containers that you deploy and address locally, so redaction happens entirely inside your boundary.

A filter that strips personal data only helps if it runs before the data leaves your boundary. If guardrails execute in the vendor’s cloud, the vendor has already received whatever you wanted redacted.
Who else touches the data Subprocessor list
Not published Not verified
The published list of third parties the vendor passes your data to. No list means you cannot know the full chain, which most data-protection agreements require you to.
SOC 2 audited SOC 2 audited
Not published Not verified
An independent audit of security controls. Enterprise buyers and their procurement teams routinely require it.
Will sign a HIPAA agreement HIPAA BAA
Not published Not verified
Required before you may send protected health information through the service. Without a signed BAA, healthcare data is off limits.
GDPR commitments GDPR commitments
Not published Not verified
Published data processing terms for handling personal data of people in the EU and UK.
Can keep data in the EU EU data residency
Not published Not verified
Requests can be processed inside the EU rather than routed to US infrastructure. Often the deciding constraint for European customers.
Does not retain your data Zero data retention
Not published Not verified
Prompts and responses are not stored after the request completes. Sometimes a paid add-on rather than the default.
Strips personal data PII redaction
Yes Verified 2 days ago
Detects and removes identifiers such as names, emails, and card numbers before the request reaches the model provider.
Content guardrails Content guardrails
Yes Verified 2 days ago
Policy checks on inputs and outputs — blocking unsafe content, enforcing formats, or catching prompt-injection attempts.
Runs fully disconnected Air-gapped deployment
Yes Verified 2 days ago
Can be deployed in a network with no internet access, which some regulated and defence environments require.
Blocks personal data in prompts PII / DLP enforcement
Can block the request Not verified

Presidio integration supports per-entity `MASK` or `BLOCK`, runs in `pre_call` mode, and can skip system/tool messages; masked data is forwarded downstream with `pass_data: true` ([Guardrails quick start](https://docs.litellm.ai/docs/proxy/guardrails/quick_start), [Policy flow builder](https://docs.litellm.ai/docs/proxy/guardrails/policy_flow_builder))

Whether personal data detection sits on the request path and can stop the call, merely inspects and forwards it, or is not documented. A control that only reports is a logging feature, not a policy control.
Blocks prompt injection Injection / jailbreak enforcement
Can block the request Not verified

`sync_block` via third parties — e.g. Lakera configured as `guardrail: lakera, mode: pre_call`, plus PromptGuard, Gray Swan, and others; detection is delegated to the vendor's classifier ([Policy flow builder](https://docs.litellm.ai/docs/proxy/guardrails/policy_flow_builder), [Guardrail providers](https://docs.litellm.ai/docs/guardrail_providers))

Whether injection and jailbreak detection can stop a request. Most products offering this call a partner classifier rather than shipping their own.
Blocks harmful content Toxicity / moderation enforcement
Can block the request Not verified

`sync_block` — moderation guardrails (e.g. OpenAI moderation, Azure Content Safety, Bedrock Guardrails) block on policy failure such as toxic content ([Guardrail providers](https://docs.litellm.ai/docs/guardrail_providers), [Policy flow builder](https://docs.litellm.ai/docs/proxy/guardrails/policy_flow_builder))

Whether hate, violence, sexual and self-harm categories are checked inline and can stop a request, in either direction.
Your own policy rules Custom policy hooks
Can block the request Not verified

Policies are declared in YAML pipelines with `on_pass`, `on_fail`, `on_error` branches per step (`allow`, `block`, `next`, `modify_response` with a custom message), and custom guardrail classes can be registered; key-level `permissions` control which guardrails apply ([Policy flow builder](https://docs.litellm.ai/docs/proxy/guardrails/policy_flow_builder), [Guardrails quick start](https://docs.litellm.ai/docs/proxy/guardrails/quick_start))

Whether you can add your own rule — a regex, a webhook, or your own classifier — rather than choosing from the vendor library.
Where guardrails run Guardrail execution location
Either, your choice Not verified
Whether guardrail evaluation happens inside your infrastructure or on the vendor’s servers. This decides whether the prompt you are trying to protect leaves your network in order to be checked.
If the guardrail itself fails Guardrail failure mode
You choose Not verified

**Configurable, defaults to the policy branch**: technical errors (timeouts, network errors, guardrail not loaded, connection failure) are handled by `on_error`, and "If `on_error` is omitted, the pipeline uses `on_fail` for error outcomes". The docs' own example fails open: "If `backup_scanner` errors → allow the request (set `on_error` to `block` if you prefer fail-closed)" ([Policy flow builder](https://docs.litellm.ai/docs/proxy/guardrails/policy_flow_builder))

What happens when the guardrail service times out or errors: does the request proceed unchecked, or is it blocked? This is the worst-documented field in the entire catalogue — only two vendors state it plainly, which means most teams are running a control whose failure behaviour they cannot know.
Third-party guardrail vendors Guardrail integrations
Microsoft Presidio, AWS Bedrock Guardrails, Azure Content Safety, OpenAI Moderation, Lakera, Aporia, Pangea, Guardrails AI, Pillar Security Not verified
Named external guardrail services the product can call. A long list means the product is a router for policy engines rather than a policy engine itself — which also means another vendor bill and another hop.

Compliance evidence

Graded by how strong the evidence is, not whether the word appears on the vendor’s website. An audited report and a marketing claim are different things, and only one of them will satisfy your own auditor.

  • SOC 2 Vendor’s own pages disagree docs say Type II in progress with an ETA of 15 Sep 2026; the enterprise page markets it as done
  • ISO 27001 Claimed, no evidence published marketed on the enterprise page, unverified
  • GDPR DPA Not published
  • HIPAA BAA Not published
  • FedRAMP Not published
  • ITAR Not published

Vendor source

Security incident

Publicly documented incidents affecting this product. An incident here is not by itself a reason to rule a product out — what matters is what failed, whether it could recur, and what it means for the way you would deploy it.

Malicious releases published to PyPI (CVE-2026-33634)

March 24, 2026 · Critical · Resolved

Two releases, 1.82.7 and 1.82.8, were published straight to PyPI carrying a credential stealer. Neither version was ever tagged on GitHub — the attacker bypassed the project's CI/CD entirely and uploaded to PyPI directly, using an API token exposed through the compromised Trivy releases in the wider TeamPCP campaign.

The payload collected environment variables, model API keys, SSH keys, AWS, GCP and Azure credentials, Kubernetes tokens, database passwords, shell history and CI/CD secrets, encrypted them with AES-256 under an RSA-4096 key, and posted them to models.litellm[.]cloud — a domain registered the day before the upload and unconnected to the project. It then polled checkmarx[.]zone for follow-on code and persisted via a systemd user unit named sysmon.service.

1.82.8 is the more dangerous of the two. It shipped a litellm_init.pth file, which Python executes at interpreter startup, so the stealer ran on any Python process on the machine whether or not anything imported LiteLLM. In 1.82.7 the payload sat in proxy_server.py and only ran when litellm.proxy was imported.

Accounts differHow long the packages were installable is disputed by the parties involved. LiteLLM says they were live from 10:39 UTC and quarantined after roughly 40 minutes, while PyPI's own incident report puts LiteLLM's exposure at 2 hours 32 minutes from upload to quarantine, and the GitHub issue tracking it records the upload at around 08:30 UTC. LiteLLM's own advice is the safest reading: treat any install between 10:39 and 16:00 UTC on 24 March 2026 as suspect.

Package-level download counts were never published. CloudSEK put the campaign's captured dataset at roughly 434,000 files, mapping potential exposure to more than 2,500 organisations — that is the whole TeamPCP campaign, not LiteLLM alone, and the file count is not a deduplicated count of victims.

PyPI quarantined the project and both versions were removed; they no longer resolve. 1.82.6 and 1.83.0 are unaffected and remain available.

What to do

If you installed LiteLLM from PyPI inside the 24 March window, treat every credential reachable from that host as compromised and rotate it — model keys, cloud credentials, SSH keys and CI tokens. Check for litellm_init.pth in site-packages, a sysmon.service user unit, and outbound traffic to models.litellm[.]cloud or checkmarx[.]zone.

Our read

This was a compromise of the distribution channel, not a flaw in the gateway's own code, and the maintainers did not write the malware. It is recorded here because the failure mode it exposes is structural: a self-hosted gateway installed from a public package index inherits that index's supply-chain risk, and this gateway is one that sits directly on your model keys and cloud credentials. Pin versions and verify hashes rather than tracking latest.

Fit & integration

Work to try it Evaluation work shape
Run something locally first Not verified
The shape of the work on the vendor’s own quickstart, from swapping one base URL through to deploying infrastructure. An ordinal class rather than a duration, because elapsed time depends on accounts and quota we cannot see.
Work to run it Production work shape
Deploy it on your infrastructure Not verified
The same scale applied to the vendor’s recommended production path. For several products this is much heavier than the quickstart, which is exactly why both are recorded.
Steps on the quickstart Numbered quickstart steps
0 Not verified

The getting-started page is prose plus commands rather than a numbered procedure.

A literal count of numbered steps on the vendor’s quickstart, recorded as evidence beside the work shape. Zero means the page publishes no numbered procedure at all. Large counts usually mean interleaved language tracks rather than more work.
Can you self-host it today Self-host install documentation
Install command published Not verified

`uv tool install 'litellm[proxy]'` / `pip install 'litellm[proxy]'` for local ([Quick Start - LiteLLM Proxy CLI](https://docs.litellm.ai/docs/proxy/quick_start)); production images `ghcr.io/berriai/litellm` and `docker.litellm.ai/berriai/litellm:v1.90.2`, plus `helm install litellm oci://ghcr.io/berriai/litellm-helm -f values.yaml` and `helm upgrade --install litellm oci://ghcr.io/berriai/litellm/chart/litellm --version 1.89.2 -f values.yaml`, with `kubectl create secret generic litellm-masterkey|litellm-db|litellm-env ...` ([Production Deployment - LiteLLM](https://docs.litellm.ai/docs/proxy/deploy))

Whether an install command is actually published. Several products advertise self-hosting while publishing no command to start from, which a plain yes/no would hide.
Works with the OpenAI SDK OpenAI SDK drop-in
Yes Not verified

Yes. LiteLLM documents OpenAI Python SDK drop-in usage with `api_key="anything"` and the exact proxy base URL `base_url="http://0.0.0.0:4000"`, using `openai.OpenAI`. ([LiteLLM documentation home](https://docs.litellm.ai/))

Whether an existing OpenAI-compatible client can be pointed at it by changing the base URL and key.
Vercel AI SDK support AI SDK provider package
Not documented Not verified

No official Vercel AI SDK provider package is documented; instead LiteLLM documents Vercel AI Gateway as an upstream provider route (`vercel_ai_gateway/`), and Vercel documents LiteLLM as a framework integration using the Python `litellm` package. ([Vercel AI Gateway LiteLLM framework integration](https://vercel.com/docs/ai-gateway/ecosystem/framework-integrations/litellm))

Whether a first-party AI SDK provider package exists, or only a community package, a documented workaround, or the generic OpenAI provider pointed at a custom base URL.
Python framework integrations Documented Python frameworks
LangChain, LlamaIndex Not verified

LiteLLM documents integrations with LangChain and LlamaIndex. Examples include LangChain classes such as `ChatOpenAI` and `OpenAIEmbeddings`, and LlamaIndex classes including `AzureOpenAI`, `AzureOpenAIEmbedding`, `VectorStoreIndex`, and `SimpleDirectoryReader`. ([LiteLLM user keys and integrations](https://docs.litellm.ai/docs/proxy/user_keys))

LangChain, LangGraph, LlamaIndex and similar orchestration frameworks with a documented integration.
Callable from Cloudflare Workers Cloudflare Workers support
Workers AI as an upstream model Not verified

LiteLLM documents Cloudflare Workers AI support using the `cloudflare/` model prefix, including `cloudflare/@cf/meta/llama-2-7b-chat-int8` and `cloudflare/@hf/thebloke/codellama-7b-instruct-awq`. ([LiteLLM Cloudflare Workers AI provider](https://docs.litellm.ai/docs/providers/cloudflare_workers))

Whether the docs show calling this product from your own Worker. Deliberately separated from the several products whose own gateway runs on Workers, which is a fact about their infrastructure and not about your edge compatibility.
Kubernetes install Helm chart availability
Official Helm chart Not verified

LiteLLM deployment documentation lists the Helm charts `litellm-helm` at `oci://ghcr.io/berriai/litellm-helm` and `litellm` at `oci://ghcr.io/berriai/litellm/chart/litellm`. ([LiteLLM deployment documentation](https://docs.litellm.ai/docs/proxy/deploy))

Whether a named, published Helm chart exists, versus Helm being referenced with no chart named, versus generic cluster documentation that has nothing to do with this product.
Terraform support Terraform provider or modules
Official modules Not verified

LiteLLM deployment documentation lists official Terraform modules `BerriAI/litellm/aws` and `BerriAI/litellm/google`, along with the resource-management provider `terraform-provider-litellm`. ([LiteLLM deployment documentation](https://docs.litellm.ai/docs/proxy/deploy))

Whether you can declare this in code: an official provider, official modules, resources inside a hyperscaler’s provider, a community provider, or only Terraform code shipped in a repo.
Reuses your cloud identity Cloud IAM reuse
Reuses your cloud identity Not verified

For AWS Bedrock AgentCore MCP servers, LiteLLM supports AWS SigV4 and can use AWS access keys or the boto3 credential chain, including IAM roles and environment variables. The documented configuration includes `auth_type: "aws_sigv4"`, `aws_region_name`, and `aws_service_name: bedrock-agentcore`. ([LiteLLM MCP documentation](https://docs.litellm.ai/docs/mcp))

Whether you can authenticate with IAM roles, workload identity or managed identities instead of another long-lived API key. Distinguished from products that only accept static upstream provider credentials.
Fits behind your API gateway API gateway integration
Not documented Not verified

n.a. (not documented)

Whether AI traffic can go through a gateway you already run, and who documented that — the product’s vendor or the gateway’s.
MCP support MCP surface shape
MCP gateway or registry Not verified

LiteLLM Proxy provides an MCP Gateway with endpoints including `/mcp-rest/tools/list` and `/mcp-rest/tools/call`. It supports listing and calling tools, prompts, resources, Streamable HTTP, SSE, and stdio, with MCP permissions by Key, Team, Organization, and end user. ([LiteLLM MCP documentation](https://docs.litellm.ai/docs/mcp))

Which kind of MCP support this is: a gateway that governs many MCP servers, a hosted MCP server you connect to, MCP tools accepted inside the completion API, or client tooling. These are different products behind one acronym.
Needs your own provider key Upstream provider key required
Not needed Not verified

No for the documented first call: it uses `huggingface/bigcode/starcoder` and states `HUGGINGFACE_API_KEY` is "[OPTIONAL]" ([Quick Start - LiteLLM Proxy CLI](https://docs.litellm.ai/docs/proxy/quick_start)). Production secrets do include a provider key, e.g. `--from-literal=OPENAI_API_KEY="<provider-key>"` ([Production Deployment - LiteLLM](https://docs.litellm.ai/docs/proxy/deploy))

Whether an upstream provider account and key must exist before your first call works. A prerequisite rather than a step, and it can differ between the hosted and self-hosted forms of the same product.
Gate before models work Model access gate
Not documented Not verified

n.a. (not documented) ([Quick Start - LiteLLM Proxy CLI](https://docs.litellm.ai/docs/proxy/quick_start))

Whether an enablement click, a quota grant, a paid tier or an approval form stands between a valid key and a working model call.
Official client languages First-party client SDK languages
Python Not verified

Documented clients and integrations include the OpenAI SDK with `openai.OpenAI`, the Anthropic SDK with `anthropic.Anthropic`, the LiteLLM Python SDK, Mistral SDK, LlamaIndex, and LangChain for JavaScript and Python. The quick-start examples also show `langchain.chat_models.ChatOpenAI` and `langchain.embeddings.OpenAIEmbeddings`. ([LiteLLM proxy quick start](https://docs.litellm.ai/docs/proxy/quick_start))

Languages with a first-party client library. An empty list can still mean the product is usable from any language via an OpenAI-compatible SDK.

How pricing actually works

OSS is free (MIT); you pay only for your own containers plus Postgres and Redis. Enterprise features (SSO, RBAC, JWT auth, SCIM, audit logs, support SLAs) require a paid LiteLLM commercial license whose price is not published.

Back to top ↑

Official links

57,500 GitHub stars — a proxy for community size, not for quality.

Independent coverage

Third-party analysis, walkthroughs and operator threads. We link criticism as readily as praise. Nothing here is written or published by the vendor, by a competitor listed on this site, or by an SEO content farm — how we vet these.

Written reviews and analysis 1

  • Using Gemini CLI Through LiteLLM Proxy DEV Community - polar3130 · Nov 2025 Hands-on writeup of putting Gemini CLI behind LiteLLM Proxy on Cloud Run with Cloud SQL, including the operational gotchas found in testing.

Video 2

  • One API to Rule Them All? LiteLLM in Production YouTube - PyData · Nov 2025 Conference talk by Alina Dallmann on a production LiteLLM deployment for a privacy-preserving university chatbot: proxy setup, Postgres budgets, Redis load balancing, key management.
  • LiteLLM Crash Course | For Complete Beginners YouTube - Data Science Basics · Jun 2025 Sudarshan Koirala's hands-on course covering the SDK, proxy server, Docker deploy, admin UI, teams and virtual keys against OpenAI, Ollama and Groq.

Practitioner discussion 2

What has changed here

No tracked value on this page has moved since it was first researched. Every figure still carries its original citation and check date.

Back to top ↑

Read the head-to-head

These pairs have a written verdict, not just a table.

Usually weighed against