Self-host only MIT

LiteLLM Open source

LiteLLM is an open-source LLM gateway with an OpenAI-compatible API; it publishes no model count. It charges no token markup, credit-purchase fee or per-seat fee. It runs only self-hosted under MIT. It does not publish a HIPAA BAA. You can point it at your own provider accounts. Beyond chat it also serves embeddings, image generation, audio and batch jobs. It handles failover, load balancing, guardrails and request logging.

· 44 dated entries · 60 source references

Built by BerriAI, founded 2023 · US company

Access at a glance

Whether this product can work for you at all, before features matter: who pays the model bill, where it can run, and which of your existing API calls keep working. Every value is the vendor’s own claim, linked to the page it came from.

Titled "LiteLLM AI Gateway (LLM Proxy)" and elsewhere "LiteLLM Proxy (LLM Gateway)", it is a self-hosted proxy that maps many provider APIs to OpenAI-format input/output (LiteLLM AI Gateway, LiteLLM providers). Not a model platform: "input, output and exceptions are mapped to the OpenAI format for all supported models" (LiteLLM proxy client examples).

Who pays the model bill

Your keys only

You contract with each model provider directly and hold those accounts. The gateway never resells inference.

The proxy is configured with the customer's own upstream provider keys, and it can additionally "allow users to pass in their OpenAI API key"; callers authenticate with LiteLLM virtual keys (e.g. sk-test-proxy-key-123) (LiteLLM proxy client examples).

Merchant of record: n.a. as an explicit statement: the LiteLLM pages fetched do not state who invoices model usage; LiteLLM Enterprise sells the software licence while the deployment runs on the customer's infrastructure and keys (LiteLLM Enterprise, LiteLLM proxy client examples).

Key handling: Keys stay in the customer's deployment: Kubernetes manifests use a Secret for keys and a ConfigMap for config.yaml (LiteLLM production deployment), and Enterprise adds secret-manager integrations — AWS Secrets Manager-style vaults, Google KMS, Google Secret Manager, HashiCorp Vault, CyberArk and custom secret managers — plus key rotations and read/write to the secret manager (LiteLLM Enterprise). Access is organized as "Organizations → Teams → Projects → Keys"; OSS includes request/response logging, Enterprise adds per-key/per-team routing to Langfuse, Langsmith and Arize, log export to GCS/Azure Blob, and "disable logging per team — GDPR-friendly opt-out at the team level" (LiteLLM Enterprise). Since it is self-hosted, no vendor plane sees request content by default (LiteLLM production deployment).

Where it can run

3 of 5 shapes documented
  • Vendor-hosted
  • Self-host
  • Your VPC
  • On-premise
  • Air-gapped

Dimmed shapes are not documented by the vendor, which is not the same as unsupported.

self-host via Docker, Helm, Terraform, Kubernetes, or a local binary (LiteLLM production deployment, LiteLLM proxy client examples); hybrid/VPC in the customer's own cloud (the AWS Terraform module reuses an existing VPC, subnets, database and Redis) (LiteLLM production deployment); air-gapped on Enterprise — "self-hosted or air-gapped, and yours to run" (LiteLLM Enterprise). Hosted SaaS: n.a. (not documented on the pages fetched).

Official images publish to ghcr.io/berriai mirrored at docker.litellm.ai/berriai (e.g. docker.litellm.ai/berriai/litellm:v1.90.2); the litellm-helm chart covers monolithic deployments on EKS/GKE/AKS and a componentized litellm chart splits gateway (LLM traffic, port 4000), backend (management API/UI, 4001) and ui (3000); official Terraform modules exist for AWS ECS Fargate and Google Cloud Run, and plain Kubernetes manifests are documented (LiteLLM production deployment). Licensing is a flag, not a build: litellm_license is "optional, omit for open source" (LiteLLM production deployment); Enterprise adds SSO+SCIM and support, self-served as "an instant key, emailed to you" (LiteLLM Enterprise).

API surfaces your code can keep using

7 of 7 documented
  • OpenAI chat POST /v1/chat/completions Yes

    /chat/completions (examples use curl http://0.0.0.0:4000/v1/chat/completions) (LiteLLM proxy client examples).

  • Anthropic messages POST /v1/messages Yes

    "LiteLLM Proxy is Anthropic-compatible" with /messages (LiteLLM proxy client examples).

  • OpenAI Responses POST /v1/responses Yes *

    Yes, referenced as a supported surface: Manus agents are called "through LiteLLM's OpenAI-compatible Responses API" (LiteLLM providers). Explicit /responses path: n.a.

  • Embeddings POST /v1/embeddings Yes

    /embeddings (LiteLLM proxy client examples).

  • Images POST /v1/images/generations Yes

    /image/generations (LiteLLM proxy client examples); provider-level image generation and editing are documented for Black Forest Labs, Fal AI and RunwayML (LiteLLM providers).

  • Audio POST /v1/audio/* Yes *

    Yes, both directions: /audio/transcriptions (STT) and /audio/speech (TTS) (LiteLLM proxy client examples); provider examples include Deepgram's /listen and Nvidia Riva transcription (LiteLLM providers).

  • Batch jobs POST /v1/batches Yes

    "Batches API endpoints" are exposed, plus a separate beta "batch completions" that fans one request out to N models via a comma-separated model string ("model"="llama3,gpt-3.5-turbo") (LiteLLM proxy client examples).

An asterisk marks a qualified verdict: support that is indirect (SDK compatibility or provider passthrough rather than a native endpoint), or a gap that is narrower or wider than the label suggests. Read the note before porting.

Drop-in base-URL swap: "Just change the base_url, api_key and model… LiteLLM Proxy is OpenAI-Compatible, it works with any project that calls OpenAI", and it is also Azure OpenAI-, Anthropic- and Vertex AI-compatible (/chat/completion, /embedding for Vertex) (LiteLLM proxy client examples). No proprietary SDK required — documented clients are the OpenAI SDK, LangChain, LlamaIndex, Instructor and curl (LiteLLM proxy client examples, LiteLLM production deployment). Streaming works with "stream": true (LiteLLM proxy client examples); Assistants and Fine-Tuning endpoints are also passed through (LiteLLM proxy client examples).

How much it reaches

Models Not published
Upstream providers Not published

n.a.: no aggregate model total is stated. The README says "100+ LLMs"; the providers page states no total and lists coverage per provider (LiteLLM providers; BerriAI/litellm).

n.a.: the providers index states no total, though it enumerates a long list of provider sections (OpenAI, Azure OpenAI, Azure AI, Vertex AI, Bedrock, HuggingFace, Nvidia NIM, Perplexity, RunwayML, WatsonX, xAI, Databricks, Snowflake Cortex, Together AI, vLLM, Ollama and many more) (LiteLLM providers, undated).

Whose models: All third-party or customer-run: LiteLLM ships no models and integrates provider APIs plus self-hosted servers (vLLM, Ollama, LM Studio, Llamafile, Lemonade, Nvidia Riva, custom torch-serve) (LiteLLM providers).

Your own endpoints: Yes: "Call your custom torch-serve / internal LLM APIs via LiteLLM", with documented support for vLLM ("all models on VLLM"), Ollama ("all models from Ollama"), AWS SageMaker ("All Sagemaker Huggingface Jumpstart Models"), LM Studio, Llamafile, and OpenAI-compatible endpoints selected by choosing openai as provider with an upstream URL (LiteLLM providers).

How it behaves in production

What happens when an upstream model is slow, wrong, or down — and what you can see and stop while it happens. Reliability features are recorded as where you configure them, not whether the vendor lists them, because almost every product here lists all of them.

6 of 6 reachable from code 4 of 4 can block 20 documented destinations

When something goes wrong

Each row says where the knob is, not whether the feature is on the marketing page. A control you can only reach by hand in someone else’s dashboard cannot be reviewed or version-controlled.

  • Request timeout Per request

    Your application decides per call, so one noisy endpoint can have its own timeout without a redeploy.

    Also available: In config

    per_request and config_file. Config: litellm_settings.request_timeout: 10 ("raise Timeout error if call takes longer than 10s") (LiteLLM reliability); router_settings.timeout: 30, per-deployment litellm_params.timeout: 300, and per-request "timeout": 1 in the JSON body (or extra_body={"timeout": 1} from OpenAI SDKs). Streaming has its own key stream_timeout (first-chunk deadline). No default value is stated (LiteLLM timeouts).

  • Retries Per request

    Also available: In config

    per_request and config_file. num_retries (e.g. litellm_settings.num_retries: 3), with precedence x-litellm-num-retries header → request body → deployment litellm_params → litellm_settings. Default count is not stated: n.a. Backoff: exponential for RateLimitError, immediate retry for generic errors; minimum delay configurable via retry_after. Per-error counts via RetryPolicy (e.g. TimeoutErrorRetries: 2, RateLimitErrorRetries: 3). Note max_retries is the provider SDK's internal counter and is forced to 0 on router paths to avoid (1 + N) ** 2 upstream calls (LiteLLM routing).

  • Fallback to another model In config

    Changing it means editing configuration and shipping it, so behaviour is uniform across traffic until you redeploy.

    config_file (also per-request). ORDERED: fallbacks: [{"gpt-3.5-turbo": ["gpt-4"]}], and "Fallbacks are done in-order". Specialized variants: default_fallbacks, content_policy_fallbacks, context_window_fallbacks. Fallback fires only after num_retries is exhausted (LiteLLM reliability). WEIGHTED failover also exists inside a model group via enable_weighted_failover (requires routing_strategy="simple-shuffle", uses weight/rpm/tpm, capped by max_fallbacks, default 5; async entrypoints only) (LiteLLM routing).

  • Load balancing In config

    Deployments sharing a model_name form a load-balanced model group; routing_strategy selects the algorithm (e.g. "least-busy", "simple-shuffle"), and weights are supported through weight/rpm/tpm in the weighted-failover path (LiteLLM routing, LiteLLM timeouts).

  • Upstream health tracking In config

    config_file, and it is cooldown-based ejection (LiteLLM does not call it circuit breaking): allowed_fails (default 3), cooldown_time (default 5s), disable_cooldowns, plus AllowedFailsPolicy per error type. Cooldown triggers: 429, >50% failures in the current minute, and non-retryable 401/404/408. Cooled deployments are removed from the pool and automatically reintroduced (LiteLLM routing).

  • Cross-region failover In config

    "LiteLLM will handle retrying in different regions if a call fails", with region_name (e.g. "eu") auto-inferred for Vertex AI, Bedrock and WatsonX, and weighted failover documented across Azure regional copies (eastus2, swedencentral). LiteLLM does not run the regions itself — you register per-region deployments (LiteLLM routing).

Fallback chain: Weighted split — Traffic splits by percentage across targets, so you can shift 5% to a new model and watch it before committing.

Deepest reliability surface of any self-hostable option here (retry precedence chain, per-error-type retry and failure policies, cooldowns, ordered + weighted failover, region inference), and the docs explicitly warn about retry amplification if the provider SDK's own max_retries is left on (LiteLLM routing). Kubernetes probes are documented at /health/readiness and /health/liveliness (LiteLLM production deployment).

How fast the hop is

Interpreted proxy

Runs on an interpreted or JIT runtime (Lua, Python, Node). Overhead is higher than a compiled binary and more sensitive to concurrency, though a Lua-on-nginx proxy and a Python one are far apart.

Currently in transition and the repo states both: the repository description is "The fastest, litest AI Gateway. Rust core with Python SDK", while the language breakdown is Python 84%, TypeScript 13.2%, Rust 0.7%. Best single label from the evidence: interpreted_proxy with a stated Rust core (BerriAI/litellm).

You can run the request path yourself Yes
Streaming Yes

Docker images ghcr.io/berriai/litellm:<release-tag> (cosign-signed; -stable tags after 12-hour load tests) plus componentized litellm-gateway/-backend/-ui images, Helm charts litellm-helm and litellm (oci://ghcr.io/berriai/litellm-helm), and Terraform modules for AWS ECS Fargate and GCP Cloud Run (BerriAI/litellm, LiteLLM production deployment).

Streaming caveats: Supported via "stream": true. Documented streaming machinery/caveats: stream_timeout bounds only the first chunk and can be used to abort hanging providers and retry another model; SSE keepalives via keepalive_seconds (clamped 1–300 s; 0 at deployment level is a hard disable) or proxy-wide sse_keepalive_ping_interval_seconds, sending : ping comment frames; client-side override requires allow_client_keepalive_override: true (header x-litellm-keepalive-seconds) (LiteLLM timeouts).

Published figures, grouped by what each one measured. Figures in different groups are different quantities and cannot be compared with one another — nor, in most cases, with another vendor’s figure in the same group.

Overhead added by the gateway

The only figures that speak to whether this product slows your application down. Still not comparable between vendors: each was measured on different hardware, at a different load, with a different payload.

  • 8 ms p95 Vendor-published

    At 1k RPS. The README does not say whether this is added overhead or end-to-end, and gives no payload or hardware.

    Source

Sustained capacity

Requests or queries per second sustained on the stated hardware.

  • 1,000 RPS load condition Vendor-published

    Stated only as the load the 8ms p95 figure was taken at; hardware not stated.

    Source

The 8 ms / 1k RPS figure is vendor self-published (BerriAI/litellm). Flag: the widely circulated LiteLLM latency numbers (P50 38.65 s, ~40 ms overhead, 44.84 req/s) are published by a RIVAL, Maxim AI's Bifrost, not by LiteLLM (Bifrost benchmarks) — they are not comparable to LiteLLM's own claim and should not be reported as LiteLLM data.

What it will stop

4 of 4 can block

Two separate questions per control: can it stop a request at all, and what does it do before you change any settings? A control that inspects and forwards is a logging feature, however it is named.

  • Personal data in prompts Can block the request

    Out of the box: You pick the action when configuring

    Presidio integration supports per-entity MASK or BLOCK, runs in pre_call mode, and can skip system/tool messages; masked data is forwarded downstream with pass_data: true (Guardrails quick start, Policy flow builder)

  • Prompt injection and jailbreaks Can block the request

    Out of the box: You pick the action when configuring

    sync_block via third parties — e.g. Lakera configured as guardrail: lakera, mode: pre_call, plus PromptGuard, Gray Swan, and others; detection is delegated to the vendor's classifier (Policy flow builder, Guardrail providers)

  • Harmful content Can block the request

    Out of the box: You pick the action when configuring

    sync_block — moderation guardrails (e.g. OpenAI moderation, Azure Content Safety, Bedrock Guardrails) block on policy failure such as toxic content (Guardrail providers, Policy flow builder)

  • Your own policies Can block the request

    Out of the box: You pick the action when configuring

    Policies are declared in YAML pipelines with on_pass, on_fail, on_error branches per step (allow, block, next, modify_response with a custom message), and custom guardrail classes can be registered; key-level permissions control which guardrails apply (Policy flow builder, Guardrails quick start)

Where checks run Either, your choice
If the guardrail itself fails You choose

Configurable, defaults to the policy branch: technical errors (timeouts, network errors, guardrail not loaded, connection failure) are handled by on_error, and "If on_error is omitted, the pipeline uses on_fail for error outcomes". The docs' own example fails open: "If backup_scanner errors → allow the request (set on_error to block if you prefer fail-closed)" (Policy flow builder)

Calls out to: Microsoft Presidio, AWS Bedrock Guardrails, Azure Content Safety, OpenAI Moderation, Lakera, Aporia, Pangea, Guardrails AI, Pillar Security. Each is a separate vendor relationship and a separate hop on the request path.

LiteLLM is the only product in this set that gives an explicit, documented three-way distinction between pass, policy failure, and technical error, with independent branch actions for each — the cleanest fail-open/fail-closed control available (Policy flow builder).

What you can see

Exports widely
What gets logged Your choice

You decide whether bodies are captured, by setting or by header.

You can turn bodies off Yes

yes — leave "Store Prompts in Spend Logs" off or set general_settings.store_prompts_in_spend_logs: false; additionally turn_off_message_logging, per-request header x-litellm-enable-message-redaction, and the no-log flag suppress message content on callbacks (Spend log settings, Logging)

Traces OpenTelemetry

OpenTelemetry is a first-class callback, alongside Langfuse, Arize, Langtrace, MLflow, and Langsmith for chain/agent-level traces (Logging)

"Store Prompts in Spend Logs" controls whether request messages and response content are stored; when disabled only metadata (tokens, cost, model) is kept (Spend log settings)

Where telemetry can go

  • OpenTelemetry
  • Datadog
  • Langfuse
  • Langsmith
  • Arize Phoenix
  • Prometheus
  • S3
  • GCS
  • Athina
  • Braintrust
  • Helicone
  • Lago
  • Logfire
  • Sentry
  • Slack
  • Webhook
  • Galileo
  • OpenMeter
  • DynamoDB
  • Azure Blob Storage

Langfuse, OpenTelemetry, GCS buckets, Amazon S3, Azure Blob Storage, GCS Pub/Sub, AWS SQS, DynamoDB, Datadog, Azure Sentinel, Lunary, MLflow, Deepeval, Langsmith, Arize, Langtrace, Galileo, OpenMeter, Sentry, Athina (Logging)

Records user feedback No
Scores live traffic Partly

n.a. — no native feedback/score endpoint on the fetched pages; scoring is delegated to callback platforms (Logging)

Indirectly, and without LiteLLM SaaS: Deepeval, MLflow, Langsmith, Galileo, Athina and Arize callbacks receive traffic for offline/online evaluation in those tools (Logging)

Depends on the vendor’s SaaS: No — the proxy is self-hosted, spend logs live in the customer's database, and callbacks ship to destinations the customer chooses (Logging, Spend log settings)

Retention: Configurable via maximum_spend_logs_retention_period (e.g. 7d, 30d) settable from the Admin UI without a restart; if unset, logs follow config/default behaviour and no fixed default is documented (Spend log settings)

Whether it fits how you work

How much work stands between you and a first call, how different that is from running it in production, and whether it slots into the stack you already have. Recorded as the shape of the work rather than a number of minutes — how long it takes you depends on which accounts and quota you already hold, which no comparison can know.

to try: cli or container to run: infrastructure rollout fits 5 of 10 common stacks

Getting to a first call

No numbered procedure published
Shape of the work Run something locally first

Nothing works until you have a process running. Fine on a laptop, but it means there is no zero-install way to try it.

Read off: the vendor’s own quickstart — no numbered procedure published.

Why the count is not the work: The getting-started page is prose plus commands rather than a numbered procedure.

Before step one

  • Your own provider key Not needed

    You can make a first call with only this product’s key. No upstream provider account needed.

    For the documented first call: it uses huggingface/bigcode/starcoder and states HUGGINGFACE_API_KEY is "[OPTIONAL]" (Quick Start - LiteLLM Proxy CLI). Production secrets do include a provider key, e.g. --from-literal=OPENAI_API_KEY="<provider-key>" (Production Deployment - LiteLLM)

  • Payment method Not documented

    n.a. (not documented) on either page; no enterprise/licence requirement is stated and litellm_license is optional in the Terraform examples (Quick Start - LiteLLM Proxy CLI, Production Deployment - LiteLLM)

  • Gate before models answer Not documented

    The docs do not say, so budget for a surprise on the first model you actually want.

    n.a. (not documented) (Quick Start - LiteLLM Proxy CLI)

Everything you need first: No account, credit card, cloud account, cluster, or provider key is stated; the Hugging Face key in the example is explicitly "[OPTIONAL]". Python 3.10+ is required for LiteLLM 1.84.0+ (Quick Start - LiteLLM Proxy CLI)

Running it in production

The same scale applied to the path the vendor recommends for production traffic. Kept separate from the quickstart because for several products here the two are barely related pieces of work.

Shape of the work Deploy it on your infrastructure

This is an infrastructure project, not an integration. Expect charts or Terraform, networking, secrets and someone who owns the deployment.

Getting to production is a step up in kind from the quickstart, not just more of the same.

What production needs: A Kubernetes cluster (EKS, GKE, AKS or other), external PostgreSQL (required for authentication/tracking) and Redis (required for multiple instances), cloud ingress/load balancer, and secrets management; the Terraform path provisions networking, databases, Redis, object storage, secrets, load balancer and migrations (AWS: Aurora PostgreSQL, ElastiCache, S3, ALB; GCP: Cloud SQL, Memorystore, GCS, HTTPS load balancer) (Production Deployment - LiteLLM)

Can you run it yourself

Install command published Install command published

There is a command you can copy and run, so you can evaluate the self-hosted path yourself today.

uv tool install 'litellm[proxy]' / pip install 'litellm[proxy]' for local (Quick Start - LiteLLM Proxy CLI); production images ghcr.io/berriai/litellm and docker.litellm.ai/berriai/litellm:v1.90.2, plus helm install litellm oci://ghcr.io/berriai/litellm-helm -f values.yaml and helm upgrade --install litellm oci://ghcr.io/berriai/litellm/chart/litellm --version 1.89.2 -f values.yaml, with kubectl create secret generic litellm-masterkey|litellm-db|litellm-env ... (Production Deployment - LiteLLM)

How it fits your stack

5 of 10

Each row is a thing you might already run. “With a caveat” means it works but not the way the vendor’s marketing implies — read the reason, because that is usually where the surprise lives.

  • Fits The OpenAI SDK Drop-in once set up — but first-call work is cli or container.
  • No The Vercel AI SDK No AI SDK route documented.
  • No Cloudflare Workers Workers AI is an upstream model here, which is the opposite direction.
  • Fits Kubernetes litellm-helm at oci://ghcr.io/berriai/litellm-helm — the GitHub page describes it as community maintained
  • With a caveat Terraform or OpenTofu Official modules
  • No An existing API gateway Nothing published about running behind your gateway.
  • Fits Cloud IAM I already run Reuses IAM roles, workload identity or managed identities.
  • Fits LangChain or LlamaIndex LangChain, LlamaIndex
  • Fits MCP servers to govern Acts as an MCP gateway or registry.
  • With a caveat Nothing — plain Node or Python You have to run a process locally before any call works.

Reading this the other way round — pick what you already run and see every product scored against it.

The integration surfaces behind those answers

  • Vercel AI SDK Not documented

    Nothing published. Assume the OpenAI-compatible route and verify it yourself.

    No official Vercel AI SDK provider package is documented; instead LiteLLM documents Vercel AI Gateway as an upstream provider route (vercel_ai_gateway/), and Vercel documents LiteLLM as a framework integration using the Python litellm package. (Vercel AI Gateway LiteLLM framework integration)

  • Cloudflare Workers Workers AI as an upstream model

    Cloudflare Workers AI is reachable as a model behind this product. That is the opposite direction from running your code on Workers.

    LiteLLM documents Cloudflare Workers AI support using the cloudflare/ model prefix, including cloudflare/@cf/meta/llama-2-7b-chat-int8 and cloudflare/@hf/thebloke/codellama-7b-instruct-awq. (LiteLLM Cloudflare Workers AI provider)

  • Kubernetes Official Helm chart

    A named, published chart. You can read its values file before committing to anything.

    Named: litellm-helm at oci://ghcr.io/berriai/litellm-helm — the GitHub page describes it as community maintained

    Documented by: community maintainers, per the chart's own GitHub page

    LiteLLM deployment documentation lists the Helm charts litellm-helm at oci://ghcr.io/berriai/litellm-helm and litellm at oci://ghcr.io/berriai/litellm/chart/litellm. (LiteLLM deployment documentation)

  • Terraform Official modules

    Vendor-published modules exist. Some are off-Registry Git references, so pin them explicitly.

    Named: BerriAI/litellm/aws and BerriAI/litellm/google, plus a terraform-provider-litellm

    LiteLLM deployment documentation lists official Terraform modules BerriAI/litellm/aws and BerriAI/litellm/google, along with the resource-management provider terraform-provider-litellm. (LiteLLM deployment documentation)

  • Existing API gateway Not documented

    Nothing published about sitting behind an existing gateway. Treat it as a separate hop you route to yourself.

    n.a. (not documented)

  • Cloud identity Reuses your cloud identity

    Authenticate with the identity you already run — IAM roles, workload identity or managed identities. No long-lived key to rotate.

    For AWS Bedrock AgentCore MCP servers, LiteLLM supports AWS SigV4 and can use AWS access keys or the boto3 credential chain, including IAM roles and environment variables. The documented configuration includes auth_type: "aws_sigv4", aws_region_name, and aws_service_name: bedrock-agentcore. (LiteLLM MCP documentation)

  • MCP MCP gateway or registry

    It sits in front of many MCP servers and governs access to them. This is the shape you want if MCP sprawl is the problem you are solving.

    LiteLLM Proxy provides an MCP Gateway with endpoints including /mcp-rest/tools/list and /mcp-rest/tools/call. It supports listing and calling tools, prompts, resources, Streamable HTTP, SSE, and stdio, with MCP permissions by Key, Team, Organization, and end user. (LiteLLM MCP documentation)

Python frameworks
  • LangChain
  • LlamaIndex

LiteLLM documents integrations with LangChain and LlamaIndex. Examples include LangChain classes such as ChatOpenAI and OpenAIEmbeddings, and LlamaIndex classes including AzureOpenAI, AzureOpenAIEmbedding, VectorStoreIndex, and SimpleDirectoryReader. (LiteLLM user keys and integrations)

First-party client libraries
  • Python

Documented clients and integrations include the OpenAI SDK with openai.OpenAI, the Anthropic SDK with anthropic.Anthropic, the LiteLLM Python SDK, Mistral SDK, LlamaIndex, and LangChain for JavaScript and Python. The quick-start examples also show langchain.chat_models.ChatOpenAI and langchain.embeddings.OpenAIEmbeddings. (LiteLLM proxy quick start)

Agent features: LiteLLM supports OpenAI-compatible tool/function calls for MCP tools and can automatically execute returned MCP tool calls when require_approval: "never", then feed the results back to the model. Multi-step tool-result-to-model flow is documented through litellm.experimental_mcp_client.call_openai_tool; sessions, threads, and A2A are not documented. (LiteLLM MCP documentation)

Version trap documented: on Python 3.9 pip silently resolves to 1.83.9 without an error, so you must check python --version, upgrade to 3.10+ and reinstall; uv tool install provisions a compatible Python automatically. Config via your_config.yaml and litellm --config your_config.yaml. Production adds LITELLM_SALT_KEY (set once, never change), a /health/readiness check, and an Admin UI at /ui; the proxy claims "1.5k+ requests/second during load tests" (Quick Start - LiteLLM Proxy CLI, Production Deployment - LiteLLM)

LiteLLM is documented as either a Proxy Server or a Python SDK; the Proxy Server can run via the LiteLLM CLI or Docker and listens at http://0.0.0.0:4000. The product documentation also describes self-hosted deployment on AWS, Google Cloud, Azure, and Kubernetes clusters. (LiteLLM documentation home)

Silence in the docs: 4 of the integration questions on this card have no published answer either way. That is recorded as undocumented, not as a no — but it does mean you would be verifying it yourself.

What it does well

  • Widest provider and model coverage of any gateway here (140+ providers)
  • MIT-licensed with no usage fees, plus Helm/Terraform/Docker install paths and air-gapped support
  • Virtual keys, team budgets, RPM/TPM limits and guardrails in the free tier
  • Response and semantic caching backed by Redis, S3 or GCS
  • Very broad API surface: chat, responses, embeddings, images, audio, batches, rerank, MCP

Where it falls short

  • Third-party operators report the Python proxy hitting a GIL bottleneck past ~300 RPS per instance with P99 blowing out to seconds and memory growth to OOM
  • Independent 2026 comparison recommends staying under 250-300 RPS per instance and disputes vendor cache/MCP feature claims
  • Vendor benchmark numbers are self-run and conflict sharply with rivals' self-run numbers (Kong measured LiteLLM 859% slower than Kong)
  • SSO, RBAC, audit logs and SCIM sit behind an unpublished-price commercial license

Choose it when

Platform teams that want a free, self-hosted, maximally broad provider abstraction with virtual keys and budgets, at moderate request volumes.

Look elsewhere when

You need a single instance to sustain thousands of RPS with tight tail latency, or you need certified compliance artifacts from a vendor rather than your own deployment.

Open source: You run the gateway yourself. Full control over the data path and no per-request vendor fee, in exchange for operating it.

How hard is it to leave?

Derived from six published facts, not from an opinion. The weights are fixed and the same for every product — see the arithmetic.

100 /100 Easy to leave
Portability score breakdown for LiteLLM
What helps you leave Points Source
Works with standard OpenAI code Switching away is a base-URL change rather than a rewrite of every call site. 22 /22 vendor page
No vendor-specific SDK required A proprietary client library spreads through your codebase and has to be torn out again. 10 /10 vendor page
Can use your own provider accounts Your keys and billing relationship stay yours, so removing the gateway does not cut off model access. 20 /20 vendor page
Can be self-hosted You can run it yourself instead of accepting a pricing or policy change. 20 /20 —
Configuration lives in version control Routing and budget rules are a file you keep, not dashboard state you would have to rebuild. 16 /16 vendor page
Your request history can be exported You leave with your own logs instead of abandoning them. 12 /12 vendor page

All six inputs are published, so this is scored against the full 100. This measures technical switching cost only. It does not price the engineering time to re-test prompts against a different routing stack.

LiteLLM models & pricing

Browse every imported listing from this provider, with published token rates and a link to compare other providers for the same model. This is provider-reported coverage; an absent listing does not mean unsupported.

Loading model listings…

Official model coverage source ↗ · Model source coverage and limitations

Full specification

Every field we track. Blank fields say "Not published" rather than "No" — we do not infer an absence from silence. Switch to Technical in the header for the precise field names and the low-level details.

Overview

What kind of product Category
Open source
Marketplaces resell many providers behind one key. Gateways add governance on top. Open-source projects you run yourself. Cloud platforms are hyperscaler surfaces. Inference providers host models on their own hardware.
Who runs it Deployment model
Self-host only
Managed means the vendor operates it. Self-host means you run it on your own infrastructure. Both means you can choose.
Licence Licence
MIT
Proprietary products cannot be inspected or forked. Open licences such as MIT and Apache-2.0 let you audit, modify, and run the code without permission.
Company Company
BerriAI
The organisation that maintains the product.
Who you would be signing with Vendor status
Independent company
Whether the product is still an independent company, has been acquired, is a large cloud vendor’s product line, is run by a software foundation, or has been put into maintenance mode. Maintenance mode means bug fixes and security patches only — no new features.
Last shipped an update Latest release
2026-08-23

Latest stable release v1.98.0. A v1.100.0 release candidate was tagged on 30 August 2026; release candidates are excluded here.

The date of the most recent release or version tag. A product that has not shipped in a year is a different risk from one that shipped last week, regardless of what its marketing site says.
GitHub stars GitHub stars
60,244
A rough proxy for community size on open-source projects. Not a quality measure.

Cost

Markup on model prices Token markup
None
How much the product adds on top of what the underlying model provider charges. Zero means you pay the same per-token price you would pay the model provider directly.
Fee to add funds Credit purchase fee
None
A percentage charged when you top up your balance, separate from token prices. It is easy to miss because it does not appear on the per-token price list.
Monthly cost per person Seat fee
None
A recurring per-user platform charge that applies regardless of how much you use the models.
Can use your own provider accounts BYOK supported
Yes
Bring Your Own Key: you keep direct contracts with OpenAI, Anthropic and others, and the gateway only routes traffic. This preserves negotiated rates and committed-spend discounts.
Cost of using your own accounts BYOK terms
Self-hosted only: you configure your own upstream provider keys and pay providers directly; no gateway token fee.
What the product charges to route traffic through your own provider keys.
Free tier Free tier
Open-source gateway is $0 forever: 140+ providers, virtual keys, teams, budgets, load balancing, RPM/TPM limits, guardrails.
What you can do without paying, useful for evaluation.
Enterprise plan from Enterprise plan from
Not published
Annual entry price for the enterprise tier, where one is published or credibly reported.
How the vendor makes money Pricing model
Open source with a managed tier
The shape of the vendor’s bill: does the routing layer charge a percentage on top of tokens, a flat monthly fee, both, neither (because inference is the product), or nothing at all (open source with no paid tier).
How pricing works, briefly Pricing model detail
OSS $0 free forever, self-hosted. Enterprise is talk-to-sales, sized to annual gateway request capacity, deployment architecture and support needs — 'never per token'. Volume discount tiers exist but no minimum amount is published.
A one-paragraph description that covers the caveats a pricing category cannot: introductory rates, per-feature meters, tier gating, and pricing that resets on a specific date.
Minimum commitment Minimum commitment
Enterprise is annual term; no stated minimum amount.
Whether the vendor requires a minimum contract term, a minimum spend, or a provisioned-capacity purchase to get its published rate.
Charges that fire after you go over an allowance Overage terms
None stated for OSS or Enterprise. Practical exposure is self-hosted infrastructure and database cost.
The line items that scale with usage after an included allowance is exhausted — log storage, extra requests, per-feature meters, data export — which is where cost estimates usually go wrong.
Prompt cache offered Cache mechanism
Both exact and semantic
Whether the gateway offers its own response cache, what kind of match it does (exact request, prefix, semantic), or simply passes provider caching through unchanged.
Discount on cached input Cache-read discount
Not published
How much cheaper cached tokens are than fresh input, when the vendor publishes a single figure. Bundled-inference clouds usually price this per model instead of as one number.
Premium on cache writes Cache-write premium
Not published
How much more the first write of a cached prefix costs versus a plain input token. A high write premium and a low hit rate can leave you paying more than you save, so this matters as much as the read discount.
Who captures the cache saving Cache economics
Exact-match and semantic self-hosted caches: In-Memory, Disk, Redis, Qdrant Semantic, Redis Semantic, Valkey Semantic, S3 bucket, GCS bucket. LiteLLM prices no tokens; provider cached-token rates flow through in its cost calculations. Saving is 100% the customer's; the cost you add is running Redis/Qdrant yourself.
Whether the customer keeps the full saving from caching or the vendor captures part of it — and any conditions attached (write premium, storage fees, best-effort hits).
What you can split spend by Cost attribution
The strongest of any product here: spend logged per hashed api_key, internal user, end-user/customer, team_id, tags, model_group, api_base, User-Agent (tracked as a tag) and arbitrary custom headers/metadata.
The dimensions the vendor documents for splitting spend — per key, per user, per team, per tag, per customer. Matters if you need to chargeback internally or bill an end customer.
How you get cost data out Cost export
API + database: /spend/logs, /user/info, /user/daily/activity, /gateway/daily/activity, /global/spend/report?group_by=customer|team|api_key|internal_user, x-litellm-response-cost header, and the LiteLLM_SpendLogs table you own. CSV, webhook, S3 and warehouse export not stated as first-class features.
The mechanisms the vendor publishes for exporting cost and usage data: CSV, an API, webhooks, S3, a data warehouse, or nothing at all. Any per-unit price is included.
Who pays the model bill BYOK mode
Your keys only
Whether you bring your own provider accounts (BYOK), buy inference from this vendor, or can do either. This is the single biggest commercial difference between these products: it decides who holds the contract with the model provider and who carries the spend.

Spend governance in detail

Seven signals matter when a bill starts to hurt: who can spend, how much, on what, and who gets paged when it goes wrong. Everything below is drawn from the vendor’s own pricing and docs pages — how we read these.

  • Virtual or scoped keys Yes — oss

    Keys that carry their own budget and rate-limit policy, so an intern experiment cannot spend against a production budget.

    Free. /key/generate accepts models, user_id, team_id, metadata, aliases, budgets and limits.

  • Budget caps per key Yes — oss

    A dollar or token ceiling attached to an individual key. Where enforcement is soft, one over-limit request still completes before the block kicks in.

    Free, pre-request. max_budget + budget_duration, with admin-side upperbound_key_generate_params.

  • Budget caps per team or workspace Yes — oss

    A ceiling applied at a higher scope than one key — a team, a workspace, a customer, or an entire environment.

    Teams carry max_budget; owner budgets apply to keys with no team.

  • Rate limiting as a cost control Yes — oss

    Configurable request-per-time-window caps. Platform-set rate limits do not count as spend controls; user-configurable ones do.

    Free, pre-request. tpm_limit, rpm_limit, max_parallel_requests, /key/block and /key/unblock.

  • Model allowlists Yes — oss

    A policy that constrains which models a key or team can call, keeping expensive frontier models out of the wrong hands.

    Free. Keys specify allowed models, evaluated against the key row itself.

  • Spend alerts Not published

    Alerts fired as spend approaches a threshold. Alerts that only fire after the meter has rolled over are marked as such.

    Not stated on the pages fetched.

  • Webhook notifications Not published

    Programmatic notifications on spend events, so budget breaches can page an on-call or open a ticket.

    Not stated on the pages fetched.

Enforcement: Enforced before each request

Catalog

Models available Models available
Not published
How many models you can call, shown as a range because several vendors publish different totals on different pages. Vendor-reported either way, so counts are not directly comparable — some count every provider variant of the same model separately.
Model providers reachable Upstream providers
Not published
How many distinct model providers or labs you can reach, shown as a range where the vendor’s own pages disagree. More providers usually means better redundancy when one has an outage.
Works with standard OpenAI code OpenAI-compatible API
Yes
If yes, you can usually switch to it by changing one base URL, and switch away just as easily. This is the main defence against lock-in.
OpenAI chat endpoint POST /v1/chat/completions
Yes
The endpoint almost every application ports first. "Not documented" means the vendor never states it, which is different from a documented no.
Anthropic messages endpoint POST /v1/messages
Yes
Whether Anthropic-shaped calls work without rewriting them. Several products support this only as SDK compatibility or provider passthrough rather than a native endpoint — the detail page says which.
OpenAI Responses endpoint POST /v1/responses
Yes
The newer stateful OpenAI surface. Support is much thinner across this market than chat completions.
Embeddings endpoint POST /v1/embeddings
Yes
Whether you can generate vectors through the same gateway, or need a second integration for retrieval workloads.
Image generation endpoint POST /v1/images/generations
Yes
Whether image models are reachable through the same surface as text.
Audio endpoints POST /v1/audio/*
Yes
Speech-to-text and text-to-speech. Frequently the first gap in an otherwise complete gateway.
Batch jobs endpoint POST /v1/batches
Yes
Asynchronous bulk processing, usually at a discount. Commonly undocumented, and commonly the reason a migration stalls late.
Needs the vendor’s own code library Requires a vendor-specific SDK
No
A proprietary client library spreads through your codebase and has to be torn out again if you leave. “No” is the better answer here, and it means the standard OpenAI client works.
You can export your request history Logs / usage data export
Yes
Whether you can get your own request logs, traces, or usage records back out — through an API, a bulk export, or a download. Decides whether you leave with your history or abandon it.
Settings can live in version control Declarative config-as-code
Yes
Whether routing, fallback, and budget rules can be declared in a file you keep in Git, rather than existing only as settings clicked into a hosted dashboard.
Embeddings Embeddings
Yes
Text-to-vector models, needed for search and retrieval features.
Image generation Image generation
Yes
Whether image models are reachable through the same interface.
Speech and audio Speech and audio
Yes
Text-to-speech or transcription models through the same interface.
Video generation Video generation
Not published
Whether video models are reachable through the same interface.
Batch processing Batch processing
Yes
Submitting large jobs for cheaper, slower processing. Often 50% off for work that is not time-sensitive.

Routing & reliability

Uptime it promises in writing Contractual SLA uptime
Not published
The uptime percentage in a published, contractual service level agreement. A public status page is not an SLA — it reports what happened, it does not promise anything or pay you back when it breaks.
Automatic failover Automatic failover
Yes
When a model provider goes down or rate-limits you, traffic moves to a backup automatically instead of returning errors to your users.
Load balancing Load balancing
Yes
Spreads requests across several providers or keys to raise your effective rate limit.
Rule-based routing Conditional routing
Yes
Send different requests to different models based on rules — for example a cheap model for free users and a strong model for paying ones.
Response caching Response caching
Yes
Reuses the answer when the exact same request comes in again, which cuts both cost and latency.
Similar-question caching Semantic cache
Yes
Reuses an answer when a new question means roughly the same thing as an earlier one. Saves far more than exact-match caching but can return subtly wrong answers if tuned loosely.
Where you set the timeout Request timeout surface
Per request

`per_request` and `config_file`. Config: `litellm_settings.request_timeout: 10` ("raise Timeout error if call takes longer than 10s") ([LiteLLM reliability](https://docs.litellm.ai/docs/proxy/reliability)); `router_settings.timeout: 30`, per-deployment `litellm_params.timeout: 300`, and per-request `"timeout": 1` in the JSON body (or `extra_body={"timeout": 1}` from OpenAI SDKs). Streaming has its own key `stream_timeout` (first-chunk deadline). No default value is stated ([LiteLLM timeouts](https://docs.litellm.ai/docs/proxy/timeout)).

Where a request timeout can be set: per request, in a config file, in the vendor dashboard, or nowhere. Nine of the twenty products documented here do not describe a request timeout at all, so the worst case of a hung upstream call is unknowable from the docs.
Where you set retries Retry policy surface
Per request

`per_request` and `config_file`. `num_retries` (e.g. `litellm_settings.num_retries: 3`), with precedence `x-litellm-num-retries` header → request body → deployment `litellm_params` → `litellm_settings`. Default count is not stated: `n.a.` Backoff: exponential for `RateLimitError`, immediate retry for generic errors; minimum delay configurable via `retry_after`. Per-error counts via `RetryPolicy` (e.g. `TimeoutErrorRetries: 2`, `RateLimitErrorRetries: 3`). Note `max_retries` is the provider SDK's internal counter and is forced to `0` on router paths to avoid `(1 + N) ** 2` upstream calls ([LiteLLM routing](https://docs.litellm.ai/docs/routing)).

Where retry count and backoff are configured. Worth knowing alongside billing: a retried streaming call can be charged more than once.
Where you set fallbacks Fallback surface
In config

`config_file` (also per-request). ORDERED: `fallbacks: [{"gpt-3.5-turbo": ["gpt-4"]}]`, and "Fallbacks are done in-order". Specialized variants: `default_fallbacks`, `content_policy_fallbacks`, `context_window_fallbacks`. Fallback fires only after `num_retries` is exhausted ([LiteLLM reliability](https://docs.litellm.ai/docs/proxy/reliability)). WEIGHTED failover also exists inside a model group via `enable_weighted_failover` (requires `routing_strategy="simple-shuffle"`, uses `weight`/`rpm`/`tpm`, capped by `max_fallbacks`, default `5`; async entrypoints only) ([LiteLLM routing](https://docs.litellm.ai/docs/routing)).

Where the fallback chain is defined. Almost every product claims fallback; the useful question is whether you can change it from code or only by hand in a dashboard.
Shape of the fallback chain Fallback shape
Weighted split
An ordered list tries targets in sequence; a weighted split sends a percentage of traffic to each, which is what you need to trial a new model on 5% of requests. Weighted splits are much rarer than the marketing implies.
Upstream health tracking Health checks / circuit breaking
In config

`config_file`, and it is cooldown-based ejection (LiteLLM does not call it circuit breaking): `allowed_fails` (default `3`), `cooldown_time` (default `5s`), `disable_cooldowns`, plus `AllowedFailsPolicy` per error type. Cooldown triggers: `429`, >50% failures in the current minute, and non-retryable `401`/`404`/`408`. Cooled deployments are removed from the pool and automatically reintroduced ([LiteLLM routing](https://docs.litellm.ai/docs/routing)).

Whether the product notices a failing upstream and stops sending traffic to it, and whether you can tune the thresholds. This is what turns a provider outage into a blip rather than a sustained error rate, and only four of the twenty expose it.
Cross-region failover you control Multi-region failover surface
In config

"LiteLLM will handle retrying in different regions if a call fails", with `region_name` (e.g. `"eu"`) auto-inferred for Vertex AI, Bedrock and WatsonX, and weighted failover documented across Azure regional copies (`eastus2`, `swedencentral`). LiteLLM does not run the regions itself — you register per-region deployments ([LiteLLM routing](https://docs.litellm.ai/docs/routing)).

Whether you can define what happens when a region degrades. A vendor running many regions is not the same as a vendor letting you configure failover between them; only three document a user-controlled mechanism.
Where you set load balancing Load balancing surface
In config

Deployments sharing a `model_name` form a load-balanced model group; `routing_strategy` selects the algorithm (e.g. `"least-busy"`, `"simple-shuffle"`), and weights are supported through `weight`/`rpm`/`tpm` in the weighted-failover path ([LiteLLM routing](https://docs.litellm.ai/docs/routing), [LiteLLM timeouts](https://docs.litellm.ai/docs/proxy/timeout)).

Where traffic distribution across upstreams or keys is configured.

Operations

Usage dashboards and logs Observability
Yes
Built-in visibility into what was sent, what came back, what it cost, and how long it took.
Spending limits Budget controls
Yes
Hard caps that stop spend before it becomes a surprise invoice. The single most valuable control for a small team.
Rate limits Rate limits
Yes
Caps on request volume per key or per user, useful for protecting against abuse and runaway loops.
Separate keys per team or app Virtual keys
Yes
Issue scoped keys with their own budgets and permissions so you can attribute cost and revoke access without rotating everything.
Prompt versioning Prompt management
Not published
Store and version prompts outside your code so they can be changed without a deploy.
Quality testing Evals
Not published
Built-in tooling to score model output against test cases, so you can tell whether a model swap made things better or worse.
MCP support MCP support
Yes
Native support for the Model Context Protocol, the emerging standard for connecting models to external tools.
What gets logged Logged content
Your choice

"Store Prompts in Spend Logs" controls whether request messages and response content are stored; when disabled only metadata (tokens, cost, model) is kept ([Spend log settings](https://docs.litellm.ai/docs/proxy/ui_spend_log_settings))

Whether full prompts and responses are stored, only metadata, or nothing. Full-body logging is the most useful debugging feature here and the one most likely to need a conversation with your compliance team.
You can turn logging off Body-logging opt-out
Yes

`yes` — leave "Store Prompts in Spend Logs" off or set `general_settings.store_prompts_in_spend_logs: false`; additionally `turn_off_message_logging`, per-request header `x-litellm-enable-message-redaction`, and the `no-log` flag suppress message content on callbacks ([Spend log settings](https://docs.litellm.ai/docs/proxy/ui_spend_log_settings), [Logging](https://docs.litellm.ai/docs/proxy/logging))

Whether prompt and response bodies can be suppressed while still keeping usage metrics. Nineteen of the twenty document a way to do this; the mechanisms range from a per-request header to an organisation-wide setting.
Traces you can take elsewhere Distributed tracing
OpenTelemetry

OpenTelemetry is a first-class callback, alongside Langfuse, Arize, Langtrace, MLflow, and Langsmith for chain/agent-level traces ([Logging](https://docs.litellm.ai/docs/proxy/logging))

Whether the product emits OpenTelemetry, a proprietary format, or nothing. OpenTelemetry means the traces land in the tooling you already run instead of only in the vendor’s dashboard.
Where telemetry can go Export destinations
OpenTelemetry, Datadog, Langfuse, Langsmith, Arize Phoenix, Prometheus, S3, GCS, Athina, Braintrust, Helicone, Lago, Logfire, Sentry, Slack, Webhook, Galileo, OpenMeter, DynamoDB, Azure Blob Storage

Langfuse, OpenTelemetry, GCS buckets, Amazon S3, Azure Blob Storage, GCS Pub/Sub, AWS SQS, DynamoDB, Datadog, Azure Sentinel, Lunary, MLflow, Deepeval, Langsmith, Arize, Langtrace, Galileo, OpenMeter, Sentry, Athina ([Logging](https://docs.litellm.ai/docs/proxy/logging))

Documented sinks for logs and metrics. This is a good proxy for how replaceable the vendor’s own dashboard is: around twenty destinations means you never have to depend on it, while a CSV download means you do.
Can record user feedback Feedback capture API
No

`n.a.` — no native feedback/score endpoint on the fetched pages; scoring is delegated to callback platforms ([Logging](https://docs.litellm.ai/docs/proxy/logging))

Whether there is an API to attach a rating or score to a logged request, which is what lets production traffic feed quality work later.
Scores live traffic Online eval hooks
Partly

Indirectly, and without LiteLLM SaaS: Deepeval, MLflow, Langsmith, Galileo, Athina and Arize callbacks receive traffic for offline/online evaluation in those tools ([Logging](https://docs.litellm.ai/docs/proxy/logging))

Whether automated scorers can run against real production requests, rather than only against a test set you assemble yourself.

Performance

Delay it adds Proxy overhead
0.66 ms
Extra time the product itself adds to each request, on top of however long the model takes. Usually irrelevant next to multi-second model latency, but it matters for high-volume or streaming-sensitive workloads.
Requests per second ceiling Throughput
2,800 rps
Published sustained request rate before the product becomes the bottleneck. Only relevant at genuinely high volume.
What the request path runs on Architecture class
Interpreted proxy

Currently in transition and the repo states both: the repository description is "The fastest, litest AI Gateway. Rust core with Python SDK", while the language breakdown is Python 84%, TypeScript 13.2%, Rust 0.7%. Best single label from the evidence: `interpreted_proxy` with a stated Rust core ([BerriAI/litellm](https://github.com/BerriAI/litellm)).

The comparable way to talk about latency here. An edge worker, a compiled Go or Rust binary, and a Python proxy have different overhead floors no matter which figures each vendor publishes. Products that never disclose their runtime are recorded as undisclosed rather than assumed.
You can run the request path yourself Self-hostable data plane
Yes

Docker images `ghcr.io/berriai/litellm:<release-tag>` (cosign-signed; `-stable` tags after 12-hour load tests) plus componentized `litellm-gateway`/`-backend`/`-ui` images, Helm charts `litellm-helm` and `litellm` (`oci://ghcr.io/berriai/litellm-helm`), and Terraform modules for AWS ECS Fargate and GCP Cloud Run ([BerriAI/litellm](https://github.com/BerriAI/litellm), [LiteLLM production deployment](https://docs.litellm.ai/docs/proxy/deploy)).

Whether the component that actually carries your prompts can run on your own infrastructure. Distinct from a vendor offering a self-hosted control plane while still proxying traffic through their network.
Streaming responses Streaming support
Yes

Supported via `"stream": true`. Documented streaming machinery/caveats: `stream_timeout` bounds only the first chunk and can be used to abort hanging providers and retry another model; SSE keepalives via `keepalive_seconds` (clamped 1–300 s; `0` at deployment level is a hard disable) or proxy-wide `sse_keepalive_ping_interval_seconds`, sending `: ping` comment frames; client-side override requires `allow_client_keepalive_override: true` (header `x-litellm-keepalive-seconds`) ([LiteLLM timeouts](https://docs.litellm.ai/docs/proxy/timeout)).

Whether token-by-token streaming is documented. The caveats matter more than the yes: some products cannot cancel a stream without still being billed, and several timeout and fallback mechanisms stop applying once the first token has been sent.

Security & compliance

Does your prompt reach their servers Prompt transits vendor
No

Self-hosted: no personal data is collected or transmitted to LiteLLM's servers, and enterprise supports air-gapped deployment so neither your data nor your keys leave your infrastructure.

Whether the text you send passes through this company’s own infrastructure. If it does, every other promise on this page is a policy commitment rather than a physical impossibility. Self-hosted products can answer no outright.
What they keep if you change nothing Logging default
Metadata only, not content

Content logging is off by default — only metadata such as tokens, cost and model is stored. Storing request and response bodies requires opting in with store_prompts_in_spend_logs.

Defaults matter more than options. A product that stores full prompts and replies unless you find the right header will have stored them by the time you read the docs.
How long they keep it Default content retention (days)
Not published

No default deletion. Logs persist according to your configuration until you set something like maximum_spend_logs_retention_period.

Default retention for request content, in days. Zero means nothing is kept. Read the note: several products keep nothing as a rule but make timed exceptions for abuse review or specific models.
Could they train on your prompts Training on customer data
No

True by construction rather than by policy: no data or telemetry reaches LiteLLM's servers when self-hosted. No separate training policy is published.

Whether the vendor may use your prompts and outputs to train models. “Not published” means we could not find any position, which is not the same as a no — ask for it in writing.
Where it runs, and what you can pin Region and residency control
Your own infrastructure, including air-gapped. Enterprise offers a multi-region control plane.
Which regions are offered and whether you can force processing to stay in one. A global endpoint that silently picks a region is a different compliance story from an endpoint you pin yourself.
Where safety filters run Guardrail execution location
In your own infrastructure

PII masking requires Presidio analyzer and anonymizer containers that you deploy and address locally, so redaction happens entirely inside your boundary.

A filter that strips personal data only helps if it runs before the data leaves your boundary. If guardrails execute in the vendor’s cloud, the vendor has already received whatever you wanted redacted.
Who else touches the data Subprocessor list
Not published
The published list of third parties the vendor passes your data to. No list means you cannot know the full chain, which most data-protection agreements require you to.
SOC 2 audited SOC 2 audited
Not published
An independent audit of security controls. Enterprise buyers and their procurement teams routinely require it.
Will sign a HIPAA agreement HIPAA BAA
Not published
Required before you may send protected health information through the service. Without a signed BAA, healthcare data is off limits.
GDPR commitments GDPR commitments
Not published
Published data processing terms for handling personal data of people in the EU and UK.
Can keep data in the EU EU data residency
Not published
Requests can be processed inside the EU rather than routed to US infrastructure. Often the deciding constraint for European customers.
Does not retain your data Zero data retention
Not applicable
Prompts and responses are not stored after the request completes. Sometimes a paid add-on rather than the default.
Strips personal data PII redaction
Yes
Detects and removes identifiers such as names, emails, and card numbers before the request reaches the model provider.
Content guardrails Content guardrails
Yes
Policy checks on inputs and outputs — blocking unsafe content, enforcing formats, or catching prompt-injection attempts.
Runs fully disconnected Air-gapped deployment
Yes
Can be deployed in a network with no internet access, which some regulated and defence environments require.
Blocks personal data in prompts PII / DLP enforcement
Can block the request

Presidio integration supports per-entity `MASK` or `BLOCK`, runs in `pre_call` mode, and can skip system/tool messages; masked data is forwarded downstream with `pass_data: true` ([Guardrails quick start](https://docs.litellm.ai/docs/proxy/guardrails/quick_start), [Policy flow builder](https://docs.litellm.ai/docs/proxy/guardrails/policy_flow_builder))

Whether personal data detection sits on the request path and can stop the call, merely inspects and forwards it, or is not documented. A control that only reports is a logging feature, not a policy control.
Blocks prompt injection Injection / jailbreak enforcement
Can block the request

`sync_block` via third parties — e.g. Lakera configured as `guardrail: lakera, mode: pre_call`, plus PromptGuard, Gray Swan, and others; detection is delegated to the vendor's classifier ([Policy flow builder](https://docs.litellm.ai/docs/proxy/guardrails/policy_flow_builder), [Guardrail providers](https://docs.litellm.ai/docs/guardrail_providers))

Whether injection and jailbreak detection can stop a request. Most products offering this call a partner classifier rather than shipping their own.
Blocks harmful content Toxicity / moderation enforcement
Can block the request

`sync_block` — moderation guardrails (e.g. OpenAI moderation, Azure Content Safety, Bedrock Guardrails) block on policy failure such as toxic content ([Guardrail providers](https://docs.litellm.ai/docs/guardrail_providers), [Policy flow builder](https://docs.litellm.ai/docs/proxy/guardrails/policy_flow_builder))

Whether hate, violence, sexual and self-harm categories are checked inline and can stop a request, in either direction.
Your own policy rules Custom policy hooks
Can block the request

Policies are declared in YAML pipelines with `on_pass`, `on_fail`, `on_error` branches per step (`allow`, `block`, `next`, `modify_response` with a custom message), and custom guardrail classes can be registered; key-level `permissions` control which guardrails apply ([Policy flow builder](https://docs.litellm.ai/docs/proxy/guardrails/policy_flow_builder), [Guardrails quick start](https://docs.litellm.ai/docs/proxy/guardrails/quick_start))

Whether you can add your own rule — a regex, a webhook, or your own classifier — rather than choosing from the vendor library.
Where guardrails run Guardrail execution location
Either, your choice
Whether guardrail evaluation happens inside your infrastructure or on the vendor’s servers. This decides whether the prompt you are trying to protect leaves your network in order to be checked.
If the guardrail itself fails Guardrail failure mode
You choose

**Configurable, defaults to the policy branch**: technical errors (timeouts, network errors, guardrail not loaded, connection failure) are handled by `on_error`, and "If `on_error` is omitted, the pipeline uses `on_fail` for error outcomes". The docs' own example fails open: "If `backup_scanner` errors → allow the request (set `on_error` to `block` if you prefer fail-closed)" ([Policy flow builder](https://docs.litellm.ai/docs/proxy/guardrails/policy_flow_builder))

What happens when the guardrail service times out or errors: does the request proceed unchecked, or is it blocked? This is the worst-documented field in the entire catalogue — only two vendors state it plainly, which means most teams are running a control whose failure behaviour they cannot know.
Third-party guardrail vendors Guardrail integrations
Microsoft Presidio, AWS Bedrock Guardrails, Azure Content Safety, OpenAI Moderation, Lakera, Aporia, Pangea, Guardrails AI, Pillar Security
Named external guardrail services the product can call. A long list means the product is a router for policy engines rather than a policy engine itself — which also means another vendor bill and another hop.

Compliance evidence

Graded by how strong the evidence is, not whether the word appears on the vendor’s website. An audited report and a marketing claim are different things, and only one of them will satisfy your own auditor.

  • SOC 2 Vendor’s own pages disagree docs say Type II in progress with an ETA of 15 Sep 2026; the enterprise page markets it as done
  • ISO 27001 Claimed, no evidence published marketed on the enterprise page, unverified
  • GDPR DPA Not published
  • HIPAA BAA Not published
  • FedRAMP Not published
  • ITAR Not published

Vendor source

Security incident

Publicly documented incidents affecting this product. An incident here is not by itself a reason to rule a product out — what matters is what failed, whether it could recur, and what it means for the way you would deploy it.

Malicious releases published to PyPI (CVE-2026-33634)

March 24, 2026 · Critical · Resolved

Two releases, 1.82.7 and 1.82.8, were published straight to PyPI carrying a credential stealer. Neither version was ever tagged on GitHub — the attacker bypassed the project's CI/CD entirely and uploaded to PyPI directly, using an API token exposed through the compromised Trivy releases in the wider TeamPCP campaign.

The payload collected environment variables, model API keys, SSH keys, AWS, GCP and Azure credentials, Kubernetes tokens, database passwords, shell history and CI/CD secrets, encrypted them with AES-256 under an RSA-4096 key, and posted them to models.litellm[.]cloud — a domain registered the day before the upload and unconnected to the project. It then polled checkmarx[.]zone for follow-on code and persisted via a systemd user unit named sysmon.service.

1.82.8 is the more dangerous of the two. It shipped a litellm_init.pth file, which Python executes at interpreter startup, so the stealer ran on any Python process on the machine whether or not anything imported LiteLLM. In 1.82.7 the payload sat in proxy_server.py and only ran when litellm.proxy was imported.

Accounts differ How long the packages were installable is disputed by the parties involved. LiteLLM says they were live from 10:39 UTC and quarantined after roughly 40 minutes, while PyPI's own incident report puts LiteLLM's exposure at 2 hours 32 minutes from upload to quarantine, and the GitHub issue tracking it records the upload at around 08:30 UTC. LiteLLM's own advice is the safest reading: treat any install between 10:39 and 16:00 UTC on 24 March 2026 as suspect.

Package-level download counts were never published. CloudSEK put the campaign's captured dataset at roughly 434,000 files, mapping potential exposure to more than 2,500 organisations — that is the whole TeamPCP campaign, not LiteLLM alone, and the file count is not a deduplicated count of victims.

PyPI quarantined the project and both versions were removed; they no longer resolve. 1.82.6 and 1.83.0 are unaffected and remain available.

What to do

If you installed LiteLLM from PyPI inside the 24 March window, treat every credential reachable from that host as compromised and rotate it — model keys, cloud credentials, SSH keys and CI tokens. Check for litellm_init.pth in site-packages, a sysmon.service user unit, and outbound traffic to models.litellm[.]cloud or checkmarx[.]zone.

Our read

This was a compromise of the distribution channel, not a flaw in the gateway's own code, and the maintainers did not write the malware. It is recorded here because the failure mode it exposes is structural: a self-hosted gateway installed from a public package index inherits that index's supply-chain risk, and this gateway is one that sits directly on your model keys and cloud credentials. Pin versions and verify hashes rather than tracking latest.

Fit & integration

Work to try it Evaluation work shape
Run something locally first
The shape of the work on the vendor’s own quickstart, from swapping one base URL through to deploying infrastructure. An ordinal class rather than a duration, because elapsed time depends on accounts and quota we cannot see.
Work to run it Production work shape
Deploy it on your infrastructure
The same scale applied to the vendor’s recommended production path. For several products this is much heavier than the quickstart, which is exactly why both are recorded.
Steps on the quickstart Numbered quickstart steps
0

The getting-started page is prose plus commands rather than a numbered procedure.

A literal count of numbered steps on the vendor’s quickstart, recorded as evidence beside the work shape. Zero means the page publishes no numbered procedure at all. Large counts usually mean interleaved language tracks rather than more work.
Can you self-host it today Self-host install documentation
Install command published

`uv tool install 'litellm[proxy]'` / `pip install 'litellm[proxy]'` for local ([Quick Start - LiteLLM Proxy CLI](https://docs.litellm.ai/docs/proxy/quick_start)); production images `ghcr.io/berriai/litellm` and `docker.litellm.ai/berriai/litellm:v1.90.2`, plus `helm install litellm oci://ghcr.io/berriai/litellm-helm -f values.yaml` and `helm upgrade --install litellm oci://ghcr.io/berriai/litellm/chart/litellm --version 1.89.2 -f values.yaml`, with `kubectl create secret generic litellm-masterkey|litellm-db|litellm-env ...` ([Production Deployment - LiteLLM](https://docs.litellm.ai/docs/proxy/deploy))

Whether an install command is actually published. Several products advertise self-hosting while publishing no command to start from, which a plain yes/no would hide.
Works with the OpenAI SDK OpenAI SDK drop-in
Yes

Yes. LiteLLM documents OpenAI Python SDK drop-in usage with `api_key="anything"` and the exact proxy base URL `base_url="http://0.0.0.0:4000"`, using `openai.OpenAI`. ([LiteLLM documentation home](https://docs.litellm.ai/))

Whether an existing OpenAI-compatible client can be pointed at it by changing the base URL and key.
Vercel AI SDK support AI SDK provider package
Not documented

No official Vercel AI SDK provider package is documented; instead LiteLLM documents Vercel AI Gateway as an upstream provider route (`vercel_ai_gateway/`), and Vercel documents LiteLLM as a framework integration using the Python `litellm` package. ([Vercel AI Gateway LiteLLM framework integration](https://vercel.com/docs/ai-gateway/ecosystem/framework-integrations/litellm))

Whether a first-party AI SDK provider package exists, or only a community package, a documented workaround, or the generic OpenAI provider pointed at a custom base URL.
Python framework integrations Documented Python frameworks
LangChain, LlamaIndex

LiteLLM documents integrations with LangChain and LlamaIndex. Examples include LangChain classes such as `ChatOpenAI` and `OpenAIEmbeddings`, and LlamaIndex classes including `AzureOpenAI`, `AzureOpenAIEmbedding`, `VectorStoreIndex`, and `SimpleDirectoryReader`. ([LiteLLM user keys and integrations](https://docs.litellm.ai/docs/proxy/user_keys))

LangChain, LangGraph, LlamaIndex and similar orchestration frameworks with a documented integration.
Callable from Cloudflare Workers Cloudflare Workers support
Workers AI as an upstream model

LiteLLM documents Cloudflare Workers AI support using the `cloudflare/` model prefix, including `cloudflare/@cf/meta/llama-2-7b-chat-int8` and `cloudflare/@hf/thebloke/codellama-7b-instruct-awq`. ([LiteLLM Cloudflare Workers AI provider](https://docs.litellm.ai/docs/providers/cloudflare_workers))

Whether the docs show calling this product from your own Worker. Deliberately separated from the several products whose own gateway runs on Workers, which is a fact about their infrastructure and not about your edge compatibility.
Kubernetes install Helm chart availability
Official Helm chart

LiteLLM deployment documentation lists the Helm charts `litellm-helm` at `oci://ghcr.io/berriai/litellm-helm` and `litellm` at `oci://ghcr.io/berriai/litellm/chart/litellm`. ([LiteLLM deployment documentation](https://docs.litellm.ai/docs/proxy/deploy))

Whether a named, published Helm chart exists, versus Helm being referenced with no chart named, versus generic cluster documentation that has nothing to do with this product.
Terraform support Terraform provider or modules
Official modules

LiteLLM deployment documentation lists official Terraform modules `BerriAI/litellm/aws` and `BerriAI/litellm/google`, along with the resource-management provider `terraform-provider-litellm`. ([LiteLLM deployment documentation](https://docs.litellm.ai/docs/proxy/deploy))

Whether you can declare this in code: an official provider, official modules, resources inside a hyperscaler’s provider, a community provider, or only Terraform code shipped in a repo.
Reuses your cloud identity Cloud IAM reuse
Reuses your cloud identity

For AWS Bedrock AgentCore MCP servers, LiteLLM supports AWS SigV4 and can use AWS access keys or the boto3 credential chain, including IAM roles and environment variables. The documented configuration includes `auth_type: "aws_sigv4"`, `aws_region_name`, and `aws_service_name: bedrock-agentcore`. ([LiteLLM MCP documentation](https://docs.litellm.ai/docs/mcp))

Whether you can authenticate with IAM roles, workload identity or managed identities instead of another long-lived API key. Distinguished from products that only accept static upstream provider credentials.
Fits behind your API gateway API gateway integration
Not documented

n.a. (not documented)

Whether AI traffic can go through a gateway you already run, and who documented that — the product’s vendor or the gateway’s.
MCP support MCP surface shape
MCP gateway or registry

LiteLLM Proxy provides an MCP Gateway with endpoints including `/mcp-rest/tools/list` and `/mcp-rest/tools/call`. It supports listing and calling tools, prompts, resources, Streamable HTTP, SSE, and stdio, with MCP permissions by Key, Team, Organization, and end user. ([LiteLLM MCP documentation](https://docs.litellm.ai/docs/mcp))

Which kind of MCP support this is: a gateway that governs many MCP servers, a hosted MCP server you connect to, MCP tools accepted inside the completion API, or client tooling. These are different products behind one acronym.
Needs your own provider key Upstream provider key required
Not needed

No for the documented first call: it uses `huggingface/bigcode/starcoder` and states `HUGGINGFACE_API_KEY` is "[OPTIONAL]" ([Quick Start - LiteLLM Proxy CLI](https://docs.litellm.ai/docs/proxy/quick_start)). Production secrets do include a provider key, e.g. `--from-literal=OPENAI_API_KEY="<provider-key>"` ([Production Deployment - LiteLLM](https://docs.litellm.ai/docs/proxy/deploy))

Whether an upstream provider account and key must exist before your first call works. A prerequisite rather than a step, and it can differ between the hosted and self-hosted forms of the same product.
Gate before models work Model access gate
Not documented

n.a. (not documented) ([Quick Start - LiteLLM Proxy CLI](https://docs.litellm.ai/docs/proxy/quick_start))

Whether an enablement click, a quota grant, a paid tier or an approval form stands between a valid key and a working model call.
Official client languages First-party client SDK languages
Python

Documented clients and integrations include the OpenAI SDK with `openai.OpenAI`, the Anthropic SDK with `anthropic.Anthropic`, the LiteLLM Python SDK, Mistral SDK, LlamaIndex, and LangChain for JavaScript and Python. The quick-start examples also show `langchain.chat_models.ChatOpenAI` and `langchain.embeddings.OpenAIEmbeddings`. ([LiteLLM proxy quick start](https://docs.litellm.ai/docs/proxy/quick_start))

Languages with a first-party client library. An empty list can still mean the product is usable from any language via an OpenAI-compatible SDK.

How pricing actually works

OSS is free (MIT); you pay only for your own containers plus Postgres and Redis. Enterprise features (SSO, RBAC, JWT auth, SCIM, audit logs, support SLAs) require a paid LiteLLM commercial license whose price is not published.

Back to top ↑

Common questions

Answered from the fields above, so these move when the catalog moves. Every figure quoted here appears in the specification with its source.

Does LiteLLM charge a markup on model prices?

LiteLLM adds no percentage markup to model prices. Adding funds is free. The cost estimator on this site itemises these mechanisms against your own volume, because which one is cheapest depends entirely on the numbers you put in. The cost estimator itemises this against your own volume.

Can LiteLLM be self-hosted?

Yes — self-hosting is the only way to run LiteLLM; there is no vendor-hosted option. The licence is MIT. Air-gapped installation is supported. Running it yourself means you supply the infrastructure and the upstream model accounts, so the bill is your own hosting plus the providers' own rates.

Is LiteLLM SOC 2 audited, and will it sign a HIPAA BAA?

LiteLLM publishes neither a SOC 2 report nor a HIPAA business associate agreement. Each of these is linked to the vendor's own page in the compliance section below. Neither absence means a refusal: both are things a vendor either publishes or does not, and smaller products often hold the certification without advertising it.

Does LiteLLM retain your prompts?

Zero data retention does not apply to LiteLLM: it runs inside your own infrastructure, so prompts never reach a vendor. Whether prompt and response bodies are logged is configurable. Logging can be turned off. It states that it does not train on customer data.

Can you use your own provider keys with LiteLLM?

Yes. LiteLLM can route through your own accounts with the underlying model providers, so inference is billed to you directly. Self-hosted only: you configure your own upstream provider keys and pay providers directly; no gateway token fee. The billing route is worth confirming with the vendor before committing.

Back to top ↑

What has changed here

  1. GitHub stars GitHub stars 59000 60244 source ↗
  2. GitHub stars GitHub stars 57500 59000 source ↗
  3. Models available Models available 1892 Not published source ↗
  4. catalog entry catalog entry Not published Added to the catalog source ↗
See this in the full changelog Back to top ↑

Read the head-to-head

These pairs have a written verdict, not just a table.

Usually weighed against

Follow updates about LiteLLM