Managed only Proprietary

Groq Inference provider

Groq is an inference provider: an OpenAI-compatible API in front of 11 models. You pay for inference itself rather than a routing fee. It cannot be self-hosted. Zero data retention and SOC 2 are published; a HIPAA BAA is not. Beyond chat it also serves audio and batch jobs. Published throughput is 1,000 requests per second.

· 34 dated entries · 50 source references

US company · no EU region

Compare with others Estimate cost Visit site ↗

Access at a glance

Whether this product can work for you at all, before features matter: who pays the model bill, where it can run, and which of your existing API calls keep working. Every value is the vendor’s own claim, linked to the page it came from.

Groq presents GroqCloud as "fast LLM inference, OpenAI-compatible", where "hosted models are directly accessible through the GroqCloud Models API endpoint" (Groq docs overview; Supported models). It is a raw first-party inference API on Groq's own silicon — "we pioneered the LPU" (Groq pricing).

Who pays the model bill

Their credits only

You buy inference from this vendor. There is no path to bring your own provider account.

In effect: usage is authenticated with a Groq API key (GROQ_API_KEY, Authorization: Bearer $GROQ_API_KEY) and priced per 1M tokens / per 1M characters / per hour by Groq, with no upstream provider keys to supply (Groq OpenAI compatibility; Supported models).

Merchant of record: Groq: it publishes and charges the token, character and hourly prices for its hosted models (e.g. "$0.15 input $0.60 output" per 1M tokens, "$40.00 per 1M characters", "$0.111 per hour"), with batch at a 50% discount and enterprise items shown as "Contact Sales" (Supported models; Groq Batch API). The pricing page itself does not state markup, BYOK or credit mechanics: n.a. (Groq pricing).

Key handling: Single-vendor key model: requests carry a Groq API key in the Authorization: Bearer header, sourced from GROQ_API_KEY and created in the Groq Console ("you can find your API key here") (Groq OpenAI compatibility; Supported models). No third-party provider secrets exist. Because inference is hosted, Groq necessarily processes request content; explicit retention or vault/KMS statements: n.a. on the pages fetched.

Where it can run

1 of 5 shapes documented
  • Vendor-hosted
  • Self-host
  • Your VPC
  • On-premise
  • Air-gapped

Dimmed shapes are not documented by the vendor, which is not the same as unsupported.

hosted SaaS only ("hosted models are directly accessible through the GroqCloud Models API endpoint") (Supported models). Self-host, hybrid/VPC, on-prem, air-gapped: n.a. on the pages fetched.

No customer-installed component; usage is against https://api.groq.com/openai/v1 with a Groq API key (Groq docs overview). Capacity is Groq-operated: "we're building hundreds of megawatts of capacity, with many more on the way", and "with LPX, it works alongside NVIDIA's next-generation GPUs" (Groq pricing). Tier gating shows up as rate limits under a "DEVELOPER PLAN" column, with some models' prices and limits marked "Contact Sales" (Supported models).

API surfaces your code can keep using

4 of 7 documented
  • OpenAI chat POST /v1/chat/completions Yes

    OpenAI-compatible base URL https://api.groq.com/openai/v1, with /v1/chat/completions used as a batch endpoint target too (Groq OpenAI compatibility; Groq Batch API).

  • Anthropic messages POST /v1/messages Not documented

    n.a. (not documented on the pages fetched).

  • OpenAI Responses POST /v1/responses Yes

    POST https://api.groq.com/openai/v1/responses, described as "a more advanced interface for generating model responses" supporting text and image inputs, stateful conversations using previous responses as context, and function calling (Groq docs overview; Groq OpenAI compatibility).

  • Embeddings POST /v1/embeddings Not documented

    n.a. (no embeddings endpoint documented on the pages fetched).

  • Images POST /v1/images/generations Not documented *

    n.a. as generation; image inputs are supported by the Responses API and vision chat requests (Groq OpenAI compatibility; Groq Batch API).

  • Audio POST /v1/audio/* Yes

    "Audio Transcription and Translation" (STT) via /v1/audio/transcriptions and /v1/audio/translations, served by whisper-large-v3 / whisper-large-v3-turbo; TTS-style models appear in the catalog (canopylabs/orpheus-v1-english, priced per 1M characters) (Groq OpenAI compatibility; Groq Batch API; Supported models).

  • Batch jobs POST /v1/batches Yes

    The Groq Batch API takes a JSONL file where "each line in this file represents a single API call", POST only, supporting /v1/chat/completions, /v1/audio/transcriptions and /v1/audio/translations, up to 50,000 lines and 200MB per file, up to 200 batch IDs per status query, multiple concurrent jobs, priced "at a 50% cost discount compared to synchronous API pricing", charging only successfully completed requests, and with rate limits separate from synchronous per-model limits (Groq Batch API).

An asterisk marks a qualified verdict: support that is indirect (SDK compatibility or provider passthrough rather than a native endpoint), or a gap that is narrower or wider than the label suggests. Read the note before porting.

Drop-in base-URL swap: "We designed Groq API to be mostly compatible with OpenAI's client libraries" — pass the Groq key as api_key and set base_url to https://api.groq.com/openai/v1; Groq also ships "our own Groq Python and Groq TypeScript libraries that we encourage you to use" (Groq OpenAI compatibility). Documented compatibility gaps: logprobs, logit_bias, top_logprobs and messages[].name are unsupported and "will result in a 400 error"; if N is supplied it must equal 1; temperature of 0 "will be converted to 1e-8" (a float32 value > 0 and <= 2 is recommended); and audio vtt and srt response formats are not supported (Groq OpenAI compatibility). Model discovery is GET https://api.groq.com/openai/v1/models, returning "a JSON list of all active models" (Supported models).

How much it reaches

Models 11
Upstream providers Not published

Models: Counted from the vendor’s own published list; no aggregate total is published.

Count of entries returned by the models API on 2026-09-26. Includes every entry exposed by that endpoint; not a count of unique base models.

First-party only: Groq runs its own inference on its own hardware ("we pioneered the LPU") and does not route to third-party provider APIs, so no provider count is published (Groq pricing; Supported models).

Whose models: Vendor hardware serving open-weight third-party models: Groq hosts models such as Llama, GPT-OSS, Whisper, Qwen and MiniMax on its own LPU/LPX infrastructure and exposes them through its API; there is no third-party routing (Supported models; Groq pricing).

Your own endpoints: n.a. (no customer endpoint registration or custom-model upload documented on the pages fetched) (Supported models).

How it behaves in production

What happens when an upstream model is slow, wrong, or down — and what you can see and stop while it happens. Reliability features are recorded as where you configure them, not whether the vendor lists them, because almost every product here lists all of them.

0 of 6 reachable from code nothing documented on the request path no documented export

When something goes wrong

Each row says where the knob is, not whether the feature is on the marketing page. A control you can only reach by hand in someone else’s dashboard cannot be reviewed or version-controlled.

Groq is an inference provider with essentially no published reliability control surface: timeouts, retries, fallback, health and balancing are all your responsibility (or your gateway's). Also note the benchmark figures are dated Llama 2 70B-era results (Groq LPU independent benchmark).

How fast the hop is

Undisclosed vendor service

The vendor does not disclose what the request path runs on, so no overhead floor can be inferred at all.

vendor_saas on the software side, with disclosed custom hardware: the LPU Inference Engine served through GroqCloud and the Groq API. No proxy runtime is disclosed (Groq LPU independent benchmark).

You can run the request path yourself No
Streaming Yes

Only client SDKs are distributed (pip install groq, Groq Python and Groq TypeScript libraries) (Groq text generation docs, Groq OpenAI compatibility).

Streaming caveats: Supported via stream=True, returning "an iterator of completion deltas rather than a single, full completion"; async streaming supported with AsyncGroq. Non-streaming caveats on the OpenAI-compatible surface: logprobs, logit_bias, top_logprobs and messages[].name are unsupported and return 400; N must equal 1; temperature: 0 is converted to 1e-8; audio transcription/translation do not support vtt or srt (Groq text generation docs, Groq OpenAI compatibility).

Published figures, grouped by what each one measured. Figures in different groups are different quantities and cannot be compared with one another — nor, in most cases, with another vendor’s figure in the same group.

Full round trip, including the model

Dominated by the upstream model, not the gateway. Useful as a sanity check, useless for comparing routing layers.

  • 0.8 s total for 100 output tokens Third-party benchmark

    Llama 2 Chat 70B, ~100-token prompts generating ~200 tokens; estimated from latency and throughput, refreshed every three hours.

    Source

Model output speed

How fast the model emits tokens. This is an inference-provider metric and has nothing to do with routing overhead.

  • 241 tokens/s output speed Third-party benchmark

    Llama 2 Chat 70B, measured by ArtificialAnalysis.ai. Groq separately claims 300 tokens/s internally.

    Source

Mixed, and the distinction matters: the 241 tok/s and 0.8 s figures originate from independent third party ArtificialAnalysis.ai but are reported on Groq's own pages; the 300 tok/s figure is vendor internal. The same third-party benchmark includes competitors (Microsoft Azure, Amazon Bedrock, Perplexity, Together.ai, Anyscale, Deepinfra, Fireworks, Lepton) — those rival numbers reach you via Groq's retelling, so verify against each rival before quoting (Groq LPU benchmark blog).

What it will stop

nothing documented on the request path

No request-path policy controls are documented. That is not a fault in a product built purely for routing — but it means anything you need blocked has to be blocked before the call reaches here.

What you can see

No documented export
What gets logged Metadata only

Token counts, latency and model names are stored, but not the text itself.

You can turn bodies off Yes

Zero Data Retention is available through Data Controls; metadata retention is not optional (Your data)

Traces OpenTelemetry via an integration

Via third-party OTel instrumentation: Arize Phoenix tracing is documented for Groq calls (Arize + Groq)

metadata_only — request metadata is always retained; customer prompt/completion data is not retained by default, though abuse-detection logs may be held for up to 30 days (Your data)

Where telemetry can go

No documented export. Whatever this product records stays in its own interface, so it cannot become part of the monitoring you already run.

Observability integrations are client-side: Arize, MLflow, and LangSmith (Integrations); Arize Phoenix specifically for OTel traces (Arize + Groq)

Records user feedback No
Scores live traffic No

n.a. (Integrations)

n.a. from Groq; evaluation happens in the connected observability platform (Integrations)

Retention: Metadata retained (duration unspecified); abuse-related logs up to 30 days; no customer content retention by default (Your data)

Whether it fits how you work

How much work stands between you and a first call, how different that is from running it in production, and whether it slots into the stack you already have. Recorded as the shape of the work rather than a number of minutes — how long it takes you depends on which accounts and quota you already hold, which no comparison can know.

to try: sdk install to run: sdk install fits 4 of 10 common stacks

Getting to a first call

No numbered procedure published
Shape of the work Install a package

You add a dependency and construct the vendor’s client. Small, but it is a code change and a package you now maintain.

Read off: the vendor’s own quickstart — no numbered procedure published.

Why the count is not the work: The quickstart is prose plus a code sample with no numbered procedure.

Before step one

Everything you need first: A Groq API key, recommended as the GROQ_API_KEY environment variable. No account, credit card, cloud account, cluster or provider key is mentioned (Quickstart - GroqDocs)

The vendor’s own time claim: Vendor claim, verbatim: "Get up and running with the Groq API in a few minutes" (Quickstart - GroqDocs) Quoted, not verified. Marketing time claims assume every account and approval is already in place.

Running it in production

The same scale applied to the path the vendor recommends for production traffic. Kept separate from the quickstart because for several products here the two are barely related pieces of work.

Shape of the work Install a package

You add a dependency and construct the vendor’s client. Small, but it is a code change and a package you now maintain.

What production needs: n.a. (not documented) beyond rate-limit tiering: "Need higher rate limits? Upgrade to Developer plan to access higher limits, Batch and Flex processing, and more." (Rate Limits - GroqDocs)

Can you run it yourself

Install command published No self-hosting

This runs on the vendor’s infrastructure only.

How it fits your stack

4 of 10

Each row is a thing you might already run. “With a caveat” means it works but not the way the vendor’s marketing implies — read the reason, because that is usually where the surprise lives.

  • Fits The OpenAI SDK Drop-in once set up — but first-call work is sdk install.
  • Fits The Vercel AI SDK @ai-sdk/groq
  • With a caveat Cloudflare Workers Documented by Cloudflare, not the vendor.
  • No Kubernetes No Kubernetes deployment published.
  • No Terraform or OpenTofu Nothing published for Terraform.
  • No An existing API gateway Nothing published about running behind your gateway.
  • No Cloud IAM I already run No identity integration published.
  • Fits LangChain or LlamaIndex LangChain, LlamaIndex
  • With a caveat MCP servers to govern MCP tools in the API — governs nothing on your side.
  • Fits Nothing — plain Node or Python Install a package.

Reading this the other way round — pick what you already run and see every product scored against it.

The integration surfaces behind those answers

  • Vercel AI SDK Official provider package

    Install the package, swap the model factory, done. Maintained by a party with a stake in it.

    Named: @ai-sdk/groq

    Yes. Groq provides Vercel AI SDK integration through @ai-sdk/groq and ai; the provider can be imported from @ai-sdk/groq and used as groq('llama-3.3-70b-versatile'). (Groq AI SDK integration docs)

  • Cloudflare Workers Documented by Cloudflare, not the vendor

    The integration is documented by Cloudflare rather than by this vendor, so treat it as a working recipe without a vendor support commitment.

    Documented by: Cloudflare

    Documented by Cloudflare, not Groq: Groq is a supported Cloudflare AI Gateway provider, with base URL https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/groq replacing https://api.groq.com/openai/v1; no Workers binding is documented on that page. (Cloudflare AI Gateway - Groq provider)

  • Kubernetes Not documented

    No Kubernetes story published.

    n.a. (not documented)

  • Terraform Not documented

    No Terraform surface published. Configuration is API or dashboard work.

    n.a. (not documented)

  • Existing API gateway Not documented

    Nothing published about sitting behind an existing gateway. Treat it as a separate hop you route to yourself.

    n.a. (not documented)

  • Cloud identity Not documented

    No identity integration published. Expect API keys in a secret store.

    n.a. (not documented)

  • MCP MCP tools in the API

    The completion API accepts MCP tool definitions, so the model can call MCP tools. A model capability, not an MCP control plane.

    Yes. Groq’s Responses API supports remote MCP over HTTPS using tools such as type: "mcp", server_label, server_url, headers, require_approval, and allowed_tools; remote MCP is currently in beta. Groq also documents a groq-mcp MCP server package usable with Claude Desktop and other MCP clients. (Groq remote MCP docs)

Python frameworks
  • LangChain
  • LlamaIndex

Groq documents LangChain integration through langchain-groq and the ChatGroq class. LlamaIndex integration is also documented for Python and JavaScript, with the LlamaIndex page additionally documenting installation of LlamaIndex. (Groq LangChain integration docs)

First-party client libraries
  • Python
  • TypeScript

Groq provides official Python and JavaScript/TypeScript libraries: Python package groq with the Groq class, and JavaScript/TypeScript package groq-sdk with the Groq class; both offer synchronous and asynchronous clients. Groq is also compatible with OpenAI’s client libraries, and its documentation shows Vercel AI SDK packages ai and @ai-sdk/groq. (Groq client libraries docs)

Agent features: The Responses API documents function calling, code_interpreter, browser_search, MCP tools, multi-step orchestration, exposed reasoning, tool discovery, and MCP approval flows using mcp_approval_request and mcp_approval_response. Groq’s compound-beta agentic tools system also supports code generation and execution, API calls, and webpage data retrieval; sessions, threads, and A2A are not documented. (Groq remote MCP docs)

Rate limits are the practical gate rather than approvals; the Developer plan unlocks higher limits plus Batch and Flex processing (Rate Limits - GroqDocs). The quickstart also points to an onboarding video, the Playground, the developer community and the Groq API Cookbook (Quickstart - GroqDocs)

Groq’s documentation lists Google Workspace Connectors for Gmail, Google Calendar, and Google Drive, along with integrations for external services. Its ecosystem also includes Vercel deployment documentation, the Groq developer community and API Cookbook, and MCP integrations with clients such as Claude Desktop, Cursor, and Windsurf. (Groq platform overview)

Silence in the docs: 6 of the integration questions on this card have no published answer either way. That is recorded as undocumented, not as a no — but it does mean you would be verifying it yourself.

What it does well

  • Highest published output speeds of the group, up to 1,000 tokens/sec on GPT-OSS-20B on its own LPU hardware
  • Zero Data Retention is self-serve for every customer, globally or per feature, with no retention of inference data by default
  • Very low per-token prices, for example gpt-oss-120b at $0.15 input / $0.60 output per 1M tokens
  • Mostly OpenAI-compatible API plus speech-to-text and text-to-speech endpoints
  • SOC 2 Type II maintained with reports published for 2024 and 2025

Where it falls short

  • Smallest catalog here: open-weight models only, with no frontier closed models and no published model count
  • Single-source hardware provider — no cross-provider routing, load balancing or failover, so an outage has no in-platform fallback
  • All retained data sits in US Google Cloud buckets, with no EU residency option and no documented HIPAA BAA
  • Free-tier rate limits are tight (about 30 requests/min on gpt-oss-120b) and some OpenAI parameters are unsupported

Choose it when

Latency-sensitive, user-facing applications on open-weight models where output speed matters more than catalog breadth.

Look elsewhere when

You need a broad multi-vendor model catalog, EU data residency, or a signed HIPAA BAA.

Inference provider: Hosts open-weight models on its own hardware. Often the cheapest or fastest route to a specific open model, but it is one source, not a router.

How hard is it to leave?

Derived from six published facts, not from an opinion. The weights are fixed and the same for every product — see the arithmetic.

32 /68 Hard to leave
Portability score breakdown for Groq
What helps you leave Points Source
Works with standard OpenAI code Switching away is a base-URL change rather than a rewrite of every call site. 22 /22 vendor page
No vendor-specific SDK required A proprietary client library spreads through your codebase and has to be torn out again. 10 /10 vendor page
Can use your own provider accounts Your keys and billing relationship stay yours, so removing the gateway does not cut off model access. not published —
Can be self-hosted You can run it yourself instead of accepting a pricing or policy change. 0 /20 —
Configuration lives in version control Routing and budget rules are a file you keep, not dashboard state you would have to rebuild. 0 /16 vendor page
Your request history can be exported You leave with your own logs instead of abandoning them. not published —

Read the fine print: Invoice downloads only; no documented usage or log export.

2 of the 6 inputs are not published, so the highest reachable score here is 68 rather than 100. That is a gap in the public documentation, not a mark against the product — no points are deducted, they simply cannot be claimed. This measures technical switching cost only. It does not price the engineering time to re-test prompts against a different routing stack.

Groq models & pricing

Browse every imported listing from this provider, with published token rates and a link to compare other providers for the same model. This is provider-reported coverage; an absent listing does not mean unsupported.

Loading model listings…

Official model coverage source ↗ · Model source coverage and limitations

Full specification

Every field we track. Blank fields say "Not published" rather than "No" — we do not infer an absence from silence. Switch to Technical in the header for the precise field names and the low-level details.

Overview

Interpret these fields: Self-hosted vs managed LLM gateways · Who still owns your LLM gateway?

What kind of product Category
Inference provider
Marketplaces resell many providers behind one key. Gateways add governance on top. Open-source projects you run yourself. Cloud platforms are hyperscaler surfaces. Inference providers host models on their own hardware.
Who runs it Deployment model
Managed only
Managed means the vendor operates it. Self-host means you run it on your own infrastructure. Both means you can choose.
Licence Licence
Proprietary
Proprietary products cannot be inspected or forked. Open licences such as MIT and Apache-2.0 let you audit, modify, and run the code without permission.
Company Company
Groq
The organisation that maintains the product.
Who you would be signing with Vendor status
Independent company
Whether the product is still an independent company, has been acquired, is a large cloud vendor’s product line, is run by a software foundation, or has been put into maintenance mode. Maintenance mode means bug fixes and security patches only — no new features.
Last shipped an update Latest release
2026-04-18

Most recent GroqCloud changelog entry. Four months without a public update is an outlier among the actively developed products here.

The date of the most recent release or version tag. A product that has not shipped in a year is a different risk from one that shipped last week, regardless of what its marketing site says.
GitHub stars GitHub stars
Not published
A rough proxy for community size on open-source projects. Not a quality measure.

Cost

Interpret these fields: How LLM gateway pricing works · LLM gateway spending limits: stop a runaway agent bill? · Choosing a gateway for realtime voice agents

Markup on model prices Token markup
Not published
How much the product adds on top of what the underlying model provider charges. Zero means you pay the same per-token price you would pay the model provider directly.
Fee to add funds Credit purchase fee
Not published
A percentage charged when you top up your balance, separate from token prices. It is easy to miss because it does not appear on the per-token price list.
Monthly cost per person Seat fee
Not published
A recurring per-user platform charge that applies regardless of how much you use the models.
Can use your own provider accounts BYOK supported
Not published
Bring Your Own Key: you keep direct contracts with OpenAI, Anthropic and others, and the gateway only routes traffic. This preserves negotiated rates and committed-spend discounts.
Cost of using your own accounts BYOK terms
Not published
What the product charges to route traffic through your own provider keys.
Free tier Free tier
Free tier with per-model rate limits (for example gpt-oss-120b at 30 requests/min, 1,000 requests/day, 8,000 tokens/min); the paid Developer plan raises this to about 1,000 RPM and 250,000 TPM and adds Batch and Flex processing.
What you can do without paying, useful for evaluation.
Enterprise plan from Enterprise plan from
Not published
Annual entry price for the enterprise tier, where one is published or credibly reported.
Cost to run it yourself Self-host cost
No self-host option; Groq sets its own per-token prices on its own LPU/LPX hardware rather than marking up another vendor's list price. Representative rates: gpt-oss-120b $0.15 input / $0.60 output per 1M tokens, gpt-oss-20b $0.075/$0.30, Qwen3.6-27B $0.60/$3.00.
What self-hosting actually costs once you account for infrastructure and any paid tier.
How the vendor makes money Pricing model
Bundled — inference is the product
The shape of the vendor’s bill: does the routing layer charge a percentage on top of tokens, a flat monthly fee, both, neither (because inference is the product), or nothing at all (open source with no paid tier).
How pricing works, briefly Pricing model detail
Pay-for-own-inference on a free/Developer/Enterprise ladder; pay-as-you-go per token with progressive billing. No separate gateway or platform fee — routing/API layer is $0.
A one-paragraph description that covers the caveats a pricing category cannot: introductory rates, per-feature meters, tier gating, and pricing that resets on a specific date.
Minimum commitment Minimum commitment
None stated. Bills in arrears; progressive billing thresholds at $1, $10, $100, $500 and $1,000 (India: $1, $10, then recurring $100). $0.50 minimum before billing.
Whether the vendor requires a minimum contract term, a minimum spend, or a provisioned-capacity purchase to get its published rate.
Charges that fire after you go over an allowance Overage terms
None stated. Downgrading stops usage charges immediately and returns the account to Free tier rate limits and restrictions.
The line items that scale with usage after an included allowance is exhausted — log storage, extra requests, per-feature meters, data export — which is where cost estimates usually go wrong.
Prompt cache offered Cache mechanism
Prefix cache
Whether the gateway offers its own response cache, what kind of match it does (exact request, prefix, semantic), or simply passes provider caching through unchanged.
Discount on cached input Cache-read discount
50%
How much cheaper cached tokens are than fresh input, when the vendor publishes a single figure. Bundled-inference clouds usually price this per model instead of as one number.
Premium on cache writes Cache-write premium
None
How much more the first write of a cached prefix costs versus a plain input token. A high write premium and a low hit rate can leave you paying more than you save, so this matters as much as the read discount.
Who captures the cache saving Cache economics
Prefix prompt caching, free to use. 50% discount on cached input tokens; pricing discount applies only on successful cache hits. No cache-write charge. Clean win for the customer — no feature fee, no write premium — with one caveat: batch requests already receive a 50% discount on all tokens, no additional discount applied to cached tokens in batch requests. The two discounts do not stack.
Whether the customer keeps the full saving from caching or the vendor captures part of it — and any conditions attached (write premium, storage fees, best-effort hits).
What you can split spend by Cost attribution
Not stated.
The dimensions the vendor documents for splitting spend — per key, per user, per team, per tag, per customer. Matters if you need to chargeback internally or bill an end customer.
How you get cost data out Cost export
Not stated.
The mechanisms the vendor publishes for exporting cost and usage data: CSV, an API, webhooks, S3, a data warehouse, or nothing at all. Any per-unit price is included.
Who pays the model bill BYOK mode
Their credits only
Whether you bring your own provider accounts (BYOK), buy inference from this vendor, or can do either. This is the single biggest commercial difference between these products: it decides who holds the contract with the model provider and who carries the spend.

Spend governance in detail

Seven signals matter when a bill starts to hurt: who can spend, how much, on what, and who gets paged when it goes wrong. Everything below is drawn from the vendor’s own pricing and docs pages — how we read these.

  • Virtual or scoped keys Not published

    Keys that carry their own budget and rate-limit policy, so an intern experiment cannot spend against a production budget.

    Not stated.

  • Budget caps per key Not published

    A dollar or token ceiling attached to an individual key. Where enforcement is soft, one over-limit request still completes before the block kicks in.

    Not stated as per-key.

  • Budget caps per team or workspace Yes

    A ceiling applied at a higher scope than one key — a team, a workspace, a customer, or an entire environment.

    Account-level Spend Limits supported: automated spending limits and proactive usage alerts as you approach budget thresholds. Listed among Developer-tier benefits.

  • Rate limiting as a cost control Yes — developer

    Configurable request-per-time-window caps. Platform-set rate limits do not count as spend controls; user-configurable ones do.

    Developer tier gives 'significantly increased rate limits for production workloads'. Platform-set; configurable limits not stated.

  • Model allowlists Not published

    A policy that constrains which models a key or team can call, keeping expensive frontier models out of the wrong hands.

    Not stated.

  • Spend alerts Yes

    Alerts fired as spend approaches a threshold. Alerts that only fire after the meter has rolled over are marked as such.

    Usage alerts against budget thresholds.

  • Webhook notifications Not published

    Programmatic notifications on spend events, so budget breaches can page an on-call or open a ticket.

    Not stated.

Enforcement: Some controls pre-request, others after the fact

Catalog

Interpret these fields: LLM gateway model counts: what “500+” means · Model aliases and deprecations: build an endpoint inventory · Choosing a provider for Qwen, DeepSeek, GLM or Kimi

Models available Models available
11
How many models you can call, shown as a range because several vendors publish different totals on different pages. Vendor-reported either way, so counts are not directly comparable — some count every provider variant of the same model separately.
Model providers reachable Upstream providers
Not published
How many distinct model providers or labs you can reach, shown as a range where the vendor’s own pages disagree. More providers usually means better redundancy when one has an outage.
Works with standard OpenAI code OpenAI-compatible API
Yes
If yes, you can usually switch to it by changing one base URL, and switch away just as easily. This is the main defence against lock-in.
OpenAI chat endpoint POST /v1/chat/completions
Yes
The endpoint almost every application ports first. "Not documented" means the vendor never states it, which is different from a documented no.
Anthropic messages endpoint POST /v1/messages
Not documented
Whether Anthropic-shaped calls work without rewriting them. Several products support this only as SDK compatibility or provider passthrough rather than a native endpoint — the detail page says which.
OpenAI Responses endpoint POST /v1/responses
Yes
The newer stateful OpenAI surface. Support is much thinner across this market than chat completions.
Embeddings endpoint POST /v1/embeddings
Not documented
Whether you can generate vectors through the same gateway, or need a second integration for retrieval workloads.
Image generation endpoint POST /v1/images/generations
Not documented
Whether image models are reachable through the same surface as text.
Audio endpoints POST /v1/audio/*
Yes
Speech-to-text and text-to-speech. Frequently the first gap in an otherwise complete gateway.
Batch jobs endpoint POST /v1/batches
Yes
Asynchronous bulk processing, usually at a discount. Commonly undocumented, and commonly the reason a migration stalls late.
Needs the vendor’s own code library Requires a vendor-specific SDK
No
A proprietary client library spreads through your codebase and has to be torn out again if you leave. “No” is the better answer here, and it means the standard OpenAI client works.
You can export your request history Logs / usage data export
Not published
Whether you can get your own request logs, traces, or usage records back out — through an API, a bulk export, or a download. Decides whether you leave with your history or abandon it.
Settings can live in version control Declarative config-as-code
No
Whether routing, fallback, and budget rules can be declared in a file you keep in Git, rather than existing only as settings clicked into a hosted dashboard.
Embeddings Embeddings
Not published
Text-to-vector models, needed for search and retrieval features.
Image generation Image generation
Not published
Whether image models are reachable through the same interface.
Speech and audio Speech and audio
Yes
Text-to-speech or transcription models through the same interface.
Video generation Video generation
Not published
Whether video models are reachable through the same interface.
Batch processing Batch processing
Yes
Submitting large jobs for cheaper, slower processing. Often 50% off for work that is not time-sensitive.

Routing & reliability

Interpret these fields: How LLM gateway failover actually works · Does your LLM gateway promise any uptime? · Is routing destroying your prompt cache? · Changing models without breaking production · Choosing a gateway for realtime voice agents

Uptime it promises in writing Contractual SLA uptime
Not published
The uptime percentage in a published, contractual service level agreement. A public status page is not an SLA — it reports what happened, it does not promise anything or pay you back when it breaks.
Automatic failover Automatic failover
Not published
When a model provider goes down or rate-limits you, traffic moves to a backup automatically instead of returning errors to your users.
Load balancing Load balancing
Not published
Spreads requests across several providers or keys to raise your effective rate limit.
Rule-based routing Conditional routing
Not published
Send different requests to different models based on rules — for example a cheap model for free users and a strong model for paying ones.
Response caching Response caching
Not published
Reuses the answer when the exact same request comes in again, which cuts both cost and latency.
Similar-question caching Semantic cache
Not published
Reuses an answer when a new question means roughly the same thing as an earlier one. Saves far more than exact-match caching but can return subtly wrong answers if tuned loosely.
Where you set the timeout Request timeout surface
Not documented

`not_documented` on the fetched Groq docs ([Groq text generation docs](https://console.groq.com/docs/text-chat)).

Where a request timeout can be set: per request, in a config file, in the vendor dashboard, or nowhere. Nine of the twenty products documented here do not describe a request timeout at all, so the worst case of a hung upstream call is unknowable from the docs.
Where you set retries Retry policy surface
Not documented

`not_documented`; default count and backoff: `n.a.` ([Groq text generation docs](https://console.groq.com/docs/text-chat)).

Where retry count and backoff are configured. Worth knowing alongside billing: a retried streaming call can be charged more than once.
Where you set fallbacks Fallback surface
Not documented

`not_documented` ([Groq OpenAI compatibility](https://console.groq.com/docs/openai)).

Where the fallback chain is defined. Almost every product claims fallback; the useful question is whether you can change it from code or only by hand in a dashboard.
Shape of the fallback chain Fallback shape
Shape not documented
An ordered list tries targets in sequence; a weighted split sends a percentage of traffic to each, which is what you need to trial a new model on 5% of requests. Weighted splits are much rarer than the marketing implies.
Upstream health tracking Health checks / circuit breaking
Not documented

`not_documented` ([Groq OpenAI compatibility](https://console.groq.com/docs/openai)).

Whether the product notices a failing upstream and stops sending traffic to it, and whether you can tune the thresholds. This is what turns a provider outage into a blip rather than a sustained error rate, and only four of the twenty expose it.
Cross-region failover you control Multi-region failover surface
Not documented

`not_documented` ([Groq OpenAI compatibility](https://console.groq.com/docs/openai)).

Whether you can define what happens when a region degrades. A vendor running many regions is not the same as a vendor letting you configure failover between them; only three document a user-controlled mechanism.
Where you set load balancing Load balancing surface
Not documented

`not_documented` ([Groq text generation docs](https://console.groq.com/docs/text-chat)).

Where traffic distribution across upstreams or keys is configured.

Operations

Interpret these fields: LLM gateway observability: traces, logs and export · Changing models without breaking production · Running coding agents through an LLM gateway · Model aliases and deprecations: build an endpoint inventory

Usage dashboards and logs Observability
Not published
Built-in visibility into what was sent, what came back, what it cost, and how long it took.
Spending limits Budget controls
Not published
Hard caps that stop spend before it becomes a surprise invoice. The single most valuable control for a small team.
Rate limits Rate limits
Yes
Caps on request volume per key or per user, useful for protecting against abuse and runaway loops.
Separate keys per team or app Virtual keys
Not published
Issue scoped keys with their own budgets and permissions so you can attribute cost and revoke access without rotating everything.
Prompt versioning Prompt management
Not published
Store and version prompts outside your code so they can be changed without a deploy.
Quality testing Evals
Not published
Built-in tooling to score model output against test cases, so you can tell whether a model swap made things better or worse.
MCP support MCP support
Not published
Native support for the Model Context Protocol, the emerging standard for connecting models to external tools.
What gets logged Logged content
Metadata only

`metadata_only` — request metadata is always retained; customer prompt/completion data is not retained by default, though abuse-detection logs may be held for up to 30 days ([Your data](https://console.groq.com/docs/your-data))

Whether full prompts and responses are stored, only metadata, or nothing. Full-body logging is the most useful debugging feature here and the one most likely to need a conversation with your compliance team.
You can turn logging off Body-logging opt-out
Yes

Zero Data Retention is available through Data Controls; metadata retention is not optional ([Your data](https://console.groq.com/docs/your-data))

Whether prompt and response bodies can be suppressed while still keeping usage metrics. Nineteen of the twenty document a way to do this; the mechanisms range from a per-request header to an organisation-wide setting.
Traces you can take elsewhere Distributed tracing
OpenTelemetry via an integration

Via third-party OTel instrumentation: Arize Phoenix tracing is documented for Groq calls ([Arize + Groq](https://console.groq.com/docs/arize))

Whether the product emits OpenTelemetry, a proprietary format, or nothing. OpenTelemetry means the traces land in the tooling you already run instead of only in the vendor’s dashboard.
Where telemetry can go Export destinations
Not published

Observability integrations are client-side: Arize, MLflow, and LangSmith ([Integrations](https://console.groq.com/docs/integrations)); Arize Phoenix specifically for OTel traces ([Arize + Groq](https://console.groq.com/docs/arize))

Documented sinks for logs and metrics. This is a good proxy for how replaceable the vendor’s own dashboard is: around twenty destinations means you never have to depend on it, while a CSV download means you do.
Can record user feedback Feedback capture API
No

`n.a.` ([Integrations](https://console.groq.com/docs/integrations))

Whether there is an API to attach a rating or score to a logged request, which is what lets production traffic feed quality work later.
Scores live traffic Online eval hooks
No

`n.a.` from Groq; evaluation happens in the connected observability platform ([Integrations](https://console.groq.com/docs/integrations))

Whether automated scorers can run against real production requests, rather than only against a test set you assemble yourself.

Performance

Delay it adds Proxy overhead
Not published
Extra time the product itself adds to each request, on top of however long the model takes. Usually irrelevant next to multi-second model latency, but it matters for high-volume or streaming-sensitive workloads.
Requests per second ceiling Throughput
1,000 rps
Published sustained request rate before the product becomes the bottleneck. Only relevant at genuinely high volume.
What the request path runs on Architecture class
Undisclosed vendor service

`vendor_saas` on the software side, with disclosed custom hardware: the LPU Inference Engine served through GroqCloud and the Groq API. No proxy runtime is disclosed ([Groq LPU independent benchmark](https://groq.com/newsroom/groq-lpu-inference-engine-leads-in-first-independent-llm-benchmark)).

The comparable way to talk about latency here. An edge worker, a compiled Go or Rust binary, and a Python proxy have different overhead floors no matter which figures each vendor publishes. Products that never disclose their runtime are recorded as undisclosed rather than assumed.
You can run the request path yourself Self-hostable data plane
No

Only client SDKs are distributed (`pip install groq`, Groq Python and Groq TypeScript libraries) ([Groq text generation docs](https://console.groq.com/docs/text-chat), [Groq OpenAI compatibility](https://console.groq.com/docs/openai)).

Whether the component that actually carries your prompts can run on your own infrastructure. Distinct from a vendor offering a self-hosted control plane while still proxying traffic through their network.
Streaming responses Streaming support
Yes

Supported via `stream=True`, returning "an iterator of completion deltas rather than a single, full completion"; async streaming supported with `AsyncGroq`. Non-streaming caveats on the OpenAI-compatible surface: `logprobs`, `logit_bias`, `top_logprobs` and `messages[].name` are unsupported and return `400`; `N` must equal `1`; `temperature: 0` is converted to `1e-8`; audio transcription/translation do not support `vtt` or `srt` ([Groq text generation docs](https://console.groq.com/docs/text-chat), [Groq OpenAI compatibility](https://console.groq.com/docs/openai)).

Whether token-by-token streaming is documented. The caveats matter more than the yes: some products cannot cancel a stream without still being billed, and several timeout and fallback mechanisms stop applying once the first token has been sent.

Security & compliance

Interpret these fields: LLM gateway compliance: SOC 2, HIPAA and evidence · How LLM gateway guardrails fail · Which LLM gateways store your prompts? · Do you need an MCP gateway as well?

Does your prompt reach their servers Prompt transits vendor
Yes

Hosted inference. Where retention does occur, the data sits in Google Cloud buckets in the United States.

Whether the text you send passes through this company’s own infrastructure. If it does, every other promise on this page is a policy commitment rather than a physical impossibility. Self-hosted products can answer no outright.
What they keep if you change nothing Logging default
Nothing stored by default

Groq does not retain customer data for inference requests by default. Usage metadata is collected for all users and contains no inputs or outputs. Temporary logging happens only to troubleshoot reliability-degrading errors or investigate suspected abuse.

Defaults matter more than options. A product that stores full prompts and replies unless you find the right header will have stored them by the time you read the docs.
How long they keep it Default content retention (days)
Nothing kept by default

Zero by default. Exceptions: up to 30 days for the reliability and abuse logs, 30 days for batch inputs and outputs, and fine-tuning artifacts held until you delete them.

Default retention for request content, in days. Zero means nothing is kept. Read the note: several products keep nothing as a rule but make timed exceptions for abuse review or specific models.
Could they train on your prompts Training on customer data
Not published — silence, not a no

The data page is silent, and the privacy policy explicitly routes the question to the services agreement and data-processing addendum. Read that as silence, not a no.

Whether the vendor may use your prompts and outputs to train models. “Not published” means we could not find any position, which is not the same as a no — ask for it in writing.
Where it runs, and what you can pin Region and residency control
Retained data is US-only, in Google Cloud buckets. Inference itself spans 13 data centres on four continents including Finland, the UK, Saudi Arabia and Australia, but no customer region pinning is documented.
Which regions are offered and whether you can force processing to stay in one. A global endpoint that silently picks a region is a different compliance story from an endpoint you pin yourself.
Where safety filters run Guardrail execution location
No guardrails offered

No Groq-operated guardrail or PII-redaction feature was found.

A filter that strips personal data only helps if it runs before the data leaves your boundary. If guardrails execute in the vendor’s cloud, the vendor has already received whatever you wanted redacted.
Who else touches the data Subprocessor list
https://trust.groq.com/subprocessors
The published list of third parties the vendor passes your data to. No list means you cannot know the full chain, which most data-protection agreements require you to.
SOC 2 audited SOC 2 audited
Yes
An independent audit of security controls. Enterprise buyers and their procurement teams routinely require it.
Will sign a HIPAA agreement HIPAA BAA
Not published
Required before you may send protected health information through the service. Without a signed BAA, healthcare data is off limits.
GDPR commitments GDPR commitments
Not published
Published data processing terms for handling personal data of people in the EU and UK.
Can keep data in the EU EU data residency
No
Requests can be processed inside the EU rather than routed to US infrastructure. Often the deciding constraint for European customers.
Does not retain your data Zero data retention
Yes
Prompts and responses are not stored after the request completes. Sometimes a paid add-on rather than the default.
Strips personal data PII redaction
Not published
Detects and removes identifiers such as names, emails, and card numbers before the request reaches the model provider.
Content guardrails Content guardrails
Not published
Policy checks on inputs and outputs — blocking unsafe content, enforcing formats, or catching prompt-injection attempts.
Runs fully disconnected Air-gapped deployment
Not published
Can be deployed in a network with no internet access, which some regulated and defence environments require.
Blocks personal data in prompts PII / DLP enforcement
Not documented

`not_documented` — no PII guardrail feature ([Your data](https://console.groq.com/docs/your-data))

Whether personal data detection sits on the request path and can stop the call, merely inspects and forwards it, or is not documented. A control that only reports is a logging feature, not a policy control.
Blocks prompt injection Injection / jailbreak enforcement
Not documented

Llama Prompt Guard 2 is offered as a model you call yourself ([Content moderation](https://console.groq.com/docs/content-moderation))

Whether injection and jailbreak detection can stop a request. Most products offering this call a partner classifier rather than shipping their own.
Blocks harmful content Toxicity / moderation enforcement
Not documented

`not_documented` as an inline platform control — moderation is available only as callable models: GPT-OSS-Safeguard 20B, Llama Guard 4, and Llama Prompt Guard 2. The application must make a second call and act on the verdict; Groq does not filter the request path ([Content moderation](https://console.groq.com/docs/content-moderation))

Whether hate, violence, sexual and self-harm categories are checked inline and can stop a request, in either direction.
Your own policy rules Custom policy hooks
Not documented

GPT-OSS-Safeguard accepts a custom policy in its prompt, but this is a model capability rather than a gateway policy engine ([Content moderation](https://console.groq.com/docs/content-moderation))

Whether you can add your own rule — a regex, a webhook, or your own classifier — rather than choosing from the vendor library.
Where guardrails run Guardrail execution location
Not documented
Whether guardrail evaluation happens inside your infrastructure or on the vendor’s servers. This decides whether the prompt you are trying to protect leaves your network in order to be checked.
If the guardrail itself fails Guardrail failure mode
Not documented

`not_documented` ([Content moderation](https://console.groq.com/docs/content-moderation))

What happens when the guardrail service times out or errors: does the request proceed unchecked, or is it blocked? This is the worst-documented field in the entire catalogue — only two vendors state it plainly, which means most teams are running a control whose failure behaviour they cannot know.
Third-party guardrail vendors Guardrail integrations
Not published
Named external guardrail services the product can call. A long list means the product is a router for policy engines rather than a policy engine itself — which also means another vendor bill and another hop.

Compliance evidence

Graded by how strong the evidence is, not whether the word appears on the vendor’s website. An audited report and a marketing claim are different things, and only one of them will satisfy your own auditor.

  • SOC 2 Claimed, no evidence published the DPA commits to annual Type II audits; no report or portal retrieved
  • ISO 27001 Not published
  • GDPR DPA Certified DPA published, with standard contractual clauses
  • HIPAA BAA Not published
  • FedRAMP Not published
  • ITAR Not published

Vendor source

Fit & integration

Interpret these fields: How much does an LLM gateway lock you in? · Changing models without breaking production · Running coding agents through an LLM gateway · Do you need an MCP gateway as well? · Choosing a gateway for realtime voice agents · Choosing a provider for Qwen, DeepSeek, GLM or Kimi

Work to try it Evaluation work shape
Install a package
The shape of the work on the vendor’s own quickstart, from swapping one base URL through to deploying infrastructure. An ordinal class rather than a duration, because elapsed time depends on accounts and quota we cannot see.
Work to run it Production work shape
Install a package
The same scale applied to the vendor’s recommended production path. For several products this is much heavier than the quickstart, which is exactly why both are recorded.
Steps on the quickstart Numbered quickstart steps
0

The quickstart is prose plus a code sample with no numbered procedure.

A literal count of numbered steps on the vendor’s quickstart, recorded as evidence beside the work shape. Zero means the page publishes no numbered procedure at all. Large counts usually mean interleaved language tracks rather than more work.
Can you self-host it today Self-host install documentation
No self-hosting
Whether an install command is actually published. Several products advertise self-hosting while publishing no command to start from, which a plain yes/no would hide.
Works with the OpenAI SDK OpenAI SDK drop-in
Yes

Yes. Groq supports OpenAI’s client libraries when configured with the base URL `https://api.groq.com/openai/v1` and a Groq API key. ([Groq OpenAI compatibility docs](https://console.groq.com/docs/openai))

Whether an existing OpenAI-compatible client can be pointed at it by changing the base URL and key.
Vercel AI SDK support AI SDK provider package
Official provider package

Yes. Groq provides Vercel AI SDK integration through `@ai-sdk/groq` and `ai`; the provider can be imported from `@ai-sdk/groq` and used as `groq('llama-3.3-70b-versatile')`. ([Groq AI SDK integration docs](https://console.groq.com/docs/ai-sdk/))

Whether a first-party AI SDK provider package exists, or only a community package, a documented workaround, or the generic OpenAI provider pointed at a custom base URL.
Python framework integrations Documented Python frameworks
LangChain, LlamaIndex

Groq documents LangChain integration through `langchain-groq` and the `ChatGroq` class. LlamaIndex integration is also documented for Python and JavaScript, with the LlamaIndex page additionally documenting installation of LlamaIndex. ([Groq LangChain integration docs](https://console.groq.com/docs/langchain))

LangChain, LangGraph, LlamaIndex and similar orchestration frameworks with a documented integration.
Callable from Cloudflare Workers Cloudflare Workers support
Documented by Cloudflare, not the vendor

Documented by Cloudflare, not Groq: Groq is a supported Cloudflare AI Gateway provider, with base URL `https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/groq` replacing `https://api.groq.com/openai/v1`; no Workers binding is documented on that page. ([Cloudflare AI Gateway - Groq provider](https://developers.cloudflare.com/ai-gateway/usage/providers/groq/))

Whether the docs show calling this product from your own Worker. Deliberately separated from the several products whose own gateway runs on Workers, which is a fact about their infrastructure and not about your edge compatibility.
Kubernetes install Helm chart availability
Not documented

n.a. (not documented)

Whether a named, published Helm chart exists, versus Helm being referenced with no chart named, versus generic cluster documentation that has nothing to do with this product.
Terraform support Terraform provider or modules
Not documented

n.a. (not documented)

Whether you can declare this in code: an official provider, official modules, resources inside a hyperscaler’s provider, a community provider, or only Terraform code shipped in a repo.
Reuses your cloud identity Cloud IAM reuse
Not documented

n.a. (not documented)

Whether you can authenticate with IAM roles, workload identity or managed identities instead of another long-lived API key. Distinguished from products that only accept static upstream provider credentials.
Fits behind your API gateway API gateway integration
Not documented

n.a. (not documented)

Whether AI traffic can go through a gateway you already run, and who documented that — the product’s vendor or the gateway’s.
MCP support MCP surface shape
MCP tools in the API

Yes. Groq’s Responses API supports remote MCP over HTTPS using tools such as `type: "mcp"`, `server_label`, `server_url`, `headers`, `require_approval`, and `allowed_tools`; remote MCP is currently in beta. Groq also documents a `groq-mcp` MCP server package usable with Claude Desktop and other MCP clients. ([Groq remote MCP docs](https://console.groq.com/docs/tool-use/remote-mcp))

Which kind of MCP support this is: a gateway that governs many MCP servers, a hosted MCP server you connect to, MCP tools accepted inside the completion API, or client tooling. These are different products behind one acronym.
Needs your own provider key Upstream provider key required
Not needed

No. The first call uses `GROQ_API_KEY` ([Quickstart - GroqDocs](https://console.groq.com/docs/quickstart))

Whether an upstream provider account and key must exist before your first call works. A prerequisite rather than a step, and it can differ between the hosted and self-hosted forms of the same product.
Gate before models work Model access gate
Not documented

n.a. (not documented) ([Quickstart - GroqDocs](https://console.groq.com/docs/quickstart), [Rate Limits - GroqDocs](https://console.groq.com/docs/rate-limits))

Whether an enablement click, a quota grant, a paid tier or an approval form stands between a valid key and a working model call.
Official client languages First-party client SDK languages
Python, TypeScript

Groq provides official Python and JavaScript/TypeScript libraries: Python package `groq` with the `Groq` class, and JavaScript/TypeScript package `groq-sdk` with the `Groq` class; both offer synchronous and asynchronous clients. Groq is also compatible with OpenAI’s client libraries, and its documentation shows Vercel AI SDK packages `ai` and `@ai-sdk/groq`. ([Groq client libraries docs](https://console.groq.com/docs/libraries))

Languages with a first-party client library. An empty list can still mean the product is usable from any language via an OpenAI-compatible SDK.

Additional charges

These are the fees that do not appear on a per-token price list, and they are where estimates usually go wrong.

  • Speech-to-text (whisper-large-v3) $0.111 per hour of audio ($0.04/hour for v3-turbo)
  • Text-to-speech (Orpheus) $22-$40 per 1M characters

How pricing actually works

No self-host option; Groq sets its own per-token prices on its own LPU/LPX hardware rather than marking up another vendor's list price. Representative rates: gpt-oss-120b $0.15 input / $0.60 output per 1M tokens, gpt-oss-20b $0.075/$0.30, Qwen3.6-27B $0.60/$3.00.

Back to top ↑

Common questions

Answered from the fields above, so these move when the catalog moves. Every figure quoted here appears in the specification with its source.

Does Groq charge a markup on model prices?

Groq does not resell other providers' models with a fee on top: inference is the product, so you pay its own published per-token rates. There is no separate routing charge to add to them. The cost estimator on this site itemises these mechanisms against your own volume, because which one is cheapest depends entirely on the numbers you put in.

Can Groq be self-hosted?

No. Groq is available only as a service the vendor operates; there is no self-hosted build. The licence is Proprietary. Prompts therefore leave your network and reach the vendor, which makes its retention and residency terms the control that matters here rather than deployment.

Is Groq SOC 2 audited, and will it sign a HIPAA BAA?

A SOC 2 report is available for Groq but it does not publish a HIPAA business associate agreement. EU data residency is not offered. Each of these is linked to the vendor's own page in the compliance section below. Ask for both documents directly before committing, because a published claim and a countersigned agreement are not the same thing.

Does Groq retain your prompts?

Groq publishes a zero-data-retention position. Only metadata is logged — not prompt or response bodies. Logging can be turned off. Stated retention is zero days. Retention, logging and training on customer data are three separate questions, and a vendor can answer one of them without answering the others.

Can you use your own provider keys with Groq?

Groq does not publish whether you can supply your own upstream provider keys. Bringing your own key decides whether inference is billed to you by the model provider or resold by the gateway, which is worth confirming before committing. The billing route is worth confirming with the vendor before committing.

How many models does Groq support?

Groq publishes no aggregate total; counting its catalog gives 11 models. Count of entries returned by the models API on 2026-09-26. Includes every entry exposed by that endpoint; not a count of unique base models. The figure on this page is dated and carries its source.

Back to top ↑

What has changed here

  1. Models available Models available Not published 11 source ↗
  2. Traces you can take elsewhere Distributed tracing otel otel_integration source ↗
  3. catalog entry catalog entry Not published Added to the catalog source ↗
See this in the full changelog Back to top ↑

Read the head-to-head

These pairs have a written verdict, not just a table.

Groq vs Together AI Read the verdict → Groq vs Fireworks AI Read the verdict →

Usually weighed against

Groq vs Fireworks AI Compare → Groq vs Together AI Compare →

Follow updates about Groq