Guide
LLM gateway compliance: SOC 2, HIPAA and evidence
Short answer
Evaluate an LLM gateway using the evidence for the exact service and deployment: the SOC 2 report, its scope and period, applicable agreements, subprocessors, retention and processing locations. A marketing badge or catalogue flag is not approval. A completed report supports a security review but cannot guarantee its outcome.
One word, several grades of evidence
Compliance is recorded here as an evidence grade rather than a tick, because a tick would flatten genuinely different situations into one. The distinction that matters most to a reviewer is between the first two grades: an audited report you can request, and a sentence on a pricing page. The catalogue records which of the two was found, and quotes the vendor’s own hedge where there is one.
- 12 Certified
- Evidence of a completed audit: a report under NDA, a trust portal, or a named auditor and period. Ask for the report and the scope, and check the period covers the service you are buying.
- 7 Claimed
- The vendor states it; no report, portal or auditor was found. A report may exist privately; request it and assess its scope rather than inferring either approval or failure.
- 2 Conflicting
- The vendor’s own pages disagree — a blog post and a trust centre naming different report types, for instance. The most useful grade of the set, because you would not find it by reading either page alone.
- 1 In progress
- An audit under way, sometimes with a published target date. Worth asking for the date in writing, because it decides whether you are waiting weeks or quarters.
- 2 Not applicable
- No vendor-operated service exists to audit. The compliance boundary is your own infrastructure, so the relevant certificate is yours. This is an honest answer and not a gap.
- 7 Not published
- Nothing was found either way. It is not a no. It does mean the evidence has to come from the vendor directly, which is a lead time you can measure now rather than discover during the review.
2 of the 31 read as not applicable across all 6 frameworks: Apache APISIX AI Gateway and MLflow AI Gateway. Reading those rows as failures is the most common mistake made with a table like this one. A software project and the service you operate have different audit boundaries, and a row of crosses would imply an assessment that never took place.
4 products contradict themselves somewhere across the 6 frameworks: Bifrost, LiteLLM, Respan and Together AI. That is not an accusation of bad faith; the usual cause is a trust centre and a blog post updated a long way apart. It is, though, the one finding you should raise before your own auditor does, because the answer determines which of the two statements you are entitled to rely on.
One more divergence is worth knowing about before you skim any summary column, including the one below. 6 of the 16 products whose SOC 2 flag is set do not have a completed audit recorded against them. 2 run the other way, with audit evidence found and no summary flag set. The flag answers “does this vendor address SOC 2 at all”; the grade answers “what could be verified”. Quote the grade.
Health data, personal data, and where the request lands
SOC 2 reports, HIPAA business associate agreements and GDPR processing agreements answer different questions. 10 of 31 catalogue rows record BAA availability. That is not a signed agreement for your account or evidence of a compliant configuration. Ask which services, upstream processors and purposes the actual agreement covers.
Data residencyData residency: A guarantee about which countries your data is processed in. Usually matters for EU obligations or public-sector contracts. is the narrowest of the three and the one most often decided by a single answer. 9 products document that requests can be processed inside the EU. 2 state outright that they cannot. 20 publish nothing either way, which for a self-hosted project is simply because residency is a property of where you deploy it, and for a vendor-operated service is a question you will have to ask.
Do two things with this. First, ask for the agreement rather than the badge: a signed BAA and a DPA with a named transfer mechanism are documents, and a compliance page is not. Second, check that the residency answer covers the model as well as the gateway. A gateway pinned to an EU region that forwards to a US-hosted model has moved the request across the boundary anyway, and that detail belongs in the review rather than after it.
What a retention answer can honestly cover
Zero data retentionZDR — zero data retention: A commitment that your prompts and responses are not stored after the request completes. Frequently a paid add-on or an enterprise-only option rather than the default. is the line reviewers reach for, and a gateway can only commit to half of it. 13 products document a zero-retention position. 6 record that it depends on how you deploy them, which is the honest answer for a router: the gateway can hold nothing itself while retention on the forwarded request remains the upstream model provider’s policy, under whichever account key is used. Check whether an upstream commitment is backed by the applicable agreement, and the ones that say so are being precise rather than evasive.
7 record it as not applicable, almost all of them products with no vendor-operated service in the request path — there is no third-party store to negotiate about, because nothing transits the vendor. 5 publish nothing. For the review, the useful question is not whether the word appears but which party the commitment binds: the gateway, the model provider, or you. What is actually written to a log, and for how long, is the subject of the prompt-logging guide rather than this one.
The two artefacts a review actually consumes
A subprocessor list identifies additional parties involved in processing. 15 of 31 catalogue rows have one recorded. Request the current list for the applicable service and its change-notification terms. Where no public source is recorded, ask for the document rather than assuming no list exists.
The second artefact is a disclosure record. 3 products have published security incidents recorded here, 9 in total, including advisories with assigned CVE identifiers. Read that the right way round. A published incident with a fix, a date and an advisory is evidence that the product discloses, and disclosure is the behaviour you want from anything in your request path. An empty record means nothing was found, not that nothing happened, and the products least likely to appear in such a list are the ones with no public advisory process at all. This asymmetry is the reason the count is presented and not scored.
| Product | SOC 2 | HIPAA BAA | EU residency | ZDR | Subprocessors |
|---|---|---|---|---|---|
| agentgateway | not publishedNo SOC 2 statement, trust portal or report reference on the project pages checked Checked date not recorded | Not published Check date not recorded | Not published Check date not recorded | Not applicable Checked 2026-09-02 | Not published Check date not recorded |
| AI Gateway HQ | not publishedVendor explicitly does not claim SOC 2 certification. Checked 2026-09-17 | Not published Checked 2026-09-17 | No Checked 2026-09-17 | Depends how you deploy it Checked 2026-09-17 | Published |
| Amazon Bedrock | Completed audit evidencein scope for SOC 1, 2 and 3 Checked date not recorded | Yes Checked 2026-08-29 | Not published Check date not recorded | Yes Check date not recorded | Published |
| Apache APISIX AI Gateway | not applicableNo detail recorded Checked date not recorded | Not published Check date not recorded | Not published Check date not recorded | Not applicable Check date not recorded | Not published Check date not recorded |
| Azure AI Foundry | not publishedthe Azure SOC offering page did not load during research Checked date not recorded | Yes Checked 2026-08-29 | Yes Checked 2026-08-29 | Yes Check date not recorded | Published |
| Bifrost | Completed audit evidenceType 2 for Maxim AI Checked date not recorded | Not published Check date not recorded | Not published Check date not recorded | Not published Check date not recorded | Not published Check date not recorded |
| Braintrust Gateway | Completed audit evidenceType II Checked date not recorded | Yes Checked 2026-08-29 | Yes Checked 2026-08-29 | Depends how you deploy it Check date not recorded | Published |
| Cloudflare AI Gateway | Completed audit evidenceType II covering security, confidentiality and availability Checked date not recorded | Not published Check date not recorded | Not published Check date not recorded | Not published Check date not recorded | Published |
| Eden AI | claimed"Eden AI is SOC 2 and ISO 27001 certified" on the security page and "SOC 2 & ISO 27001 certified" on the home page; no type, audit period, report or trust portal is named Checked 2026-09-02 | Not published Check date not recorded | Yes Checked 2026-09-02 | Yes Checked 2026-09-02 | Not published Check date not recorded |
| Envoy AI Gateway | not publishedNo SOC 2 statement on any page fetched; the project is Apache-2.0 software you run yourself and publishes no compliance attestations (security, SECURITY.md). Checked 2026-09-02 | Not published Check date not recorded | Not published Check date not recorded | Not applicable Checked 2026-09-02 | Not published Check date not recorded |
| Fireworks AI | Completed audit evidenceType II Checked date not recorded | Yes Checked 2026-08-29 | Not published Check date not recorded | Yes Check date not recorded | Not published Check date not recorded |
| Google Vertex AI | Completed audit evidenceType II, issued quarterly Checked date not recorded | Yes Checked 2026-08-29 | Yes Checked 2026-08-29 | Yes Check date not recorded | Published |
| Groq | claimedthe DPA commits to annual Type II audits; no report or portal retrieved Checked date not recorded | Not published Check date not recorded | No Checked 2026-08-29 | Yes Check date not recorded | Published |
| Helicone | Completed audit evidenceType II, report on request Checked date not recorded | Yes Checked 2026-08-29 | Yes Checked 2026-08-29 | Depends how you deploy it Check date not recorded | Not published Check date not recorded |
| Higress | not publishedNo SOC 2 claim on higress.ai, the documentation index or the CNCF project page. Higress is self-hosted software; certification would attach to your own deployment. Checked 2026-09-02 | Not published Check date not recorded | Not published Check date not recorded | Not applicable Checked 2026-09-02 | Not published Check date not recorded |
| Hugging Face Inference Providers | claimedScoped to the Hub, not to the gateway: "The Hugging Face Hub, which Inference Providers is a feature of, is SOC2 Type 2 certified" (https://huggingface.co/docs/inference-providers/security). No report, audit period, auditor or trust portal is named, and no Inference-Providers-specific attestation exists, so the soc2 field is left unset. Checked 2026-09-03 | Not published Check date not recorded | Not published Check date not recorded | Yes Check date not recorded | Published |
| Kong AI Gateway | Completed audit evidenceType 2, via trust center Checked date not recorded | Not published Check date not recorded | Not published Check date not recorded | Not published Check date not recorded | Published |
| LiteLLM | conflictingdocs say Type II in progress with an ETA of 15 Sep 2026; the enterprise page markets it as done Checked date not recorded | Not published Check date not recorded | Not published Check date not recorded | Not applicable Check date not recorded | Not published Check date not recorded |
| LLM Gateway | Completed audit evidenceType II, independently examined Checked date not recorded | Not published Check date not recorded | Not published Check date not recorded | Depends how you deploy it Check date not recorded | Published |
| Merge Gateway | claimedCompany-level claim only: "Merge adheres to industry-standard compliance frameworks, including SOC 2 Type II, ISO 27001, HIPAA, GDPR, and CCPA" on merge.dev/security, a page written around Merge Unified integration data (Linked Accounts, selective sync). No Gateway page, the Gateway Terms or the pricing page repeats it, and no trust portal or report is named. Checked 2026-09-02 | Not published Check date not recorded | Not published Check date not recorded | Yes Checked 2026-09-02 | Published |
| MLflow AI Gateway | not applicableNo detail recorded Checked 2026-09-02 | Not published Check date not recorded | Not published Check date not recorded | Not applicable Checked 2026-09-02 | Not published Check date not recorded |
| New API | not publishedSelf-host-only open-source project; no attestation, trust portal or certification page found on the docs site (checked Acceptable Use, Business Cooperation, API Reference). Checked 2026-09-02 | Not published Check date not recorded | Not published Check date not recorded | Not applicable Check date not recorded | Not published Check date not recorded |
| OpenRouter | claimedType II, asserted on the enterprise page Checked date not recorded | Not published Check date not recorded | Yes Checked 2026-08-29 | Yes Check date not recorded | Published |
| Orq.ai Router | Completed audit evidenceType 2, monitored via Vanta Checked date not recorded | Yes Checked 2026-08-29 | Yes Checked 2026-08-29 | Yes Check date not recorded | Not published Check date not recorded |
| Portkey | claimedType II said to be available via an unnamed trust portal Checked date not recorded | Yes Checked 2026-08-29 | Not published Check date not recorded | Yes Check date not recorded | Not published Check date not recorded |
| Requesty | in progressType II expected Q3 2026 Checked date not recorded | Not published Check date not recorded | Yes Checked 2026-08-29 | Yes Check date not recorded | Published |
| Respan | claimedVendor states Type II; report under NDA, not independently examined. Checked 2026-09-15 | Not published Checked 2026-09-15 | Yes Checked 2026-09-15 | Not published Checked 2026-09-15 | Not published Check date not recorded |
| Together AI | conflictinga blog post reports a completed Type 2 examination; the trust center lists only Type 1 Checked date not recorded | Yes Checked 2026-08-29 | Not published Check date not recorded | Depends how you deploy it Checked 2026-09-05 | Not published Check date not recorded |
| TrueFoundry AI Gateway | Completed audit evidenceType II programme for TrueFoundry SaaS Checked date not recorded | Not published Check date not recorded | Not published Check date not recorded | Not published Check date not recorded | Published |
| Velokey | not publishedNo detail recorded Checked date not recorded | Not published Check date not recorded | Not published Check date not recorded | Depends how you deploy it Checked 2026-09-19 | Not published Check date not recorded |
| Vercel AI Gateway | Completed audit evidenceType 2, third-party audited Checked date not recorded | Yes Checked 2026-08-29 | Not published Check date not recorded | Yes Check date not recorded | Published |
The SOC 2 column uses the graded evidence and recorded detail, not the older capability flag. “Completed audit evidence” describes the source found; it does not certify your application or promise review approval. Check the linked profile for framework-specific sources and obtain the current report.
Where you stand before the review opens
Ready for a security review if
- The actual report covers the service, period and criteria your review requires.
- A subprocessor list is public and dated.
- The DPA names a transfer mechanism you can accept.
- The residency answer covers the model, not only the gateway.
Expect a longer review if
- The compliance claim is present but the evidence is not.
- Two vendor pages state different report types.
- No subprocessor list exists, so the chain cannot be enumerated.
- Retention is described without saying which party it binds.
Ask for in writing
- The report itself, or trust-portal access, with the period and scope.
- The signed BAA if health data will pass through.
- The subprocessor list and how changes are notified.
- The default retention window, and who holds the upstream key.
Build a service-specific review packet
Record the legal entity and product edition, report type (Type I or Type II), covered date or period, system boundary, relevant exceptions, customer responsibilities and any bridge letter. The AICPA describes SOC reports as evidence for assessing outsourced-service risks. A BAA instead establishes obligations for protected health information; consult HHS business associate contract guidance. These sources were reviewed September 16, 2026. Neither document alone establishes the compliance of an entire application.
For each gateway → model → logging destination, attach the applicable agreement, processing region, retention setting and subprocessor entry. Mark gaps as pending questions for the responsible reviewer, not as automatic approval or rejection.
Common questions
What is the difference between completed SOC 2 evidence and a claim?
The catalogue grade certified means evidence of a completed audit was found: a report available under NDA, a trust portal, or a named auditor and period. Claimed means the vendor asserts compliance and no such evidence was located. Both read as “SOC 2 compliant” on a marketing page, but the reviewer must obtain and assess the actual report, scope, period and exceptions.
A product has no compliance data at all. Should I rule it out?
No. An empty field records that nothing was found published, not that the answer is no, and several products with silent fields are widely deployed. What it does mean is that the evidence has to come from the vendor directly, on your timetable rather than theirs, so raise it at the start of the evaluation instead of at the end.
Why is “not applicable” a valid answer rather than a failure?
Because some products are software you run yourself with no vendor-operated service in the request path. Check whether any vendor-operated control plane, telemetry or support service receives data, so the compliance boundary is your own infrastructure and your own certifications. Marking that as a missing certificate would penalise the products that never asked for your data.
Can a gateway promise zero data retention on behalf of the model provider?
A gateway may arrange upstream retention commitments, but verify the contractual chain. Retention depends on model-provider terms, account credentials and enabled features as well as gateway storage. That is why “depends how you deploy it” is recorded as an answer rather than as evasion, and why the useful question is which party the commitment binds.
What should I ask for before the security review starts?
The SOC 2 report or trust-portal access rather than the badge, the subprocessor list, the DPA and its transfer mechanism, the retention default in writing, and confirmation of which region processes the request. Ask for all five at once, in writing. Each one that already exists in public shortens the review; each one that does not is a lead time you can measure now instead of discovering later.