How LLM gateway guardrails fail

Fail open, fail closed, or configurable

For context, 26 of 31 catalogue rows lack a recorded failure policy across all products. That denominator includes products without recorded guardrail evidence; it is different from the guardrail subset in the answer above. Neither count estimates the whole market.

A guardrail is an extra service in the request path. It has to decide something before the prompt goes to the model, so it can be slow, it can error, and it can be unreachable. Whoever built the gateway had to pick what happens then, and the two answers protect opposite things.

Fail open — the request continues

The checker errors or times out, the prompt goes to the model unchecked, and your users notice nothing. Availability is preserved and the control is silently absent for the duration. This is the sensible default for a low-risk endpoint and the dangerous one for a regulated workload, because there is no failure a user can report.

Fail closed — the request is rejected

The control is preserved and the feature is down. Every part of your product that calls a model through the gateway starts returning errors when one classifier has a bad afternoon. Correct for anything you would have to disclose, and expensive everywhere else.

Configurable — 3 document the choice

The most useful answer, and the one to ask for. What matters is the granularity: per deployment is a compromise, per route or per policy lets a support summariser fail open while a claims workflow fails closed. Check the shipped default too, because a configurable setting still arrives set to something.

Not documented — 26 of 31

Recorded as unpublished rather than as a no. Several of these products almost certainly do have a defined behaviour; it has simply never been written down where a buyer can read it. What it means for you is that the answer will come from a salesperson under a deadline, or from an incident.

The silence is not explained by products that ship no checks at all. 24 of the 31 products here offer guardrails of some kind, and 19 of those 24 still publish nothing about failure behaviour. The remaining 7 have no positive guardrail evidence in these two fields; that does not settle whether a feature exists for them.

The 5 products that do state a behaviour are worth reading closely, because the note is more specific than the value: Cloudflare AI Gateway, Higress, LiteLLM, Merge Gateway and Orq.ai Router. Each of their pages carries the vendor’s own wording, with the timeout, the setting name and the exception where one exists. If you are shortlisting on this question, start there rather than with a feature matrix.

Where the check actually runs

The failure question has a companion that people usually ask second and should ask first: whose computer is doing the checking. A filter that strips personal data helps only if it runs before the data leaves your boundary, and the deployment shape of the proxy does not settle it — a self-hosted router can still call a hosted classifier.

10 Vendor’s cloud

Checks are offered only as part of the hosted service. The prompt must leave your boundary in order to be inspected for having left your boundary.

7 Your infrastructure

Checks run where you run the gateway. This is the group to look at if the point of the control is that data does not travel.

6 Either

Both shapes are documented, so the answer depends on how you deploy it. Worth confirming which one your chosen shape gives you.

7 No checks offered

No guardrail feature recorded. Not a criticism — a fast routing layer that does one job is a legitimate product, and the control belongs elsewhere.

Where evaluation happens is recorded separately, and the split is different. Of the 24 products with checks, 10 document that evaluation can run either side, 11 that it runs on the vendor’s servers, and 3 publish nothing. Not one documents evaluation that runs only inside your own infrastructure, which is the shape a reader worried about data movement is usually hoping for.

Say plainly what that means for a deployment. If evaluation is on the vendor’s side, the failure mode you care about is a network failure between two third parties, and the data-protection assessment has an extra processor in it. That is a legitimate architecture and a common one. It is simply not the architecture most people picture when they read that a gateway can redact personal data, and it is worth writing down before the security review does it for you.

What is switched on before anyone configures it

A guardrail that exists and a guardrail that is running are different facts, and the catalogue records them in different fields. For each check type there is a value for what the product can do and a value for what a fresh deployment does. The second describes the recorded default, not necessarily your configured production traffic.

Personal data in prompts

5 block out of the box, 8 ask you to choose the action during setup, 3 log without blocking until you change it, 1 ship switched off, and 7 do not publish the shipped state. The check most often bought for a compliance obligation, and the one whose shipped state matters most.

Prompt injection and jailbreaks

7 block out of the box, 7 ask you to choose the action during setup, 1 log without blocking until you change it, 1 ship switched off, and 8 do not publish the shipped state. Almost always a partner classifier, which means an extra network call in the request path.

Harmful content

8 block out of the box, 5 ask you to choose the action during setup, 1 log without blocking until you change it, 1 ship switched off, and 9 do not publish the shipped state. The category vendors are most comfortable blocking by default, because a false positive is cheap.

Your own policies

0 block out of the box, 14 ask you to choose the action during setup, 2 log without blocking until you change it, 0 ship switched off, and 8 do not publish the shipped state. A rule you wrote has no sensible default, so the vendor rightly declines to pick one for you.

Blocking and observing read alike on a feature list and behave nothing alike in production. A check that blocks rejects the request. A check in observe mode inspects the prompt, records what it found, and passes it to the model anyway — it is observing that check rather than blocking on it; other controls may still protect the request. 4 products ship at least one check in that state: Azure AI Foundry, Higress, Portkey and Requesty. That is a reasonable place to start a rollout, and a bad place to still be six months later, so the practical action is to check who reads the log.

Among the recorded configuration states: 15 of the 24 products with checks leave at least one action to be chosen when you configure it, and 3 ship at least one check switched off. No product here ships every check blocking by default. These counts do not prove that every unconfigured deployment lacks protection. They show why each check and its action must be inspected. The gap is measurable: 15 products document a personal-data check that can block a request, but only 5 of them block by default, and for prompt injection it is 7 of 15.

Whose engine is actually deciding

Most gateways do not write their own classifiers. They call someone else’s, and the catalogue records which: 12 of the 31 products name at least one bought-in engine, while 19 name none. Across the catalogue 23 distinct engines appear, the most frequently integrated being AWS Bedrock Guardrails (5), Azure Content Safety (5) and Microsoft Presidio (5). The longest list belongs to Portkey, which names 12.

This matters for the failure question specifically. A product that routes to a third-party engine inherits that engine’s timeouts, rate limits and outages, not its own, so “what does the gateway do when the guardrail fails” becomes a pair of questions: what the engine does, and what the gateway does when the engine does it. 9 of the 12 products that name an engine publish nothing about the second half. A long integration list is a strength — it means you can bring a policy engine you already trust — but it moves the control’s reliability into a supplier you have not evaluated.

A pair of adjacent fields is easy to mistake for an answer here. 17 products document that you can restrict which models a key may reach, and 16 document personal-data redaction as a capability. Both are useful; neither says what happens when the check breaks. For what the vendor is certified to do, see the compliance guide; for where the prompt is stored afterwards, see the prompt-logging guide.

Test the checker failing, not only the prompt failing

Proposed staging test, not a reported benchmark: record the gateway version, edition and check configuration. With synthetic content, compare a normal allow, a normal block, a checker timeout and a checker error. Record whether the upstream model received the request, what the caller saw and whether an alert was emitted. Repeat for input and output checks and streaming; an output block cannot retract tokens already delivered. Set an explicit timeout and failure policy for each check.

The 24 products with checks, side by side

Every product in the catalogue that documents a guardrail of any kind, with where evaluation runs, what the two most-requested checks can do, and what happens when the check itself fails. Each value links to the vendor page it was read from.

A blank reads as not published rather than as no. On this table that distinction is the whole point: a product with no documented failure mode may well fail closed, and a reader who treats the gap as a negative would be wrong about it. Treat it as the question to put to the vendor in writing.

Common questions