Guide
Which LLM gateways store your prompts?
Short answer
Some LLM gateways record prompt and completion bodies by default; others store metadata, require opt-in or leave storage to your deployment. Check the exact plan and settings for logs and trace exports separately. Disabling gateway storage does not stop transmission to the gateway or model provider, establish a training policy, or delete copies already exported elsewhere.
Follow every copy of the request
Primary path: application → gateway → model provider → gateway → application. Possible copies: application logs; gateway request logs or caches; model-provider records; exported traces and log sinks; backups and support diagnostics.
For each destination, record what is transmitted, what is stored, who can access it, retention and deletion behavior, and whether data may be used for training. These are separate controls. Capture the product version, edition and account settings; defaults in a catalogue are not proof of your deployed configuration.
Use a synthetic canary with no sensitive data to inspect accessible dashboards, exports and collectors before and after a logging change. Finding it proves a copy exists there. Not finding it does not prove absence from internal logs, backups or delayed exports. Combine the test with configuration review, deletion checks and the applicable contract. No canary test results are claimed here.
What ships if you change nothing
A gatewayGateway: A single endpoint you send all your AI requests to, which then forwards them to whichever model you asked for. One integration instead of one per vendor. sits in the request path of everything your product does with a model, which makes its logging default the single most consequential setting nobody chooses. The catalogue records that default in four states, and they sum to the whole catalogue with no product declining to answer.
- 11 Prompts and replies
- Bodies are stored unless somebody turns it off. By the time the first person reads the documentation, the text is already in the vendor’s account.
- 12 Metadata only
- Token counts, latency, model, cost and the calling key, with no prompt text. The group where content logging is a feature you switch on rather than off.
- 6 Nothing by default
- No content retention as a documented rule. Read each note anyway: several keep nothing as policy while making a timed exception for abuse review.
- 2 You own the logs
- Recorded as not applicable, which is the honest value for a binary you run yourself: it writes to a sink you chose, so there is no vendor default to report and no vendor promise to obtain.
A second field records what is stored rather than whether anything is, and its modal answer is the reason the default matters at all. 14 of 31 make content logging configurable, 4 store full request and response, 11 store metadata only and 2 store nothing. Configurable is not a reassurance. It means the position you end up in is whichever one the product shipped with, until an engineer changes it deliberately — so the thing to check before you ship is not whether the switch exists but which way it is currently pointing in your account.
The combination worth acting on is the intersection: the products that store bodies by default and document no supported way to keep the metrics while dropping the text. Those are Requesty and Respan. 7 of 31 publish nothing about an opt-out at all. That silence is recorded as unpublished rather than as a refusal, because a header may well exist and never have been written down. Treat it as the first question for the vendor, not as an answer.
“Not stored” is not “not sent”
Suppressing storage and suppressing transmission are two different controls, and one does not imply the other. Reading the logging notes across the catalogue, 2 products state plainly that their mechanism stops storage while the request and response still travel to the vendor: Helicone and Orq.ai Router.
This is worth saying without any suggestion of bad faith. Documenting the limit of your own feature is the opposite of hiding it, and both of these vendors did. The point is that the control answers an operational question — is there a copy of the text anywhere — and a reader with a regulatory concern is asking a different one. For a data residencyData residency: A guarantee about which countries your data is processed in. Usually matters for EU obligations or public-sector contracts. assessment, where the bytes travelled matters as much as where they came to rest, because the transfer itself is the event the rules describe. A processing agreement covers a transfer that happened whether or not anything was written down at the far end.
So separate the two questions when you ask them. Does the body leave my network. Does it reach your infrastructure. Is it written to storage there. How long does it stay. Four answers, and only the last two are what most vendors mean by a logging setting. Where the answer to the second question has to be no, the control you need is a product that never receives the text — which is a deployment decision rather than a configuration one, and is covered in the self-hosting guide.
Where each product stands
All 31 products, sorted by name. The first column is the default, the second is what the default stores, the third is whether the vendor documents turning it off, and the fourth is the retention position a reviewer will ask about. Every value links to the vendor page it was read from.
A blank reads as not published rather than as no. A vendor who has not documented an opt-out may still have one; a vendor who has not documented a retention window may still hold nothing. Collapsing silence into a negative would flatter the products that publish least, and collapsing it into a positive would mislead the reader who has to sign something.
How long it stays, and why the number improves as you pay
Retention is recorded twice: as a number of days, and as the vendor’s own prose. 6 products state a default of nothing kept, 9 state a window in days, and 16 of 31 have no number at all — mostly because retention is a property of the sink you chose rather than of the product. 7 publish no retention prose worth quoting.
4 products tie the retention window to the plan you are on: Helicone, LLM Gateway, Portkey and Respan. The ladder runs the direction buyers do not expect. The cheapest tier keeps your prompts for the shortest time and the most expensive keeps them longest, up to indefinitely, because retention is being sold as a debugging and audit feature rather than rationed as a liability. If your reason for caring about retention is compliance rather than convenience, a longer default may conflict with your retention policy and you should read the enterprise tier’s window before you sign for it.
The other shape to watch for is a minimum rather than a maximum. OpenRouter records a default of nothing kept, and a stated floor on how long the logs are held once you do opt in. Both facts are true and only the first one appears in a numeric column, which is a good reason to read the note beside any zero.
Two fields describing one thing will sometimes disagree, and where they do this site reports both rather than choosing. On 4 products the number in the retention field appears nowhere in the prose that describes the same setting: Braintrust Gateway, Kong AI Gateway, Requesty and TrueFoundry AI Gateway. For most of them the prose simply declines to state a window, so the number is the only figure available and should be treated as provisional.
Two controls that look like this one and are not
Zero data retentionZDR — zero data retention: A commitment that your prompts and responses are not stored after the request completes. Frequently a paid add-on or an enterprise-only option rather than the default. is the phrase a reviewer will bring to the conversation, and it answers a narrower question than a logging default does. 13 products document a zero-retention position, 6 record that it depends on how you deploy them — the honest answer for a router, since what happens to the forwarded request is the model provider’s policy and not the gateway’s to promise — 7 record it as not applicable for the recorded deployment scope, and 5 publish nothing. The compliance guide states each product’s position as a reviewer needs it stated, and this page does not repeat it.
Redaction is the other near-neighbour. Stripping identifiers before storage is a different control from not storing, and it is the one to reach for when you need the logs and cannot hold personal data in them. 16 products document personal-data redaction as a capability and 2 document that they do not have it, while 13 of 31 say nothing. 15 run a blocking personal-data guardrail in the request path and 2 run one that observes without blocking. The two fields do not describe the same set: 19 products appear in one or the other, which is more than either field marks on its own, so check which of the two a product means before you rely on it. What a guardrail does when it trips, and what happens to the request while it decides, is the subject of the guardrails guide.
Which side you are on
Default logging is fine if
- Prompts contain no personal or customer data, and you can show that.
- Full bodies in a dashboard are how your team debugs a bad completion.
- Your processing agreement already covers this vendor as a processor.
- Retention on the vendor’s default window is shorter than your own policy.
Turn it off before you ship if
- Anything a user types could be personal, health or payment data.
- A regulator or customer contract limits where that text may be processed.
- You cannot say who inside the vendor can open a log entry.
- Retention is tied to your plan and nobody has read the enterprise window.
Verify it yourself
- Send a canary string that appears nowhere else in your systems.
- Look for it in the dashboard, then in a data export, then in the log sink.
- Repeat with storage disabled and inspect whether new entries are absent or truncated; separately check deletion of old copies.
- Watch the request leave: check whether disabling storage also stopped transmission.
Common questions
Does an opt-out mean my prompts never reach the vendor?
Not necessarily, and this is the most common misreading on the subject. Some opt-out mechanisms are a header on a request that has already been sent: the body arrives at the vendor, is processed, and is not written to storage. That is a genuine control and it is not the same control as never transmitting the text. If your concern is regulatory rather than operational, ask which of the two the vendor is offering, and ask for the answer in writing.
Why is “not applicable” recorded as a logging default for some products?
Because it is the honest answer for a binary you run yourself. A self-hosted gateway writes to a sink you configured, on infrastructure you own, so there is no vendor-side default to report and nothing for a vendor to promise about. It is not evasion and it is not a null. It does mean the question moves rather than disappears: you now own the retention decision, and nothing prunes a table you never set a policy on.
Is metadata-only logging safe to leave on?
Inspect the actual fields before deciding. Model name, token counts, latency, cost, status codes and the identity of the calling key are typical, and none of them contain prompt text. They can still be sensitive in aggregate — which internal team is calling which model, how often, and at what volume is commercially meaningful information. Treat it as telemetry with an access-control question attached rather than as nothing.
How do I actually verify what a gateway stores?
Send a request containing a distinctive string that appears nowhere else in your systems, then look for it in the vendor’s dashboard, in an export, and in whatever log sink the product ships to. Repeat the test with storage disabled. Finding the canary proves storage in that location; absence does not prove there are no other copies. Record plan, version and settings, and review internal retention and backup terms separately. To find out what was transmitted, watch the request leave — the presence of a full body on the wire with an empty log entry at the far end is precisely the case worth knowing about.
Does redaction remove the need to turn logging off?
They solve different halves of the problem. Redaction detects and strips identifiers before the text is stored or forwarded, which helps when you need the logs for debugging and cannot hold personal data in them. Disabling storage removes the copy entirely. Redaction depends on a detector being right about what it is looking at, so it reduces exposure rather than eliminating it, and the two controls are frequently worth having together.