Choosing a gateway for realtime voice agents

Choose a realtime voice gateway by testing the complete conversation: secure session creation, live audio transport, interruption handling, tool execution, reconnection and usage accounting. Support for speech generation or text streaming alone does not establish support for a bidirectional voice session.

For teams building interactive voice agents. Vercel’s voice integration and Google’s Gemini Live documentation are selected examples; endpoint and SDK behavior must be checked for the chosen model. Primary documentation reviewed 18 September 2026. Vendor capabilities below are documented, not independently benchmarked. The evaluation procedures are GatewayScore recommendations.

Separate three kinds of audio support

An application can assemble speech recognition, a text model and speech synthesis into a useful voice pipeline. A native live model is another architecture. Compare them using the same user task instead of assuming that one label means lower latency or better conversation. A gateway may support one architecture without supporting all its transports or model features.

Vercel’s June 2026 voice documentation describes realtime, speech generation and transcription, with beta AI SDK 7 interfaces. Its realtime example mints a short-lived credential on the server and gives that credential to the browser. Treat the documented SDK and model combination as the unit to evaluate, not a promise that every listed text model accepts live audio. Vercel realtime voice guide.

Authenticate session creation and the session itself

Keep the long-lived gateway credential on the server. The endpoint that mints a short-lived session credential should authenticate the user, enforce allowed models and apply your session policy before issuing it. Otherwise an attacker may use an unprotected minting endpoint even though the browser never sees the permanent key. This is a recommended application control, not an assertion that a documentation snippet implements your entire authorization design.

Write down token lifetime, permitted model, scope, session identity and revocation behavior. Test what happens when the token expires before connection and while a session is already established; those are different cases. Decide how concurrent sessions and long conversations consume a user’s budget. A connection count by itself is not a cost limit.

Treat interruption as a state transition

A user saying “stop” while audio is playing is more than a cancelled text completion. The app has incoming audio, an output buffer, conversation history and possibly a pending tool action. Specify what each component should do. Audio already heard and audio generated but discarded should not be treated as the same user experience.

Google’s Gemini Live best-practices documentation tells clients to discard buffered audio on an interruption signal and describes session resumption for reconnecting WebSockets. Those are provider-specific mechanisms; test how the gateway exposes them rather than assuming a transport reconnect preserves state. Gemini Live best practices.

For a tool that changes external state, separate “stop speaking” from “cancel the action.” If an order was already submitted, muting playback does not undo it. Use an application operation ID and an authoritative tool result to avoid repeating a write when a connection drops. Design a clear user response for an action whose outcome is still unknown.

Measure conversational delay at the user boundary

For an illustrative sequential path, define response delay from the user’s last speech sample to the first audible response. Suppose end-of-turn detection takes 300 ms, dispatch plus model startup 180 ms, initial audio generation 170 ms, and client buffering 100 ms. The total is 750 ms. These invented values explain the method; they are not measurements or a recommended service-level objective.

Real pipelines can overlap stages, so do not sum overlapping spans as independent delays. Retain a trace of the critical path. Report median and tail latency by client network, device and task type, and measure interruption-to-silence separately. A gateway’s HTTP time to first byte does not describe all of these outcomes.

Run a voice-specific acceptance suite

1. Complete a normal conversation and a tool-assisted task using the exact client, SDK, gateway and model versions. Record session and operation IDs.
2. Interrupt during speech, before a tool dispatch and after a tool completes. Confirm the visible/audible behavior matches the action state.
3. Disconnect during input and output, then reconnect. Verify what context is restored, what is lost and whether a pending write is duplicated.
4. Expire or revoke session authority and attempt another action. Record the actual enforcement point.
5. Run the intended maximum session duration and concurrency. Test quota exhaustion and provider errors with a clear user recovery path.
6. Reconcile complete and interrupted sessions to the returned usage categories. Keep unknown usage flagged and compare provider billing records when available.

Use a test account and harmless tool targets. For rollout acceptance, choose supported languages, noise conditions and accessibility needs that reflect the audience. A quiet desktop demo does not establish mobile reliability or speech quality in the environment your users occupy.

A practical selection rule

Shortlist a gateway only after it supports your exact live transport and model. Then compare identity controls, interruption fidelity, failure recovery, observability and total successful-conversation cost. If a gateway only records session creation while media or tools travel elsewhere, document that visibility boundary explicitly.

Read the observability guide for tracing, spending limits for long-running sessions, and prompt logging before deciding which audio, transcripts or metadata to retain. Compare deployment details in the Vercel AI Gateway profile.

Evidence and maintenance

Retest when voice SDK interfaces, model transport, session duration, resumption behavior or usage categories change. The worked latency budget is illustrative and must be replaced by measurements from the deployed path.

This guide uses the selected sources cited above, not a census or ranking of every gateway. Undocumented behavior remains unknown. Review the exact product edition, deployment and configuration before relying on a control.