Voice & Realtime
Voice assistants that answer fast, every turn.
Stream each conversational turn to your speech layer, keep the line alive with failover, and log every turn for observability.
Speed, resilience, and a turn-level trail.
Every conversational turn gets streaming, failover, and request logging.
Streaming turns
With stream: true the gateway opens the connection at once and sends tokens as server-sent events.
- Your TTS layer can start speaking before the answer is finished
- x-nr-latency-ms on a stream is time to first byte
Mid-conversation failover
If a provider rate-limits or stalls during a live call, that turn moves to your backup model.
- Triggers: 429, 503, a connection that never opened, no first byte in time
- Once response bytes arrive, the turn is not replayed elsewhere
Turn-level observability
Every conversational turn logs end-to-end latency, the model that answered, token counts, and exact cost.
Explore observabilityRealtime model catalog
Pick chat models for the reasoning turn and speech models for synthesis, all behind one key.
Explore realtime modelsThe reasoning turn is a chat call.
Use the same client for each turn; add stream: true to receive tokens as server-sent events.
pip install openai| 1 | # Cache: enabled (org default). Pass nrouter_cache: false to skip. |
| 2 | from openai import OpenAI |
| 3 | import os |
| 4 |
|
| 5 | client = OpenAI( |
| 6 | api_key=os.environ["NROUTER_API_KEY"], |
| 7 | base_url="https://api.nrouter.ai/v1", |
| 8 | ) |
| 9 |
|
| 10 | response = client.chat.completions.create( |
| 11 | model="gpt-5.4-mini", |
| 12 | temperature=1, |
| 13 | max_completion_tokens=1024, |
| 14 | messages=[ |
| 15 | {"role": "user", "content": "Hello! What models do you support?"}, |
| 16 | ], |
| 17 | extra_body={ |
| 18 | # "nrouter_cache": False, # Uncomment to skip cache |
| 19 | }, |
| 20 | ) |
| 21 |
|
| 22 | print(response.choices[0].message.content) |
FAQs
How does nRouter keep latency low for voice applications?
Routing decisions happen in-memory before dispatch, so LLM inference time dominates. With stream: true the gateway opens the connection immediately and sends tokens as server-sent events, so your TTS layer can start speaking before the answer is finished.
What happens to a live voice call if a provider degrades or rate-limits?
Failover is per conversational turn. If the primary foundation model rate-limits (429), is overloaded (503), fails to connect, or does not start responding in time, that turn retries on the backup provider in your routing chain instead of erroring, so the turn fails over to your next model when a provider is rate limited or unavailable.
Can inline guardrails operate without inducing audio stutter or jitter?
Yes. Guardrails evaluate the incoming transcript inline, before the request is dispatched — a blocked prompt never reaches the model, and a passed prompt streams back token by token with no extra buffering stage in between.
Can I monitor turn-level latency for every conversational turn?
Yes. Every turn records end-to-end latency, the model that answered, token counts, and exact cost in the request log, so your voice engineering team can find slow turns and the provider behind them.
Does nRouter handle speech-to-text and text-to-speech directly?
nRouter provides the LLM intelligence layer between your STT (e.g. Whisper, Deepgram) and TTS (e.g. ElevenLabs, Cartesia) services. By focusing on LLM routing, budget enforcement, and failover, nRouter keeps live conversational pipelines resilient.