Response Cache: Repeat the request. Reuse the answer.
LLM caching reduces API costs by reusing a saved response for a matching request instead of generating it again — a reused answer costs no new inference. In nRouter it applies to eligible non-streaming text requests, stays inside your organization, and can be expired or bypassed per request.
Watch the same question twice.
How response caching works. Exact match within your organization.
Reuse matching answers.
Eligible repeats can reuse a saved answer on non-streaming text calls when model, prompt, and settings match.
- Covers /v1/chat/completions, /v1/messages, /v1/responses and /v1/completions
- Matches identical model, messages, tools and generation settings
- Responses up to 256 KiB that pass post-call guardrails can be stored
Keep responses within your organization.
Logical access isolation scopes saved responses to your organization, team permissions, and active guardrails.
- Scoped to your organization, team model permissions and guardrail policy
- One organization cannot match or read another organization’s saved responses
- Re-evaluates active post-call guardrails on every served cache hit
Expire saved answers.
Response bodies are stored briefly and expire automatically so saved answers do not linger indefinitely.
- Saved response bodies expire after 300 seconds by default (900 seconds max)
- Reports x-nr-response-cache: hit and x-nr-response-cache-age in seconds
- If the cache store times out after 200 ms, lookups fail open to the model
Choose when to bypass.
On by default. Turn off for your organization or bypass individual requests when freshness matters.
- Pass "nrouter_cache": false in the request body to skip a single call
- Streaming requests ("stream": true) and non-text routes always bypass
- Turn off Response Caching for your organization in Router Settings
Gateway response cache vs provider prompt cache.
| Capability | Gateway response cacheReuses a saved response in the gateway when an eligible request repeats. | Provider prompt cacheReuses eligible prompt prefixes upstream while generating a new answer. |
|---|---|---|
| What is reused | The complete saved response for an eligible matching request within your organization | Eligible shared prompt prefixes on the upstream provider, even when the user question changes |
| Whether generation runs | 0 new model generations on a hit; returns the saved answer directly | Runs a new model generation upstream on every request |
| What is billed | Model cache-read input rate with no output charge when published, or fallback to the stored original total cost (0% platform fee) | Provider prompt-cache rates on eligible prefix tokens plus generated output tokens; eligibility and pricing vary by provider and model |
| Streaming support | Non-streaming text routes only (/v1/chat/completions, /v1/messages, /v1/responses, /v1/completions); streaming requests bypass | Works on both non-streaming and streaming requests where supported by the provider and model |
Take a closer look. How the SDK works.
- 01
Install
One package for your language. Python and TypeScript shown; every official SDK works the same way.
pip install nrouter-sdknpm install @nrouter_ai/sdk - 02
Create a client
nRouter() reads NROUTER_API_KEY from your environment and sends each call to api.nrouter.ai. Provider keys are ours to manage.
client = nRouter() - 03
Call any model
The OpenAI-compatible calls you already know. nRouter’s options ride on the same request; each topic below shows one.
client.chat.completions.create(…)
Already on the OpenAI SDK? Point its base URL at https://api.nrouter.ai/v1 and keep your code.
Eligible non-streaming text responses report x-nr-response-cache as hit, miss, or bypass. On a hit, the gateway returns the saved response body, adds x-nr-response-cache-age in seconds, records the charge in x-nr-request-cost, and re-evaluates active post-call guardrails before returning.
# Example request |
POST https://api.nrouter.ai/v1/chat/completions HTTP/1.1 |
Host: api.nrouter.ai |
Authorization: Bearer $NROUTER_API_KEY |
Content-Type: application/json |
|
{ |
"model": "gpt-6-luna", |
"messages": [{ "role": "user", "content": "Ping" }], |
"temperature": 0 |
} |
FAQs
What requests are eligible for response caching?
Response caching is on by default for non-streaming text requests on /v1/chat/completions, /v1/messages, /v1/responses, and /v1/completions when the serialized JSON response body is 256 KiB or smaller and passes post-call guardrails. Streaming requests ("stream": true), responses larger than 256 KiB, and embeddings, audio, image, and video routes always bypass the cache.
How does the gateway decide whether two requests match?
The gateway matches only identical requests within your organization—never semantic or vector similarity. A request matches when your organization, team permissions, requested and resolved model, messages, system prompt, tool definitions, generation settings (such as temperature, top_p, max_tokens, seed, and response_format), active guardrail policy, and web-search grounding settings are the same.
When should I bypass the cache, and how do I turn it off?
A response-cache hit returns an earlier saved answer rather than running a fresh model generation, so bypass the cache whenever a request requires live data, fresh sampling variation, or a unique one-time output. Pass "nrouter_cache": false in the JSON request body (or extra_body in OpenAI SDKs) to bypass a single request, use streaming ("stream": true), or turn off Response Caching for your organization in Router Settings.
How are response-cache hits billed?
When the served model publishes a cache-read input rate for that service tier, a hit bills input tokens at the model’s cache-read rate with no output token charge. When no cache-read rate is published for that model or tier, billing falls back to the stored original total cost of the response, which can include output tokens. Every served hit is subject to a $0.000001 minimum charge floor with a 0% platform fee.
How long are saved responses retained, and what happens when I turn caching off?
Saved response bodies expire after 300 seconds (5 minutes) by default and cannot exceed 900 seconds (15 minutes). When an Owner or Organization Admin turns off Response Caching in Router Settings, gateway policy refreshes within a few seconds to stop serving cached responses and triggers an asynchronous purge of your organization’s stored entries. Policy refresh and background purge run as separate steps, and a purge can fail if the cache store is unreachable, so turning off the setting is not an immediate deletion guarantee.
How are cached responses isolated between organizations?
Cached responses use logical access isolation scoped to your organization, team model permissions, and active guardrail settings, so one organization’s requests cannot match or read another organization’s saved responses. Replayed hits also re-evaluate active post-call guardrails before returning.
What happens if the cache store is slow or concurrent identical requests arrive?
The response cache fails open: if a cache lookup times out after 200 ms or the cache store is unavailable, the gateway treats the request as a miss and calls the model provider so traffic is not blocked. When concurrent misses arrive for the same request, the gateway attempts to coalesce them behind an in-flight request for up to 60 seconds, though a waiting request can still fall through to a normal provider call if the first attempt does not store a reusable response.
- Response caching is on by default for eligible non-streaming text requests on /v1/chat/completions, /v1/messages, /v1/responses, and /v1/completions with JSON response bodies up to 256 KiB that pass post-call guardrails. Streaming requests (stream: true) and embeddings, audio, image, and video routes always bypass the cache.
- When the served model publishes a cache-read input rate for the request lane, a response-cache hit bills input tokens at that rate with no output token charge. When no cache-read rate exists, billing falls back to the stored original total cost of the response, which can include output tokens. Every served hit is subject to a $0.000001 minimum charge floor with a 0% platform fee.
- Saved response bodies are retained for 300 seconds by default (up to 900 seconds maximum). Pass "nrouter_cache": false in the JSON request body to bypass per request, or turn off Response Caching for your organization in Router Settings. Organization policy changes refresh within seconds and trigger an asynchronous purge of stored responses, though policy refresh and purge run separately and purge can fail. If the cache store times out (200 ms) or is unavailable, lookups fail open and call the model provider.