RAG
Retrieval-augmented generation, one key for embed and chat.
Route embedding and chat synthesis calls through one nRouter endpoint, cache repeat queries, and track exact cost per pipeline stage.
The four things a RAG pipeline needs.
Two model families, repeat queries, stage-level cost attribution, and multi-provider failover.
Embeddings and chat, one key
The same endpoint serves the embeddings call that builds your index and the chat call that answers.
- /v1/embeddings and /v1/chat/completions with one virtual key
- Restrict a key to /embeddings when it should only index
Exact-match response caching
Gateway response caching serves byte-identical repeat queries without a new provider call.
- A hit is billed at the cache-read input rate, never free
- x-nr-response-cache reports hit, miss, or bypass on every response
- Send nrouter_cache: false for a fresh generation
Cost per pipeline stage
Tag indexing and query synthesis requests in metadata to attribute exact list-price cost per stage.
Explore observabilityFailover keeps retrieval answering
A 429, 503, or no first byte in time moves embedding or synthesis calls to your next fallback model.
Explore failover chainsPoint your client at one endpoint.
The synthesis call is a standard chat completion; embeddings use the same client and key.
pip install openai| 1 | # Cache: enabled (org default). Pass nrouter_cache: false to skip. |
| 2 | from openai import OpenAI |
| 3 | import os |
| 4 |
|
| 5 | client = OpenAI( |
| 6 | api_key=os.environ["NROUTER_API_KEY"], |
| 7 | base_url="https://api.nrouter.ai/v1", |
| 8 | ) |
| 9 |
|
| 10 | response = client.chat.completions.create( |
| 11 | model="gpt-5.4-mini", |
| 12 | temperature=1, |
| 13 | max_completion_tokens=1024, |
| 14 | messages=[ |
| 15 | {"role": "user", "content": "Hello! What models do you support?"}, |
| 16 | ], |
| 17 | extra_body={ |
| 18 | # "nrouter_cache": False, # Uncomment to skip cache |
| 19 | }, |
| 20 | ) |
| 21 |
|
| 22 | print(response.choices[0].message.content) |
FAQs
Does nRouter support embedding models alongside chat completions?
Yes. The same OpenAI-compatible gateway serves embeddings (/v1/embeddings) and chat synthesis (/v1/chat/completions). Your corpus indexing pipelines and query-time retrieval both authenticate with one virtual key, eliminating the need to configure separate embedding vendors.
How does caching lower RAG operational costs?
RAG workloads frequently repeat themselves: popular questions, re-asked queries, identical context windows. nRouter caches responses by exact match, so a byte-identical repeat is served from the cache instead of a new provider call, and settles at the cache-read input rate. Send nrouter_cache: false on a request when you need a fresh generation instead.
How do guardrails prevent indirect prompt injection in RAG contexts?
Retrieved third-party documents often contain adversarial instructions, untrusted links, or hidden jailbreaks. nRouter’s inline guardrails screen retrieved context before passing it to the synthesis model, detecting and blocking indirect prompt injections before the provider call.
Can I track and attribute cost for each stage of a RAG pipeline?
Yes. nRouter reports exact provider token usage and dollar spend for every call. By tagging indexing requests and query synthesis requests with stage metadata, you can analyze cost-per-search, embedding spend, and generation costs directly in the dashboard.
What happens if an embedding or chat provider goes down during production?
If your primary embedding or chat model returns 429 or 503, never opens a connection, or sends no first byte in time, the request moves to the next model in your fallback list, so your retrieval pipelines and user-facing answers keep working.