LlamaIndex
Connect LlamaIndex to nRouter for retrieval-augmented generation (RAG) and agent workflows. Configure LLM and embedding endpoints with enterprise guardrails.
Connect LlamaIndex to nRouter (https://api.nrouter.ai/v1) to bring unified model access, real-time cost attribution, exact-match caching, and server-side guardrails to your RAG pipelines. Configuring Settings.llm and Settings.embed_model routes generation and embedding operations through nRouter with zero token markup.
For framework details, see the official LlamaIndex documentation.
Installation & Setup
Install LlamaIndex core, OpenAI extensions, and export your virtual API key:
pip install llama-index-core llama-index-llms-openai llama-index-embeddings-openai nrouter-sdk
export NROUTER_API_KEY="sk-nrouter-xxxxxxxxxxxx"Configure global defaults on Settings:
import os
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
Settings.llm = OpenAI(
model="claude-sonnet-4-5-20250929",
api_key=os.environ["NROUTER_API_KEY"],
api_base="https://api.nrouter.ai/v1",
timeout=60.0,
max_retries=2,
additional_kwargs={"nrouter_cache": True},
)
Settings.embed_model = OpenAIEmbedding(
model_name="text-embedding-3-small",
api_key=os.environ["NROUTER_API_KEY"],
api_base="https://api.nrouter.ai/v1",
)Document Indexing & Streaming RAG
Build an in-memory vector index and stream guarded query responses:
from llama_index.core import VectorStoreIndex, Document, Settings
documents = [
Document(text="nRouter routes inference across OpenAI, Anthropic, and Bedrock models."),
Document(text="Virtual keys enforce budget limits and rate limits per team or project."),
]
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(streaming=True)
response = query_engine.query("How do virtual keys protect spend?")
for text in response.response_gen:
print(text, end="", flush=True)
print()Routing & Observability
Configure model selection, caching, and error handling:
- Routing & fallbacks: Target specific models or use
nrouter/auto(see Router Settings). For per-call fallbacks, passnrouter_fallbacksinadditional_kwargs(see Per-Request Options). - FinOps telemetry: Headers include
x-nr-request-id,x-nr-model,x-nr-cost-status, andx-nr-request-cost. - Exact-match caching: Set
"nrouter_cache": Trueinadditional_kwargsto answer repeat queries directly from edge cache. - Error handling: Gateway errors map to HTTP status codes (see Errors & Rate Limits):
import openai
from llama_index.core import Settings
try:
response = Settings.llm.complete("Summarize corporate data.")
except openai.BadRequestError as e:
print(f"Blocked by guardrails or invalid request: {e}")
except openai.RateLimitError:
print("Rate limit reached. Backing off before retry.")Next Steps
- LangChain Integration — Multi-step chains and agent systems
- Python SDK Guide — Official nRouter Python SDK documentation
- Chat Completions API — HTTP endpoint specifications