Operating production LLMs without granular observability turns generative workflows into opaque black boxes. Full-stack LLM observability binds edge request parameters, prompt token counts, completion latencies, guardrail inspection scores, and exact monetary costs into a unified distributed trace. High-cardinality telemetry empowers engineering teams to detect latency regressions, diagnose provider errors, and audit regulatory compliance.
Tracing requests across multiple microservices, proxy layers, guardrail sidecars, and third-party model providers often leaves fragmented log files with no common correlation identifier, complicating root-cause debugging.
Real-time token streaming generates massive log volumes. Systems must capture precise tokenomics and timing metrics without creating disk I/O bottlenecks or slowing down inference.
Indiscriminately storing full prompt and response payloads creates severe privacy risks for GDPR and HIPAA compliance, requiring granular data retention policies.
Calculating exact dollar spend in real time requires reconciling differing input, output, cached, and multimodal token rates across dozens of foundation model providers.
Request identifier binding edge WAF logs, the dashboard request trace, and immutable database spend rows.
High-precision financial audit records capturing exact token counts, provider rates, and funding sources.
Dashboard request logs with a per-request trace, exportable as CSV.
Every inference request routed through nRouter receives an immutable x-nr-request-id generated at the edge WAF. As requests traverse Preflight Phases 1 through 4 and stream from downstream providers, nRouter records token metrics, latency milestones, and exact costs into a 22-column spend row. Request metadata is always stored because it is the billing record; prompt and response text are never stored. Request logs are viewable in the dashboard and exportable as CSV, and every response carries x-nr-request-id, x-nr-request-cost, and x-nr-model headers.
Explore our observability guides to discover how to debug agentic loops, monitor p99 latency percentiles, and set up real-time spend anomaly alerts.
Send an nrouter/auto request with the Python SDK, read routing, guardrails and cost from its headers, fetch the request back by id, print every gateway step with its start, end and duration, then measure TTFT and latency over N runs.
nRouter Team
What a reused provider connection saves an LLM request, what it cannot save, and how to measure the gateway's share of your latency instead of trusting a headline number.
Inspect end-to-end inference traces, exact-cent spend rows, and multi-provider routing decisions under one request ID, with response headers on every call.
Follow a repeatable LLM gateway benchmark methodology for cost, latency, reliability, failover, and output quality before publishing measured results.
Understand where LLM streaming failover stops being safe, why partial output cannot be replayed silently, and how clients recover without duplicated text.
Request count and dollar cost tell different stories, and the gap between them is where the savings are. Here are the four shapes an overlay of cost and usage produces, which one to chase first, and what makes the numbers trustworthy enough to act on.
nRouter does not deliver request logs to Datadog or Langfuse yet. Here is what works today: response headers, request tags, CSV export and alert channels.
Learn how write-time redaction, typed placeholders, and metadata scrubbing keep LLM request logs fully debuggable without storing sensitive customer PII.
The fields that make an LLM request log worth keeping, the content that turns it into a liability, and how a gateway's built-in logs compare with running your own self-hosted trace store.
Why average LLM latency misleads, how to analyze p50/p95/p99 distributions and time-to-first-token, and how to isolate tail delays across model providers.
"The AI bill went up" becomes a query once spend carries structure. Three attribution layers — virtual keys, teams, and the OpenAI-spec user field — turn one opaque total into a breakdown you can group, filter, and cap.
Head-to-head: nRouter vs Langfuse. A self-hostable observability + prompt + evals specialist vs a hosted LLM gateway that bundles observability, routing, and governance — every feature included on every plan. A 0% platform fee on pay as you go. Models available in your live catalog behind one API key.
nRouter Team
Attribute LLM spend across multi-agent workflows by role, run, and step, reconcile usage against the credit ledger, and enforce hard gateway budget caps.
