A/B Testing

Split live production traffic between model variants to compare output quality, latency, and cost across providers in real time with zero code changes.

nRouter A/B testing transparently splits live production traffic across model variants to benchmark latency, output quality, and real-world provider costs with zero client-side code changes. By defining variant weights and parameter overrides in the dashboard, the gateway intercepts requests, deterministically assigns traffic, records granular telemetry, and settles usage at the served model's exact list price.

Plane Separation & Credentials

Plane	Endpoint	Authentication	Protocol Rules
Control Plane (Management)	`https://app.nrouter.ai/api/ab-tests?orgId=<uuid>`	Dashboard Session Cookie (`-b`)	Requires browser session and `orgId`; mutations require JSON bodies. Rejects virtual keys.
Data Plane (Inference)	`https://api.nrouter.ai/v1/*`	Bearer Token (`Authorization: Bearer`)	Authenticates via virtual key (`sk-nrouter-*`). Rejects session cookies.

Evaluation & Routing Conditions

Variant selection intercepts buffered requests on /v1/chat/completions, /v1/messages, /v1/responses, and /v1/completions under five conditions:

Prompts Enabled: The organization Prompts toggle is active.
Active Status: Test is currently running.
Model Matching: Request matches the test's model filter.
Non-Streaming Only: stream: true requests return 400 because terminal stream telemetry cannot be captured mid-flight.
Traffic Allocation: Request falls within the test traffic_percentage (0.0 to 1.0); remaining traffic serves the base model.

Execution & Bucketing

Overrides: The gateway replaces body.model and applies config_override (temperature, top_p, max_tokens, max_completion_tokens, seed). Unknown keys return 400.
Refusals: Pre-call failures (auth, rates, credit holds, guardrails) incur $0 spend on blocked requests and do not increment counters.
Sticky Hashing: Traffic splits deterministically by hashing test_id and caller identity (virtual_key_id → user_id → organization_id).

Lifecycle & Configuration

Tests transition across: draft → running ↔ paused → completed. Non-draft configurations are immutable.

Create tests via POST https://app.nrouter.ai/api/ab-tests?orgId=<uuid>:

Field	Type	Required	Constraints
`name`	string	Yes	1–255 characters, unique within organization.
`variants`	array	Yes	Minimum 2 variants; weights must sum to `1.0`.
`traffic_percentage`	number	No	`> 0.0` and `≤ 1.0` (default: 1.0).
`model`	string	No	Intercept target; omit to intercept all models.

Query metrics via GET /api/ab-tests?orgId=<uuid>&testId=<uuid>.

Endpoint Reference

Operation	Method & Path	Access Control
List / Inspect Tests	`GET /api/ab-tests[?testId=<uuid>]`	All Members
Create / Update / Delete	`POST` or `DELETE /api/ab-tests[?action=...]`	Owner / `org_admin`

Visual management is in the dashboard under Prompts → A/B Tests.

Frequently asked questions

How is traffic split between variants?

Traffic splits deterministically by hashing the test ID and virtual key ID, pinning callers to the same variant.

Which variant's cost am I billed for?

You are billed for the exact model that served the response; keep variant costs aligned with requested models to prevent credit under-reservation.

Next Steps

Playground — Verify variant interception before scaling traffic.
Budget Controls — Set per-key spend caps during testing.
Chat Completions — Core inference endpoint.