Backend Engineer — LLM Infrastructure

You will work on the Rust gateway that sits between every customer request and every provider. That means guardrails, routing, cost attribution, rate limits, credit holds, and streaming. You will own the parts of the request lifecycle that have to stay correct under real production load.

What you'll do

• Extend the nRouter Rust gateway with new guardrails and features
• Own streaming, retries, and provider failover correctness
• Build and maintain cost attribution from x-nr-request-cost
• Profile and tune hot paths — every millisecond is in the user-facing latency budget
• Harden multi-tenancy isolation at the request layer

Required experience

• 5+ years of backend systems work in production, with Rust in production
• Deep experience with async Rust (Tokio) and high-concurrency services
• Comfortable with Postgres, connection pooling, and query optimization
• Production experience with streaming APIs or proxies

Nice to have

• Prior work on LLM APIs, model gateways, or SSE streaming
• Experience with LLM routing engines or model gateways
• Performance profiling and flame graph literacy

nRouter is an equal opportunity employer. We welcome builders from all backgrounds and non-traditional career paths.

If you need reasonable accommodations at any stage of our interview process, reach out directly to careers@nrouter.ai. Requests are kept confidential and have zero adverse impact on your application.