Warm Connection Pools in an LLM Gateway, and What They Do Not Fix

The short answer: a fresh HTTPS connection costs at least two network round trips — one for the TCP handshake and one for the TLS 1.3 handshake — before the first byte of a prompt is sent. A gateway that keeps provider connections open and reuses them removes that cost from every request after the first. It does not make the gateway's own work free: authenticating the key, rate limits, guardrails and the credit hold all take measurable time, which you can read back per request instead of taking anyone's word for it.

Adding an intermediary between your application and the model provider looks like a latency tax, and part of it is one. The useful question is not "is there overhead" but "which overhead, and how big, for my traffic". This post separates the cost a warm connection pool genuinely removes from the cost it cannot touch, then shows the numbers from one real production request so the split is concrete.

The problem in one request

Consider a serverless function that receives a user prompt, builds a new SDK client, and calls a model provider. Before the provider sees a single prompt token, the client does this:

Client (cold)                                   Provider edge
   │                                                 │
   ├── 1. DNS lookup for the provider host ─────────>│  (resolver round trip, unless cached)
   ├── 2. TCP handshake: SYN, SYN-ACK, ACK ─────────>│  1 round trip
   ├── 3. TLS 1.3 handshake ────────────────────────>│  1 round trip
   ├── 4. HTTP request headers + prompt body ───────>│  1+ round trips for a large body
   │                                                 │
   └── first byte of the answer comes back after the model starts generating

Steps 2 and 3 are fixed by the protocols: TCP needs a three-way handshake (RFC 9293), and a full TLS 1.3 handshake needs one more round trip before application data (RFC 8446). How many milliseconds that is depends entirely on the distance between your client and the provider: a client in the same region pays little, one on another continent pays a lot.

Step 4 has a second, quieter cost. A new TCP connection starts with a small congestion window — the standard initial window is ten segments (RFC 6928) — and grows it only as acknowledgements arrive (RFC 5681). A prompt carrying a large retrieved context can therefore need several round trips just to finish sending on a fresh connection, where a warmed one sends it at once.

If the next user turn builds another client, it pays the whole sequence again.

Why the naive approach breaks

The usual first fix is client-side keep-alive. It helps, and it stops helping in three predictable ways:

The first row is the structural one. An HTTP/1.1 connection carries one request at a time (RFC 9112); a streamed completion holds it for as long as tokens keep coming. Twenty concurrent chats need twenty connections, and a twenty-first request either waits or opens a new one. HTTP/2 interleaves many streams on one connection (RFC 9113), up to the concurrency limit the server advertises, which is why it matters for streaming workloads.

Client-side pools also cannot coordinate anything across processes: two containers cannot know they are jointly about to exceed a rate limit or a budget until the provider, or the invoice, tells them.

The mechanism

nRouter sends every provider call through one shared, long-lived HTTP client per gateway instance rather than a client per request. In customer terms, that means:

1. Connections are reused. After a provider connection is opened, it stays in the pool and later requests to the same provider reuse it, skipping the TCP and TLS handshakes. Where the provider negotiates HTTP/2, concurrent requests share connections as multiplexed streams, and the flow-control window adapts to the link.
2. The pool is bounded. The number of idle connections kept per provider host is capped, and idle connections are retired after a timeout, so a burst cannot leave the gateway holding an unbounded pile of sockets it needs for everything else.
3. Dead connections are detected. TCP keepalive probes catch a connection that a NAT or load balancer dropped without telling anyone, so a request does not pick up a socket that will never answer. Small writes are sent immediately rather than batched.
4. Every wait has a deadline. Connecting and reading from a provider are both bounded, so a provider that accepts a connection and then says nothing cannot hold a request, or its credit hold, indefinitely.

Before the provider call, the gateway runs its checks in a fixed order. They are real work against shared state, not free:

Your app ──> nRouter gateway ───────────────────────────────────────> Provider
              │ key_auth            authenticate the virtual key
              │ prompt_apply        apply your organization's prompt template
              │ rate_limit          your RPM and TPM windows
              │ guardrails_scoring  pre-call content inspection
              │ concurrency_permit  your concurrency slot
              │ auto_decision       only for nrouter/auto: choose the model
              │ reserve             the credit hold for this request
              │ compression         prompt compression decision
              │ provider ──────────────── reused connection ────────────>│
              │ attempt_record      record the provider attempt
              │ post_call           post-call guardrails on the response

The step names are the ones each request's trace records. Tokens are relayed to you as they arrive, subject to the post-call guardrail chain, and the cost is settled after the response finishes — the accounting behind that is in Reserve-and-Settle: Never Overspend a Credit Balance.

Worked example

Here is one real production request, measured on 2026-10-15: a single nrouter/auto chat completion with the response cache off, read back with its own key through the per-step trace. It is one request, not a benchmark, and not a promise about yours.

The arithmetic: 1,053 + 4,465 + 113 ≈ 5,631 ms. Three things follow from it.

1. The provider was most of the wait — about 79% of this request. Connection reuse shortens the time to reach the provider; it does nothing about generation time.
2. The gateway's own work was about a second here, not microseconds. The biggest single step was auto_decision, which runs only when you ask nrouter/auto to choose the model; a request that names a model directly skips it.
3. Two gaps totalling about 106 ms belong to no declared step. A trace that shows its own gaps is more useful than one that quietly folds them into a neighbour, and those gaps are where we look next.

To measure your own share, compare the gateway's figure with your client's wall clock on a buffered call:

import time
from nroutersdk import nRouter

with nRouter() as client:                       # reads NROUTER_API_KEY
    t0 = time.perf_counter()
    client.nrouter.chat(
        messages=[{"role": "user", "content": "One sentence on TCP slow start."}],
        model="nrouter/auto", cache=False, max_tokens=64,
    )
    wall_ms = (time.perf_counter() - t0) * 1000
    meta = client.last_response
    print("request id      ", meta.request_id)
    print("served by       ", meta.model, "| routing", meta.routing, "| attempts", meta.attempts)
    print("gateway (ms)    ", meta.latency_ms, "  arrival to response headers")
    print("client wall (ms)", round(wall_ms, 1))
    if meta.latency_ms is not None:
        print("outside gateway ", round(wall_ms - meta.latency_ms, 1), "ms: your network and client")

For the per-step breakdown, TTFT on streams, and percentiles over many runs, follow Measure TTFT and Trace an LLM Request Step by Step in Python.

Edge cases we had to decide

1. When a provider accepts the connection and then says nothing, we stop waiting, because a silent socket would otherwise hold your credit hold forever. Connect and read deadlines bound every provider call. A request that fails before the provider answered releases its hold; you are not charged for it.

2. When a provider answers with a redirect, we refuse it rather than follow it, because following it would send the provider credential to whatever host the redirect named. A model API has no legitimate reason to redirect a request, so a redirect is treated as the failed upstream response it is.

3. When a provider has not sent its first byte within the attempt limit on a text endpoint and another candidate can serve the request, we move to that candidate inside the same credit hold, because a stalled provider should cost you neither money nor the whole request deadline. You are charged only for the candidate that answers. A lone or last candidate is never cut at that limit — there is nothing better to move to.

4. When a burst opens many connections to one provider, we keep only a bounded number idle afterwards, because an unbounded idle pool exhausts the sockets the gateway needs to settle or release holds it already took. Extra connections are closed once the burst passes.

5. When the answering model is not the primary, we tell you, because a latency number from a fallback is a different measurement. x-nr-routing reads fallback:<n> and x-nr-attempts counts the provider calls, so you can separate those samples. The behaviour of the chain itself is in Provider Fallback Chains: Surviving an OpenAI Outage.

What you see from the outside

Every response carries x-nr-* headers, and every request leaves a record you can read back with the same key:

HTTP/1.1 200 OK
x-nr-request-id: <id>
x-nr-model: <model that answered>
x-nr-routing: direct
x-nr-attempts: 1
x-nr-latency-ms: <integer ms, arrival to response headers>
x-nr-cost-status: exact
x-nr-request-cost: <USD>

The record adds what headers cannot carry: the settled cost of a streamed request, the server-side first-token time, and the per-step trace used in the worked example. Reading that record is walked through in What an LLM Request Log Should Contain — and What to Leave Out.

Limits

1. Distance is not negotiable. A reused connection removes handshakes, not the round trip itself. If your application and the provider are far apart, every request still pays that distance; placing compute near the provider matters more than any pool.
2. Generation time belongs to the model. In the worked example the provider was most of the wait. Choosing a faster model for the task is the lever there — compare candidates on models.
3. HTTP/2 over TCP still shares one loss domain. When a packet is lost, every stream on that connection waits for the retransmission; that head-of-line effect is the problem QUIC was designed around (RFC 9000).
4. The first request to a provider is still cold. Pools warm on use. A gateway instance that has not yet talked to a provider pays the handshakes once.
5. The gateway's checks have a cost. Authentication, rate limits, guardrails and the credit hold protect your budget and your data, and they take measurable time. Measure them for your traffic; latency figures for percentiles are explained in LLM Latency: p50, p95, p99, and Time-to-First-Token.

Try it

1. Create a key at app.nrouter.ai/signup. Pay as you go starts from a $5 credit purchase, 0% platform fee; a purchase charges exactly the credits bought.
2. Measure your own split with the script in Measure TTFT and Trace an LLM Request Step by Step in Python.
3. Talk to us about placement. If distance dominates your latency, review nRouter deployment options for dedicated capacity.

See also

• Measure TTFT and Trace an LLM Request Step by Step in Python — the script that produces the per-step numbers used in this post, for your own requests.
• LLM Latency: p50, p95, p99, and Time-to-First-Token — how to read the tail once you have more than one sample.
• How to Benchmark an LLM Gateway in Production: Cost, Latency, Reliability, and Quality — how to turn measurements into a benchmark you can defend.
• Provider Fallback Chains: Surviving an OpenAI Outage — what happens to a request when the first provider cannot serve it.
• Reserve-and-Settle: Never Overspend a Credit Balance — what the reserve step does and why a failed call costs nothing.
• RPM and TPM Rate Limiting Per Key, Team, and Org — the windows the rate_limit step checks.
• Pricing — pay as you go, 0% platform fee, usage at the served model's exact list price.

Sources

Verified 2026-10-15; if any specification has drifted, email hello@nrouter.ai.

• TCP (RFC 9293): datatracker.ietf.org/doc/html/rfc9293
• TLS 1.3 (RFC 8446): datatracker.ietf.org/doc/html/rfc8446
• TCP initial window (RFC 6928): datatracker.ietf.org/doc/html/rfc6928
• TCP congestion control (RFC 5681): datatracker.ietf.org/doc/html/rfc5681
• HTTP/1.1 (RFC 9112): datatracker.ietf.org/doc/html/rfc9112
• HTTP/2 (RFC 9113): datatracker.ietf.org/doc/html/rfc9113
• QUIC (RFC 9000): datatracker.ietf.org/doc/html/rfc9000
• OpenAI production best practices: platform.openai.com/docs/guides/production-best-practices
• Anthropic API overview: docs.anthropic.com/en/api/overview
• Production timings in the worked example: one nrouter/auto request measured on 2026-10-15 and read back through its per-step trace; a single example, not a benchmark.