Measuring gateway overhead honestly

Put a router in front of a model API and you have added a hop. The only interesting question is how much that hop costs, and the answer that gets published is almost always a P50 in single-digit milliseconds. Ours is one of those numbers. Before you believe it — ours included — it is worth knowing what such a number can be made to exclude.

• Measure request handling only. Start the clock when the request is parsed, stop it when the upstream socket is written. This omits TLS, queueing and the response path, and can cut a real number in half.
• Report the median of a warm loop. Run a thousand requests to the same model from the same region with connections already established, and you have measured your connection pool, not your service.
• Exclude the failure path. If the number only covers requests that succeeded first try, it hides exactly the cases a router exists to handle.
• Compare against a cold direct call. Benchmarking your warm gateway against an unpooled direct connection to the provider is a comparison of connection reuse, not of routing.

None of these are fraud. Each is a defensible measurement of something. They are just not measurements of the thing a user cares about, which is: how much later does my first token arrive because I went through you?

Overhead is added time to first token, measured client-side, against a control. Concretely: from a fixed client, alternate requests between the router and a direct call to the same upstream provider for the same model with the same prompt, interleaved in the same run so that provider-side load is shared between both arms. Record TTFT for both. Overhead is the difference of the distributions, not the difference of the means.

Interleaving matters more than anything else in that paragraph. Model endpoints have load that moves on a minutes-to-hours scale. Run your gateway arm at 09:00 and your control arm at 09:40 and you have measured the provider's morning, not your own code.

# A/B interleaved, same prompt, same model, 500 pairs.
# Alternating within the run is the point — it shares provider load
# between both arms instead of assigning it to one.
for i in $(seq 1 500); do
  bench --arm router --model claude-haiku-4-5 --prompt fixtures/short.txt
  bench --arm direct --model claude-haiku-4-5 --prompt fixtures/short.txt
done | percentiles --field ttft_ms --by arm --p 50,90,99

A P50 tells you about the boring case. Every operational problem lives in the tail — a cold pool, a retry, a DNS refresh, a region failing over. We publish a median overhead of about 11 ms, and we say median rather than latency because the P99 is a different and much larger number, as it is for everyone. A router that claims a tight tail is either not carrying enough traffic to have found one yet, or is not looking.

If your latency page has no P99 on it, the P50 on it is decoration.

In our own traces, the routing decision itself is not the expensive part. Selecting a provider from the model-by-provider matrix is a lookup against an in-memory table and costs microseconds. The real costs are TLS establishment to a provider we have not talked to recently, request-body re-serialization when the upstream dialect differs from the OpenAI-compatible shape we accept, and the queue we impose on ourselves when a burst arrives.

Payment adds its own step, and we are not going to hide it. On a paid tier, the first request of a session carries a 402 round trip and a facilitator verification before the stream can open. That is real added latency on that request. It is amortized across subsequent calls in the same authorization window, but the first one pays it, and any benchmark that quietly warms the payment path before starting the clock is doing the same trick as the ones above.

Here is the uncomfortable framing for anyone selling latency: for most workloads, routing overhead is not the dominant term. Model providers run at roughly 99 to 99.5 percent uptime, six to fourteen times worse than the cloud infrastructure underneath them. Two providers with automatic failover take expected downtime from around 41 hours a year to about 12 minutes.

Forty-one hours is 2,460 minutes. Trading a handful of milliseconds per request for that is not a close call, and any honest overhead discussion should end by saying so. Publish the milliseconds, keep them small, and be clear that they are not why the router is there.