Transparency
How Relixr routes requests
This page describes the routing algorithm, how latency is measured, and what happens during provider failures — including known limitations. We publish this so you can verify our claims, audit the behavior, and hold us to it.
On this page
Provider selection
When a request arrives, the router scores every eligible provider and forwards to the highest-scoring one. Scoring accounts for recent latency and reliability together — a provider that is fast but unreliable is scored worse than one that is slightly slower but consistent.
The router continuously learns from observed traffic. Providers that have performed well recently score higher; providers with little recent data are given a fair chance to demonstrate their current performance. No manual configuration is required — the allocation adjusts automatically as provider conditions change.
Latency measurement
Latency is measured as time-to-first-token: the interval from when Relixr begins forwarding the request to the upstream provider, to when the first token byte arrives in the response stream.
This is the latency that matters for streaming applications — it is the delay before your user sees anything. Total generation time is not used because it scales with output length, which is outside the provider's control and varies per request.
The measured upstream latency is returned on every response as X-Relixr-Latency-Ms. This is the raw upstream figure — it excludes Relixr's own overhead. See Relixr's overhead for how to measure the end-to-end total.
Reliability scoring
A provider's routing score is not based on raw latency alone. Recent error rates are factored in, so a provider experiencing elevated failures is scored as effectively slower — even if its time-to-first-token on successful responses looks fine.
The practical effect: during a partial outage, a degraded provider is automatically deprioritized in proportion to how degraded it is, without any manual threshold or intervention. As the provider recovers and errors fall, its score recovers accordingly.
Failover logic
If the selected provider returns a 5xx or a connection timeout, Relixr retries the request with the next-best candidate from the same selection round.
4xx errors are not retried — they indicate a problem with the request itself (invalid model id, quota exceeded at the key level, malformed payload) where switching providers will not help.
When a failover occurs, the response includes X-Relixr-Failover: true and X-Relixr-Providerreflects the provider that actually served the request. Your dashboard counts these as “outages absorbed.”
Relixr's overhead
Relixr runs on serverless edge nodes co-located with major provider infrastructure. Typical overhead is under 10ms. You can measure it yourself: subtract the X-Relixr-Latency-Ms header value from your client-side round-trip time.
This comparison works against any OpenAI-compatible gateway using equivalent methodology.
Known limitations
The routing system has known limitations we document openly. Some are fundamental tradeoffs; others are areas of active improvement.
Success-only latency recording
Latency is recorded only on successful responses. A provider that fails immediately (returning a quick 503) contributes no latency signal — only an error. The reliability scoring partially compensates for this, but the compensation is imperfect at very high error rates. We are tracking an improvement to capture connection-failure timing separately.
Cold-start period
When a model is requested for the first time in a workspace, the router has no observation history. Early selections during this period are less optimized. Performance stabilizes as traffic accumulates.
Prompt cache affinity
Provider-side prompt caches are per-provider and sometimes per-region. The router does not currently factor cache affinity into routing decisions, so workloads with large repeated prefixes may not benefit from cache hits as consistently as they would with a fixed provider assignment.
Single-provider models
Some models are only available from one provider. For these, the router sends directly to that provider without selection. Automatic failover is not available for single-provider models.
Verifying these claims yourself
Every routing decision is recorded in the response. A single curl command reveals the routing metadata:
Every request your workspace sends is also visible in the Log explorer with the full routing decision, provider, upstream latency, Relixr overhead, and whether failover occurred. Expanding any row shows the candidate list, fallback chain, and cache activity. Your dashboard's failover count corresponds exactly to requests where X-Relixr-Failover: true was returned.
Questions or discrepancies: support@relixr.com or open an issue referencing the request id.