Skip to navigation

LLM Router plugin

The LLM Router plugin decides which of a route’s connections handle each LLM request and what happens when one fails. It can send every request to one target, walk an ordered fallback chain, pick a target by weight and spill over to the rest, or, in lightyear mode, send requests through Lightyear. It only routes across connections that are already on the route.

A client can send the x-gateway-service header to pick one connection on the route for a request. That request goes to that connection only and doesn’t fail over. A header that names a connection that isn’t on the route gets 400. The router ignores the header in lightyear mode.

Where it runs

ProtocolsLLM routes. On other protocols the plugin does nothing.
Scope kindsroute only
PhaseRequest

Configuration

mode
stringDefaults to single

single sends requests to exactly one target. fallback tries targets by order, lowest first, and moves on when one fails. loadbalance picks the first target at random in proportion to weight, and then tries the others in list order. lightyear sends requests through Lightyear and takes no targets. For more information, see Route requests through Lightyear.

credentials
string

Valid only in lightyear mode. Set it to client to forward the caller’s own provider credential instead of the one saved on the connection. The console shows this as Use the client’s provider credentials.

targets
array of objects

The connections to route to. The control plane refuses an empty list on save, except in lightyear mode, which takes no targets.

targets[].service
stringRequired

A connection on this route. Each connection can appear once.

targets[].order
integerDefaults to List order

The position in a fallback chain. Equal values keep list order.

targets[].weight
integer

The share of first attempts in loadbalance mode. Must not be negative, and must be positive in loadbalance mode. The control plane doesn’t currently catch an omitted weight, so such a route returns 500.

retry.max_attempts
integerDefaults to No retries

Attempts on one target before moving to the next, counting the first, from 2 to 10.

retry.initial_backoff_ms
integerDefaults to Gateway default

The first retry delay in milliseconds: 0, or 50 to 10,000.

on_status_codes
array of integers

Extra status codes, from 400 to 599, that trigger a retry or fallback. 429 and 503 always do. A 4xx code other than 429 triggers a fallback but isn’t retried on the same target.

on_status_match
object

A range-aware alternative to on_status_codes. You can’t set both. It has an operator of in or not_in, and a codes array of single codes or inclusive ranges between 400 and 599, such as [429, "500-599"]. With in, the router falls back on the listed codes and on the codes that always trigger fallback. With not_in, it falls back on every code from 400 to 599 except the listed ones, and you can’t exclude 429 or 503. You also can’t exclude 500, 502, or 504 while allow_nonidempotent_failover is on. The console shows this as Fall back on and Status codes.

allow_nonidempotent_failover
booleanDefaults to true

Also fails over on 500, 502, and 504. A fallback can repeat a request that the first provider already processed.

enrolled_consumers
array of stringsDefaults to Every caller

Applies the targets only to these consumers. Other callers get the gateway’s default routing.

A configuration that fails these checks makes every request on the route return 500. The control plane rejects such a configuration when you save it. A deployment that has no control plane doesn’t check it on save.

Example

{
"name": "llm-router-fallback",
"plugin_id": "llm-router",
"scope": { "kind": "route", "name": "openai-route" },
"config": {
"mode": "fallback",
"targets": [
{ "service": "openai", "order": 1 },
{ "service": "azure-openai", "order": 2 }
],
"on_status_codes": [408]
}
}