LLM Router plugin
The LLM Router plugin decides which of a route’s connections handle each LLM request and what happens when one fails. It can send every request to one target, walk an ordered fallback chain, pick a target by weight and spill over to the rest, or, in lightyear mode, send requests through Lightyear. It only routes across connections that are already on the route.
A client can send the x-gateway-service header to pick one connection on the route for a request. That request goes to that connection only and doesn’t fail over. A header that names a connection that isn’t on the route gets 400. The router ignores the header in lightyear mode.
Where it runs
Configuration
single sends requests to exactly one target. fallback tries targets by order, lowest first, and moves on when one fails. loadbalance picks the first target at random in proportion to weight, and then tries the others in list order. lightyear sends requests through Lightyear and takes no targets. For more information, see Route requests through Lightyear.
Valid only in lightyear mode. Set it to client to forward the caller’s own provider credential instead of the one saved on the connection. The console shows this as Use the client’s provider credentials.
The connections to route to. The control plane refuses an empty list on save, except in lightyear mode, which takes no targets.
A connection on this route. Each connection can appear once.
The position in a fallback chain. Equal values keep list order.
The share of first attempts in loadbalance mode. Must not be negative, and must be positive in loadbalance mode. The control plane doesn’t currently catch an omitted weight, so such a route returns 500.
Attempts on one target before moving to the next, counting the first, from 2 to 10.
The first retry delay in milliseconds: 0, or 50 to 10,000.
Extra status codes, from 400 to 599, that trigger a retry or fallback. 429 and 503 always do. A 4xx code other than 429 triggers a fallback but isn’t retried on the same target.
A range-aware alternative to on_status_codes. You can’t set both. It has an operator of in or not_in, and a codes array of single codes or inclusive ranges between 400 and 599, such as [429, "500-599"]. With in, the router falls back on the listed codes and on the codes that always trigger fallback. With not_in, it falls back on every code from 400 to 599 except the listed ones, and you can’t exclude 429 or 503. You also can’t exclude 500, 502, or 504 while allow_nonidempotent_failover is on. The console shows this as Fall back on and Status codes.
Also fails over on 500, 502, and 504. A fallback can repeat a request that the first provider already processed.
Applies the targets only to these consumers. Other callers get the gateway’s default routing.
A configuration that fails these checks makes every request on the route return 500. The control plane rejects such a configuration when you save it. A deployment that has no control plane doesn’t check it on save.

