Fail over across providers with the LLM Router
The LLM Router controls which connections handle requests on a route and what happens when one fails. This topic shows you how to configure fallback across providers. To compare it with the other routing plugins, see Route LLM requests across models and providers.
Prerequisites
Before you begin, make sure you have the following:
- A route that accepts authenticated requests. For the steps, see Connect an LLM provider.
- Permission to create plugin instances.
- At least two connections on the route.
- No model router or Adaptive Model Router on the route. A route can have only one routing plugin.
Configure fallback
Fallback routing keeps requests from failing when a provider has an outage. You list connections in priority order. The gateway tries the first one. If it fails, the gateway moves down the list until one of them answers.
For example, a route can list OpenAI first, and then the same OpenAI models hosted on Azure or on Amazon Bedrock Mantle. If a request to OpenAI fails, the gateway sends it to the next connection, which serves the same models. Fabric sends every target the same request, including the model name, so each target must accept that name. To send each provider its own model ID, use model aliases instead.
To create an LLM Router instance for fallback, do the following:
-
Select Plugins, find LLM Router in the Routing category, and select Use.
-
Enter a Name, for example
llm-router-fallback. The LLM Router can be scoped only to a route, so set Scope kind torouteand Scope name to the route you want to configure. -
For Mode, select
fallback. -
Configure the following settings:
-
Select Save.
429 and 503 always trigger fallback, so you can’t exclude them. You can’t exclude 500, 502, or 504 while nonidempotent failover is on. Set it to Never first.
Fabric retries a network error, a 429, or a 5xx response that triggers fallback on the same target, up to Attempts per target. It doesn’t retry any other 4xx response on the same target, because the same target would give the same answer. It falls back to the next target instead.
For streaming responses, Fabric retries a retryable first event before it sends any bytes. After the first valid event reaches the client, Fabric doesn’t retry or fall back.
A client that sends the x-gateway-service header reaches only the service it names, with no fallback. If the header names a service that isn’t on the route, the client gets 400.
If every target fails, the client gets the last target’s answer as the provider sent it. If the last target can’t be reached, the client gets 502, or 504 if it timed out. If no target is available to try, the client gets 503.
To confirm which model answered, see Check which model answered. The response doesn’t name the service that answered. To see which target served a request, open the request in Traces. For details, see Inspect traces.
Next steps
- Pick a model for each request based on its difficulty. For the steps, see Route requests with the Adaptive Model Router.

