Skip to navigation

Fail over across providers with the LLM Router

The LLM Router controls which connections handle requests on a route and what happens when one fails. This topic shows you how to configure fallback across providers. To compare it with the other routing plugins, see Route LLM requests across models and providers.

Prerequisites

Before you begin, make sure you have the following:

  • A route that accepts authenticated requests. For the steps, see Connect an LLM provider.
  • Permission to create plugin instances.
  • At least two connections on the route.
  • No model router or Adaptive Model Router on the route. A route can have only one routing plugin.

Configure fallback

Fallback routing keeps requests from failing when a provider has an outage. You list connections in priority order. The gateway tries the first one. If it fails, the gateway moves down the list until one of them answers.

For example, a route can list OpenAI first, and then the same OpenAI models hosted on Azure or on Amazon Bedrock Mantle. If a request to OpenAI fails, the gateway sends it to the next connection, which serves the same models. Fabric sends every target the same request, including the model name, so each target must accept that name. To send each provider its own model ID, use model aliases instead.

To create an LLM Router instance for fallback, do the following:

  1. Select Plugins, find LLM Router in the Routing category, and select Use.

  2. Enter a Name, for example llm-router-fallback. The LLM Router can be scoped only to a route, so set Scope kind to route and Scope name to the route you want to configure.

  3. For Mode, select fallback.

  4. Configure the following settings:

    FieldDescription
    TargetsThe connections to try, in priority order. The top target has the highest priority. Select the arrows to reorder targets, Add target to add one, and the trash icon to remove one.
    Attempts per targetHow many times Fabric tries a provider before it moves to the next target. The default is No retries (1 attempt): Fabric moves to the next target unless the service has its own retry settings. If you allow retries, set between 2 and 10 attempts, counting the first request. A 4xx status other than 429 moves to the next target without a retry. Each extra attempt adds latency, so use retries sparingly.
    Initial retry delay (ms)Appears when you allow retries. How long Fabric waits before the first retry on the same target, from 50 to 10,000 ms. The default is 100. The delay doubles after each retry.
    Fall back onWhich responses make Fabric retry on the current target and then fall back. Default status codes uses 429 and 503, and also 500, 502, and 504 when nonidempotent failover is on. You can also select Any 4xx or 5xx, Any 4xx, Any 5xx, Rate limits (429) and any 5xx, These status codes…, or All status codes except….
    Status codesAppears when you select These status codes… or All status codes except…. Enter codes or ranges between 400 and 599, separated by commas, for example 404, 500-599. With All status codes except…, Fabric falls back on every code from 400 to 599 that you don’t list, for example 400-401, 404, 422.
    Allow nonidempotent failoverControls whether Fabric also falls back on 500, 502, and 504 responses, and on timeouts after the request was sent, when the first provider may already have processed the request. It applies to every request on the route. Choose Default (LLM: fail over), Always fail over, or Never. For LLM routes, the default is to fail over, which adds 500, 502, and 504 to the status codes that trigger fallback. A fallback can repeat a request that the first provider already processed, so select Never if duplicate requests are a problem.
  5. Select Save.

429 and 503 always trigger fallback, so you can’t exclude them. You can’t exclude 500, 502, or 504 while nonidempotent failover is on. Set it to Never first.

Fabric retries a network error, a 429, or a 5xx response that triggers fallback on the same target, up to Attempts per target. It doesn’t retry any other 4xx response on the same target, because the same target would give the same answer. It falls back to the next target instead.

For streaming responses, Fabric retries a retryable first event before it sends any bytes. After the first valid event reaches the client, Fabric doesn’t retry or fall back.

A client that sends the x-gateway-service header reaches only the service it names, with no fallback. If the header names a service that isn’t on the route, the client gets 400.

If every target fails, the client gets the last target’s answer as the provider sent it. If the last target can’t be reached, the client gets 502, or 504 if it timed out. If no target is available to try, the client gets 503.

To confirm which model answered, see Check which model answered. The response doesn’t name the service that answered. To see which target served a request, open the request in Traces. For details, see Inspect traces.

Next steps