Adaptive Model Routing and Provider Failover

Routing plugins decide which model or provider serves each request on a route. The Routing category of the plugin catalog includes three plugins. The Adaptive Model Router picks a model and reasoning effort for each request. The Adaptive Router (Meta, OpenAI) does cost-aware OpenAI model routing. The LLM Router controls which services handle requests and what happens when one fails. This topic shows you how to configure the Adaptive Model Router and the LLM Router.

Prerequisites

Before you begin, make sure you have the following:

• A route that accepts authenticated requests. For the steps, see Connect an LLM provider.
• Permission to create plugin instances.
• For adaptive model routing, a decision service. This topic explains how to set one up.
• For fallback routing, at least two services on the route.

Understand the plugins

Each routing plugin attaches to a route. To find them, select Plugins, filter by the Routing category, and select Use next to the plugin.

Add the Adaptive Model Router

Models differ in how much they reason before they answer. Smaller models answer quickly and cost less. Larger models reason more, which uses more tokens and costs more, even for a request that doesn’t need that effort. With adaptive model routing, the gateway rates each prompt before it reaches the provider and sends the prompt to the model that fits its difficulty. A simple greeting goes to a small model even if the client asked for a large one.

The plugin does this with a decision service: a model that returns probabilities instead of text. For example, a decision service can read a support ticket and report how likely it is to be about billing, technical problems, or something else. The Adaptive Model Router uses the same approach to rate how difficult a prompt is.

To set up adaptive model routing, connect a decision service, and then create the plugin instance.

Connect a decision service

You bring your own decision service. You can connect any decision service that you configure on the gateway, including the following:

• A hosted decision model, such as Jev.
• A service that supports the OpenAI Decisions API. Fabric supports its /v1/decisions endpoint.
• A decision model that you host yourself.

If your security requirements don’t allow traffic to leave your infrastructure, host both the decision model and the LLM yourself. Then the gateway routes everything through your own infrastructure, and no data goes to another provider. To connect an endpoint that you host, specify your own target URL when you create the connection.

To connect a hosted decision service such as Jev, do the following:

1. In the sidebar, select LLMs > Providers, search for the provider, and click Connect. For details about the provider setup wizard, see Connect an LLM provider.
2. Enter the provider’s API key, and then continue.
3. In the model filter step, allow the models that the decision service offers.
4. Expose the service on a route. You can reuse a gateway key that you already created.
5. Click Finish setup.

Fabric stores the decision service’s API key, so you don’t enter it on the plugin or on the route.

Create the plugin instance

To create the plugin instance, do the following:

1. Select Plugins, find Adaptive Model Router in the Routing category, and click Use.

2. Enter the following values:

   Field	Value	Description
   Name	adaptive-router	Must be unique within your organization. You can’t change it later.
   Scope kind	route	The Adaptive Model Router can be scoped only to a route.
   Scope name	anthropic-route	The route whose requests the plugin routes.
   Connection	anthropic	The service that receives the routed request. The list shows the services that the route exposes. If the route has several services, choose the one to use.
   Decision connection	jev	The service that rates each prompt. Fabric resolves it across your organization, so you don’t need to attach it to this route.

3. Click Advanced settings, and then set the following options:

   Field	Value	Description
   Jev model	jev-1.13.0	The model identifier that the gateway sends to the decision service. Pin a version so routing doesn’t change under your tuned thresholds when a newer version is released.
   Max input characters	32000	How much of the request the gateway sends to the decision service. The default is 32000.
   Default max output tokens	4096	The output limit for the decision service’s own call. The default is 4096.
   Session stickiness	Off	When on, a conversation stays on the model chosen in its first turn. This keeps the provider’s prompt cache warm and skips a decision call on later turns. It applies within one session, such as a back-and-forth with a coding agent. A new conversation starts a new session.
   Include system messages	Off	When on, the gateway also sends system messages to the decision service. Many clients add long system instructions to every request, which a decision service usually doesn’t need. Turn it on if the instructions affect how a prompt should be rated.
   Include tool descriptions	Off	When on, the gateway also sends tool definitions to the decision service. Clients that have several MCP servers attached send many tool definitions with each request. Turn it on only if tool details affect how a prompt should be rated.

4. Click Save.

Test the plugin

To test the routing, send the same request that you used to test the route, and ask for a larger model. For a simple prompt such as Hello, world!, the response comes back from a smaller model than the one in your request. The gateway sent the prompt to the decision service, which rated it as simple, and the plugin changed the model before the request reached the provider.

Configure fallback

Fallback routing keeps requests from failing when a provider has an outage. You list services in priority order. The gateway tries the first one. If it fails, the gateway moves down the list until one of them answers.

For example, a route can list OpenAI first, and then the same OpenAI models hosted on Azure or on Amazon Bedrock Mantle. If a request to OpenAI fails, the gateway sends it to the next service, which serves the same models.

To create an LLM Router instance for fallback, do the following:

1. Click Plugins, find LLM Router in the Routing category, and click Use.

2. Enter a Name, for example llm-router-fallback. The LLM Router can be scoped only to a route, so set Scope kind to route and Scope name to the route you want to configure.

3. For Mode, select fallback.

4. Configure the following settings:

   Field	Description
   Targets	The services to try, in priority order. The top target has the highest priority. Select the arrows to reorder targets, Add target to add one, and the trash icon to remove one.
   Attempts per target	How many times Fabric tries a provider before it moves to the next target. The default is No retries (1 attempt), which moves on immediately. If you allow retries, set between 2 and 10 attempts, counting the first request. Each extra attempt adds latency, so use retries sparingly.
   On status codes	The HTTP status codes (400–599) that trigger a retry or fallback, entered as a comma-separated list, for example 429, 503. Leave the field empty to use the plugin default, which retries on 429 and 503. When nonidempotent failover is on, the default also covers 500, 502, and 504.
   Allow nonidempotent failover	Controls whether Fabric fails over on requests that change state, such as POST and PATCH. For LLM routes, the default is to fail over, which adds 500, 502, and 504 to the status codes that trigger fallback. A fallback can repeat a request that the first provider already processed, so turn this off if duplicate requests are a problem.

5. Click Save.

For streaming responses, Fabric retries a retryable first event before it sends any bytes. After the first valid event reaches the client, Fabric doesn’t retry or fall back.

Check which model answered

After you save a routing plugin, send test requests to the route. To confirm which model answered, check the model field in the response. In Traces, a request that uses adaptive model routing appears with its own request to the decision service, followed by the request to the provider. For the steps, see Inspect traces.

Next steps

• Add a rate limit to the route with the LLM Rate Limit plugin. For the steps, see Control traffic with plugins.
• Add a second connection for another provider, and use it as a fallback target.
• Compare cost and token usage across models on the LLM tab of Review analytics.