> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.fabricgateway.ai/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.fabricgateway.ai/_mcp/server. # Adaptive Model Routing and Provider Failover > Choose a model for each LLM request or configure failover across providers. Routing plugins decide which model or provider serves each request on a route. The **Routing** category of the plugin catalog includes three plugins. The Adaptive Model Router picks a model and reasoning effort for each request. The Adaptive Router (Meta, OpenAI) does cost-aware OpenAI model routing. The LLM Router controls which services handle requests and what happens when one fails. This topic shows you how to configure the Adaptive Model Router and the LLM Router. ## Prerequisites Before you begin, make sure you have the following: * A route that accepts authenticated requests. For the steps, see [Connect an LLM provider](/fabric/gateways/llm-providers). * Permission to create plugin instances. * For adaptive model routing, a decision service. This topic explains how to set one up. * For fallback routing, at least two services on the route. ## Understand the plugins Each routing plugin attaches to a route. To find them, select **Plugins**, filter by the **Routing** category, and select **Use** next to the plugin. | Plugin | What it does | | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Adaptive Model Router** | Selects an LLM model and reasoning effort for each request. It uses a decision service, which rates the request and returns probabilities that the plugin uses to choose. | | **Adaptive Router (Meta, OpenAI)** | Cost-aware OpenAI model routing that selects a model and reasoning effort for each request. This topic doesn't cover its settings. | | **LLM Router** | Builds the routing directive for the matched route and request. Its modes control how Fabric chooses among the services you list. | ## Add the Adaptive Model Router Models differ in how much they reason before they answer. Smaller models answer quickly and cost less. Larger models reason more, which uses more tokens and costs more, even for a request that doesn't need that effort. With adaptive model routing, the gateway rates each prompt before it reaches the provider and sends the prompt to the model that fits its difficulty. A simple greeting goes to a small model even if the client asked for a large one. The plugin does this with a *decision service*: a model that returns probabilities instead of text. For example, a decision service can read a support ticket and report how likely it is to be about billing, technical problems, or something else. The Adaptive Model Router uses the same approach to rate how difficult a prompt is. To set up adaptive model routing, connect a decision service, and then create the plugin instance. ### Connect a decision service You bring your own decision service. You can connect any decision service that you configure on the gateway, including the following: * A hosted decision model, such as Jev. * A service that supports the OpenAI Decisions API. Fabric supports its `/v1/decisions` endpoint. * A decision model that you host yourself. If your security requirements don't allow traffic to leave your infrastructure, host both the decision model and the LLM yourself. Then the gateway routes everything through your own infrastructure, and no data goes to another provider. To connect an endpoint that you host, specify your own target URL when you create the connection. To connect a hosted decision service such as Jev, do the following: 1. In the sidebar, select **LLMs** > **Providers**, search for the provider, and click **Connect**. For details about the provider setup wizard, see [Connect an LLM provider](/fabric/gateways/llm-providers). 2. Enter the provider's API key, and then continue. 3. In the model filter step, allow the models that the decision service offers. 4. Expose the service on a route. You can reuse a gateway key that you already created. 5. Click **Finish setup**. Fabric stores the decision service's API key, so you don't enter it on the plugin or on the route. ### Create the plugin instance To create the plugin instance, do the following: 1. Select **Plugins**, find **Adaptive Model Router** in the **Routing** category, and click **Use**. 2. Enter the following values: | Field | Value | Description | | ----------------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name** | `adaptive-router` | Must be unique within your organization. You can't change it later. | | **Scope kind** | `route` | The Adaptive Model Router can be scoped only to a route. | | **Scope name** | `anthropic-route` | The route whose requests the plugin routes. | | **Connection** | `anthropic` | The service that receives the routed request. The list shows the services that the route exposes. If the route has several services, choose the one to use. | | **Decision connection** | `jev` | The service that rates each prompt. Fabric resolves it across your organization, so you don't need to attach it to this route. | 3. Click **Advanced settings**, and then set the following options: | Field | Value | Description | | ----------------------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Jev model** | `jev-1.13.0` | The model identifier that the gateway sends to the decision service. Pin a version so routing doesn't change under your tuned thresholds when a newer version is released. | | **Max input characters** | `32000` | How much of the request the gateway sends to the decision service. The default is 32000. | | **Default max output tokens** | `4096` | The output limit for the decision service's own call. The default is 4096. | | **Session stickiness** | Off | When on, a conversation stays on the model chosen in its first turn. This keeps the provider's prompt cache warm and skips a decision call on later turns. It applies within one session, such as a back-and-forth with a coding agent. A new conversation starts a new session. | | **Include system messages** | Off | When on, the gateway also sends system messages to the decision service. Many clients add long system instructions to every request, which a decision service usually doesn't need. Turn it on if the instructions affect how a prompt should be rated. | | **Include tool descriptions** | Off | When on, the gateway also sends tool definitions to the decision service. Clients that have several MCP servers attached send many tool definitions with each request. Turn it on only if tool details affect how a prompt should be rated. | 4. Click **Save**. > **Note** > > The **Include system messages** and **Include tool descriptions** options send more of your request content to the decision service. Leave them off unless your routing needs that content. ### Test the plugin To test the routing, send the same request that you used to test the route, and ask for a larger model. For a simple prompt such as `Hello, world!`, the response comes back from a smaller model than the one in your request. The gateway sent the prompt to the decision service, which rated it as simple, and the plugin changed the model before the request reached the provider. ## Configure fallback Fallback routing keeps requests from failing when a provider has an outage. You list services in priority order. The gateway tries the first one. If it fails, the gateway moves down the list until one of them answers. For example, a route can list OpenAI first, and then the same OpenAI models hosted on Azure or on Amazon Bedrock Mantle. If a request to OpenAI fails, the gateway sends it to the next service, which serves the same models. To create an LLM Router instance for fallback, do the following: 1. Click **Plugins**, find **LLM Router** in the **Routing** category, and click **Use**. 2. Enter a **Name**, for example `llm-router-fallback`. The LLM Router can be scoped only to a route, so set **Scope kind** to `route` and **Scope name** to the route you want to configure. 3. For **Mode**, select `fallback`. 4. Configure the following settings: | Field | Description | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Targets** | The services to try, in priority order. The top target has the highest priority. Select the arrows to reorder targets, **Add target** to add one, and the trash icon to remove one. | | **Attempts per target** | How many times Fabric tries a provider before it moves to the next target. The default is **No retries (1 attempt)**, which moves on immediately. If you allow retries, set between 2 and 10 attempts, counting the first request. Each extra attempt adds latency, so use retries sparingly. | | **On status codes** | The HTTP status codes (400–599) that trigger a retry or fallback, entered as a comma-separated list, for example `429, 503`. Leave the field empty to use the plugin default, which retries on `429` and `503`. When nonidempotent failover is on, the default also covers `500`, `502`, and `504`. | | **Allow nonidempotent failover** | Controls whether Fabric fails over on requests that change state, such as `POST` and `PATCH`. For LLM routes, the default is to fail over, which adds `500`, `502`, and `504` to the status codes that trigger fallback. A fallback can repeat a request that the first provider already processed, so turn this off if duplicate requests are a problem. | 5. Click **Save**. For streaming responses, Fabric retries a retryable first event before it sends any bytes. After the first valid event reaches the client, Fabric doesn't retry or fall back. ## Check which model answered After you save a routing plugin, send test requests to the route. To confirm which model answered, check the `model` field in the response. In **Traces**, a request that uses adaptive model routing appears with its own request to the decision service, followed by the request to the provider. For the steps, see [Inspect traces](/fabric/observability/traces). ## Next steps * Add a rate limit to the route with the **LLM Rate Limit** plugin. For the steps, see [Control traffic with plugins](/fabric/plugins). * Add a second connection for another provider, and use it as a fallback target. * Compare cost and token usage across models on the **LLM** tab of [Review analytics](/fabric/observability/analytics).