Route requests with the Adaptive Model Router
The Adaptive Model Router picks a model and reasoning effort for each LLM request on a route. This topic shows you how to connect a decision service, create the plugin instance, and test it. To compare it with the other routing plugins, see Route LLM requests across models and providers.
Prerequisites
Before you begin, make sure you have the following:
- A route that accepts authenticated requests. For the steps, see Connect an LLM provider.
- No other routing plugin on the route. If a route has two, such as the LLM Router and the Adaptive Model Router, requests fail with
503. - A provider connection with no model allowlist, for the plugin to route to. Fabric refuses an Adaptive Model Router whose Connection has a model allowlist, and refuses adding an allowlist to a connection that an Adaptive Model Router uses. The decision connection can have one.
- Permission to create plugin instances.
- A decision service. This topic explains how to set one up.
Add the Adaptive Model Router
Models differ in how much they reason before they answer. Smaller models answer quickly and cost less. Larger models reason more, which uses more tokens and costs more, even for a request that doesn’t need that effort. With adaptive model routing, the gateway rates each prompt before it reaches the provider and sends the prompt to the model that fits its difficulty. A simple greeting goes to a small model even if the client asked for a large one.
The plugin does this with a decision service: a model that returns probabilities instead of text. For example, a decision service can read a support ticket and report how likely it is to be about billing, technical problems, or something else. The Adaptive Model Router uses the same approach to rate how difficult a prompt is.
To set up adaptive model routing, connect a decision service, and then create the plugin instance.
Connect a decision service
You bring your own decision service. It must be a Jev endpoint. The gateway sends a request to the endpoint’s /v1/systemone path, so you can connect Jev, or a Jev endpoint that you host yourself. Other decision APIs aren’t supported.
If your security requirements don’t allow traffic to leave your infrastructure, host both the decision model and the LLM yourself. Then the gateway routes everything through your own infrastructure, and no data goes to another provider. To connect an endpoint that you host, specify your own target URL when you create the connection. The target URL must be an HTTPS origin with no path.
To connect a hosted decision service such as Jev, do the following:
- In the sidebar, select LLMs, select the Providers tab, search for the provider, and click Connect. For details about the Create LLM connection page, see Connect an LLM provider.
- In Upstream auth, enter the provider’s API key.
- In Models, allow the models that the decision service offers.
- In Gateway route, keep Create gateway route selected to expose the connection on a route. You can reuse a gateway key that you already created.
- Click Create connection.
Fabric stores the decision service’s API key, so you don’t enter it on the plugin or on the route.
Create the plugin instance
To create the plugin instance, do the following:
-
Select Plugins, find Adaptive Model Router in the Routing category, and click Use.
-
Enter the following values:
-
Click Advanced settings, and then set the following options:
-
Click Save.
The Include system messages and Include tool descriptions options send more of your request content to the decision service. By default, the gateway sends the conversation text up to Max input characters, without system messages or tool descriptions, and waits up to 5 seconds for an answer. Leave them off unless your routing needs that content.
Test the plugin
To test the routing, send the same request that you used to test the route, and ask for a larger model. For a simple prompt such as Hello, world!, the response comes back from a smaller model than the one in your request. The gateway sent the prompt to the decision service, which rated it as simple, and the plugin changed the model before the request reached the provider.
To confirm which model answered, see Check which model answered.
Next steps
- Keep requests working during a provider outage. For the steps, see Fail over across providers with the LLM Router.
- Compare cost and token usage across models on the LLM tab of Review analytics.

