Skip to navigation

Route requests with the Adaptive Model Router

The Adaptive Model Router picks a model and reasoning effort for each LLM request on a route. This topic shows you how to connect a decision service, create the plugin instance, and test it. To compare it with the other routing plugins, see Route LLM requests across models and providers.

Prerequisites

Before you begin, make sure you have the following:

  • A route that accepts authenticated requests. For the steps, see Connect an LLM provider.
  • No other routing plugin on the route. If a route has two, such as the LLM Router and the Adaptive Model Router, requests fail with 503.
  • A provider connection with no model allowlist, for the plugin to route to. Fabric refuses an Adaptive Model Router whose Connection has a model allowlist, and refuses adding an allowlist to a connection that an Adaptive Model Router uses. The decision connection can have one.
  • Permission to create plugin instances.
  • A decision service. This topic explains how to set one up.

Add the Adaptive Model Router

Models differ in how much they reason before they answer. Smaller models answer quickly and cost less. Larger models reason more, which uses more tokens and costs more, even for a request that doesn’t need that effort. With adaptive model routing, the gateway rates each prompt before it reaches the provider and sends the prompt to the model that fits its difficulty. A simple greeting goes to a small model even if the client asked for a large one.

The plugin does this with a decision service: a model that returns probabilities instead of text. For example, a decision service can read a support ticket and report how likely it is to be about billing, technical problems, or something else. The Adaptive Model Router uses the same approach to rate how difficult a prompt is.

To set up adaptive model routing, connect a decision service, and then create the plugin instance.

Connect a decision service

You bring your own decision service. It must be a Jev endpoint. The gateway sends a request to the endpoint’s /v1/systemone path, so you can connect Jev, or a Jev endpoint that you host yourself. Other decision APIs aren’t supported.

If your security requirements don’t allow traffic to leave your infrastructure, host both the decision model and the LLM yourself. Then the gateway routes everything through your own infrastructure, and no data goes to another provider. To connect an endpoint that you host, specify your own target URL when you create the connection. The target URL must be an HTTPS origin with no path.

To connect a hosted decision service such as Jev, do the following:

  1. In the sidebar, select LLMs, select the Providers tab, search for the provider, and click Connect. For details about the Create LLM connection page, see Connect an LLM provider.
  2. In Upstream auth, enter the provider’s API key.
  3. In Models, allow the models that the decision service offers.
  4. In Gateway route, keep Create gateway route selected to expose the connection on a route. You can reuse a gateway key that you already created.
  5. Click Create connection.

Fabric stores the decision service’s API key, so you don’t enter it on the plugin or on the route.

Create the plugin instance

To create the plugin instance, do the following:

  1. Select Plugins, find Adaptive Model Router in the Routing category, and click Use.

  2. Enter the following values:

    FieldValueDescription
    Nameadaptive-routerMust be unique within your organization. You can’t change it later.
    Scope kindrouteThe Adaptive Model Router can be scoped only to a route.
    Scope nameanthropic-routeThe route whose requests the plugin routes.
    ConnectionanthropicThe connection that receives the routed request. The list shows the connections that the route exposes. If the route has several connections, choose the one to use. It can’t have a model allowlist.
    Decision connectionjevThe connection that rates each prompt. Fabric resolves it across your organization, so you don’t need to attach it to this route.
  3. Click Advanced settings, and then set the following options:

    FieldValueDescription
    Jev modeljev-1.13.0The model identifier that the gateway sends to the decision service. Pin a version so routing doesn’t change under your tuned thresholds when a newer version is released.
    Max input characters32000How much of the request the gateway sends to the decision service. The default is 32000.
    Default max output tokens4096The output token limit that the gateway applies to the routed request to the chosen model, not to the decision service’s call. The gateway applies the smaller of this value and the client’s own limit, and a small cap can also lower the reasoning effort. The default is 4096.
    Session stickinessOffWhen on, a conversation stays on the model chosen in its first turn. This keeps the provider’s prompt cache warm and skips a decision call on later turns. It applies within one session, such as a back-and-forth with a coding agent, and it needs the client to send a session ID header, such as X-Amr-Session-ID, X-Claude-Code-Session-ID, Thread-ID, Session-ID, or X-Session-ID. Requests without one aren’t pinned. A pin expires 24 hours after its last use. If the data plane has no distributed store, requests with a session ID return 503.
    Include system messagesOffWhen on, the gateway also sends system messages to the decision service. Many clients add long system instructions to every request, which a decision service usually doesn’t need. Turn it on if the instructions affect how a prompt should be rated.
    Include tool descriptionsOffWhen on, the gateway also sends tool definitions to the decision service. Clients that have several MCP servers attached send many tool definitions with each request. Turn it on only if tool details affect how a prompt should be rated.
  4. Click Save.

The Include system messages and Include tool descriptions options send more of your request content to the decision service. By default, the gateway sends the conversation text up to Max input characters, without system messages or tool descriptions, and waits up to 5 seconds for an answer. Leave them off unless your routing needs that content.

Test the plugin

To test the routing, send the same request that you used to test the route, and ask for a larger model. For a simple prompt such as Hello, world!, the response comes back from a smaller model than the one in your request. The gateway sent the prompt to the decision service, which rated it as simple, and the plugin changed the model before the request reached the provider.

To confirm which model answered, see Check which model answered.

Next steps