Skip to navigation

Adaptive Model Router plugin

The Adaptive Model Router plugin picks a model and a reasoning effort for each LLM request, instead of the model that the client named. It asks a decision service to rate the request, turns the rating into a tier (fast, balanced, advanced, or reasoning), picks that tier’s model for the request’s provider, and rewrites the request body for it. It only chooses among the request’s own provider’s models. Adaptive Model Router is a Preview plugin.

Where it runs

ProtocolsLLM routes, for OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages requests
Scope kindsroute. The console offers only route scope, and provider_services must name connections on the route the request matched. A route can use only one routing plugin. A second one, such as the LLM Router or the LLM Model Router, makes requests fail with 503.
PhaseRequest

If the decision service fails or can’t be found, for example because it was renamed or deleted, the request goes on unchanged. A request with the x-gateway-service header is also left unchanged.

Configuration

The table lists the top-level fields. For a working setup, follow the guide.

provider_services
objectRequired

Maps a provider, openai or anthropic, to a connection on this route. Without it, the plugin does nothing.

decision.service
stringRequired

The connection that runs the decision model. It can be any connection in your organization, not only one on this route.

decision.model
stringRequired

The decision model.

decision.max_input_chars
integerDefaults to 32000

How much of the conversation the decision service reads.

default_max_output_tokens
integerDefaults to 4096

The output token limit that the gateway applies to the routed request. The gateway applies the smaller of this value and the client’s own limit, and a small cap can also lower the reasoning effort. The console labels this Default max output tokens.

decision.include_system_messages
booleanDefaults to false

Sends system messages to the decision service.

decision.include_tool_descriptions
booleanDefaults to false

Sends tool descriptions to the decision service.

tier_models
objectDefaults to Built-in

The model for each tier, per provider. The fast, balanced, and advanced tiers each need a model in the pricing catalog.

pricing
array of objectsDefaults to Built-in catalog

Extra or changed model entries. Each entry has id, provider (openai, anthropic, or gemini), tier (fast, balanced, advanced, or reasoning), input_per_mtok, output_per_mtok, cached_input_per_mtok, context_window, max_output, reasoning, and capability_score (from 0 to 1), and optionally the reasoning efforts and endpoints the model supports. Prices are in USD per million tokens.

execution_mode
stringDefaults to predict

predict rewrites the request and sends it once. cascade is a legacy mode that calls the provider itself and can retry at a higher effort. It doesn’t stream, and each retry is another billed model call. The console shows Execution mode only for instances that are already on cascade.

session.stickiness
booleanDefaults to false

Keeps the model chosen on a conversation’s first turn. Can’t be combined with cascade. It needs the client to send a session ID header, such as X-Amr-Session-ID, X-Claude-Code-Session-ID, Thread-ID, Session-ID, or X-Session-ID. A pin expires 24 hours after its last use. Without a distributed store on the data plane, requests with a session ID return 503.

A configuration that fails its checks makes every request on the route return 500. When you save, the control plane checks only that no connection in provider_services has a model allowlist. It also refuses adding an allowlist to a connection that an instance maps. The data plane runs the other checks.

Example

{
"name": "adaptive-router",
"plugin_id": "adaptive-model-router",
"scope": { "kind": "route", "name": "openai-route" },
"config": {
"provider_services": { "openai": "openai" },
"decision": { "service": "decision-service", "model": "jev-1.13.0" }
}
}