Skip to navigation

Route requests by model name

The model router (llm-model-router) gives clients one base URL for every provider on a route, the way hosted LLM routers work. The client names a model as <provider>/<model>, for example openai/gpt-5.5 or anthropic/claude-sonnet-5-5, and the gateway sends the request to a connection that serves that model. If requests to that connection fail, the gateway tries the next one. This topic shows you how to group a route’s connections by provider, choose a strategy for each group, accept the gateway key from either SDK, and test the route. To compare the model router with the other routing plugins, see Choose a routing approach.

Prerequisites

Before you begin, make sure you have the following:

  • An LLM route with the connections you want to route across, one connection for each provider account or deployment. In the console, select LLMs, select the Routes tab, select Create route, and select every connection in Connections. Include the GET /v1/models endpoint. The console can’t put connections with different API formats, such as OpenAI and Anthropic, on one route. To route across both, create the route with the admin API, as described in the Admin API tab of Create the model router.
  • An allowed models list on every connection the router uses. When you connect the provider, clear All current and future models under Models, and select the models the connection serves. The router serves exactly the models on these lists.
  • Permission to create plugin instances.
  • No LLM Router or Adaptive Model Router on the route. A route can have only one routing plugin.
  • The llm-model-router plugin in your plugin catalog. If it isn’t there, ask a plugin administrator to make it available. If it’s missing for everyone, contact support.

Understand the configuration

The model router has two parts: provider groups, which this topic covers, and aliases, which Define model aliases covers.

A provider group puts some of the route’s connections under a name. The client uses that name as the prefix of the model, so with a group named openai, the model openai/gpt-5.5 goes to the connections in that group. You choose the name. It doesn’t have to match the provider’s name, but it can’t be empty or contain /.

Each group has a strategy:

StrategyHow the gateway orders the group’s connections
fallbackIn the order you list them. The first connection gets every request, and the others are tried only when it fails.
loadbalanceThe first connection is picked at random in proportion to each connection’s weight. If it fails, the rest are tried in the order you list them. Every connection needs a positive weight.

A connection serves only the models on its allowed models list. When a client asks for openai/gpt-5.5, the gateway tries only the openai connections that list gpt-5.5, in the group’s order. The router changes one thing in the request body: it sets model to the name the provider expects, gpt-5.5 in this example. It doesn’t translate between APIs, so a request goes only to connections whose provider serves the endpoint the client called.

The gateway moves to the next connection when the current one answers 429, 500, 502, 503, or 504, or when the network connection to it fails or times out. It also moves on when a connection that signs its requests, such as an AWS connection, answers 401 or 403. Any other answer, success or error, goes back to the client as the provider sent it. If the last connection answers one of the failover statuses, the client gets that answer as the provider sent it, with the x-fabric-model header. Before the gateway moves to the next connection, it retries the current one as that connection’s retry settings allow. If the last connection can’t be reached, the client gets 502, or 504 if it timed out, with the message no candidate for model `<name>` is available. If no connection is available to try, the client gets 503 with the same message. A streamed answer can fail over only until its first event reaches the client.

An OpenAI Responses request that carries previous_response_id goes only to the first connection, with no failover, because the stored response lives at the provider account that created it. In a loadbalance group, that first connection is a weighted pick, so the request can reach an account that doesn’t have the response and get a 404 from the provider. Clients that need failover there send the full input with store: false. The gateway applies this choice before it checks access, so if your key has no grant for that first connection, the request gets 403 even when a later connection would have allowed it.

Create the model router

You can create the model router in the console or with the admin API.

The example below routes across OpenAI and Anthropic, so its route must be created with the admin API. On a route that you created in the console, every group uses the same API format.

The console has no form for this plugin, so you enter its configuration as JSON. To create the instance, do the following:

  1. Select Plugins, find llm-model-router, and select Use.

  2. Enter a Name, for example model-router.

  3. Set Scope kind to route, and set Scope name to your route. The model router can be scoped only to a route.

  4. In Configuration, select Edit as JSON, and then enter the configuration. The following example has an openai group that splits traffic three to one between two OpenAI connections, and an anthropic group that falls back from one Anthropic connection to another:

    {
    "providers": {
    "openai": {
    "strategy": "loadbalance",
    "services": [
    { "name": "openai-us", "weight": 3 },
    { "name": "openai-eu", "weight": 1 }
    ]
    },
    "anthropic": {
    "strategy": "fallback",
    "services": [
    { "name": "anthropic-prod" },
    { "name": "anthropic-backup" }
    ]
    }
    },
    "aliases": {
    "claude-sonnet-5-5": [
    { "provider": "anthropic", "model": "claude-sonnet-5-5" },
    { "provider": "openai", "model": "gpt-5.5" }
    ]
    }
    }

    Each name is the name of a connection on the route. aliases is optional. For details, see Define model aliases.

  5. Select Save.

If the configuration fails a check, Fabric doesn’t save it and shows the reason. For the list, see Fix a configuration that Fabric refuses.

Accept the gateway key from either SDK

SDKs send their API key in their own header: the OpenAI SDK sends Authorization: Bearer <key>, and the Anthropic SDK and Claude Code send x-api-key. To let clients put their gateway key where their SDK already puts it, add a headers list to the route’s Key Auth instance:

  1. Select Plugins, select the Configured tab, and then open the Key Auth instance that’s scoped to your route.

  2. Select Edit as JSON.

  3. Keep the existing fields, and add the headers list:

    {
    "header": "fg-api-key",
    "headers": ["Authorization", "X-API-Key"]
    }
  4. Select Save.

With this list, Key Auth reads the key from the headers on it instead of from fg-api-key, so a caller that sends only fg-api-key gets 401. The list can name any headers, so to accept all three, add fg-api-key to it. A request that carries two different values in these headers gets 401 with conflicting credentials.

On a model router route, the gateway removes every provider credential header the client sent (Authorization, x-api-key, and x-goog-api-key) before it calls the provider, so a client that sends its own provider key instead of a gateway key is refused, not passed through. It also removes the x-gateway-service and x-amr-session-id headers and every header that starts with x-fabric-.

The gateway picks a request’s namespace from the fg-api-key header, and serves a request without that header from the Default namespace. A key that the gateway doesn’t know gets 401, never the Default namespace. So a route that callers reach with their SDK’s own header must be in the Default namespace. For namespaces, see Manage namespaces.

Fix a configuration that Fabric refuses

Fabric checks the configuration when you save the model router, its route, or one of the route’s connections, and refuses a save that breaks any of the following rules:

RuleWhat to fix
The instance is scoped to an LLM route.Set Scope kind to route and pick an LLM route.
providers has at least one group, each group has at least one connection, and each group name is non-empty with no /.Add a connection to the group, or rename or remove it.
Every connection in a group is on the route, is an LLM connection set up for a provider, and is in only one group.Add the connection to the route, or remove it from the other group.
Every connection in a group has an allowed models list, with no empty or repeated entries.Edit the connection’s allowed models.
strategy is fallback or loadbalance. Every connection in a loadbalance group has a positive weight, the weights add up to a number that fits in an integer, and a fallback group has no weight.Fix the strategy or the weights.
The route has no LLM Router or Adaptive Model Router.Remove the other routing plugin.
No two connections on the route serve the same path in different API formats.Move one of the connections to another route.
The instance is at route scope, and its failure mode is the default, which blocks a request the router didn’t handle.If you set on_error through the admin API, remove it or set it to fail_closed.
Every alias name is non-empty, has no /, and isn’t the name of a provider group.Rename the alias.
Every alias has at least one target and no target twice, and each target names a provider group and a model that a connection in that group allows.Fix the target, or allow the model on a connection.

A connection on the route can be in no group. The router never picks it, and it doesn’t appear in the models list. To add a connection, first add it to the route, and then add it to a group in the router’s configuration. To remove one, take it out of its group first, and then out of the route.

Test the route

Replace BASE_URL with your data plane URL followed by the route’s base path, for example https://gateway.example.com/router. To copy it, see Get the base URL. Point the OpenAI SDK at BASE_URL/v1, and the Anthropic SDK at BASE_URL.

To list the models the route serves, send a request to /v1/models with your gateway key. The examples send the key in Authorization. If you didn’t add the headers list to Key Auth, send it in fg-api-key instead.

curl "BASE_URL/v1/models" \
-H "Authorization: Bearer GATEWAY_KEY"

The gateway answers this itself, without calling a provider. The list has one entry for each <provider>/<model> and each alias that the caller can use, for example openai/gpt-5.5 and claude-sonnet-5-5.

For the example configuration, the response looks like this:

{
"object": "list",
"data": [
{ "id": "anthropic/claude-sonnet-5-5", "object": "model", "created": 0, "owned_by": "anthropic" },
{ "id": "claude-sonnet-5-5", "object": "model", "created": 0, "owned_by": "fabric" },
{ "id": "openai/gpt-5.5", "object": "model", "created": 0, "owned_by": "openai" }
]
}

An alias has owned_by set to fabric. If you can use none of the models, data is empty.

Then send a chat completion request that names one of the models:

curl -i "BASE_URL/v1/chat/completions" \
-H "Authorization: Bearer GATEWAY_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "openai/gpt-5.5",
"messages": [{ "role": "user", "content": "Hello!" }]
}'

A successful request returns 200 OK with a chat completion from the provider. The response headers include the following:

HTTP/1.1 200 OK
x-fabric-model: openai/gpt-5.5

The x-fabric-model header names the <provider>/<model> that answered. For the errors a client can get, see Choose a model.

Serve every namespace from a Default route

A model router route in the Default namespace can also serve the other namespaces in your organization, so every team uses one set of provider connections. This happens when the route meets all of the following conditions:

  • It’s an LLM route in the Default namespace.
  • It has an enabled model router instance scoped to the route, with no match conditions.

A request from another namespace reaches this route when that namespace has no route of its own at the request’s path. The namespace’s own route always wins. The request also must not match a more specific Default route that has no model router: if it does, the request gets 404. The request works as follows:

  • Authentication. The caller sends a gateway key from its own namespace in the fg-api-key header. The namespace must authenticate it with a namespace-wide plugin, such as a Key Auth instance with Scope kind set to global. A new namespace doesn’t have one, so create it in that namespace. Without one, the request gets 401. The Default route’s own Key Auth checks only Default callers.
  • Connections. The caller reaches only the route’s Open connections, never its Gated ones. If none of the connections for the model is open, the request gets 403.
  • Plugins. The Default route’s plugins and its connections’ plugins run, except authentication plugins. The namespace’s plugins at global, consumer, and consumer group scope run too, except access control.
  • Usage and limits. Usage is recorded under the namespace that sent the request, with the Default route and connection names. Limits and budgets on the Default route and connections are one pool shared by every namespace that uses the route. Set per-namespace limits in the namespace.

To stop serving other namespaces, remove the model router instance from the route. Requests from other namespaces then get 404.

Next steps