Skip to navigation

Supported endpoints

An LLM route can expose only the endpoints that its providers offer. Each provider connection declares its endpoints, and the gateway sends a request only to a provider that declares the endpoint you called, in the format you sent. The gateway doesn’t translate between APIs, so an OpenAI-format request never reaches an Anthropic-format endpoint. This page shows which endpoints each kind of provider supports, and what the gateway does with each one.

The exact list for each provider appears in the console. When you connect a provider, Endpoints lists the endpoints that it offers. For the steps, see Connect an LLM provider.

Endpoints by provider

EndpointOpenAIAnthropicOpenAI-compatible serversAmazon Bedrock
POST /v1/chat/completionsYesNoYesVaries
POST /v1/responsesYesNoVariesNo
POST /v1/embeddingsYesNoVariesNo
GET /v1/modelsYesYesYesNo
GET /v1/models/{model}YesVariesVariesNo
POST /v1/messagesNoYesNoNo
POST /v1/messages/count_tokensNoYesNoNo
POST /model/{model_id}/invokeNoNoNoYes
POST /model/{model_id}/invoke-with-response-streamNoNoNoYes

Varies means it depends on the provider or on how the connection is set up. Check Endpoints for the connection.

To use Claude models from an OpenAI-compatible client, connect a provider whose connection declares an OpenAI-compatible chat endpoint. Otherwise, use /v1/messages with an Anthropic client.

What the gateway does with each endpoint

Endpoint familyRequest formatWhere the gateway reads the modelStreamingToken usage
OpenAI Chat CompletionsOpenAI chatmodel in the JSON bodyServer-sent events, with "stream": trueRead from the response
OpenAI ResponsesOpenAI responsesmodel in the JSON bodyServer-sent events, with "stream": trueRead from the response
Anthropic MessagesAnthropic messagesmodel in the JSON bodyServer-sent events, with "stream": trueRead from the response
Embeddings and other OpenAI-compatible endpointsPassed throughmodel in the JSON body, where presentAs the provider sends itRead from the response in the provider’s format, where present
GET /v1/modelsNo bodyNoneNoneNone
Bedrock InvokeModelAnthropic messagesThe {model_id} path segmentAWS event stream, on the streaming endpointRead from the response

Notes:

  • Model checks. If a connection allows only some models, the gateway checks the model on every endpoint that names one. A request whose model the gateway can’t read on such an endpoint is refused. For the errors, see Gateway error codes.
  • OpenAI chat streams. OpenAI includes token usage in a chat stream only when the request sets stream_options.include_usage to true. The gateway doesn’t add it for you, so without it a streamed chat request records no tokens.
  • Token counting. The gateway records token usage only when your organization has LLM observability turned on. For details, see Monitor gateway traffic.

Routes with several providers

A route can have several provider connections. On such a route:

  • A routing plugin, such as the LLM Router, chooses which connections to try and in what order. Without one, the request must name a connection in the x-gateway-service header.
  • The gateway accepts the request if any connection on the route declares the endpoint.
  • Each fallback attempt uses its own connection’s endpoint and credentials, and skips connections that don’t declare the endpoint or don’t allow the model.
  • All connections that declare the same path on a route must use the same API format. Fabric refuses to save a route that mixes them.

On a route with the llm-model-router plugin, GET /v1/models and GET /v1/models/{model} are answered by the gateway from the models that the route’s connections allow, not by a provider. For details, see Route requests by model name.

Next steps