Skip to navigation

Use the OpenAI-compatible API

An LLM route accepts the OpenAI API, so the OpenAI SDK and other OpenAI-compatible clients work with the gateway by changing their base URL. The gateway reads your request to learn the model and whether you asked for a stream, sends the same request to the provider, and relays the provider’s answer. It doesn’t translate between APIs: an OpenAI-format request goes only to a provider endpoint in the OpenAI format. This topic lists the endpoints, shows how to point an SDK at the gateway, and explains streaming, headers, and errors. To send your first request, see Authenticate and send a request.

Prerequisites

Before you begin, make sure you have the following:

Find the endpoints

Append the endpoint path to the route’s base URL. The following endpoints use the OpenAI format:

Method and pathWhat it does
POST /v1/chat/completionsChat completions.
POST /v1/responsesResponses.
POST /v1/embeddingsEmbeddings.
GET /v1/modelsThe model list. On a model router route, the gateway answers it.
GET /v1/models/<model>One model from the list. On a model router route, the gateway answers it.

A route serves an endpoint when its provider does. On a model router route, a POST endpoint is available when at least one of the route’s providers serves it, and GET /v1/models is always available. A model router route doesn’t serve /v1/completions, images, audio, files, batches, or paths without the /v1 prefix. A path that the route doesn’t serve gets 404 with unknown LLM endpoint.

Point the OpenAI SDK at the gateway

Set the SDK’s base URL to the route’s base URL, followed by /v1, and send your gateway key in the fg-api-key header. For a complete example in Python and JavaScript, see Use the OpenAI SDK with Fabric AI Gateway.

Some model router routes read the gateway key from the SDK’s own header instead of fg-api-key. On those routes, pass the gateway key as api_key and leave out default_headers. The SDK then sends it as Authorization: Bearer GATEWAY_KEY. Ask your admin which header your route reads. For details, see Accept the gateway key from either SDK.

Stream a response

To stream, set "stream": true in the request. The gateway relays the provider’s server-sent events as they arrive, unchanged. It holds the first event until it’s complete, so that if the provider fails before that event, the gateway can still try another connection.

  • Before the stream starts. If the request fails before the provider answers, you get a JSON error, not a stream, as you would from the provider.
  • After the first event. The gateway doesn’t retry or fail over. If the connection breaks, the stream ends where it stopped, without a final [DONE] event.

A streamed chat completion carries token usage only if the request sets "stream_options": {"include_usage": true}. The gateway doesn’t add this option, so without it, the request counts as zero tokens in analytics, budgets, and token rate limits.

Send headers

The gateway passes your headers to the provider, with these exceptions:

  • It removes the header that carried your gateway key, and adds the provider’s credential.
  • It removes Accept-Encoding, so responses are never compressed.

Provider headers such as OpenAI-Organization reach the provider unchanged. The response has the provider’s status and headers, plus X-Request-Id. On a model router route, it also has x-fabric-model, which names the <provider>/<model> that answered.

Understand errors

Errors that the gateway writes on an OpenAI-format endpoint use the OpenAI error envelope, so the SDK can parse them:

{
"error": {
"message": "model `openai/gpt-9` not found",
"type": "invalid_request_error",
"code": "model_not_found"
}
}

The response has Content-Type: application/json. Errors from the provider come back exactly as the provider sent them.

StatusWhen
400, 403, 404, 413The gateway can’t serve the model you named.
404The route doesn’t serve the endpoint. The message is unknown LLM endpoint.
502The last connection the gateway tried failed. The type is server_error.
503No connection was available. The type is server_error.
504The last connection timed out. The type is server_error.

Some errors from other plugins aren’t in this envelope. A 401 from Key Auth is plain text, Unauthorized or conflicting credentials, and other plugin errors, such as a 429 from a rate limit, can be plain text too. Check the status code before you parse the body. For authentication errors, see Troubleshoot authentication errors.

On a model router route, a Responses request that carries previous_response_id goes only to the first connection the gateway picks, with no failover, because the stored response lives at the provider account that created it. To keep failover, send the full input with "store": false.

Next steps