Skip to navigation

Limits and defaults

Fabric AI Gateway applies fixed size caps to the requests and responses it reads, and default timeouts and retry settings to the upstreams it calls. This page lists those values. Limits that you configure yourself, such as rate limits, token budgets, and budgets, are on their own pages: Rate limits and Budgets.

Request and response size

WhatLimitWhat happens above it
Request headers1 MiBThe request is refused with 431 before the gateway handles it.
MCP request body8 MiBRefused with 413 mcp_body_too_large.
LLM request body that the gateway reads10 MiBThe gateway doesn’t read the body, so it doesn’t know the model. If the connection allows only some models, the request is refused with 400 Could not determine a model. Otherwise it’s forwarded unread.
LLM request body on a route with the llm-model-router plugin10 MiB (10,485,760 bytes)Refused with 413 request body too large. A body of exactly 10 MiB is accepted.
Request body kept for retries and failover10 MiBLarger bodies are still sent to the upstream, but the gateway doesn’t retry them or fail over.
REST request body that a plugin reads1 MiBThe plugin runs without the body. A plugin that needs the body and is set to fail closed fails, and the client gets 502 policy plugin failure.
REST response held for transformation or validation1 MiBThe response is passed on unchanged, and the response transformation and validation are skipped.
LLM response read for token counting10 MiBThe client still gets the whole response. Usage beyond the cap isn’t read.
First event of an LLM stream1 MiBThe gateway holds the first event until it’s complete, so that it can still fail over if the event is an error.

The client always receives the full response body. The caps only limit what the gateway keeps in memory.

Timeouts

Each connection can set its own timeouts. When it doesn’t, the gateway uses these defaults:

TimeoutDefaultNotes
Connect to the upstream15 secondsA connection’s connect timeout can only shorten it.
TLS handshake with the upstream10 secondsReplaced by the connection’s connect timeout when one is set.
Wait for response headersNoneSet a read timeout on the connection to limit it. When it fires, the client gets 504.
Gap between reads of the response bodyNoneThe read timeout also applies here. An upstream that keeps sending, such as a live event stream, is never cut off by it.
Upload from the clientNoneSet a write timeout on the connection to limit how long an upload may stall. When it fires, the client gets 408.
Whole attemptNoneThe connection’s overall timeout covers one attempt, including its retries and the whole response body. When it fires before a response starts, the client gets 502 on REST routes and 504 on LLM routes.

The connect, read, and write timeouts each accept 0 to 600,000 milliseconds. A value of 0 means not set.

Retries and failover

SettingDefault
Retries against the same upstreamNone. A retry policy on the connection turns them on.
What a retry policy retriesNetwork errors, including timeouts and resets. A policy can also retry 5xx answers or resets only.
Wait between retriesStarts at 100 ms and doubles each time, up to 10 seconds.
Requests with signed credentials, such as AWSAlways get a second try with fresh credentials after a 401 or 403.
Status codes that trigger LLM Router failover429 and 503, plus 500, 502, and 504 when nonidempotent failover is on. For LLM routes it’s on by default.
Retry after the response startsNever. Once the gateway sends response headers, it doesn’t retry or fail over.

For how to configure failover, see Fail over across providers with the LLM Router.

Upstream health

Health checks and the circuit breaker run only on connections that configure them. Without them, the gateway doesn’t take a failing upstream out of rotation. When a connection configures them without setting these values, the gateway uses these defaults:

SettingDefault
Failures in a row before an upstream is marked unhealthy3
Successes in a row before it’s healthy again2
Active health check interval, when a check path is set10 seconds
Active health check timeout5 seconds
Failures in a row before the circuit breaker opens5
Time the circuit stays open30 seconds. After that, one trial request is let through.

Connections to upstreams

SettingValue
Idle connections closed after90 seconds
Idle connections kept per upstream host10
Minimum TLS version to upstreamsTLS 1.2

Streaming

WhatValue
Server-sent events and AWS event streamsPassed on event by event, as soon as each one is complete.
Newline-delimited JSON streams on LLM routesPassed on as the bytes arrive, in chunks of up to 32 KiB.
MCP notification streams per caller10 on each data plane instance. An 11th gets a stream with a 30-second retry hint, and is then closed. On revision 2026-07-28 (subscriptions/listen), the 11th request gets HTTP 503 with JSON-RPC -32603 and Retry-After: 30.
Keep-alive on MCP notification streamsA comment every 30 seconds.

MCP tool lists

WhatValue
Timeout for each MCP server when the gateway builds tools/list30 seconds. A server that doesn’t answer is left out of the list.
Tool description lengthCut to 4,096 bytes.

Other limits

WhatValue
X-Request-Id that the gateway accepts from the client256 characters. A longer one is replaced with a generated ID.
Adaptive Model Router session ID256 characters. Pins last 24 hours from the last use.

Next steps