Limit requests and tokens
Rate limits protect your providers and your spend from a single caller using too much. Fabric AI Gateway has two rate-limiting plugins in the Traffic category. Rate Limit counts requests on any route. LLM Rate Limit counts the LLM tokens that callers use. This topic shows you how to configure each one and what a client gets when it reaches a limit.
Prerequisites
Before you begin, make sure you have the following:
- A route with authentication, such as Key Auth, so the gateway can tell callers apart. For the steps, see Authenticate requests with Key Auth.
- Permission to create plugin instances.
Limit requests per caller
The Rate Limit plugin limits requests with a throttling algorithm (GCRA, the generic cell rate algorithm) instead of counting them in a rolling window. It lets a caller send up to the Limit plus one request at once, and then makes room again at an even rate, which is the Limit divided by the window. It refuses requests once the caller is out of room. It attaches to a route only.
To create a Rate Limit instance, do the following:
-
Select Plugins, find Rate Limit in the Traffic category, and select Use.
-
Enter a Name, for example
rate-limit-chat. Set Scope kind torouteand Scope name to the route. -
Configure the following settings:
-
Select Save.
A route can have one Rate Limit instance for each Count by value. For example, you can limit each consumer to 100 requests a minute and the whole route to 5,000 requests a minute. Every instance counts each request, and the first one that’s out of room refuses it.
Counts carry over when you change Limit or Window (seconds). You can’t rename an instance after you create it. Changing Count by or renaming the route starts every counter from zero.
Limit LLM tokens per caller
The LLM Rate Limit plugin gives each caller a token budget per window. After each LLM response, it adds the tokens that the model used to the caller’s count. Before each request, it checks the tokens already used in the window and refuses the request if the count has reached the budget. It counts only LLM requests and lets other traffic through.
You can attach LLM Rate Limit to a connection, a route, or a consumer. If more than one instance applies to a request, only the one at the narrowest scope runs, so a consumer limit replaces a route limit for that consumer.
To create an LLM Rate Limit instance, do the following:
-
Select Plugins, find LLM Rate Limit in the Traffic category, and select Use.
-
Enter a Name, and set Scope kind and Scope name to the connection, route, or consumer that you want to limit.
-
Configure the following settings:
-
Select Save.
Each caller has a separate count on each route. A consumer with a budget of 100,000 tokens can use 100,000 tokens on each route they call. All anonymous callers of a route share one count.
The plugin counts the tokens that the provider reports. If the provider reports none, the request isn’t counted. For OpenAI streamed chat completions, that means the client must set "stream_options": {"include_usage": true}.
LLM Rate Limit charges the tokens that the LLM Observability plugin records, so your organization needs a global LLM Observability instance. The control plane doesn’t create one for you, and without it the budget never refuses a request. After an Adaptive Model Router cascade, it charges the total of every attempt, whether or not LLM Observability is present.
Responses when a limit is reached
When a request is refused, the client gets 429 Too Many Requests, and the gateway doesn’t send the request to the provider. A refused request doesn’t count against the limit.
-
Rate Limit: The response carries
Retry-Afterand the limit headers only when the instance’sshow_headerssetting is on. The console form doesn’t offer this setting yet. For the setting, see the Rate Limit reference. -
LLM Rate Limit: The response always carries
Retry-After, the number of seconds until the caller’s token count falls below the budget.
Clients should wait for the Retry-After interval before they retry. For more about handling errors, see Handle errors and retries.
If you run your own data plane with several replicas, each replica keeps its own counts unless the data plane uses a shared Redis counter store. Without one, a caller can use up to the limit on each replica. For how to set it up, see Scale and high availability.
Verify the limit
Send requests to the route until you reach the limit. The next request returns 429. With LLM Rate Limit, check the caller’s token usage in Analytics.

