Skip to navigation

LLM Budget plugin

The LLM Budget plugin adds up what LLM requests cost, in US dollars, and compares the total with a limit for the current UTC week or month. With enforcement on, it refuses a request with 429 Too Many Requests once the limit is reached. With enforcement off, it only tracks spending.

Fabric prices each request from a rate card for the provider and model, using the tokens in the response. The count runs after the response is sent, so it adds no delay. The Budget page in the console sets up the organization and namespace budgets for you. Create an instance yourself only for other scopes. For how budgets work, see Control spend with budgets.

Where it runs

ProtocolsLLM routes. The plugin skips MCP and HTTP requests.
Scope kindsThe console’s Budget page creates global instances.
PhaseRequest, to check spending. After the response, or after a failed forward, to count it.
AvailabilityNot available on a standalone gateway. Counting needs the control plane and the data plane’s shared counter store.

A request can match more than one LLM Budget instance, such as an organization budget and a namespace budget. Each instance counts the request, and the gateway refuses it when any enforced budget is at its limit.

The plugin needs an active LLM Observability instance, because it reads the token counts that plugin records. Without one, an enforced budget refuses every request with 503.

Set up the plugin

For an organization or namespace budget, use the Budget page instead of the steps here. For the steps, see Set a budget. Use the steps on this page to attach a budget to another scope, such as one route.

Prerequisites

Before you begin, make sure you have the following:

  • An active LLM Observability instance. For the steps, see LLM Observability.
  • A control plane deployment. The plugin isn’t available on a standalone gateway.
  • Permission to create plugin instances.
  • For a budget per gateway key, Key Auth on the route, so each request carries a gateway key. For the steps, see Authenticate requests with Key Auth.

Steps

  1. In the console sidebar, select Plugins, and then open the All tab.
  2. Find LLM Budget, and then select Use. The New plugin instance form opens.
  3. In Apply to, select the scope. If you select a scope other than Global, select the target in the field below it.
  4. In Instance name, enter a name, for example team-monthly-budget.
  5. Under Configuration, in Spending limit (USD), enter the limit, as a positive amount with up to 6 decimal places, for example 100.00.
  6. For Budget period, select Weekly or Monthly. A weekly budget resets at 00:00 UTC on Monday. A monthly budget resets at 00:00 UTC on the first of the month.
  7. To keep counting without refusing requests, clear Stop requests at the limit. The setting is selected by default.
  8. (Optional) To give each gateway key its own limit of this amount, select Separate budget per gateway key.
  9. (Optional) Expand Match conditions (optional) to limit when the instance runs. The conditions are Path prefix, Methods, Consumer, and Consumer group. All conditions must match.
  10. Confirm that Enabled is selected. Fabric saves a disabled instance but doesn’t run it.
  11. Select Create instance. The form doesn’t save until you enter a spending limit.

Verify it works

To check that the budget counts spending, do the following:

  1. Send a request through an LLM route that the budget covers.

  2. Open the Budget page, and check that the amount spent goes up. The page can lag a little, because counting runs after the response.

  3. To check enforcement, set Spending limit (USD) to a small amount, send requests until the budget is used, and then send one more. The gateway returns 429 with this body, and doesn’t send the request to the provider:

    { "error": { "type": "llm_budget", "message": "LLM budget exhausted" } }

Fix common problems

ProblemCauseFix
Every request returns 503 with “budget requires active llm-observability”.The route has no active LLM Observability instance.Add one, and make sure Enabled is selected.
Every request returns 503 with “budget requires an opaque credential ID”.Separate budget per gateway key is on, and the request has no gateway key.Require a gateway key on the route with Key Auth, or clear the setting.
Requests return 503 with “budget accounting unavailable”.The gateway can’t read spending from its counter store, or its accounting isn’t ready.Retry. If it persists, check the data plane’s health.
Spending doesn’t go up.The gateway has no price for the model, or the response had no token counts.Check the data plane’s logs for “budget pricing unavailable”. The gateway logs the request and doesn’t count it.
The budget never refuses a request.Stop requests at the limit is cleared, so the budget only tracks spending.Select Stop requests at the limit.

Configuration

An instance has these fields. The console shows the first four. The rest are set by the Budget page, and the console form keeps them when you edit an instance.

limit_usd_micros
integerRequired

The spending limit in millionths of a US dollar, from 1 to 9,007,199,254,740,991. The console and the Budget page enforce that range. A plugin instance created another way only needs a positive value. For example, 100000000 is $100. The console labels this Spending limit (USD) and converts the amount for you.

period
stringRequired

week or month, on the UTC calendar. A week starts on Monday. The console labels this Budget period, with the options Weekly and Monthly.

mode
stringRequired

total keeps one balance for everything that the instance covers. per_key keeps a separate balance for each gateway key, and refuses requests with no key when enforcement is on. The console labels this Separate budget per gateway key.

enforce
booleanDefaults to true

When true, the gateway refuses requests once the limit is reached. When false, the budget only tracks spending, and it never refuses a request because the limit is used up or because counting is unavailable. The console labels this Stop requests at the limit.

partition_by
object

Splits a per_key budget into one balance for each namespace or each route, with {"source": "namespace"} or {"source": "route"}. The Budget page sets this for the organization and namespace budgets.

limit_overrides
array of objects

Up to 100 exceptions to the limit, for per_key budgets. Each item has limit_usd_micros and one selector: credential_id without a partition, namespace_id for a namespace partition, or route_name for a route partition. The Budget page manages these as allowances.

initial_usage
object

Spending to carry over when a total budget is created partway through a period. The control plane sets it at creation, and it can’t change afterward.

Responses

StatusCause
429An enforced budget is at its limit. The body is {"error":{"type":"llm_budget","message":"LLM budget exhausted"}}, with no Retry-After header.
503An enforced budget can’t check spending: the LLM Observability instance is missing, the gateway can’t read the counters, the budget needs a gateway key and has none, or the route can’t be resolved for a route partition.

The request that takes spending over the limit still completes, so spending can end slightly above the limit. Without enforcement, a missing LLM Observability instance, missing counters, or a missing price only skips counting. The gateway logs each case and serves the request.

The gateway also returns 503 with budget configuration invalid when an instance’s saved configuration doesn’t parse, whether enforcement is on or off. The console checks the configuration when you save it, so this is rare.

Example

{
"name": "team-monthly-budget",
"plugin_id": "llm-budget",
"scope": { "kind": "route", "name": "openai-prod" },
"config": {
"limit_usd_micros": 100000000,
"period": "month",
"mode": "total",
"enforce": true
}
}