--- title: Tenant Rate Limiting Reference --- # Tenant Rate Limiting Reference Precise contract for per-tenant token/request rate limiting: every configuration knob, the YAML policy schema, and the exact response shape. For how it works conceptually — the reserve/settle model, retry handling, `n>1` accounting — see [Tenant Rate Limiting](../concepts/reliability/tenant-rate-limiting.md). Tenant rate limiting is **disabled by default** and, when enabled, only enforced on the **gRPC router's** Chat, Generate, Completion, Messages, Decisions, Embeddings, Classify, Audio Transcriptions, and Responses endpoints (Harmony-mode Chat and Responses included). The gRPC router serves both gRPC- and [ZMQ-connected](../getting-started/zmq-workers.md) workers (`--connection-mode zmq`, `ipc://` or `tcp://` worker URLs); for every covered endpoint except Decisions, enforcement does not depend on the worker transport. Tenant rate limiting does not apply to the HTTP/external-provider routers. A Responses request reserves **per iteration**: every turn of an [MCP tool loop](../concepts/extensibility/mcp.md) takes a reservation of its own — the estimated input tokens plus a flat `1` from the request counter — and settles against that turn's reported usage, so an *N*-turn loop debits *N* requests from `requests_per_minute`. A request without MCP tools runs a single iteration and is charged like a chat request. Decisions is the one endpoint where the worker transport matters: the gRPC router serves it only on gRPC-connected SGLang workers, and a Decisions request for a model with no such worker is rejected with **501** (`X-SMG-Error-Code: decisions_not_supported`) before any reservation is made. --- ## Enabling it ```bash smg \ --worker-urls grpc://w1:9000 grpc://w2:9000 \ --tenant-rate-limit-enabled \ --tenant-rate-limit-config /etc/smg/tenant-rate-limit.yaml ``` ### CLI flags | Flag | Default | Description | |------|---------|--------------| | `--tenant-rate-limit-enabled` | `false` | Master switch. When unset, no rate limiter is constructed and every request skips reservation entirely. | | `--tenant-rate-limit-config` | unset | Path to the tenant-rate-limit YAML. Required when `--tenant-rate-limit-enabled` is set. | Both flags are accepted by the Rust `smg` binary and the Python launcher (`smg launch` from pip, and the container image) alike; under `smg serve` they take the `--router-` prefix (`--router-tenant-rate-limit-enabled`, `--router-tenant-rate-limit-config`). !!! warning "Fail-closed startup" With `--tenant-rate-limit-enabled`, a missing config path, or a file that is unreadable, unparsable, or fails validation, stops the gateway at startup with an error. An operator who turned the limiter on never gets a gateway that silently runs unlimited. --- ## YAML configuration ```yaml default_policy: tokens_per_minute: 100000 requests_per_minute: 600 tenants: - tenant_key: "auth:team-red" tokens_per_minute: 500000 requests_per_minute: 3000 model_rules: - rule_id: gpt4-cap matcher: type: exact value: gpt-4 tokens_per_minute: 50000 requests_per_minute: 300 - rule_id: legacy-models matcher: type: prefix value: "legacy-" tokens_per_minute: 10000 requests_per_minute: 100 - tenant_key: "anonymous" tokens_per_minute: 5000 requests_per_minute: 60 ``` A tenant not listed under `tenants` uses `default_policy`, with its own buckets: tenants never share a bucket. Tenant keys are the same canonical keys SMG resolves elsewhere in the request path — `auth:`, `header:`, `ip:
`, or `anonymous` — there's no separate tenant-identity system for this feature. A `--tenant-api-key team-red:` caller resolves to `auth:team-red`; a caller using the shared `--api-key` resolves to `auth:` followed by the hex SHA-256 of that key. ### `default_policy` / `tenants[]` fields | Field | Type | Meaning | |-------|------|---------| | `tenant_key` | string | Canonical tenant key. Must be **absent** on `default_policy` and **present** on every entry under `tenants`. | | `tokens_per_minute` | integer, `> 0` | Token-bucket capacity *and* full-minute refill rate for this scope. The bucket starts full, so the first request(s) can burst up to this value immediately. | | `requests_per_minute` | integer, `> 0` | Same shape as `tokens_per_minute`, but for request count. Each admitted reservation debits exactly `1`, regardless of how many tokens it used. | | `model_rules` | list, optional | Per-model overrides layered on top of this scope's own limits. See below. | ### `model_rules[]` fields | Field | Type | Meaning | |-------|------|---------| | `rule_id` | string, `[A-Za-z0-9._-]+` | Stable identifier, unique within the tenant (or `default_policy`). | | `matcher.type` | `exact` \| `prefix` | How `matcher.value` is compared against the request's model ID. | | `matcher.value` | string | The model ID (`exact`) or model ID prefix (`prefix`) this rule applies to. No surrounding whitespace. | | `tokens_per_minute` | integer, `> 0` | Independent token bucket for this rule. | | `requests_per_minute` | integer, `> 0` | Independent request bucket for this rule. | **At most one model rule applies per request.** An exact match wins over any prefix match; the longest matching prefix wins among competing prefixes. Rules never stack with each other — only with the tenant-global limits. When a rule applies, a reservation must be affordable in **both** the tenant-global scope and the rule's scope, or it's denied. ### Validation Checked once, at load, before the gateway ever serves traffic on this config: - `default_policy` must **not** set `tenant_key`; every entry under `tenants` **must**. - Tenant keys must be non-empty, have no surrounding whitespace, be unique across `tenants`, and be a canonical serving-path tenant key (`auth:`- or `header:`-prefixed, `ip:` followed by a valid IP address in its canonical form, or exactly `anonymous`) — a bare ID copy-pasted without its prefix, or an `ip:` key whose address doesn't parse, is rejected rather than silently never matching. - A `header:` tenant key requires `--trust-tenant-header`, since without it no request resolves to a `header:` tenant. - `tokens_per_minute` and `requests_per_minute` must be `> 0`, on every scope (`default_policy`, each tenant, each model rule). - `rule_id` must match `[A-Za-z0-9._-]+`, and be unique within its tenant (or `default_policy`). - `matcher.value` must be non-empty and have no surrounding whitespace. - No two rules within the same tenant may share the same `exact` value, or the same `prefix` value (an `exact` and a `prefix` rule *may* share the same literal string — they're different match kinds). - Unknown YAML fields anywhere in the document are rejected rather than silently ignored (so a typo like `tenant:` instead of `tenants:` fails loudly instead of compiling to an empty override list). Any validation failure is the same as an unparsable file: the gateway refuses to start. --- ## Response codes | Status | Condition | `X-SMG-Error-Code` | Extra headers | |--------|-----------|---------------------|----------------| | **429** Too Many Requests | Reservation denied — the tenant (or matching model rule) doesn't have enough budget right now | `tenant_rate_limit_exceeded` | `Retry-After: ` — present only for a finite wait | The response body is the gateway's standard JSON error envelope: ```json { "error": { "type": "rate_limit_error", "code": "tenant_rate_limit_exceeded", "message": "Tenant rate limit exceeded for this request", "param": null } } ``` !!! tip "When `Retry-After` is absent" If the request's estimated token cost exceeds the scope's *total* capacity — meaning no amount of waiting would ever admit it — the gateway omits `Retry-After` instead of sending an effectively-infinite wait time. Every other denial (the budget is just temporarily exhausted) carries a real `Retry-After` value. --- ## How admission is computed Each affected scope (tenant-global, and the matching model rule if any) is a continuous-refill bucket with two independent counters: - **Capacity** = the configured `tokens_per_minute` / `requests_per_minute` value itself. The bucket starts full. - **Refill rate** = capacity ÷ 60, applied continuously (not in discrete per-minute resets). - **Reserve** debits the estimated input-token count from the token counter and a flat `1` from the request counter, only if *both* counters can currently afford it. A request is denied if *either* can't. - **Settle** applies a signed delta — `(real input tokens + real completion tokens) − estimated tokens` — to the token counter only (an embeddings or classify response has no completion, so its real total is the backend's reported prompt count). A response that used more than estimated can push the counter temporarily negative (debt); the request counter is never trued up, since it was always exactly `1`. --- ## Observability There are **no dedicated Prometheus metrics** for tenant rate-limit decisions — admissions and settlement deltas are not exposed as counters or histograms. Denials surface through two generic counters, each with blind spots on the Responses endpoint: - `smg_http_responses_total` records a denial's `429` — `status_code="429"`, `error_code="tenant_rate_limit_exceeded"` — when the denial is the response the request gets: a denied non-streaming request fails with the `429` even partway through an [MCP tool loop](../concepts/extensibility/mcp.md), and a streaming request denied before its stream starts gets the `429` in place of the stream (for streaming Responses, a first-iteration denial). A denial on a later iteration of a streaming MCP tool loop (regular and Harmony mode alike) comes after the stream already went out as `200`: it ends the stream with an in-stream error event, and this counter records the `200`, not a `429`. - `smg_router_request_errors_total` counts a denial with `error_type="rate_limit_error"` and `reason="rate_limited"` (the gateway's tenant rate limit) under the denied endpoint's label, on every covered endpoint except Responses. The per-iteration Responses reservations record unevenly: a **streaming** Responses request outside Harmony mode runs each iteration through the chat pipeline, so its denials — first iteration and mid-loop alike — count under `endpoint="chat"`, not a Responses label; a **non-streaming** Responses request and a Harmony-mode Responses request (streaming or not) record no sample here at all. A mid-loop denial on a Harmony-mode streaming request therefore appears in neither counter, only as the in-stream error event that ends the stream. Both counters' full label sets are in the [Metrics Reference](metrics.md). --- ## See also
### :material-swap-horizontal: Tenant Rate Limiting Concept The reserve/settle model, retry handling, and `n>1` accounting. [Tenant Rate Limiting →](../concepts/reliability/tenant-rate-limiting.md)
### :material-tray-full: Rate Limiting Concept The concurrency-based limiter this feature complements. [Rate Limiting →](../concepts/reliability/rate-limiting.md)
### :material-cog: Configuration Reference All other gateway CLI flags and configuration options. [Configuration →](configuration.md)