---
title: OpenAI-Compatible API
---

# OpenAI-Compatible API Reference

SMG provides a fully OpenAI-compatible API, allowing you to use existing OpenAI client libraries with your self-hosted inference workers.

---

## Base URL

```
http://localhost:30000/v1
```

---

## Authentication

SMG supports optional API key authentication:

```bash
curl http://localhost:30000/v1/chat/completions \
  -H "Authorization: Bearer your-api-key" \
  -H "Content-Type: application/json" \
  -d '...'
```

Enable authentication with `--api-key`:

```bash
smg launch --worker-urls http://worker:8000 --api-key "your-api-key"
```

For real multi-tenant separation (e.g. per-tenant rate limiting), configure one key per
tenant instead with `--tenant-api-key tenant_id:key` (repeatable). Each key resolves to
its own tenant identity; `--api-key` remains available as a single shared fallback key
whose callers all share one identity.

```bash
smg launch --worker-urls http://worker:8000 \
  --tenant-api-key team-red:red-secret \
  --tenant-api-key team-blue:blue-secret
```

!!! note "Rust binary only"
    `--tenant-api-key` is a flag of the Rust `smg` binary (`cargo install smg` or a source build). The Python launcher behind `pip install smg` and the container images does not accept it.

Callers authenticate the same way as with `--api-key` — a `Bearer` token in the
`Authorization` header — just using their own tenant's key instead of the shared one:

```bash
curl http://localhost:30000/v1/chat/completions \
  -H "Authorization: Bearer red-secret" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
```

This request is attributed to `team-red`, distinct from a request authenticated with
`blue-secret`. A missing key, or one that doesn't match any configured `--api-key` or
`--tenant-api-key` value, is rejected with `401 Unauthorized` and an empty body. Tenant
keys authenticate serving endpoints only — `/v1/chat/completions`, `/v1/completions`,
`/v1/embeddings`, etc. They do not grant access to admin/management routes (`/workers`,
`/flush_cache`, and similar). When control-plane authentication is enabled, those
routes require a control-plane credential with the `admin` role. Otherwise, they
accept the shared `--api-key` if configured. With neither control-plane auth nor any API keys configured,
those routes are open; with only `--tenant-api-key` configured, every admin request
is rejected with `401`.

`GET /v1/models` is a public route and answers without a key (see [List Models](#list-models)).

---

## Endpoints

### Chat Completions

Create a chat completion.

```
POST /v1/chat/completions
```

#### Request Body

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `model` | string | Yes | Model identifier. A name configured with `--model-alias` is accepted and resolved to its canonical model |
| `messages` | array | Yes | Array of message objects; must not be empty |
| `max_completion_tokens` | integer | No | Upper bound on generated completion tokens (at least 1) |
| `max_tokens` | integer | No | Deprecated — use `max_completion_tokens`. Still accepted and migrated when `max_completion_tokens` is unset |
| `temperature` | number | No | Sampling temperature (0-2) |
| `top_p` | number | No | Nucleus sampling parameter, greater than 0 and at most 1 |
| `n` | integer | No | Number of completions to generate (1-10) |
| `stream` | boolean | No | Enable streaming responses |
| `stream_options` | object | No | Streaming options; only allowed with `stream: true`. See [Stream Usage](#stream-usage) |
| `stop` | string/array | No | Up to 4 non-empty stop sequences |
| `presence_penalty` | number | No | Presence penalty (-2 to 2) |
| `frequency_penalty` | number | No | Frequency penalty (-2 to 2) |
| `logprobs` | boolean | No | Return log probabilities of the output tokens |
| `top_logprobs` | integer | No | Most likely tokens to return per position (0-20); requires `logprobs: true` |
| `logit_bias` | object | No | Map of token ID to bias |
| `tools` | array | No | Function tools the model may call |
| `tool_choice` | string/object | No | `none`, `auto`, `required`, `{"type": "function", "function": {"name": "..."}}`, or `{"type": "allowed_tools", "mode": "auto", "tools": [...]}` (`mode` is `auto` or `required`). Every value except `none` and `auto` requires tools, and a named function must exist in them |
| `parallel_tool_calls` | boolean | No | Allow parallel function calls |
| `response_format` | object | No | `text`, `json_object`, or `json_schema`. See [Structured Output](#structured-output) |
| `reasoning_effort` | string/number | No | Reasoning effort; a number is converted to its decimal string. See [Thinking and Reasoning Effort](#thinking-and-reasoning-effort) |
| `thinking` | object | No | Typed thinking control (`type`, `effort`, `keep`, `clear_thinking`). See [Thinking and Reasoning Effort](#thinking-and-reasoning-effort) |
| `user` | string | No | End-user identifier |

#### Extension Fields

SMG also accepts these SGLang-style extensions on Chat Completions:

| Field | Type | Description |
|-------|------|-------------|
| `top_k` | integer | Top-k sampling; `-1` disables it, otherwise at least 1 |
| `min_p` | number | Min-p sampling threshold (0-1) |
| `min_tokens` | integer | Minimum tokens to generate; must not exceed `max_completion_tokens` |
| `repetition_penalty` | number | Repetition penalty (0-2) |
| `regex`, `ebnf` | string | Constrain the output. Only one of `regex`, `ebnf`, and a `json_schema` response format may be set, and neither combines with a JSON `response_format` |
| `stop_token_ids` | array | Token IDs that stop generation |
| `ignore_eos`, `no_stop_trim` | boolean | Keep generating past EOS; keep the stop sequence in the output |
| `skip_special_tokens` | boolean | Drop special tokens when detokenizing (default `true`) |
| `continue_final_message` | boolean | Continue the trailing assistant message instead of starting a new one |
| `separate_reasoning`, `stream_reasoning` | boolean | Return reasoning in `reasoning_content` and stream it (both default `true`) |
| `chat_template_kwargs` | object | Extra chat-template variables. On gRPC workers an explicit entry wins over the `reasoning_effort`, `tool_choice`, and `response_format` values SMG passes to the template |
| `lora_path` | string | LoRA adapter to apply |
| `sampling_seed` | integer | Seed for deterministic sampling |
| `return_hidden_states` | boolean | Return the model's hidden states |
| `rid` | string | Request ID forwarded to the backend |

Fields SMG does not model are kept and forwarded unchanged to HTTP workers, and so are
unknown `stream_options` keys such as SGLang's `step_usage_chunks`. The SGLang-native
`/generate` endpoint likewise keeps its nested `sampling_params.custom_params` object.
Typed sampling numbers are forwarded without float widening: `"top_p": 0.95` reaches
HTTP workers, PD legs, and external providers as `0.95`, not `0.949999988079071`.

#### Message Object

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `role` | string | Yes | `system`, `developer`, `user`, `assistant`, `tool`, `function`, or `root` (MiniMax only; see [Provider Profiles](#provider-profiles)) |
| `content` | string/array | Yes | Text, or an array of [content parts](#content-parts). Optional on `system` and `assistant` messages; must not be empty on `user` messages |
| `name` | string | No | Participant name |
| `tool_calls` | array | No | Tool calls made by an `assistant` message |
| `tool_call_id` | string | For `tool` | ID of the tool call this `tool` message answers |
| `reasoning_content` | string | No | Reasoning from an earlier `assistant` turn; `reasoning` is accepted as an alias |
| `tools` | array | No | Kimi only: tools declared on a `system` or `developer` message |

#### Content Parts

| `type` | Payload | Notes |
|--------|---------|-------|
| `text` | `text` | |
| `image_url` | `image_url.url`, optional `detail` | MiniMax-M3 also takes `image_url.max_long_side_pixel` |
| `video_url` | `video_url.url` | MiniMax-M3 also takes `video_url.fps` and `video_url.max_long_side_pixel` |
| `input_audio` | `input_audio.data` (base64), `input_audio.format` | |
| `audio_url` | `audio_url.url` | |
| Anything else | Forwarded as sent | HTTP workers only |

A content part whose `type` SMG does not model, or a known type with a malformed payload,
is forwarded unchanged to HTTP workers. gRPC workers reject it with `400`
`unsupported_content_part`, and the gpt-oss (Harmony) path with `400`
`harmony_build_failed`. Which workers accept which media is covered in
[Multimodal](../../concepts/architecture/multimodal.md).

#### Thinking and Reasoning Effort

Two fields control reasoning. `reasoning_effort` is OpenAI's effort level. `thinking` is
a typed object shared by the Kimi, MiniMax, z.ai, and DeepSeek dialects:

| Field | Values | Meaning |
|-------|--------|---------|
| `type` | `enabled`, `disabled`, `adaptive` | Thinking toggle. `adaptive` states no preference; any other value is rejected |
| `effort` | string | Effort level; takes precedence over `reasoning_effort` |
| `keep` | string | Kimi `keep` mode; accepted and ignored by SMG's renderers |
| `clear_thinking` | boolean | z.ai: whether earlier turns' reasoning is dropped from the prompt |

The effective effort is `thinking.effort` when present, otherwise `reasoning_effort`.
Leaving `thinking` out changes nothing: the chat template's own default applies (Kimi K3,
for example, thinks by default), subject to `reasoning_effort`. The exception is the
DeepSeek V4 profile, which turns thinking on unless the request turns it off (see
[Defaults and Rewrites](#defaults-and-rewrites)). A `thinking` object without `type`
expresses no toggle, but its `effort` still counts.

On gRPC workers SMG renders the chat template itself. The template receives the effective
effort as its `reasoning_effort` variable, and SMG decides whether the model starts in
reasoning mode from the first of these that applies:

1. an explicit thinking toggle in `chat_template_kwargs`;
2. for templates with their own effort levels, the effort (`chat_template_kwargs.reasoning_effort`, else the effective effort): `none` and `minimal` turn thinking off, a native level turns it on;
3. `thinking.type` (`enabled` on, `disabled` off);
4. the effective effort, where `none` and `minimal` turn thinking off.

If none applies, the template default stands. The template and the reasoning parser use
the same decision. On gpt-oss (Harmony) models SMG reads `reasoning_effort` and clamps it to
the three levels Harmony knows: `none` and `minimal` become `low`, `xhigh` and `max`
become `high`, and an unknown value becomes `medium`. HTTP workers receive both fields as
sent unless a [provider profile](#provider-profiles) rewrites them, and provider profiles
restrict the accepted values per model family.

#### Structured Output

`response_format` accepts:

- `{"type": "text"}`
- `{"type": "json_object"}`
- `{"type": "json_schema", "json_schema": {"name": "...", "schema": {...}, "strict": true}}`

For `json_schema`, `name` must not be empty and `schema` must be a JSON object (`{}` is
allowed). A string, array, boolean, number, or `null` schema is rejected with `400` before
the request reaches any worker, on every backend, unless the HTTP router streams the request
to the worker unparsed (see [Request Streaming](../../concepts/performance/request-streaming.md)).

#### Example Request

```bash
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "max_tokens": 100,
    "temperature": 0.7
  }'
```

#### Response

```json
{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1705312345,
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 8,
    "total_tokens": 33
  }
}
```

#### Usage

| Field | Description |
|-------|-------------|
| `prompt_tokens`, `completion_tokens`, `total_tokens` | Token counts. With `n` > 1 the shared prompt is counted once and the choices' completions are summed |
| `prompt_tokens_details.cached_tokens` | Prompt tokens served from the worker's prefix cache |
| `completion_tokens_details.reasoning_tokens` | Reasoning tokens, when the count is non-zero. On gRPC workers SGLang reports them, and SMG counts them itself for gpt-oss (Harmony) models |
| `completion_tokens_details.accepted_prediction_tokens` | Speculative decoding: draft tokens the target model accepted |
| `completion_tokens_details.rejected_prediction_tokens` | Speculative decoding: draft tokens rejected (drafted minus accepted) |

On gRPC workers SMG builds `usage` from the engine's counters; HTTP workers return the
engine's own usage object. The speculative-decoding fields appear only when the engine
reports proposed draft tokens. SMG reads them from SGLang, TokenSpeed, vLLM (started with
`--per-request-spec-decode-metrics`), and TensorRT-LLM (once its servicer fills the
field); MLX reports none.

#### Streaming Response

With `"stream": true`, responses are sent as Server-Sent Events:

```bash
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'
```

Response (abbreviated):

```
data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{"content":"Hello!"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[],"usage":{"prompt_tokens":9,"completion_tokens":2,"total_tokens":11}}

data: [DONE]
```

An error before the first chunk is an ordinary JSON error response with its own status;
when a worker rejects a streaming request, SMG keeps the worker's content type
(`application/json`, not `text/event-stream`) so SSE clients still see the error body.
If generation fails after a gRPC stream has started, SMG sends
`data: {"error": {"message": "...", "type": "internal_error"}}` followed by `data: [DONE]`.

Starting with v1.13.0, a quiet open SSE response can contain `: ping` comments
between events. SMG sends them after 15 seconds of silence by default, including
while waiting for the first token. SSE clients should ignore comment lines.
Configure or disable them with
[`--sse-keepalive-secs`](../configuration.md#streaming-keepalives).

#### Stream Usage

| `stream_options` field | Default | Effect |
|------------------------|---------|--------|
| `include_usage` | `false` (Kimi: `true`) | Send the whole request's usage in a final chunk with an empty `choices` array |
| `continuous_usage_stats` | `false` | gRPC workers, except for gpt-oss (Harmony) models: with `include_usage: true`, every chunk also carries a running `usage` snapshot (the prompt plus the tokens generated so far) |

On HTTP workers `stream_options`, including keys SMG does not model, is forwarded to the
engine, which produces the usage chunks. The Kimi profile turns `include_usage` on for
streaming requests that leave it unset, on every backend; an explicit `false` is kept.

The DeepSeek V4 profile uses DeepSeek's stream format on gRPC workers instead: aggregate
usage rides on the final finish chunk whether or not `include_usage` is set, and no
usage-only chunk follows. With `include_usage: true`, earlier chunks carry
`"usage": null`. `continuous_usage_stats` does not apply, and if the backend stream ends
before every choice completes, the finish chunk carries no aggregate usage.

---

### Completions

Create a text completion (legacy API).

```
POST /v1/completions
```

#### Request Body

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `model` | string | Yes | Model identifier |
| `prompt` | string/array | Yes | Text prompt, or a non-empty array of prompts |
| `max_tokens` | integer | No | Maximum tokens to generate |
| `temperature` | number | No | Sampling temperature (0-2) |
| `top_p` | number | No | Nucleus sampling parameter, greater than 0 and at most 1 |
| `n` | integer | No | Completions per prompt (1-128) |
| `best_of` | integer | No | Candidates to generate server-side (0-20); must be greater than `n` and cannot be combined with `stream` |
| `logprobs` | integer | No | Log probabilities of the most likely tokens (0-5) |
| `stream` | boolean | No | Enable streaming |
| `stream_options` | object | No | `include_usage`; only allowed with `stream: true` |
| `stop` | string/array | No | Up to 4 non-empty stop sequences |
| `echo` | boolean | No | Echo prompt in response |
| `suffix` | string | No | Text that comes after the completion |
| `presence_penalty`, `frequency_penalty` | number | No | Penalties (-2 to 2) |
| `seed` | integer | No | Seed for best-effort deterministic sampling |
| `user` | string | No | End-user identifier |

Completions takes the same [extension fields](#extension-fields) as Chat Completions except
the chat-only ones (`continue_final_message`, `separate_reasoning`, `stream_reasoning`,
`chat_template_kwargs`), plus a `json_schema` string constraint. On gRPC workers each
prompt in an array is its own engine request.

#### Example Request

```bash
curl http://localhost:30000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "prompt": "The quick brown fox",
    "max_tokens": 50
  }'
```

#### Response

```json
{
  "id": "cmpl-abc123",
  "object": "text_completion",
  "created": 1705312345,
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "choices": [
    {
      "text": " jumps over the lazy dog.",
      "index": 0,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 4,
    "completion_tokens": 7,
    "total_tokens": 11
  }
}
```

---

### Decisions

Ask predicate (yes/no), choice, and score questions about shared evidence. The model
scores each question's possible answers instead of generating text, and every answer
carries a probability distribution.

```
POST /v1/decisions
```

Decisions is available since v1.12.0.

Requests are always treated as non-streaming: there is no `stream` field, and SMG returns
a single JSON body.

#### Request Body

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `model` | string | Yes | Model identifier. A name configured with `--model-alias` is accepted and resolved to its canonical model |
| `input` | string/array | Yes | The evidence every question is answered against: a plain string, or an array of `user` messages (see [Decision Input](#decision-input)) |
| `questions` | array | Yes | Questions to answer; answers come back in the same order (see [Decision Questions](#decision-questions)) |
| `safety_identifier` | string | No | Opaque end-user identifier, at most 128 characters |

Fields SMG does not model — on the request and on its messages, parts, questions, choices,
and levels — are kept as backend extensions: they are forwarded unchanged to non-SGLang
HTTP workers and rejected with `400` on SGLang workers (see [Decision Workers](#decision-workers)).

#### Decision Input

As an array, each `input` message has:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `role` | string | Yes | Must be `user` |
| `type` | string | No | `message`, when present |
| `content` | string/array | Yes | Text, or an array of `input_text` and `input_image` parts |

| Part `type` | Payload | Limits |
|-------------|---------|--------|
| `input_text` | `text` | At most 10,485,760 characters per part |
| `input_image` | `image_url`, optional `detail` (`low`, `high`, `auto`, `original`) | Must be an inline base64 image data URL (`data:image/png;base64,...`); external URLs and file IDs are rejected. At most 128 images per request |

`model`, `instructions`, `name`, `description`, and `label` values each take at most
1,048,576 characters. A limit violation is rejected with `400` before routing.

#### Decision Questions

Every question has a `type`, required `instructions`, and an optional `name`. Names need
not be unique: each answer repeats its question's `name` (`null` when unset), and answers
keep question order.

| `type` | Extra fields | The model estimates |
|--------|--------------|---------------------|
| `predicate` | — | The probability that the statement in `instructions` is true |
| `choice` | `choices`: array of `{"value": ...}` with an optional `description`; `value` is a string or boolean, and `true` is distinct from `"true"` | A probability for each choice |
| `score` | `levels`: array of `{"label": "..."}` with an optional `description`, ordered from lowest to highest | A probability for each level |

#### Example Request

```bash
curl http://localhost:30000/v1/decisions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B-Instruct-2507",
    "input": "The delivery arrived a day late, but the package was intact and support was friendly.",
    "questions": [
      {"type": "predicate", "instructions": "Did the delivery arrive on time?"},
      {"type": "choice", "name": "sentiment", "instructions": "What is the overall sentiment?",
       "choices": [{"value": "positive"}, {"value": "negative"}]},
      {"type": "score", "instructions": "Rate the customer experience.",
       "levels": [{"label": "poor"}, {"label": "fair"}, {"label": "good"}]}
    ]
  }'
```

#### Response

```json
{
  "model": "Qwen/Qwen3-4B-Instruct-2507",
  "answers": [
    {"type": "predicate", "name": null, "probability": 0.11},
    {"type": "choice", "name": "sentiment", "choice": "positive", "confidence": 0.74,
     "probabilities": [
       {"value": "positive", "probability": 0.87},
       {"value": "negative", "probability": 0.13}
     ]},
    {"type": "score", "name": null, "score": 1.62, "confidence": 0.55,
     "probabilities": [
       {"value": 0, "label": "poor", "probability": 0.08},
       {"value": 1, "label": "fair", "probability": 0.22},
       {"value": 2, "label": "good", "probability": 0.70}
     ]}
  ],
  "usage": {
    "input_tokens": 182,
    "output_tokens": 0,
    "total_tokens": 182,
    "input_tokens_details": {"cached_tokens": 0, "cache_write_tokens": 0},
    "output_tokens_details": {"reasoning_tokens": 0}
  }
}
```

Each answer carries its question's `type` and `name` plus:

| `type` | Fields |
|--------|--------|
| `predicate` | `probability` (0–1) that the statement is true |
| `choice` | `choice` — the `value` of the most probable choice; `confidence` (0–1); `probabilities` — one `{value, probability}` entry per choice, in request order |
| `score` | `score` — the probability-weighted average of the level indices, not a winning level (level `i` in request order has `value: i`, so 3 levels give a score between 0 and 2); `confidence` (0–1); `probabilities` — one `{value, label, probability}` entry per level |
| `refusal` | No further fields: the worker declined to answer the question. SMG's own SGLang scoring always produces one of the three types above |

`usage` reflects scoring work: on SGLang workers (both the HTTP translation and gRPC
scoring) the answer labels are scored without generating tokens, so `output_tokens` and
`output_tokens_details.reasoning_tokens` are `0` and `total_tokens` equals `input_tokens`.

On gRPC workers each question is scored as its own prompt over the shared input —
`input_tokens` sums those prompts — and `input_tokens_details.cached_tokens` reports the
engine's prefix-cache hits. The SGLang HTTP translation returns `0` for both cache
counters, meaning unreported rather than measured. Non-SGLang HTTP workers return their
own usage object.

#### Decision Workers

What serves the request depends on the worker:

| Worker | Behavior |
|--------|----------|
| HTTP (SGLang runtime) | SMG translates the request to SGLang's native System One API (`POST /v1/systemone`, as served by SGLang 0.5.21) and maps the reply back to the Decisions shape |
| HTTP (other runtimes) | The request is forwarded to the worker's own `/v1/decisions`, unknown fields included; the worker's response — errors included — is returned as is |
| gRPC (regular mode, SGLang runtime) | SMG renders one prompt per question with the model's chat template and scores the answer labels through the engine, without generation |
| PD, EPD, ZMQ, and external providers | Not implemented (`501` `decisions_not_supported`) |

In a mixed fleet, SMG dispatches Decisions only to regular HTTP workers and regular SGLang
gRPC workers serving the model.

The SGLang paths (HTTP translation and gRPC scoring) reject these requests with `400`
`unsupported_decisions_request` before dispatch:

- `input_image` parts — SGLang Decisions takes text evidence only
- unknown fields anywhere on the request — SGLang Decisions cannot honor them
- empty or whitespace-only text evidence or `instructions`
- no questions, a `choice` question outside 1–255 choices, or a `score` question outside 1–10 levels

gRPC scoring also depends on the model: the internal answer labels SMG scores (`yes`/`no`,
option letters, level digits — your own choice values and level labels are unrestricted)
must each be one distinct token after the chat prompt, and the model's chat template must
produce plain text without leaving a reasoning block open. A model that cannot satisfy
this answers `400`.

#### Decision Errors

| Status | `code` | Returned when |
|--------|--------|---------------|
| 400 | `400` or `json_parse_error` | A length or image limit fails, or the body does not match the schema (a non-`user` role, an unknown question `type`, a numeric choice `value`, ...) |
| 400 | `unsupported_decisions_request` | An SGLang worker was selected and the request needs something SGLang Decisions does not support (see [Decision Workers](#decision-workers)) |
| 404 | `model_not_found` | No worker serves the model |
| 501 | `decisions_not_supported` | No regular HTTP worker and no regular SGLang gRPC worker serves the model |
| 502 | `invalid_decisions_response` | An SGLang worker (HTTP translation or gRPC scoring) reported success, but the result could not be mapped to a valid Decisions response |

Backend errors on the SGLang HTTP translation keep their status and OpenAI error body.

For tenant rate limiting on Decisions, see
[Tenant Rate Limiting](../tenant-rate-limiting.md).

---

### List Models

List available models.

```
GET /v1/models
```

#### Example Request

```bash
curl http://localhost:30000/v1/models
```

#### Response

```json
{
  "object": "list",
  "data": [
    {
      "id": "meta-llama/Llama-3.1-8B-Instruct",
      "object": "model",
      "created": 0,
      "owned_by": "self_hosted"
    }
  ]
}
```

The list covers the models served by self-hosted workers, each once under its canonical
ID. Names configured with `--model-alias` are accepted in requests but are not listed.
`created` is always `0`. `owned_by` is `self_hosted` for locally hosted workers, or the
provider name (for example `openai`, `anthropic`, `xai`, `gemini`) for upstream providers.
When no self-hosted worker serves any model, the endpoint returns `503` with the
plain-text body `No models available`.

The endpoint needs no key. If the caller does send a bearer token (or an `x-api-key`
header) that is not one of the gateway's own keys, SMG treats it as the caller's own
provider key: it asks the registered external providers' `/v1/models` with that token and
returns the first non-empty list, falling back to the self-hosted list. A gateway key
(`--api-key` or `--tenant-api-key`) is never forwarded to a provider.

---

### Audio Transcriptions

Transcribe an audio file (batch).

```
POST /v1/audio/transcriptions
```

The request is `multipart/form-data`:

| Field | Required | Description |
|-------|----------|-------------|
| `file` | Yes | The audio file; must not be empty |
| `model` | Yes | Model identifier |
| `language` | No | Language of the audio |
| `prompt` | No | Text to guide the transcript |
| `response_format` | No | Output format (default `json`) |
| `temperature` | No | Sampling temperature (0.0-1.0) |
| `timestamp_granularities[]` | No | `word` or `segment`; repeat the field for both |
| `stream` | No | `true`/`false` (or `1`/`0`) |

A malformed form, a missing or empty `file`, a missing `model`, an out-of-range
`temperature`, or an unparseable `stream` gets a plain-text `400` before routing. Other
form fields are ignored.

```bash
curl http://localhost:30000/v1/audio/transcriptions \
  -F file=@audio.wav \
  -F model=Qwen/Qwen3-ASR-1.7B
```

What serves the request depends on the worker:

| Worker | Behavior |
|--------|----------|
| HTTP | The form is forwarded to the worker's `/v1/audio/transcriptions`; the worker's model and options apply |
| gRPC (regular mode) | SMG runs transcription as its own pipeline endpoint: it turns the audio into a chat turn, renders it with the model's chat template, and returns the transcript. Supported family: Qwen3-ASR |
| PD, EPD, and external providers | Not implemented (`501`) |

On gRPC, a model belongs to the Qwen3-ASR family when its ID, or a serving worker's model
ID or model-identity label, contains `qwen3-asr` or `qwen3_asr` (ignoring case). Audio
input over gRPC is currently accepted by TokenSpeed workers only; other gRPC runtimes
answer `400` `multimodal_not_supported`. The gRPC endpoint is whole-file only:

| Request | gRPC result |
|---------|-------------|
| A model outside the supported families | `400` `audio_transcription_model_not_supported` |
| `stream: true` | `400` `streaming_transcription_not_supported` |
| Any `timestamp_granularities` | `400` `transcription_timestamps_not_supported` |
| `response_format` of `srt`, `vtt`, or `verbose_json` | `400` `unsupported_transcription_response_format` |
| A `language` outside the model's list (ISO code or English name, such as `en` or `English`) | `400` `unsupported_transcription_language` |
| A `prompt` over 4096 bytes | `400` `asr_prompt_too_long` |

`response_format: json` returns `{"text": "..."}` and `text` returns the plain transcript.
Sampling is greedy unless `temperature` is set, and a given `language` pins the transcript
language.

---

### Realtime API

SMG proxies the OpenAI Realtime API to a realtime-capable worker. Both the OpenAI router
(to an upstream provider) and the HTTP router (to a **local** worker labeled
[`realtime: "true"`](../../getting-started/multiple-workers.md#realtime-capable-workers))
support it. SMG relays frames verbatim, so the worker must speak the OpenAI Realtime
protocol — for local workers, for example vLLM serving an ASR model with the realtime task.

| Endpoint | Transport | Purpose |
|----------|-----------|---------|
| `GET /v1/realtime` | WebSocket | Bidirectional realtime session (e.g. live streaming transcription) |
| `POST /v1/realtime/calls` | WebRTC (SDP) | Browser/WebRTC realtime session |
| `POST /v1/realtime/sessions` | HTTP | Create a realtime session |
| `POST /v1/realtime/client_secrets` | HTTP | Mint an ephemeral client secret |
| `POST /v1/realtime/transcription_sessions` | HTTP | Create a realtime transcription session |

`GET /v1/realtime` requires the `model` query parameter and answers `400` without it.
`POST /v1/realtime/calls` takes the model from the `model` query parameter for an
`application/sdp` body, or from the session JSON in a `multipart/form-data` body.

#### WebSocket example

```python
# pip install websockets
import asyncio, websockets

async def main():
    url = "ws://localhost:30000/v1/realtime?model=Qwen/Qwen3-ASR-1.7B"
    headers = {"Authorization": "Bearer your-api-key"}
    async with websockets.connect(url, additional_headers=headers) as ws:
        # Send realtime events (session.update, input_audio_buffer.append, ...)
        # and receive transcription/response events from the worker.
        ...

asyncio.run(main())
```

---

## Provider Profiles

Some model families come with a vendor contract that differs from the OpenAI baseline.
SMG picks a provider profile from the request's `model` and applies that profile's
defaults and validation to Chat Completions requests before routing, on every backend.

### Profile Selection

A profile matches when any `/`-separated segment of the model ID starts with one of its
markers, ignoring case, so `moonshotai/Kimi-K2-Instruct` and `/models/kimi-k3` both select
Kimi.

| Profile | Selected by |
|---------|-------------|
| Kimi | `kimi*`, `moonshot*`; the K3 rules apply to `kimi-k3*` and `kimi_k3*` |
| MiniMax | `minimax*`, `abab*` |
| z.ai (GLM) | `glm*`, `zai*`, `z-ai*`; the GLM-5.3 rules apply to `glm-5.3*`, `glm5.3*`, `glm_5.3*`, `glm-5-3*`, `glm5-3*` |
| DeepSeek V4 | A segment equal to `deepseek-flash`, `deepseek-v4-flash`, `deepseek-v4-pro`, or `deepseek-v4.1-flash` |
| OpenAI baseline | Everything else, including older DeepSeek models and `THUDM/chatglm3-6b` |

Selection reads the model name as sent, before `--model-alias` resolution: an alias that
doesn't match a marker itself gets the OpenAI baseline. Message-level extension fields
that belong to another profile (such as Kimi's message `tools` sent to a non-Kimi model)
are dropped before dispatch, with one warning logged per request.

Profile defaults and validation apply to `/v1/chat/completions` requests that pass
through SMG's request validation. They are not applied to `/v1/responses`, and a request
the HTTP router streams to the worker unparsed skips them (see
[Request Streaming](../../concepts/performance/request-streaming.md)).

### Validation Rules

A violation is rejected with `400`. The response has `error.type` set to
`invalid_request_error` and `error.code` set to `400` (as for every validation failure),
and `error.message` carries the rule's message:

```json
{
  "error": {
    "message": "__all__: invalid thinking.type: only enabled or disabled is allowed for this model",
    "type": "invalid_request_error",
    "code": 400
  }
}
```

The rule code in the last column names the rule in SMG's source and tests; only
`context_length_exceeded` is returned as `error.code`.

| Profile | Rejected | Rule code |
|---------|----------|-----------|
| All except MiniMax | A `root` message | `invalid_role` |
| Kimi | `tools` on a `user` or `assistant` message | `tools_role_restricted` |
| Kimi | `tools` on a `system` or `developer` message that is not a list of tools | `tools_malformed` |
| Kimi | A non-empty message `tools` list together with non-empty content on that message | `tools_content_conflict` |
| Kimi | A message tool whose `type` is not `function` | `tool_type_unsupported` |
| Kimi | A message tool name not matching `[A-Za-z_][A-Za-z0-9_]*`, or longer than 256 characters | `tool_name_invalid` |
| Kimi | A message tool name already declared in `tools` or on another message | `tool_name_duplicate` |
| Kimi K3 | `temperature` other than 0, 0.6, or 1; `top_p` other than 0.95; a non-zero `presence_penalty` or `frequency_penalty`; `n` other than 1 | `temperature_not_allowed`, `top_p_not_allowed`, `presence_penalty_not_allowed`, `frequency_penalty_not_allowed`, `n_not_allowed` |
| Kimi K3, z.ai, DeepSeek V4 | `thinking.type: adaptive` | `thinking_type_not_supported` |
| Kimi K3 | `thinking.effort` other than `low`, `high`, or `max` | `thinking_effort_invalid` |
| MiniMax | A tool-call ID used twice in the history | `tool_call_id_duplicate` |
| MiniMax | A `tool` message whose `tool_call_id` answers no pending call | `tool_call_id_mismatch` |
| MiniMax | Historical tool-call `arguments` that are non-empty but not a JSON object | `tool_call_arguments_invalid_json` |
| MiniMax | A tool call with no `tool` message answering it | `tool_call_unanswered` |
| z.ai | `thinking.effort` or `reasoning_effort` outside `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max` (GLM-5.3: outside `low`, `high`, `max`) | `reasoning_effort_not_allowed` |
| z.ai GLM-5.3 | `thinking.type: disabled` | `thinking_disabled_not_supported` |
| z.ai | A tool whose `type` is not `function` | `tool_type_not_supported` |
| z.ai | A `file_url` content part (SMG does not fetch files for the model) | `content_part_not_supported` |
| z.ai (gRPC) | `max_completion_tokens` (or `max_tokens`) larger than the context window the selected worker advertises | `context_length_exceeded` |
| DeepSeek V4 | `reasoning_effort` or `thinking.effort` outside `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`; V4.1 also takes an integer budget from 1 to 100 | `reasoning_effort_not_allowed` |
| DeepSeek V4 | `chat_template_kwargs.thinking` or `enable_thinking` that is neither a boolean nor `null` | `invalid_thinking_override` |
| DeepSeek V4 | `chat_template_kwargs.thinking` and `enable_thinking` that disagree | `conflicting_thinking_overrides` |
| DeepSeek V4 | A forced `tool_choice` (`required`, a named function, or `allowed_tools` with mode `required`) while thinking is on | `tool_choice_not_supported` |

z.ai and DeepSeek V4 check both effort spellings, even the one that `thinking.effort`
shadows.

### Defaults and Rewrites

| Profile | Behavior |
|---------|----------|
| Kimi | Streaming requests get `stream_options.include_usage: true` unless set. Tools declared on `system` and `developer` messages join the request's `tools` for `tool_choice`, the tool-choice constraint, and tool-call parsing; a `system` message that only declares tools may omit `content` |
| Kimi K3 | Omitted fields get the contract values: `temperature` 1.0, `top_p` 0.95, `presence_penalty` 0, `frequency_penalty` 0, `n` 1 |
| MiniMax | `root` messages move, in order, to the front as `system` messages. On gRPC workers, responses are scanned for tool calls even when the request declares no tools |
| z.ai | `thinking.clear_thinking` becomes the `clear_thinking` chat-template variable unless `chat_template_kwargs` sets it. `tool_stream` is accepted and dropped (SMG streams tool-call deltas anyway). GLM-5.x, GLM-4.7, and GLM-4.6 get `temperature` 1.0 and `top_p` 0.95 when omitted |
| DeepSeek V4 | Efforts `minimal` → `low` and `medium` → `high`; `xhigh` → `high` on V4 (V4.1 keeps `xhigh`). Thinking is on unless turned off (`thinking.type: disabled`, effort `none`, or a template override), and the result is written to `chat_template_kwargs.thinking` when that is unset. Streams use DeepSeek's usage format (see [Stream Usage](#stream-usage)) |

---

## Error Responses

### Error Format

Errors that SMG generates use this envelope and repeat `code` in the `X-SMG-Error-Code`
response header:

```json
{
  "error": {
    "type": "Bad Request",
    "code": "context_length_exceeded",
    "message": "This model's maximum context length is 2048 tokens. However, your request has 2310 input tokens. Please reduce the length of the input.",
    "param": null
  }
}
```

`type` is the HTTP status's reason phrase (`Bad Request`, `Not Found`,
`Too Many Requests`, `Service Unavailable`, ...). Request parsing and validation failures
use a different shape and carry no `X-SMG-Error-Code` header:

| Failure | Body |
|---------|------|
| Body is not valid JSON, does not match the request schema, or lacks `Content-Type: application/json` | `{"error": {"message": "...", "type": "invalid_request_error", "code": "json_parse_error"}}` |
| A field range, a cross-field rule, or a [provider-profile](#provider-profiles) rule fails | `{"error": {"message": "...", "type": "invalid_request_error", "code": 400}}` |

A `401` from API-key authentication has an empty body. Errors from HTTP workers are
relayed with the worker's status and body. SMG never forwards a worker's
`X-SMG-Error-Code` header, so that header always means SMG itself produced the error.

### Error Codes

| Status | `code` | Returned when |
|--------|--------|---------------|
| 400 | `400` | Request validation failed: a field range, a cross-field rule (`stream_options` without `stream`, `top_logprobs` without `logprobs`, a `tool_choice` naming a missing tool, ...), a non-object `json_schema`, or a [provider-profile](#provider-profiles) rule |
| 400 | `json_parse_error` | The body could not be parsed into the request type |
| 400 | `context_length_exceeded` | gRPC: the longest prompt is longer than the context window the selected worker advertises (the tighter leg governs in PD, and each prompt of a batched completion counts on its own). Workers that advertise no window are not checked |
| 400 | `unsupported_content_part` | gRPC: a content part SMG cannot render |
| 400 | `audio_output_not_supported`, `unsupported_output_modality` | gRPC: `return_audio: true` or a non-`text` entry in `modalities`; gRPC workers produce text only |
| 400 | `multimodal_not_supported` | gRPC: the selected engine does not accept one of the request's input modalities |
| Engine-mapped | `start_generation_failed`, `prefill_worker_failed_to_start`, `decode_worker_failed_to_start` | gRPC: the engine did not start generation. The status follows the engine's gRPC status, for example invalid argument → `400` (not retried and not counted against the circuit breaker), resource exhausted → `429`, unavailable → `503`, deadline exceeded → `504`, internal → `500` |
| 401 | — | Missing or unknown API key (empty body) |
| 404 | `model_not_found` | No worker serves the model |
| 408 | `request_body_stalled` | HTTP router: a streamed request body stalled for `--stream-body-stall-timeout-secs` |
| 413 | `request_body_too_large` | HTTP router: a streamed request body exceeded `--max-payload-size` |
| 429 | `admission_queue_full`, `scheduler_queue_full`, `tenant_rate_limit_exceeded` | Admission control or rate limiting; see [Rate Limiting](#rate-limiting) |
| 500 | `internal_error` and stage-specific codes | A failure inside SMG; the code names the failing step |
| 502 | `upstream_response_too_large` | A worker's non-streaming response exceeded `--max-payload-size` |
| 503 | `no_available_workers` | Workers serve the model, but every one is unhealthy or has an open circuit breaker, or the routing policy selects none of them (HTTP and gRPC paths) |
| 503 | `worker_overload_protection_shed`, `admission_queue_timeout`, `scheduler_queue_timeout`, `scheduler_preempted` | Overload and admission sheds; see [Rate Limiting](#rate-limiting) |

On gRPC workers, an engine finish reason of `error` (outside the OpenAI set) becomes an
error response or stream error; clients never receive `"finish_reason": "error"`.

---

## Client Libraries

### Python (OpenAI SDK)

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="your-api-key"  # or "not-needed" if auth disabled
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "Hello!"}
    ]
)

print(response.choices[0].message.content)
```

### JavaScript/TypeScript

```typescript
import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'http://localhost:30000/v1',
  apiKey: 'your-api-key'
});

const response = await client.chat.completions.create({
  model: 'meta-llama/Llama-3.1-8B-Instruct',
  messages: [
    { role: 'user', content: 'Hello!' }
  ]
});

console.log(response.choices[0].message.content);
```

### cURL

```bash
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

---

## Request Headers

Header names are case-insensitive. SMG reads the following request headers:

| Header | Purpose | Documented in |
|--------|---------|---------------|
| `Content-Type` | `application/json`, or `multipart/form-data` for `/v1/audio/transcriptions` | — |
| `Authorization` | `Bearer <key>`; required when `--api-key` or `--tenant-api-key` is set | [Authentication](../../concepts/security/authentication.md) |
| `x-api-key`, `x-goog-api-key` | The caller's own key for Anthropic (`x-api-key`) and Gemini (`x-goog-api-key`) provider workers. `/v1/responses` and `/v1/interactions` prefer it over `Authorization`, and `/v1/messages` forwards `x-api-key` as sent; `/v1/chat/completions` uses only `Authorization` | [External Providers](../../getting-started/external-providers.md) |
| `X-Request-ID` | Request ID for log correlation. SMG checks `x-request-id`, `x-correlation-id`, `x-trace-id`, and `request-id` in that order, and generates an ID when none is present. `--request-id-headers` replaces the list | [Logging](../../getting-started/logging.md) |
| `traceparent`, `tracestate` | W3C trace context. With `--enable-trace`, SMG continues the caller's trace and propagates it to workers | [Monitoring](../../getting-started/monitoring.md) |
| `X-SMG-Routing-Key` | Routing key for sticky sessions and the key-based policies (`manual`, `consistent_hashing`, `prefix_hash`). It is the default name in `--routing-key-headers` | [Sticky Sessions and Routing Keys](../../concepts/routing/sticky-sessions.md) |
| `X-SMG-Routing-Tokens` | Routing hint carrying the prompt's leading token IDs | [Routing Hint Headers](#routing-hint-headers) |
| `X-SMG-Target-Worker` | 0-based index of the worker to use among the model's available workers. Honored only by `consistent_hashing`; an index with no available worker fails with `503` | [Load Balancing](../../concepts/routing/load-balancing.md) |
| `X-SMG-Priority` | Priority class: `system`, `interactive`, `default`, or `bulk`. Read only when the priority scheduler is enabled | [Priority Scheduler](../priority-scheduler.md) |
| `X-SMG-Tenant-ID` | Tenant identity asserted by a trusted upstream, resolved as tenant key `header:<value>`. Read only with `--trust-tenant-header`, and only when API-key auth has not already identified the caller. `--tenant-header-name` renames it | [Tenant Rate Limiting](../tenant-rate-limiting.md) |
| `X-SMG-MCP` | `enabled` makes SMG run the MCP tool loop itself for `/v1/messages` requests with MCP toolsets sent to an Anthropic provider, instead of forwarding them as-is | [Messages API](messages.md) |
| Names set with `--storage-context-headers` | Copied into the storage hook request context | [Configuration Reference](../configuration.md) |

When SMG proxies a request to an HTTP worker, it forwards only these client headers: `Authorization`, `anthropic-version`, `anthropic-beta`, `X-Request-ID`, `X-Correlation-ID`, headers starting with `X-Request-ID-`, `traceparent`, `tracestate`, and `X-SMG-Routing-Key`.

### Routing Hint Headers

A client, or a proxy in front of SMG that has already tokenized the prompt, can pass the routing signal in a header so SMG does not need the request body to choose a worker. Hints affect only worker placement. A malformed or over-limit hint is ignored and the request routes as if the header were absent; it never causes an error.

| Header | Value | Limits | Effect |
|--------|-------|--------|--------|
| `X-SMG-Routing-Tokens` | The prompt's leading token IDs as comma-separated decimal integers from `0` to `4294967295`, with no spaces, signs, or empty items | At most 512 IDs and 4096 bytes | Replaces the token IDs and text SMG would take from the body when choosing a worker. `cache_aware` matches the hinted prefix against its token tree (its placement index with `--cache-index hash`) and records it there; `prefix_hash` hashes it. Read on the regular HTTP path only (not in PD mode or for gRPC workers), and never forwarded to workers |
| `X-SMG-Routing-Key` | An opaque key | Non-empty UTF-8, at most 128 bytes | Without `--routing-key-override`, `consistent_hashing` and `prefix_hash` hash the key instead of their other inputs, and `manual` pins it. With the override, every policy except `consistent_hashing` pins the key |

```bash
# X-SMG-Routing-Tokens carries the prompt's first token IDs
curl http://localhost:30000/v1/completions \
  -H "Content-Type: application/json" \
  -H "X-SMG-Routing-Tokens: 128000,791,4062,14198,39935" \
  -d @request.json
```

With a valid `X-SMG-Routing-Tokens` hint, the text-routing policies (`cache_aware`, `bucket`) no longer need the body to place the request, so a large body can stream to the worker instead of being buffered; see [Request Streaming](../../concepts/performance/request-streaming.md). When `--routing-key-override` is enabled, SMG always buffers the body to read its `rid`, and hints do not change that.

### Response Headers

| Header | When SMG sets it | Documented in |
|--------|------------------|---------------|
| `x-request-id` | On API responses: the request ID SMG used, whether received or generated | [Logging](../../getting-started/logging.md) |
| `x-smg-routed-worker-id` | On inference responses from HTTP, gRPC, and ZMQ workers: the registered URL of the worker that served the request, with an `@<rank>` suffix for data-parallel workers. For a prefill/decode pair it names the decode worker. From gRPC and ZMQ workers, only successful responses carry it, and Responses API responses do not | [Sticky Sessions and Routing Keys](../../concepts/routing/sticky-sessions.md#routed-worker-header) |
| `X-SMG-Error-Code` | On errors the gateway generates in its standard error envelope, with the same value as `error.code`; request-validation `400` errors do not carry it. SMG drops this header from worker responses, so it always marks a gateway decision | — |
| `X-SMG-Preempted` | `true` on the `503` returned to a request preempted by the priority scheduler | [Priority Scheduler](../priority-scheduler.md) |
| `Retry-After` | On `429` and `503` responses that ask the client to back off: concurrency limits, worker overload, the priority scheduler, and tenant rate limits | [Rate Limiting](../../concepts/reliability/rate-limiting.md) |

---

## Rate Limiting

SMG does not send `X-RateLimit-*` headers. A request shed by admission control, rate
limiting, or overload protection gets `429` or `503` with the standard
[error envelope](#error-format), the `X-SMG-Error-Code` header, and usually a
`Retry-After` header in seconds:

| `code` | Status | `Retry-After` | Cause |
|--------|--------|---------------|-------|
| `admission_queue_full` | 429 | `2` | `--max-concurrent-requests` is reached and the admission queue is full or disabled |
| `admission_queue_timeout` | 503 | `2` | The request waited in the admission queue longer than `--queue-timeout-secs` |
| `scheduler_queue_full` | 429 | `2` | Priority scheduler: the request's class queue is full |
| `scheduler_queue_timeout` | 503 | `2` | Priority scheduler: the request waited longer than its class's queue timeout |
| `scheduler_preempted` | 503 | `1` | Priority scheduler: preempted by higher-priority traffic before its first token; also sets `X-SMG-Preempted: true` |
| `tenant_rate_limit_exceeded` | 429 | Seconds until the tenant's budget can admit the request; omitted when it never can | Tenant rate limiting |
| `worker_overload_protection_shed` | 503 | `--load-monitor-interval` (default `10`) | Every eligible worker is over its overload threshold (or the selected one crossed it just before dispatch), or a PD decode worker could not admit the request within `--pd-admission-wait-secs` |

Wait at least `Retry-After` seconds before retrying. For configuration, see
[Rate Limiting](../../concepts/reliability/rate-limiting.md),
[Tenant Rate Limiting](../tenant-rate-limiting.md),
[Priority Scheduling](../../concepts/reliability/priority-scheduling.md), and
[Overload Protection](../../concepts/reliability/overload-protection.md).