--- title: OpenAI-Compatible API --- # OpenAI-Compatible API Reference SMG provides a fully OpenAI-compatible API, allowing you to use existing OpenAI client libraries with your self-hosted inference workers. --- ## Base URL ``` http://localhost:30000/v1 ``` --- ## Authentication SMG supports optional API key authentication: ```bash curl http://localhost:30000/v1/chat/completions \ -H "Authorization: Bearer your-api-key" \ -H "Content-Type: application/json" \ -d '...' ``` Enable authentication with `--api-key`: ```bash smg launch --worker-urls http://worker:8000 --api-key "your-api-key" ``` For real multi-tenant separation (e.g. per-tenant rate limiting), configure one key per tenant instead with `--tenant-api-key tenant_id:key` (repeatable). Each key resolves to its own tenant identity; `--api-key` remains available as a single shared fallback key whose callers all share one identity. ```bash smg launch --worker-urls http://worker:8000 \ --tenant-api-key team-red:red-secret \ --tenant-api-key team-blue:blue-secret ``` Callers authenticate the same way as with `--api-key` — a `Bearer` token in the `Authorization` header — just using their own tenant's key instead of the shared one: ```bash curl http://localhost:30000/v1/chat/completions \ -H "Authorization: Bearer red-secret" \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}] }' ``` This request is attributed to `team-red`, distinct from a request authenticated with `blue-secret`. A missing key, or one that doesn't match any configured `--api-key` or `--tenant-api-key` value, is rejected with `401 Unauthorized` in the standard [error envelope](#error-format) with the code `invalid_api_key`, plus a `WWW-Authenticate: Bearer` challenge (`error="invalid_token"` when a key was presented but not recognized). Tenant keys authenticate serving endpoints only — `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, etc. They do not grant access to admin/management routes (`/workers`, `/flush_cache`, and similar). When control-plane authentication is enabled, those routes require a control-plane credential: the `admin` role may use every method, the `user` role only `GET` and `HEAD`. Otherwise, they accept the shared `--api-key` if configured. With neither control-plane auth nor any API keys configured, those routes are open; with only `--tenant-api-key` configured, every admin request is rejected with `401`. The model routes — `GET /v1/models` and `GET /v1/models/{model}` — are public and answer without a key (see [List Models](#list-models) and [Retrieve Model](#retrieve-model)). --- ## Endpoints ### Chat Completions Create a chat completion. ``` POST /v1/chat/completions ``` #### Request Body | Field | Type | Required | Description | |-------|------|----------|-------------| | `model` | string | Yes | Model identifier. A name configured with `--model-alias` is accepted and resolved to its canonical model | | `messages` | array | Yes | Array of message objects; must not be empty | | `max_completion_tokens` | integer | No | Upper bound on generated completion tokens (at least 1) | | `max_tokens` | integer | No | Deprecated — use `max_completion_tokens`. Still accepted: it is the cap when `max_completion_tokens` is unset, and a proxied body reaches the worker with the field as the client wrote it (not rewritten). OpenAI-named models reject the two limits together (see [Validation Rules](#validation-rules)) | | `temperature` | number | No | Sampling temperature (0-2) | | `top_p` | number | No | Nucleus sampling parameter, greater than 0 and at most 1 | | `n` | integer | No | Number of completions to generate (1-128) | | `stream` | boolean | No | Enable streaming responses | | `stream_options` | object | No | Streaming options; only allowed with `stream: true`. See [Stream Usage](#stream-usage) | | `stop` | string/array | No | Up to 4 non-empty stop sequences | | `presence_penalty` | number | No | Presence penalty (-2 to 2) | | `frequency_penalty` | number | No | Frequency penalty (-2 to 2) | | `logprobs` | boolean | No | Return log probabilities of the output tokens | | `top_logprobs` | integer | No | Most likely tokens to return per position (0-20); requires `logprobs: true` | | `logit_bias` | object | No | Map of token ID to bias | | `tools` | array | No | Function tools the model may call | | `tool_choice` | string/object | No | `none`, `auto`, `required`, `{"type": "function", "function": {"name": "..."}}`, or `{"type": "allowed_tools", "mode": "auto", "tools": [...]}` (`mode` is `auto` or `required`). Every value except `none` and `auto` requires tools, and a named function must exist in them | | `parallel_tool_calls` | boolean | No | Allow parallel function calls | | `response_format` | object | No | `text`, `json_object`, or `json_schema`. See [Structured Output](#structured-output) | | `reasoning_effort` | string/number | No | Reasoning effort; a number is converted to its decimal string. See [Thinking and Reasoning Effort](#thinking-and-reasoning-effort) | | `thinking` | object | No | Typed thinking control (`type`, `effort`, `keep`, `clear_thinking`). See [Thinking and Reasoning Effort](#thinking-and-reasoning-effort) | | `verbosity` | string | No | `low`, `medium`, or `high`; any other value is rejected | | `user` | string | No | End-user identifier | #### Extension Fields SMG also accepts these SGLang-style extensions on Chat Completions: | Field | Type | Description | |-------|------|-------------| | `top_k` | integer | Top-k sampling; `-1` disables it, otherwise at least 1 | | `min_p` | number | Min-p sampling threshold (0-1) | | `min_tokens` | integer | Minimum tokens to generate; must not exceed `max_completion_tokens` | | `repetition_penalty` | number | Repetition penalty (0-2) | | `regex`, `ebnf` | string | Constrain the output. Only one of `regex`, `ebnf`, and a `json_schema` response format may be set, and neither combines with a JSON `response_format` | | `stop_token_ids` | array | Token IDs that stop generation | | `ignore_eos`, `no_stop_trim` | boolean | Keep generating past EOS; keep the stop sequence in the output | | `skip_special_tokens` | boolean | Drop special tokens when detokenizing (default `true`) | | `continue_final_message` | boolean | Continue the trailing assistant message instead of starting a new one | | `add_generation_prompt` | boolean | Whether the prompt ends with the assistant's generation header; absent reads `true`. `false` renders the conversation without it, leaving the model to continue on its own. Set to `true` beside `continue_final_message` it is rejected with `400` `add_generation_prompt_conflicts_with_continue_final_message` | | `separate_reasoning`, `stream_reasoning` | boolean | Return reasoning in `reasoning_content` and stream it (both default `true`) | | `chat_template_kwargs` | object | Extra chat-template variables. On gRPC workers an explicit entry wins over the `tool_choice` and `response_format` values SMG passes to the template, but not over the effort: a `reasoning_effort` entry applies only when the request sets no effort of its own (see [Thinking and Reasoning Effort](#thinking-and-reasoning-effort)) | | `lora_path` | string | LoRA adapter to apply | | `sampling_seed` | integer | Seed for deterministic sampling | | `return_hidden_states` | boolean | Return the model's hidden states | | `rid` | string | Request ID forwarded to the backend | Fields SMG does not model are kept and forwarded unchanged to HTTP workers, and so are unknown `stream_options` keys such as SGLang's `step_usage_chunks`. The SGLang-native `/generate` endpoint likewise keeps its nested `sampling_params.custom_params` object. Typed sampling numbers are forwarded without float widening: `"top_p": 0.95` reaches HTTP workers, PD legs, and external providers as `0.95`, not `0.949999988079071`. #### Message Object | Field | Type | Required | Description | |-------|------|----------|-------------| | `role` | string | Yes | `system`, `developer`, `user`, `assistant`, `tool`, `function`, or `root` (MiniMax only; see [Provider Profiles](#provider-profiles)) | | `content` | string/array | Yes | Text, or an array of [content parts](#content-parts). Optional on `system` and `assistant` messages; required on `user` messages, where an empty string or array is accepted; a plain string or `null` on `function` messages (see below) | | `name` | string | No | Participant name | | `tool_calls` | array | No | Tool calls made by an `assistant` message | | `tool_call_id` | string | For `tool` | ID of the tool call this `tool` message answers | | `reasoning_content` | string | No | Reasoning from an earlier `assistant` turn; `reasoning` is accepted as an alias | | `tools` | array | No | Kimi only: tools declared on a `system` or `developer` message | On a `function` message (the deprecated predecessor of `tool`), `content` is required but nullable, and a `null` is kept rather than dropped: a body forwarded to an HTTP worker or external provider still carries `"content": null`, and on gRPC workers the chat template receives it as empty content in the template's own format (an empty string, or an empty array of parts). #### Content Parts | `type` | Payload | Notes | |--------|---------|-------| | `text` | `text` | | | `image_url` | `image_url.url`, optional `detail` | Optional `image_url.max_long_side_pixel` caps the image's long side: a MiniMax-M3 extension that SMG's own media processing applies for any model (see [Frame Sampling](../../concepts/architecture/multimodal.md#frame-sampling)) | | `video_url` | `video_url.url` | Optional `video_url.fps` and `video_url.max_long_side_pixel` set the frame sampling rate and frame size: MiniMax-M3 extensions that SMG's own media processing applies for any model (see [Frame Sampling](../../concepts/architecture/multimodal.md#frame-sampling)) | | `input_audio` | `input_audio.data` (base64), `input_audio.format` | | | `audio_url` | `audio_url.url` | | | Anything else | Forwarded as sent | HTTP workers only | A content part whose `type` SMG does not model, or a known type with a malformed payload, is forwarded unchanged to HTTP workers. gRPC workers reject it with `400` `unsupported_content_part`, and the gpt-oss (Harmony) path with `400` `harmony_build_failed`. Which workers accept which media is covered in [Multimodal](../../concepts/architecture/multimodal.md). #### Thinking and Reasoning Effort Two fields control reasoning. `reasoning_effort` is OpenAI's effort level. `thinking` is a typed object shared by the Kimi, MiniMax, z.ai, and DeepSeek dialects: | Field | Values | Meaning | |-------|--------|---------| | `type` | `enabled`, `disabled`, `adaptive` | Thinking toggle. `adaptive` states no preference; any other value is rejected | | `effort` | string | Effort level; takes precedence over `reasoning_effort` | | `keep` | string | Kimi `keep` mode; accepted and ignored by SMG's renderers | | `clear_thinking` | boolean | z.ai: whether earlier turns' reasoning is dropped from the prompt | The effective effort is `thinking.effort` when present, otherwise `reasoning_effort`. Leaving `thinking` out changes nothing: the chat template's own default applies (Kimi K3, for example, thinks by default), subject to `reasoning_effort`. The exception is the DeepSeek V4 profile, which turns thinking on unless the request turns it off (see [Defaults and Rewrites](#defaults-and-rewrites)). A `thinking` object without `type` expresses no toggle, but its `effort` still counts. On gRPC workers SMG renders the chat template itself. The template receives the effective effort as its `reasoning_effort` variable, and SMG decides whether the model starts in reasoning mode from the first of these that applies: 1. an explicit thinking toggle in `chat_template_kwargs`; 2. for templates with their own effort levels, the effort the template renders (the request's effective effort, else a `chat_template_kwargs.reasoning_effort` entry): a native level turns thinking on, and `none` and `minimal` turn it off. Three exceptions: the DeepSeek-V4 renderer keeps thinking on for every value but `none` (a model the [DeepSeek V4 profile](#defaults-and-rewrites) matches never reaches this step: the profile resolves the toggle up front and writes it as the step-1 override, reading a kwargs-only `reasoning_effort` of `none` or `minimal` as off); Kimi-K3 lets only `none` decide at this step, leaving every other value — `minimal` and native levels included — to the steps below; and Hunyuan 4 templates, whose `reasoning_effort` words carry the toggle itself, are read by those words alone: `no_think` turns thinking off, and every other value — `none` and `minimal` included — keeps thinking on; 3. `thinking.type` (`enabled` on, `disabled` off); 4. the effective effort, where `none` and `minimal` turn thinking off — except on MiniMax models, where no effort value does (MiniMax reads `none` as the model's default thinking depth and `minimal` as `low`). If none applies, the template default stands. The template and the reasoning parser use the same decision. On gpt-oss (Harmony) models SMG reads `reasoning_effort` and clamps it to the three levels Harmony knows: `none` and `minimal` become `low`, `xhigh` and `max` become `high`, and an unknown value becomes `medium`. HTTP workers receive both fields as sent unless a [provider profile](#provider-profiles) rewrites them, and provider profiles restrict the accepted values per model family. #### Structured Output `response_format` accepts: - `{"type": "text"}` - `{"type": "json_object"}` - `{"type": "json_schema", "json_schema": {"name": "...", "schema": {...}, "strict": true}}` For `json_schema`, `name` must not be empty and `schema` must be a JSON object (`{}` is allowed). A string, array, boolean, number, or `null` schema is rejected with `400` `json_schema_schema_not_object` before the request reaches any worker, on every backend, unless the HTTP router streams the request to the worker unparsed (see [Request Streaming](../../concepts/performance/request-streaming.md)). #### Example Request ```bash curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"} ], "max_tokens": 100, "temperature": 0.7 }' ``` #### Response ```json { "id": "chatcmpl-abc123", "object": "chat.completion", "created": 1705312345, "model": "meta-llama/Llama-3.1-8B-Instruct", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "The capital of France is Paris." }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 25, "completion_tokens": 8, "total_tokens": 33 } } ``` #### Usage | Field | Description | |-------|-------------| | `prompt_tokens`, `completion_tokens`, `total_tokens` | Token counts. With `n` > 1 the shared prompt is counted once and the choices' completions are summed | | `prompt_tokens_details.cached_tokens` | Prompt tokens served from the worker's prefix cache | | `completion_tokens_details.reasoning_tokens` | Reasoning tokens: a chat completion always carries the key (`0` without reasoning), while the legacy Completions endpoint includes it only when the count is non-zero. On gRPC workers SGLang reports them, and SMG counts them itself for gpt-oss (Harmony) models | | `completion_tokens_details.accepted_prediction_tokens` | Speculative decoding: draft tokens the target model accepted | | `completion_tokens_details.rejected_prediction_tokens` | Speculative decoding: draft tokens rejected (drafted minus accepted). Omitted when the drafted count is unknown — see below | On gRPC workers SMG builds `usage` from the engine's counters; HTTP workers return the engine's own usage object. SMG reads the speculative counters from SGLang, TokenSpeed, vLLM (started with `--per-request-spec-decode-metrics`), and TensorRT-LLM (once its servicer fills the fields); MLX reports none. An engine does not always report a usable drafted count next to the tokens it accepted, so the two speculative fields follow different rules. `accepted_prediction_tokens` carries the accepted count whenever the engine reports speculative activity. `rejected_prediction_tokens` is drafted minus accepted, reported only when every choice supplied a positive drafted count at least as large as its accepted count. If any choice pairs measured speculation with a zero drafted count, or reports fewer drafted than accepted tokens, the rejection count is unknown and the key is omitted rather than reported as `0`. Without any speculative decoding a chat completion reports both fields as `0`; the legacy Completions endpoint includes them only with speculative activity. A request's streamed and non-streamed answers carry the same usage object. The running snapshots requested with [`continuous_usage_stats`](#stream-usage) follow the same rules: a choice whose final counters are still outstanding counts as a zero drafted count, so measured speculation on another choice leaves the rejection count unknown. #### Streaming Response With `"stream": true`, responses are sent as Server-Sent Events: ```bash curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}], "stream": true, "stream_options": {"include_usage": true} }' ``` Response (abbreviated): ``` data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]} data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{"content":"Hello!"},"finish_reason":null}]} data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]} data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312345,"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[],"usage":{"prompt_tokens":9,"completion_tokens":2,"total_tokens":11}} data: [DONE] ``` An error before the first chunk is an ordinary JSON error response with its own status; when a worker rejects a streaming request, SMG keeps the worker's content type (`application/json`, not `text/event-stream`) so SSE clients still see the error body. If generation fails after a gRPC stream has started, SMG sends `data: {"error": {"message": "...", "type": "..."}}` followed by `data: [DONE]`. The `type` is `stream_error` when the worker fails the stream: its connection breaks, its server cuts the stream, it answers a server error or a timeout after the first chunk, or the engine itself reports that the request failed. Any other status the engine answers mid-stream keeps the `type` it would carry in an error body (see [Error Format](#error-format)): for example `invalid_request_error` when the engine rejects the request after its first chunk, or `rate_limit_error` for capacity pushback. `internal_error` means the failure was in SMG's own processing. A quiet open SSE response can contain `: ping` comments between events. SMG sends them after 15 seconds of silence by default, including while waiting for the first token. SSE clients should ignore comment lines. Configure or disable them with [`--sse-keepalive-secs`](../configuration.md#streaming-keepalives). #### Stream Usage | `stream_options` field | Default | Effect | |------------------------|---------|--------| | `include_usage` | `false` (Kimi, MiniMax: `true`) | Send the whole request's usage in a final chunk with an empty `choices` array | | `continuous_usage_stats` | `false` | gRPC workers, except for gpt-oss (Harmony) models: with `include_usage: true`, every chunk also carries a running `usage` snapshot (the prompt plus the tokens generated so far) | On HTTP workers `stream_options`, including keys SMG does not model, is forwarded to the engine, which produces the usage chunks. The Kimi and MiniMax profiles turn `include_usage` on for streaming requests that leave it unset, on every backend; an explicit `false` is kept. The DeepSeek V4 profile uses DeepSeek's stream format on gRPC workers instead: aggregate usage rides on the final finish chunk whether or not `include_usage` is set, and no usage-only chunk follows. With `include_usage: true`, earlier chunks carry `"usage": null`. `continuous_usage_stats` does not apply, and if the backend stream ends before every choice completes, the finish chunk carries no aggregate usage. --- ### Completions Create a text completion (legacy API). ``` POST /v1/completions ``` #### Request Body | Field | Type | Required | Description | |-------|------|----------|-------------| | `model` | string | Yes | Model identifier | | `prompt` | string/array | Yes | A string, an array of strings, an array of token IDs, or an array of token-ID arrays; an array must be non-empty | | `max_tokens` | integer | No | Maximum tokens to generate | | `temperature` | number | No | Sampling temperature (0-2) | | `top_p` | number | No | Nucleus sampling parameter, greater than 0 and at most 1 | | `n` | integer | No | Completions per prompt (1-128) | | `best_of` | integer | No | Candidates to generate server-side (0-20); must be greater than `n` and cannot be combined with `stream` | | `logprobs` | integer | No | Log probabilities of the most likely tokens (0-20, the bound of chat's `top_logprobs`; the public API's own limit of 5 is not imposed) | | `stream` | boolean | No | Enable streaming | | `stream_options` | object | No | `include_usage`; only allowed with `stream: true` | | `stop` | string/array | No | Up to 4 non-empty stop sequences | | `echo` | boolean | No | Echo prompt in response | | `suffix` | string | No | Text that comes after the completion | | `presence_penalty`, `frequency_penalty` | number | No | Penalties (-2 to 2) | | `seed` | integer | No | Seed for best-effort deterministic sampling | | `user` | string | No | End-user identifier | Completions takes the same [extension fields](#extension-fields) as Chat Completions except the chat-only ones (`continue_final_message`, `add_generation_prompt`, `separate_reasoning`, `stream_reasoning`, `chat_template_kwargs`), plus a `json_schema` string constraint. On gRPC workers each prompt in an array is its own engine request. A prompt can arrive already tokenized: an array of token IDs (unsigned 32-bit integers, such as those [`/v1/tokenize`](extensions.md#protected-utility-endpoints) returns), or an array of token-ID arrays for a batch. The first element of an array fixes its shape — a string makes it an array of text prompts, an integer an array of token IDs, an array a batch of token-ID prompts. A mixed array, a negative, fractional, or out-of-range ID, an empty `prompt` array, and an empty ID list in a batch are all rejected. On gRPC workers token IDs bypass the gateway's tokenizer and reach the engine exactly as sent. `usage.prompt_tokens` counts the IDs, and with `echo: true` each choice opens with the decoded text of its prompt's IDs (special tokens included; an ID the tokenizer has no text for adds none). On that path every ID must be a token of the model — in its tokenizer's vocabulary, added tokens included, or within the embedding table its workers report — otherwise the request is rejected with `invalid_prompt_token` before any worker is called (see [Error Codes](#error-codes)). On HTTP workers the prompt is forwarded as sent and the engine applies its own handling. #### Example Request ```bash curl http://localhost:30000/v1/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "The quick brown fox", "max_tokens": 50 }' ``` #### Response ```json { "id": "cmpl-abc123", "object": "text_completion", "created": 1705312345, "model": "meta-llama/Llama-3.1-8B-Instruct", "choices": [ { "text": " jumps over the lazy dog.", "index": 0, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 4, "completion_tokens": 7, "total_tokens": 11 } } ``` --- ### Decisions Ask predicate (yes/no), choice, and score questions about shared evidence. The model scores each question's possible answers instead of generating text, and every answer carries a probability distribution. ``` POST /v1/decisions ``` Requests are always treated as non-streaming: there is no `stream` field, and SMG returns a single JSON body. #### Request Body | Field | Type | Required | Description | |-------|------|----------|-------------| | `model` | string | Yes | Model identifier. A name configured with `--model-alias` is accepted and resolved to its canonical model | | `input` | string/array | Yes | The evidence every question is answered against: a plain string, or an array of `user` messages (see [Decision Input](#decision-input)) | | `questions` | array | Yes | Questions to answer; answers come back in the same order (see [Decision Questions](#decision-questions)) | | `safety_identifier` | string | No | Opaque end-user identifier, at most 128 characters | Fields SMG does not model — on the request and on its messages, parts, questions, choices, and levels — are kept as backend extensions: they are forwarded unchanged to non-SGLang HTTP workers and rejected with `400` on SGLang workers (see [Decision Workers](#decision-workers)). #### Decision Input As an array, each `input` message has: | Field | Type | Required | Description | |-------|------|----------|-------------| | `role` | string | Yes | Must be `user` | | `type` | string | No | `message`, when present | | `content` | string/array | Yes | Text, or an array of `input_text` and `input_image` parts | | Part `type` | Payload | Limits | |-------------|---------|--------| | `input_text` | `text` | At most 10,485,760 characters per part | | `input_image` | `image_url`, optional `detail` (`low`, `high`, `auto`, `original`) | Must be an inline base64 image data URL (`data:image/png;base64,...`); external URLs and file IDs are rejected. At most 128 images per request | `model`, `instructions`, `name`, `description`, and `label` values each take at most 1,048,576 characters. A limit violation is rejected with `400` before routing. #### Decision Questions Every question has a `type`, required `instructions`, and an optional `name`. Names need not be unique: each answer repeats its question's `name` (`null` when unset), and answers keep question order. | `type` | Extra fields | The model estimates | |--------|--------------|---------------------| | `predicate` | — | The probability that the statement in `instructions` is true | | `choice` | `choices`: array of `{"value": ...}` with an optional `description`; `value` is a string or boolean, and `true` is distinct from `"true"` | A probability for each choice | | `score` | `levels`: array of `{"label": "..."}` with an optional `description`, ordered from lowest to highest | A probability for each level | #### Example Request ```bash curl http://localhost:30000/v1/decisions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen/Qwen3-4B-Instruct-2507", "input": "The delivery arrived a day late, but the package was intact and support was friendly.", "questions": [ {"type": "predicate", "instructions": "Did the delivery arrive on time?"}, {"type": "choice", "name": "sentiment", "instructions": "What is the overall sentiment?", "choices": [{"value": "positive"}, {"value": "negative"}]}, {"type": "score", "instructions": "Rate the customer experience.", "levels": [{"label": "poor"}, {"label": "fair"}, {"label": "good"}]} ] }' ``` #### Response ```json { "model": "Qwen/Qwen3-4B-Instruct-2507", "answers": [ {"type": "predicate", "name": null, "probability": 0.11}, {"type": "choice", "name": "sentiment", "choice": "positive", "confidence": 0.74, "probabilities": [ {"value": "positive", "probability": 0.87}, {"value": "negative", "probability": 0.13} ]}, {"type": "score", "name": null, "score": 1.62, "confidence": 0.55, "probabilities": [ {"value": 0, "label": "poor", "probability": 0.08}, {"value": 1, "label": "fair", "probability": 0.22}, {"value": 2, "label": "good", "probability": 0.70} ]} ], "usage": { "input_tokens": 182, "output_tokens": 0, "total_tokens": 182, "input_tokens_details": {"cached_tokens": 0, "cache_write_tokens": 0}, "output_tokens_details": {"reasoning_tokens": 0} } } ``` Each answer carries its question's `type` and `name` plus: | `type` | Fields | |--------|--------| | `predicate` | `probability` (0–1) that the statement is true | | `choice` | `choice` — the `value` of the most probable choice; `confidence` (0–1); `probabilities` — one `{value, probability}` entry per choice, in request order | | `score` | `score` — the probability-weighted average of the level indices, not a winning level (level `i` in request order has `value: i`, so 3 levels give a score between 0 and 2); `confidence` (0–1); `probabilities` — one `{value, label, probability}` entry per level | | `refusal` | No further fields: the worker declined to answer the question. SMG's own SGLang scoring always produces one of the three types above | `usage` reflects scoring work: on SGLang workers (both the HTTP translation and gRPC scoring) the answer labels are scored without generating tokens, so `output_tokens` and `output_tokens_details.reasoning_tokens` are `0` and `total_tokens` equals `input_tokens`. On gRPC workers each question is scored as its own prompt over the shared input — `input_tokens` sums those prompts — and `input_tokens_details.cached_tokens` reports the engine's prefix-cache hits. The SGLang HTTP translation returns `0` for both cache counters, meaning unreported rather than measured. Non-SGLang HTTP workers return their own usage object. #### Decision Workers What serves the request depends on the worker: | Worker | Behavior | |--------|----------| | HTTP (SGLang runtime) | SMG translates the request to SGLang's native System One API (`POST /v1/systemone`, as served by SGLang 0.5.21) and maps the reply back to the Decisions shape | | HTTP (other runtimes) | The request is forwarded to the worker's own `/v1/decisions`, unknown fields included; the worker's response — errors included — is returned as is | | gRPC (regular mode, SGLang runtime) | SMG renders one prompt per question with the model's chat template and scores the answer labels through the engine, without generation | | PD, EPD, ZMQ, and external providers | Not implemented (`501` `decisions_not_supported`) | In a mixed fleet, SMG dispatches Decisions only to regular HTTP workers and regular SGLang gRPC workers serving the model. The SGLang paths (HTTP translation and gRPC scoring) reject these requests with `400` `unsupported_decisions_request` before dispatch: - `input_image` parts — SGLang Decisions takes text evidence only - unknown fields anywhere on the request — SGLang Decisions cannot honor them - empty or whitespace-only text evidence or `instructions` - no questions, a `choice` question outside 1–255 choices, or a `score` question outside 1–10 levels gRPC scoring also depends on the model: the internal answer labels SMG scores (`yes`/`no`, option letters, level digits — your own choice values and level labels are unrestricted) must each be one distinct token after the chat prompt, and the model's chat template must produce plain text without leaving a reasoning block open. A model that cannot satisfy this answers `400`. #### Decision Errors | Status | `code` | Returned when | |--------|--------|---------------| | 400 | The failure's code (see [Error Format](#error-format)) | The body is not valid JSON (`invalid_json`), does not match the schema (`invalid_value` for a non-`user` role or an unknown question `type`, `invalid_type` for a numeric choice `value`, ...), or fails a limit (`invalid_value` for a length limit, `too_many_images`, `invalid_image_url`); `param` names the offending field when one is known | | 400 | `unsupported_decisions_request` | An SGLang worker was selected and the request needs something SGLang Decisions does not support (see [Decision Workers](#decision-workers)) | | 404 | `model_not_found` | No worker serves the model | | 501 | `decisions_not_supported` | No regular HTTP worker and no regular SGLang gRPC worker serves the model | | 502 | `invalid_decisions_response` | An SGLang worker (HTTP translation or gRPC scoring) reported success, but the result could not be mapped to a valid Decisions response | Backend errors on the SGLang HTTP translation keep their status and OpenAI error body. For tenant rate limiting on Decisions, see [Tenant Rate Limiting](../tenant-rate-limiting.md). --- ### List Models List available models. ``` GET /v1/models ``` #### Example Request ```bash curl http://localhost:30000/v1/models ``` #### Response ```json { "object": "list", "data": [ { "id": "meta-llama/Llama-3.1-8B-Instruct", "object": "model", "created": 0, "owned_by": "self_hosted" } ] } ``` The list covers the models served by self-hosted workers, each once under its canonical ID, followed by one entry per alias (`--model-alias` names and model-card `aliases` entries), so every name a client may send is one it can discover. An alias is listed under the model the registry resolves it to and nowhere else: an alias two workers' cards declare appears once, and a name that is also a registered model's ID is never listed as an alias. `created` is always `0` and `owned_by` is `self_hosted`. When no self-hosted worker serves any model, the endpoint returns `503` in the gateway's JSON [error envelope](#error-format) with the code `no_models_available`. The endpoint needs no key. If the caller does send a bearer token (or an `x-api-key` header) that is not one of the gateway's own keys, SMG treats it as the caller's own provider key: it calls `/v1/models` on the registered external providers of the caller's API family with that token and returns the first non-empty list, falling back to the self-hosted list. A request carrying `anthropic-version` reaches the Anthropic providers alone; every other request reaches the OpenAI-compatible ones — OpenAI, xAI, custom providers, and external workers that name no provider — so a key presented in one family's API never reaches another family's provider, and a Gemini provider is asked by neither. A gateway key (`--api-key` or `--tenant-api-key`) is never forwarded to a provider. On that provider list, `owned_by` is the provider name for the recognized provider hosts (`openai`, `anthropic`, `xai`, `gemini`) and `self_hosted` for a provider at any other host. A request carrying an `anthropic-version` header gets its inventory in the Anthropic Messages shape instead, with that API's paging read from the query: `limit` (1 to 1000, default 20), one cursor (`after_id` or `before_id`, not both), and a `lifecycle` filter (`active`, `deprecated`, or `retired`; every self-hosted model is `active`). The page carries `has_more`, `first_id`, and `last_id`, and its `data` entries `type`, `id`, `display_name` (the model ID when no display name is known), `created_at` (`1970-01-01T00:00:00Z`, the counterpart of `created: 0`), `lifecycle`, and the model card's context length as `max_tokens` and `max_input_tokens`. An out-of-range `limit`, both cursors at once, an unknown `lifecycle`, or a cursor naming no entry is a `400` `invalid_query_parameter`. Errors then come in the Messages envelope (top-level `type: "error"` and a `request_id`). The same header selects the shape on [Retrieve Model](#retrieve-model). On the provider fallback, a Messages-shaped caller's upstream is asked for the page its query names and an Anthropic-shaped page is relayed as the upstream served it; an OpenAI-shaped inventory is paged at the gateway in id order. --- ### Retrieve Model Fetch one model's card. ``` GET /v1/models/{model} ``` Like [List Models](#list-models), this is a public route and answers without a key. `{model}` takes the whole rest of the path, so an ID with a slash in it works raw or percent-encoded: `/v1/models/meta-llama/Llama-3.1-8B-Instruct` and `/v1/models/meta-llama%2FLlama-3.1-8B-Instruct` name the same model. An alias — a name configured with `--model-alias`, or an `aliases` entry on a worker's model card (see the [Worker Spec](admin.md#worker-spec)) — resolves to its canonical model, and the response carries the canonical ID. The route answers from the self-hosted inventory only; it never asks an external provider. #### Example Request ```bash curl http://localhost:30000/v1/models/meta-llama/Llama-3.1-8B-Instruct ``` #### Response ```json { "id": "meta-llama/Llama-3.1-8B-Instruct", "object": "model", "created": 0, "owned_by": "self_hosted" } ``` With an `anthropic-version` header, the response is the Anthropic model object (`type`, `id`, `display_name`, `created_at`, `lifecycle`, and the card's context length as `max_tokens` and `max_input_tokens`) instead. An unknown ID gets the same `404` `model_not_found` as every other route, in the OpenAI [error envelope](#error-format) — or, under `anthropic-version`, in the Messages envelope (top-level `type: "error"`, `error.type` `not_found_error`, and a `request_id`), the same refusal `/v1/messages` gives that model. --- ### Audio Transcriptions Transcribe an audio file (batch). ``` POST /v1/audio/transcriptions ``` The request is `multipart/form-data`: | Field | Required | Description | |-------|----------|-------------| | `file` | Yes | The audio file; must not be empty | | `model` | Yes | Model identifier | | `language` | No | Language of the audio | | `prompt` | No | Text to guide the transcript | | `response_format` | No | Output format (default `json`) | | `temperature` | No | Sampling temperature (0.0-1.0) | | `timestamp_granularities[]` | No | `word` or `segment`; repeat the field for both | | `stream` | No | `true`/`false` (or `1`/`0`) | A malformed form, a missing or empty `file`, a missing `model`, an out-of-range `temperature`, or an unparseable `stream` is rejected with `400` in the standard [error envelope](#error-format) before routing, with `param` naming the field when one is known: `missing_required_parameter` for a missing `model` or `file`, `invalid_value` for an empty `file` or a bad `temperature` or `stream` value, and `invalid_multipart` for a form that cannot be read. Other form fields are ignored. ```bash curl http://localhost:30000/v1/audio/transcriptions \ -F file=@audio.wav \ -F model=Qwen/Qwen3-ASR-1.7B ``` What serves the request depends on the worker: | Worker | Behavior | |--------|----------| | HTTP | The form is forwarded to the worker's `/v1/audio/transcriptions`; the worker's model and options apply | | gRPC (regular mode) | SMG runs transcription as its own pipeline endpoint: it turns the audio into a chat turn, renders it with the model's chat template, and returns the transcript. Supported family: Qwen3-ASR | | PD, EPD, and external providers | Not implemented (`501`) | On gRPC, a model belongs to the Qwen3-ASR family when its ID, or a serving worker's model ID or model-identity label, contains `qwen3-asr` or `qwen3_asr` (ignoring case). Audio input over gRPC is currently accepted by TokenSpeed workers only; other gRPC runtimes answer `400` `multimodal_not_supported`. The gRPC endpoint is whole-file only: | Request | gRPC result | |---------|-------------| | A model outside the supported families | `400` `audio_transcription_model_not_supported` | | `stream: true` | `400` `streaming_transcription_not_supported` | | Any `timestamp_granularities` | `400` `transcription_timestamps_not_supported` | | `response_format` of `srt`, `vtt`, or `verbose_json` | `400` `unsupported_transcription_response_format` | | A `language` outside the model's list (ISO code or English name, such as `en` or `English`) | `400` `unsupported_transcription_language` | | A `prompt` over 4096 bytes | `400` `asr_prompt_too_long` | `response_format: json` returns `{"text": "..."}` and `text` returns the plain transcript. Sampling is greedy unless `temperature` is set, and a given `language` pins the transcript language. --- ### Realtime API SMG proxies the OpenAI Realtime API to a realtime-capable worker. Both the OpenAI router (to an upstream provider) and the HTTP router (to a **local** worker labeled [`realtime: "true"`](../../getting-started/multiple-workers.md#realtime-capable-workers)) support it. The worker must speak the OpenAI Realtime protocol — for local workers, for example vLLM serving an ASR model with the realtime task. SMG relays frames as the worker sent them; error frames are the one exception (see [Upstream Errors](#upstream-errors)). | Endpoint | Transport | Purpose | |----------|-----------|---------| | `GET /v1/realtime` | WebSocket | Bidirectional realtime session (e.g. live streaming transcription) | | `POST /v1/realtime/calls` | WebRTC (SDP) | Browser/WebRTC realtime session | | `POST /v1/realtime/sessions` | HTTP | Create a realtime session | | `POST /v1/realtime/client_secrets` | HTTP | Mint an ephemeral client secret | | `POST /v1/realtime/transcription_sessions` | HTTP | Create a realtime transcription session | `GET /v1/realtime` requires the `model` query parameter and answers `400` without it. `POST /v1/realtime/calls` takes the model from the `model` query parameter for an `application/sdp` body, or from the session JSON in a `multipart/form-data` body. #### Upstream Errors A refusal loses the account identifiers in a JSON error's `error.message` before it reaches the caller: phrases naming an organization (`organization org-…`) or a project (`project proj_…`) are removed, as on every provider relay (see [Provider Error Responses](../../getting-started/external-providers.md#provider-error-responses)). Both routers do this, so error frames from local realtime workers are sanitized too. - A non-success answer to a realtime REST call or to the `/v1/realtime/calls` SDP exchange is returned under the upstream's status and content type with its body sanitized this way. - On a WebSocket session, `error` events and `conversation.item.input_audio_transcription.failed` events are the only frames SMG rewrites, with the same rule; every other frame is relayed as it came, never re-serialized. #### WebSocket example ```python # pip install websockets import asyncio, websockets async def main(): url = "ws://localhost:30000/v1/realtime?model=Qwen/Qwen3-ASR-1.7B" headers = {"Authorization": "Bearer your-api-key"} async with websockets.connect(url, additional_headers=headers) as ws: # Send realtime events (session.update, input_audio_buffer.append, ...) # and receive transcription/response events from the worker. ... asyncio.run(main()) ``` --- ## Provider Profiles Some model families come with a vendor contract that differs from the OpenAI baseline. SMG picks a provider profile from the request's `model` and applies that profile's defaults and validation to Chat Completions requests before routing, on every backend. ### Profile Selection A profile matches when any `/`-separated segment of the model ID starts with one of its markers, ignoring case, so `moonshotai/Kimi-K2-Instruct` and `/models/kimi-k3` both select Kimi. | Profile | Selected by | |---------|-------------| | Kimi | `kimi*`, `moonshot*`; the K3 rules apply to `kimi-k3*` and `kimi_k3*` | | MiniMax | `minimax*`, `abab*` | | z.ai (GLM) | `glm*`, `zai*`, `z-ai*`; the GLM-5.3 rules apply to `glm-5.3*`, `glm5.3*`, `glm_5.3*`, `glm-5-3*`, `glm5-3*` | | DeepSeek V4 | A segment equal to `deepseek-flash`, `deepseek-v4-flash`, `deepseek-v4-pro`, `deepseek-v4.1-flash`, or `deepseek-v4.1-flash-1017` | | OpenAI | A model none of the above matched that names one of OpenAI's own hosted models: an `openai` segment (`openai/gpt-4.1`), a segment starting with `gpt-` (`gpt-oss*` is self-hosted and excluded), `chatgpt-`, `codex-`, `computer-use-preview`, `text-embedding-` or `omni-moderation`, or the `o1`/`o3`/`o4` reasoning series (`o3`, `o4-mini`) | | Baseline (default) | Everything else, including older DeepSeek models and `THUDM/chatglm3-6b`: only the core request validation applies | Names configured with `--model-alias` are read through the alias table. An alias takes the profile of the served model it resolves to, so the profile is the same whichever of its names a request carries. A served model whose own name selects no vendor contract takes the contract of a vendor-named (Kimi, MiniMax, z.ai, DeepSeek) alias it is served under, series rules included: an opaque alias of a Kimi K3, GLM-5.3, or DeepSeek V4.1 model keeps that series' rules. When two aliases of one served model name different vendors, the first configured wins. An OpenAI model name binds no contract as an alias: with `gpt-4o=local-model`, a request for `gpt-4o` is judged as the self-hosted `local-model`, not under the OpenAI rules. Message-level extension fields that belong to another profile (such as Kimi's message `tools` sent to a non-Kimi model) are dropped before dispatch, with one warning logged per request. Profile defaults and validation apply to `/v1/chat/completions` requests that pass through SMG's request validation. They are not applied to `/v1/responses`, and a request the HTTP router streams to the worker unparsed skips them (see [Request Streaming](../../concepts/performance/request-streaming.md)). ### Validation Rules A violation is rejected with `400` in the validation envelope (see [Error Format](#error-format)): `error.type` is `invalid_request_error`, `error.code` is the rule code from the last column, and `error.message` carries the rule's message. Profile rules judge the request as a whole, so `error.param` is null: ```json { "error": { "message": "invalid thinking.type: only enabled or disabled is allowed for this model", "type": "invalid_request_error", "param": null, "code": "thinking_type_not_supported" } } ``` | Profile | Rejected | Rule code | |---------|----------|-----------| | All except MiniMax | A `root` message | `invalid_role` | | Kimi | `tools` on a `user` or `assistant` message | `tools_role_restricted` | | Kimi | `tools` on a `system` or `developer` message that is not a list of tools | `tools_malformed` | | Kimi | A non-empty message `tools` list together with non-empty content on that message | `tools_content_conflict` | | Kimi | A message tool whose `type` is not `function` | `tool_type_unsupported` | | Kimi | A message tool name not matching `[A-Za-z_][A-Za-z0-9_]*`, or longer than 256 characters | `tool_name_invalid` | | Kimi | A message tool name already declared in `tools` or on another message | `tool_name_duplicate` | | Kimi K3 | `temperature` other than 0, 0.6, or 1; `top_p` other than 0.95; a non-zero `presence_penalty` or `frequency_penalty`; `n` other than 1 | `temperature_not_allowed`, `top_p_not_allowed`, `presence_penalty_not_allowed`, `frequency_penalty_not_allowed`, `n_not_allowed` | | Kimi K3, z.ai, DeepSeek V4 | `thinking.type: adaptive` | `thinking_type_not_supported` | | Kimi K3 | `thinking.effort` other than `low`, `high`, or `max` | `thinking_effort_invalid` | | MiniMax | A tool-call ID used twice in the history | `tool_call_id_duplicate` | | MiniMax | A `tool` message whose `tool_call_id` answers no pending call | `tool_call_id_mismatch` | | MiniMax | Historical tool-call `arguments` that are non-empty but not a JSON object | `tool_call_arguments_invalid_json` | | MiniMax | A tool call with no `tool` message answering it | `tool_call_unanswered` | | z.ai | `thinking.effort` or `reasoning_effort` outside `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max` (GLM-5.3 also rejects `none`: the series cannot switch thinking off) | `reasoning_effort_not_allowed` | | z.ai GLM-5.3 | `thinking.type: disabled` | `thinking_disabled_not_supported` | | z.ai | A tool whose `type` is not `function` | `tool_type_not_supported` | | z.ai | A `file_url` content part (SMG does not fetch files for the model) | `content_part_not_supported` | | z.ai (gRPC) | `max_completion_tokens` (or `max_tokens`) larger than the context window the selected worker advertises | `context_length_exceeded` | | DeepSeek V4 | `reasoning_effort` or `thinking.effort` outside `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`; V4.1 also takes an integer budget from 1 to 100 | `reasoning_effort_not_allowed` | | DeepSeek V4 | `chat_template_kwargs.thinking` or `enable_thinking` that is neither a boolean nor `null` | `invalid_thinking_override` | | DeepSeek V4 | `chat_template_kwargs.thinking` and `enable_thinking` that disagree | `conflicting_thinking_overrides` | | OpenAI, DeepSeek V4 | A `tool` message with no assistant `tool_calls` turn right before it; a `tool_call_id` that turn did not issue; an assistant `tool_calls` turn whose calls are not each answered by a `tool` message before the conversation goes on | `tool_message_without_tool_calls`, `tool_call_id_not_found`, `tool_calls_without_tool_messages` | | OpenAI | `max_tokens` and `max_completion_tokens` both set | `invalid_parameter_combination` | | OpenAI | `response_format: json_object` without the word "json" (any case) in a message's text | `json_object_requires_json_in_messages` | | OpenAI | A `json_schema` response format with `strict: true` whose schema contains an object without `"additionalProperties": false` | `invalid_json_schema` | | OpenAI | A tool with `function.strict: true` whose parameter schema contains an object without `"additionalProperties": false` | `invalid_function_parameters` | z.ai and DeepSeek V4 check both effort spellings, even the one that `thinking.effort` shadows. The z.ai (gRPC) context-window rule needs the selected worker's advertised window, so it is enforced in the gRPC pipeline rather than in request validation: alone in this table, its `400` carries the `X-SMG-Error-Code` header. ### Defaults and Rewrites | Profile | Behavior | |---------|----------| | Kimi | Streaming requests get `stream_options.include_usage: true` unless set. Tools declared on `system` and `developer` messages join the request's `tools` for `tool_choice`, the tool-choice constraint, and tool-call parsing; a `system` message that only declares tools may omit `content` | | Kimi K3 | Omitted fields get the contract values: `temperature` 1.0, `top_p` 0.95, `presence_penalty` 0, `frequency_penalty` 0, `n` 1 | | MiniMax | Streaming requests get `stream_options.include_usage: true` unless set. `root` messages are hoisted above all other messages and merged into one leading message that keeps the `root` role: texts are joined by a blank line in their original order, a parts list keeps its parts, and the merged message carries the first one's `name`. On gRPC workers, responses are scanned for tool calls even when the request declares no tools | | z.ai | `thinking.clear_thinking` becomes the `clear_thinking` chat-template variable unless `chat_template_kwargs` sets it. `tool_stream` is accepted and dropped (SMG streams tool-call deltas anyway). GLM-5.x, GLM-4.7, and GLM-4.6 get `temperature` 1.0 and `top_p` 0.95 when omitted | | z.ai GLM-5.3 | Effort `minimal` → `low`, `medium` → `high`, and `xhigh` → `max` in both `reasoning_effort` and `thinking.effort`, matching z.ai's own API; workers and external providers see only the level | | DeepSeek V4 | Effort `minimal` → `low`; on V4, `medium` and `xhigh` → `high` (V4.1 keeps both as native levels, though a request forwarded to an HTTP worker or external provider still sends `medium` as `high`, matching DeepSeek's own API). Thinking is on unless turned off (`thinking.type: disabled`, effort `none`, or a template override), and the result is written to `chat_template_kwargs.thinking` when that is unset. Streams use DeepSeek's usage format (see [Stream Usage](#stream-usage)) | --- ## Error Responses ### Error Format Errors that SMG generates use this envelope and repeat `code` in the `X-SMG-Error-Code` response header: ```json { "error": { "type": "invalid_request_error", "code": "context_length_exceeded", "message": "This model's maximum context length is 2048 tokens. However, your request has 2310 input tokens. Please reduce the length of the input.", "param": null } } ``` `type` is the public API's vocabulary, which clients switch on: `invalid_request_error` for a 4xx (the detail is in `code`; a 403 is `permission_error` and a 429 `rate_limit_error`) and `server_error` for a 5xx. Request parsing and validation failures use the same envelope but carry no `X-SMG-Error-Code` header. Their `code` names the failure, and `param` names the offending field when one is known (for example `messages[0].content`); it is null for a syntax error or a rule over the whole request: | Failure | `code` | |---------|--------| | The body is not valid JSON | `invalid_json` | | The body does not match the request schema | `missing_required_parameter`, `unknown_parameter`, `invalid_type` (a value of the wrong shape, including a non-object body), `invalid_value` (a value outside an enumeration), or `integer_below_min_value` (a negative integer where the field cannot be negative) | | A field crosses a documented bound | The kind of value and the bound crossed: `integer_above_max_value`, `decimal_below_min_value`, `string_above_max_length`, `empty_array`, ... | | A cross-field or [provider-profile](#provider-profiles) rule fails | The rule's own code: `stream_options_requires_stream`, `tool_choice_requires_tools`, `mutually_exclusive_parameters`, ... | | The `Content-Type: application/json` header is missing, or the body fails to parse for any other reason | `json_parse_error` | A `401` from API-key authentication uses the envelope (`code` `invalid_api_key`) with a `WWW-Authenticate: Bearer` challenge, but carries no `X-SMG-Error-Code` header either. Errors from HTTP workers are relayed with the worker's status and body. An external provider's error is relayed under its status too, minus the account identifiers in its message (see [Provider Error Responses](../../getting-started/external-providers.md#provider-error-responses)). SMG never forwards a worker's `X-SMG-Error-Code` header, so that header always means SMG itself produced the error. ### Error Codes | Status | `code` | Returned when | |--------|--------|---------------| | 400 | `invalid_json` | The body is not valid JSON | | 400 | `missing_required_parameter`, `unknown_parameter`, `invalid_type`, `invalid_value`, `integer_below_min_value` | The body does not match the request schema; `param` names the field when one is known | | 400 | `integer_above_max_value`, `decimal_below_min_value`, `string_above_max_length`, `empty_array`, ... | A field is outside its documented bounds; the code names the kind of value and the bound crossed, `param` the field | | 400 | The rule's code | A cross-field rule (`stream_options_requires_stream` for `stream_options` without `stream`, `top_logprobs_requires_logprobs`, `tool_choice_function_not_found` for a `tool_choice` naming a missing tool, `json_schema_schema_not_object` for a non-object `json_schema`, ...) or a [provider-profile](#provider-profiles) rule failed | | 400 | `json_parse_error` | The `Content-Type: application/json` header is missing, or the body failed to parse for a reason outside the set above | | 400 | `context_length_exceeded` | gRPC: the longest prompt is longer than the context window the selected worker advertises (the tighter leg governs in PD, and each prompt of a batched completion counts on its own). Workers that advertise no window are not checked | | 400 | `invalid_prompt_token` | gRPC: a completion's token-ID prompt holds an ID that is not a token of the model. The message names the position and value (`prompt[2] = ...`, or `prompt[1][1] = ...` in a batch); an invalid ID that only the decode catches is reported as a failed decode instead. Refused before any worker is called | | 400 | `unsupported_content_part` | gRPC: a content part SMG cannot render | | 400 | `audio_output_not_supported`, `unsupported_output_modality` | gRPC: `return_audio: true` or a non-`text` entry in `modalities`; gRPC workers produce text only | | 400 | `multimodal_not_supported` | gRPC: the selected engine does not accept one of the request's input modalities | | Engine-mapped | `start_generation_failed` | gRPC: the engine did not start generation. The status follows the engine's gRPC status, for example invalid argument → `400` (not retried and not counted against the circuit breaker), resource exhausted → `429`, unavailable → `503`, deadline exceeded → `504`, internal → `500` | | Engine status, or 502 | `prefill_worker_failed_to_start`, `decode_worker_failed_to_start` | PD on either transport: the prefill or decode leg never started. The engine refused it and its status is kept (over gRPC, mapped from the engine's gRPC status as above), or the call to its worker failed below HTTP (`502`) | | Engine status, or 502 | `prefill_stream_error`, `decode_stream_error` | PD on either transport: the leg's stream ended with an error after it began, with the answer to the client still whole. Over gRPC the engine's status is kept (mapped as above); over HTTP the leg's body was cut after its `2xx` head (`502`). A failure after the client's own stream has started ends it with a `stream_error` event instead (see [Streaming Response](#streaming-response)) | | 500 | `pd_rendezvous_failed` | gRPC PD: the decode leg failed before producing anything, and the prefill's bootstrap port, probed now, no longer accepts connections — the sides could not rendezvous. The message says the port did not accept a connection, with the probe's error; the gateway log names the port's address | | 401 | `invalid_api_key` | Missing or unknown API key. The response adds a `WWW-Authenticate: Bearer` challenge (`error="invalid_token"` when a key was presented) | | 404 | `model_not_found` | No worker serves the model | | 408 | `request_body_stalled` | The request body stalled for `--stream-body-stall-timeout-secs` while being read, whether it is buffered or streamed to the worker | | 408 | `request_body_timeout` | The request body was still incomplete at `--request-timeout-secs`, whether it is buffered or streamed to the worker | | 413 | `request_body_too_large` | The request body exceeds `--max-payload-size`, however it arrives: a declared `Content-Length` over the limit (refused before routing), a buffered or chunked body that crosses it while being read, or a streamed body that grows past it mid-upload | | 429 | `admission_queue_full`, `scheduler_queue_full`, `tenant_rate_limit_exceeded`, `pd_prefill_queue_full`, `pd_prefill_queue_timeout` | Admission control or rate limiting; see [Rate Limiting](#rate-limiting) | | 500 | `internal_error` and stage-specific codes | A failure inside SMG; the code names the failing step | | 502 | `upstream_response_too_large` | A worker's non-streaming response exceeded `--max-payload-size` | | 503 | `no_available_workers` | Workers serve the model, but every one is unhealthy or has an open circuit breaker, or the routing policy selects none of them (HTTP and gRPC paths) | | 503 | `no_models_available` | `GET /v1/models`: no self-hosted worker serves any model (see [List Models](#list-models)) | | 503 | `worker_overload_protection_shed`, `admission_queue_timeout`, `selection_queue_timeout`, `scheduler_queue_timeout`, `scheduler_preempted` | Overload and admission sheds; see [Rate Limiting](#rate-limiting) | | 504 | `prefill_worker_silent`, `decode_worker_silent` | gRPC PD: the leg produced nothing within its liveness bound. Retryable, and the retry window closes on that worker alone — its partner stays selectable | On gRPC workers, an engine finish reason of `error` (outside the OpenAI set) becomes an error response or stream error; clients never receive `"finish_reason": "error"`. --- ## Client Libraries ### Python (OpenAI SDK) ```python from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="your-api-key" # or "not-needed" if auth disabled ) response = client.chat.completions.create( model="meta-llama/Llama-3.1-8B-Instruct", messages=[ {"role": "user", "content": "Hello!"} ] ) print(response.choices[0].message.content) ``` ### JavaScript/TypeScript ```typescript import OpenAI from 'openai'; const client = new OpenAI({ baseURL: 'http://localhost:30000/v1', apiKey: 'your-api-key' }); const response = await client.chat.completions.create({ model: 'meta-llama/Llama-3.1-8B-Instruct', messages: [ { role: 'user', content: 'Hello!' } ] }); console.log(response.choices[0].message.content); ``` ### cURL ```bash curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer your-api-key" \ -d '{ "model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello!"}] }' ``` --- ## Request Headers Header names are case-insensitive. SMG reads the following request headers: | Header | Purpose | Documented in | |--------|---------|---------------| | `Content-Type` | `application/json`, or `multipart/form-data` for `/v1/audio/transcriptions` | — | | `Authorization` | `Bearer `; required when `--api-key` or `--tenant-api-key` is set | [Authentication](../../concepts/security/authentication.md) | | `x-api-key`, `x-goog-api-key` | The caller's own key for Anthropic (`x-api-key`) and Gemini (`x-goog-api-key`) provider workers. `/v1/responses` and `/v1/interactions` prefer it over `Authorization`, and `/v1/messages` forwards `x-api-key` as sent; `/v1/chat/completions` uses only `Authorization` | [External Providers](../../getting-started/external-providers.md) | | `X-Request-ID` | Request ID for log correlation. SMG checks `x-request-id`, `x-correlation-id`, `x-trace-id`, and `request-id` in that order, and generates an ID when none is present. `--request-id-headers` replaces the list | [Logging](../../getting-started/logging.md) | | `traceparent`, `tracestate` | W3C trace context. With `--enable-trace`, SMG continues the caller's trace and propagates it to workers | [Monitoring](../../getting-started/monitoring.md) | | `X-SMG-Routing-Key` | Routing key for sticky sessions and the key-based policies (`manual`, `consistent_hashing`, `prefix_hash`). It is the default name in `--routing-key-headers` | [Sticky Sessions and Routing Keys](../../concepts/routing/sticky-sessions.md) | | `X-SMG-Routing-Tokens` | Routing hint carrying the prompt's leading token IDs | [Routing Hint Headers](#routing-hint-headers) | | `X-SMG-Target-Worker` | 0-based index of the worker to use among the model's available workers. Honored only by `consistent_hashing`; an index with no available worker fails with `503` | [Load Balancing](../../concepts/routing/load-balancing.md) | | `X-SMG-Priority` | Priority class: `system`, `interactive`, `default`, or `bulk`. Read only when the priority scheduler is enabled | [Priority Scheduler](../priority-scheduler.md) | | `X-SMG-Tenant-ID` | Tenant identity asserted by a trusted upstream, resolved as tenant key `header:`. Read only with `--trust-tenant-header`, and only when API-key auth has not already identified the caller. `--tenant-header-name` renames it | [Tenant Rate Limiting](../tenant-rate-limiting.md) | | `X-SMG-MCP` | `enabled` makes SMG run the MCP tool loop itself for `/v1/messages` requests with MCP toolsets sent to an Anthropic provider, instead of forwarding them as-is | [Messages API](messages.md) | | `X-SMG-Parser` | `legacy` or `symphony`: which parser reads this request's output, overriding `--parser` for the one request. Read on Chat Completions, Messages API, and Responses API requests served through the gRPC pipeline (gRPC and ZMQ workers); any other value is rejected with `400` `invalid_parser_header` | [gRPC Pipeline](../../concepts/architecture/grpc-pipeline.md#per-request-switch-and-response-header) | | `x-smg-cache-trace` | `1` asks for this request's routing-decision record in the `x-smg-cache-trace` response header, on requests served through the gRPC pipeline, whatever the `SMG_CACHE_TRACE_*` variables say | [Routing Trace Variables](../configuration.md#routing-trace-variables) | | Names set with `--storage-context-headers` | Copied into the storage hook request context | [Configuration Reference](../configuration.md) | When SMG proxies a request to an HTTP worker, it forwards only these client headers: `Authorization`, `anthropic-version`, `anthropic-beta`, `X-Request-ID`, `X-Correlation-ID`, headers starting with `X-Request-ID-`, `traceparent`, `tracestate`, `X-SMG-Routing-Key`, and `x-override-priority` (SGLang's request priority, read by an engine started with `SGLANG_ENABLE_REQUEST_HEADER_OVERRIDES=true` and not by SMG). ### Routing Hint Headers A client, or a proxy in front of SMG that has already tokenized the prompt, can pass the routing signal in a header so SMG does not need the request body to choose a worker. Hints affect only worker placement. A malformed or over-limit hint is ignored and the request routes as if the header were absent; it never causes an error. | Header | Value | Limits | Effect | |--------|-------|--------|--------| | `X-SMG-Routing-Tokens` | The prompt's leading token IDs as comma-separated decimal integers from `0` to `4294967295`, with no spaces, signs, or empty items | At most 512 IDs and 4096 bytes | Replaces the token IDs and text SMG would take from the body when choosing a worker. `cache_aware` matches the hinted prefix against its token tree (its placement index with `--cache-index hash`) and records it there; `prefix_hash` hashes it. Read on the regular HTTP path only (not in PD mode or for gRPC workers), and never forwarded to workers | | `X-SMG-Routing-Key` | An opaque key | Non-empty UTF-8, at most 128 bytes | Without `--routing-key-override`, `consistent_hashing` and `prefix_hash` hash the key instead of their other inputs, and `manual` pins it. With the override, every policy except `consistent_hashing` pins the key | ```bash # X-SMG-Routing-Tokens carries the prompt's first token IDs curl http://localhost:30000/v1/completions \ -H "Content-Type: application/json" \ -H "X-SMG-Routing-Tokens: 128000,791,4062,14198,39935" \ -d @request.json ``` With a valid `X-SMG-Routing-Tokens` hint, the text-routing policies (`cache_aware`, `bucket`) no longer need the body to place the request, so a large body can stream to the worker instead of being buffered; see [Request Streaming](../../concepts/performance/request-streaming.md). When `--routing-key-override` is enabled, SMG always buffers the body to read its `rid`, and hints do not change that. ### Response Headers | Header | When SMG sets it | Documented in | |--------|------------------|---------------| | `x-request-id` | On API responses: the request ID SMG used, whether received or generated | [Logging](../../getting-started/logging.md) | | `x-smg-routed-worker-id` | On inference responses from HTTP, gRPC, and ZMQ workers: the registered URL of the worker that served the request, with an `@` suffix for data-parallel workers. For a prefill/decode pair it names the decode worker. From gRPC and ZMQ workers, only successful responses carry it, and Responses API responses do not | [Sticky Sessions and Routing Keys](../../concepts/routing/sticky-sessions.md#routed-worker-header) | | `x-smg-parser` | On successful Chat Completions, Messages API, and Responses API responses from gRPC and ZMQ workers: the parser that read the output, `symphony` or `legacy`. Responses served by the separate gpt-oss (Harmony) pipeline, and the queued object a non-streaming `background: true` Responses request returns, carry no header | [gRPC Pipeline](../../concepts/architecture/grpc-pipeline.md#per-request-switch-and-response-header) | | `x-smg-cache-trace` | On successful responses from gRPC and ZMQ workers when the request sent `x-smg-cache-trace: 1` or the gateway runs with `SMG_CACHE_TRACE_HEADER=1`: the dispatch's routing-decision record. Omitted when the record exceeds 2048 bytes or is not plain ASCII | [Routing Trace Variables](../configuration.md#routing-trace-variables) | | `X-SMG-Error-Code` | On errors the gateway generates in its standard error envelope, with the same value as `error.code`; request-validation errors do not carry it. SMG drops this header from worker responses, so it always marks a gateway decision | — | | `X-SMG-Preempted` | `true` on the `503` returned to a request preempted by the priority scheduler | [Priority Scheduler](../priority-scheduler.md) | | `Retry-After` | On `429` and `503` responses that ask the client to back off: concurrency limits, worker overload, the priority scheduler, tenant rate limits, and the `503` answered when no worker is available (`Retry-After: 1`) | [Rate Limiting](../../concepts/reliability/rate-limiting.md) | --- ## Rate Limiting SMG does not send `X-RateLimit-*` headers. A request shed by admission control, rate limiting, or overload protection gets `429` or `503` with the standard [error envelope](#error-format), the `X-SMG-Error-Code` header, and usually a `Retry-After` header in seconds: | `code` | Status | `Retry-After` | Cause | |--------|--------|---------------|-------| | `admission_queue_full` | 429 | `2` | `--max-concurrent-requests` is reached and the admission queue is full or disabled | | `admission_queue_timeout` | 503 | `2` | The request waited in the admission queue longer than `--queue-timeout-secs` | | `selection_queue_timeout` | 503 | `2` | gRPC and ZMQ workers: the request had already waited longer than `--queue-timeout-secs` inside the gateway when it reached worker selection (the prefill-admission wait of a PD or EPD request included) | | `scheduler_queue_full` | 429 | `2` | Priority scheduler: the request's class queue is full | | `scheduler_queue_timeout` | 503 | `2` | Priority scheduler: the request waited longer than its class's queue timeout | | `scheduler_preempted` | 503 | `1` | Priority scheduler: preempted by higher-priority traffic before its first token; also sets `X-SMG-Preempted: true` | | `tenant_rate_limit_exceeded` | 429 | Seconds until the tenant's budget can admit the request; omitted when it never can | Tenant rate limiting | | `pd_prefill_queue_full` | 429 | — | PD and EPD: the opt-in [prefill admission](../../concepts/routing/pd-disaggregation.md#prefill-admission) queue is full | | `pd_prefill_queue_timeout` | 429 | — | PD and EPD: the request waited out `--prefill-queue-timeout-secs` without a prefill slot | | `worker_overload_protection_shed` | 503 | `--load-monitor-interval` (default `10`) | With `--worker-overload-shed` (off by default, see [Overload Protection](../../concepts/reliability/overload-protection.md)): every eligible worker is over its overload threshold, or the selected one crossed it just before dispatch. Also, with or without `--worker-overload-shed`, on a gRPC PD or EPD dispatch whose decode worker could not admit the request within `--pd-admission-wait-secs`; HTTP PD dispatches are not gated by the decode admission window | Wait at least `Retry-After` seconds before retrying. For configuration, see [Rate Limiting](../../concepts/reliability/rate-limiting.md), [Tenant Rate Limiting](../tenant-rate-limiting.md), [Priority Scheduling](../../concepts/reliability/priority-scheduling.md), and [Overload Protection](../../concepts/reliability/overload-protection.md).