# Admin API Reference SMG provides administrative endpoints for managing tokenizers, workers, WASM modules, and engine caches, plus read-only model and server information. !!! tip "Related Documentation" For health probes, the full route list, and each route's auth tier, see [Gateway Extensions](extensions.md). --- ## Tokenizer Management Manage tokenizers for text processing and tokenization. !!! note "Authentication" These are control-plane routes. When control-plane auth is configured, use a control-plane credential: the `admin` role for every method, or the `user` role for the `GET` routes. Otherwise, use the shared `--api-key` if configured. With neither control-plane auth nor any API keys configured, the routes are open. Tenant keys never grant admin access; with only `--tenant-api-key` configured, every admin request gets `401`. See [Authentication](#authentication). ### Add Tokenizer ``` POST /v1/tokenizers ``` Adds a new tokenizer from a local path or HuggingFace model ID. **Request Body:** ```json { "name": "llama3-tokenizer", "source": "meta-llama/Meta-Llama-3-8B", "chat_template_path": "/path/to/template.jinja" } ``` | Field | Type | Required | Description | |-------|------|----------|-------------| | `name` | string | Yes | Unique tokenizer identifier | | `source` | string | Yes | HuggingFace model ID or local path | | `chat_template_path` | string | No | Path to custom Jinja2 chat template | **Response:** `202 Accepted` ```json { "id": "550e8400-e29b-41d4-a716-446655440000", "status": "pending", "message": "Tokenizer 'llama3-tokenizer' registration job submitted. Loading from: meta-llama/Meta-Llama-3-8B" } ``` Loading runs in the background. Poll [Get Tokenizer Status](#get-tokenizer-status) with the returned `id`. **Response:** `409 Conflict` when a tokenizer with this `name` is already registered. The `id` is the existing tokenizer's: ```json { "id": "550e8400-e29b-41d4-a716-446655440000", "status": "failed", "message": "Tokenizer 'llama3-tokenizer' already exists" } ``` --- ### List Tokenizers ``` GET /v1/tokenizers ``` Returns all registered tokenizers. **Response:** `200 OK` ```json { "tokenizers": [ { "id": "550e8400-e29b-41d4-a716-446655440000", "name": "llama3-tokenizer", "source": "meta-llama/Meta-Llama-3-8B", "vocab_size": 128256 } ] } ``` --- ### Get Tokenizer ``` GET /v1/tokenizers/{tokenizer_id} ``` Returns details for a specific tokenizer. `{tokenizer_id}` is the tokenizer's ID or its name. **Response:** `200 OK` ```json { "id": "550e8400-e29b-41d4-a716-446655440000", "name": "llama3-tokenizer", "source": "meta-llama/Meta-Llama-3-8B", "vocab_size": 128256 } ``` **Response:** `404 Not Found` ```json { "error": { "message": "Tokenizer 'llama3-tokenizer' not found", "type": "tokenizer_not_found" } } ``` --- ### Get Tokenizer Status ``` GET /v1/tokenizers/{tokenizer_id}/status ``` Returns the loading status of a tokenizer. A loaded tokenizer can be looked up by ID or name; a job that is still queued, loading, or failed only by the ID that [Add Tokenizer](#add-tokenizer) returned. **Response:** `200 OK` ```json { "id": "550e8400-e29b-41d4-a716-446655440000", "status": "completed", "message": "Tokenizer 'llama3-tokenizer' is loaded and ready", "vocab_size": 128256 } ``` | Status | Description | |--------|-------------| | `pending` | Tokenizer loading queued | | `processing` | Tokenizer currently loading | | `completed` | Tokenizer ready for use | | `failed` | Loading failed (see message) | `vocab_size` is present only for `completed`. A `failed` status is kept for about five minutes. After that, and for an ID that SMG does not know: **Response:** `404 Not Found` ```json { "error": { "message": "Tokenizer '550e8400-e29b-41d4-a716-446655440000' not found and no pending job", "type": "not_found" } } ``` --- ### Remove Tokenizer ``` DELETE /v1/tokenizers/{tokenizer_id} ``` Removes a tokenizer. `{tokenizer_id}` is the tokenizer's ID or its name. **Response:** `200 OK` ```json { "success": true, "message": "Tokenizer 'llama3-tokenizer' removed successfully" } ``` **Response:** `404 Not Found` ```json { "success": false, "message": "Tokenizer 'llama3-tokenizer' not found" } ``` --- ## Worker Management Register, inspect, update, and remove backend workers at runtime. | Method | Path | Purpose | |--------|------|---------| | `POST` | `/workers` | [Register a worker](#create-worker) | | `GET` | `/workers` | [List workers](#list-workers) | | `GET` | `/workers/{worker_id}` | [Get one worker and its job status](#get-worker) | | `PATCH` | `/workers/{worker_id}` | [Change priority, cost, labels, API key, or health settings](#update-worker-partial) | | `PUT` | `/workers/{worker_id}` | [Replace the spec and re-run registration](#replace-worker-full) | | `DELETE` | `/workers/{worker_id}` | [Drain and remove a worker](#delete-worker) | `worker_id` is the UUID that `POST /workers` returns. Changes are asynchronous: the gateway checks the request, queues a job, and answers `202 Accepted`. Follow the job with `GET /workers/{worker_id}`. !!! note "Authentication" These are control-plane routes. With [control-plane auth](../../getting-started/control-plane-auth.md) configured, they need a credential with the `admin` role; a `user`-role credential may call the `GET` routes (`401` without a valid credential, `403` when the role does not allow the method). Otherwise they need `Authorization: Bearer ` with the gateway `--api-key`. Per-tenant keys are not accepted, so a gateway with tenant keys but no `--api-key` answers `401` to every call. With no keys configured at all, the routes are open. ### Create Worker ``` POST /workers ``` Registers a worker. The body is a [worker spec](#worker-spec); only `url` is required. The URL scheme selects the transport: | Scheme | Transport | Backends | |--------|-----------|----------| | `http://`, `https://` | HTTP | SGLang, vLLM, or other OpenAI-compatible HTTP servers, and external providers. Hosts ending in `openai.com`, `anthropic.com`, `x.ai`, or `googleapis.com` register as external automatically. | | `grpc://`, `grpcs://` | gRPC | SGLang, vLLM, TensorRT-LLM, TokenSpeed, or MLX gRPC servers. See [gRPC Workers](../../getting-started/grpc-workers.md). | | `ipc://`, `tcp://:` | ZMQ | A vLLM, TokenSpeed, or SGLang engine core: `ipc://` for one on the same host, `tcp://` for engines that dial in from other hosts. See [ZMQ Workers](../../getting-started/zmq-workers.md). | The scheme must be lowercase. Send the spec as JSON: ```bash curl -X POST http://localhost:30000/workers \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${ADMIN_TOKEN}" \ -d @worker.json ``` === "HTTP worker" ```json { "url": "http://gpu1:8000", "api_key": "worker-secret-key", "labels": {"region": "us-east"} } ``` The gateway probes the worker, detects the engine, and reads the served model from it. === "gRPC worker" ```json { "url": "grpc://gpu2:50051", "runtime_type": "sglang", "models": [ { "id": "meta-llama/Llama-3.1-8B-Instruct", "aliases": ["llama-3.1-8b"], "tool_parser": "llama" } ] } ``` An explicit `runtime_type` skips engine detection. The model card adds an alias and pins the tool-call parser for this model. === "ZMQ worker" ```json { "url": "ipc:///tmp/smg-zmq/engine-31000", "runtime_type": "vllm", "labels": {"model_path": "Qwen/Qwen3-8B"} } ``` The gateway binds the sockets and the engine dials in. For this path the handshake listens on `tcp://127.0.0.1:22714`, a port derived from the `ipc://` path. An engine core does not report a model name, so a ZMQ worker needs `models`, a model label such as `model_path` (which also locates the tokenizer), or a gateway started with `--model-path`. For a group of data-parallel engines or a fixed handshake address, see the ZMQ fields under [Worker Spec](#worker-spec). A `tcp://:` URL serves engines on other hosts: the gateway binds the handshake at that address itself (a wildcard host such as `0.0.0.0` is refused) and its data-plane sockets on ephemeral ports of the same host. These sockets carry no authentication, so keep such a worker on a trusted network. === "Overload and HTTP/2 overrides" ```json { "url": "http://gpu3:8000", "runtime_type": "vllm", "overload": { "waiting_requests": 32, "token_usage": 0.85 }, "http_pool": { "http2": true, "connect_timeout_secs": 5 }, "health": { "drain_settle_secs": 30 } } ``` This worker leaves routing while 32 or more requests wait in its queue or its KV cache is at least 85% used, instead of at the gateway's 8 requests and 80%, and even under `--disable-worker-overload-protection`. The gateway always speaks HTTP/2 (h2c) to it instead of negotiating the version, so registration fails if the worker only speaks HTTP/1.1. When deleted while `ready`, it drains for 30 seconds. **Response:** `202 Accepted`, with a `Location` header equal to `location`. ```json { "status": "accepted", "worker_id": "01997a4e-3c2b-7f1d-9a8e-2b6c4d5e7f80", "url": "http://gpu1:8000", "location": "/workers/01997a4e-3c2b-7f1d-9a8e-2b6c4d5e7f80", "message": "Worker addition queued for background processing" } ``` `202` means the request passed the checks listed under [Worker Errors](#worker-errors). Probing, engine detection, and the remaining validation run in the background job; [Get Worker](#get-worker) shows how to follow it. --- ### Worker Spec The body of `POST /workers` and `PUT /workers/{worker_id}`. Only `url` is required. The nested blocks (`health`, `overload`, `http_pool`, `resilience`) are partial: a field you leave out falls back to the gateway default. Unknown fields are ignored rather than rejected, so check the spelling. A known field with the wrong type, or an unknown enum value, is rejected with `400`. **Identity and transport** | Field | Type | Default | Description | |-------|------|---------|-------------| | `url` | string | required | Worker address. The scheme selects the transport (see [Create Worker](#create-worker)). Each URL can be registered once. | | `runtime_type` | string | `unspecified` | `sglang`, `vllm`, `trtllm`, `mlx`, `tokenspeed`, `generic` (an OpenAI-compatible HTTP server whose engine is unknown), or `external` (a third-party API, required for a provider behind a host the gateway does not recognize). `unspecified` detects the engine: over HTTP from `/v1/models`, `/version`, and `/server_info`, registering an unidentified OpenAI-compatible server as `generic`; over gRPC by trying `sglang`, `vllm`, `trtllm`, `tokenspeed`, then `mlx`. ZMQ workers default to `vllm`. Also accepted as `runtime`. | | `worker_type` | string | `regular` | `regular`, `prefill`, `decode`, or `encode` (EPD encode worker). See [PD Disaggregation](../../concepts/routing/pd-disaggregation.md). | | `api_key` | string | none | Key the gateway presents to the worker. Write-only: never returned. Workers added through the API do not inherit the gateway `--api-key`. For provider keys, see [External Providers](../../getting-started/external-providers.md). | **Models and labels** | Field | Type | Default | Description | |-------|------|---------|-------------| | `models` | array | `[]` | Model cards the worker serves (fields below). A self-hosted worker uses the first card. With none, it takes the model the engine reports: the `served_model_name`, `model_id`, or `model_path` label, in that order. External workers ignore this field: the gateway lists the provider's models when it has a key (`api_key`, or the provider's `*_ADMIN_KEY` environment variable), and without one the worker accepts any model. | | `labels` | object | `{}` | String-to-string metadata, merged over the labels the gateway discovers from the engine (your values win). At most 128 labels; a key is 1 to 128 printable ASCII characters without spaces, a value at most 4096 bytes without control characters, and the `smg.ai/discovery-` prefix is reserved for [service discovery](../../getting-started/service-discovery.md). Some keys have built-in meaning (see below). | Model card fields (`models[]`): | Field | Description | |-------|-------------| | `id` | Model ID (required). | | `aliases` | Other model names that route to this model. | | `model_type` | Capabilities, any of `chat`, `completions`, `responses`, `embeddings`, `rerank`, `generate`, `vision`, `tools`, `reasoning`, `image_gen`, `audio`, `moderation`. Default: `["chat", "completions", "responses", "tools"]`. | | `context_length` | Context window in tokens. | | `chat_template` | Chat template path used when the gateway loads the tokenizer for a gRPC or ZMQ worker. | | `tool_parser`, `reasoning_parser` | Parser names for this model on the gRPC path, overriding `--tool-call-parser` and `--reasoning-parser`. Under `--parser symphony`, they may also name a [Symphony table or `hf`](../../concepts/architecture/grpc-pipeline.md#symphony-parsing). A name the configured `--parser` cannot read with fails registration. | Cards also accept `display_name`, `hf_model_type`, `architectures`, `provider`, `tokenizer_path`, `metadata`, `id2label`, and `num_labels`. Labels with built-in meaning: | Label | Effect | |-------|--------| | `realtime` | `"true"` lets the worker serve the [Realtime API](openai.md#realtime-api). The value must be `"true"` or `"false"`; anything else is rejected with `400`. | | `tool_parser`, `reasoning_parser` | Same as the model card fields, when the card does not set them. | | `constrained_decoding` | When the gateway hands this worker's engine a grammar: `requested` (the default) only when the request asks for one, or `always` for an engine that takes a grammar on every turn — under [Symphony parsing](../../concepts/architecture/grpc-pipeline.md#symphony-parsing), a request that allows tool calls under `auto` then gets the table's structural tag for them. Any other value fails registration or the label update. gRPC engines can report the label themselves; a spec field of the same name takes the same values and wins over the label. | | `structural_tag_vocabulary` | `older` makes [Symphony](../../concepts/architecture/grpc-pipeline.md#older-xgrammar-releases) spell this worker's structural tags without the grammar nodes that need xgrammar 0.1.33; the Rust gRPC servicers set it themselves for an older xgrammar. Any other value reads as unset. | | `served_model_name`, `model_id`, `model_path` | Model ID when `models` is empty, checked in this order. | | `tokenizer_path`, `model_path` | Where the gateway loads the tokenizer for a gRPC or ZMQ worker, ahead of `--tokenizer-path` and `--model-path`. | | `pairing_protocol` | Explicit PD pairing key: a prefill and a decode worker that both set it pair only when the values match. | | `kv_connector`, `kv_role`, `kv_engine_id` | Moved into the spec fields of the same name at registration (a non-empty `kv_connector` or `kv_engine_id` field wins over the label). | **Routing metadata** | Field | Type | Default | Description | |-------|------|---------|-------------| | `priority` | integer | `50` | Stored and returned in responses. No built-in routing policy reads it. | | `cost` | number | `1.0` | Stored and returned in responses. No built-in routing policy reads it. Must be finite and above 0. | **Health checks (`health`)**: per-worker overrides of the gateway health-check flags. See [Health Checks](../../concepts/reliability/health-checks.md). | Field | Gateway default | Description | |-------|-----------------|-------------| | `timeout_secs` | `--health-check-timeout-secs` (5) | Probe timeout, also used for the registration probes. | | `check_interval_secs` | `--health-check-interval-secs` (60) | Seconds between probes. | | `success_threshold` | `--health-success-threshold` (2) | Consecutive successful probes before the worker becomes `ready`. | | `failure_threshold` | `--health-failure-threshold` (3) | Consecutive failed probes before the worker leaves rotation. | | `disable_health_check` | `--disable-health-check` (off); `true` for external workers | Skip probing: the worker is routable as soon as it registers. Ignored for ZMQ workers. | | `drain_settle_secs` | `--drain-settle-secs` (5) | Seconds a `ready` worker stays `draining` when it is removed. `0` removes it immediately. | **Overload thresholds (`overload`)**: self-hosted workers only. See [Overload Protection](../../concepts/reliability/overload-protection.md). | Field | Type | Gateway default | Description | |-------|------|-----------------|-------------| | `waiting_requests` | integer, at least 1 | `--worker-overload-waiting-requests` (8) | Waiting requests, summed across DP ranks, at or above which the worker leaves routing. | | `token_usage` | number in (0.0, 1.0] | `--worker-overload-token-usage` (0.8) | Mean KV-cache usage across DP ranks at or above which the worker leaves routing. | Either field keeps overload protection on for this worker, even under `--disable-worker-overload-protection`; a field you leave out uses the gateway value. The worker returns to routing once a load report is below both thresholds. An out-of-range value fails registration. `PATCH` cannot change this block; use `PUT`. **HTTP client (`http_pool`)**: applies to HTTP workers, including external providers. See [Request Streaming](../../concepts/performance/request-streaming.md). | Field | Type | Default | Description | |-------|------|---------|-------------| | `http2` | boolean | `--upstream-http2` (off) | `true`: always speak HTTP/2 with prior knowledge (h2c on `http://`); registration fails if the worker only speaks HTTP/1.1. `false`: no HTTP/2 prior knowledge, so an `http://` worker stays on HTTP/1.1. Unset: under `--upstream-http2`, the gateway probes an `http://` worker with both versions and prefers HTTP/2; without the flag, an `http://` worker uses HTTP/1.1. Responses report the result as `http2`. | | `pool_max_idle_per_host` | integer | `500` | Idle connections kept per host. | | `pool_idle_timeout_secs` | integer | `--upstream-pool-idle-timeout-secs` (3) | How long an idle connection is kept. Keep it below the engine's keep-alive timeout; `0` keeps idle connections forever. | | `timeout_secs` | integer | `--request-timeout-secs` (1800) | Default request timeout. | | `connect_timeout_secs` | integer | `10` | Connection timeout. | **Resilience (`resilience`)**: per-worker circuit-breaker settings. See [Circuit Breakers](../../concepts/reliability/circuit-breakers.md). | Field | Type | Gateway default | Description | |-------|------|-----------------|-------------| | `cb_failure_threshold` | integer | `--cb-failure-threshold` (10) | Failures that open this worker's circuit. | | `cb_success_threshold` | integer | `--cb-success-threshold` (3) | Successes in the half-open state that close it. | | `cb_timeout_secs` | integer | `--cb-timeout-duration-secs` (60) | Seconds before an open circuit tries half-open. | | `cb_window_secs` | integer | `--cb-window-duration-secs` (120) | Accepted but unused: the circuit breaker counts consecutive failures, not failures in a window. | | `retryable_status_codes` | integer array | `[408, 429, 500, 502, 503, 504]` | Statuses counted as circuit-breaker failures. Replaces the default set; it does not change which responses are retried. | | `capacity_status_codes` | integer array | `[429]` | Capacity-pushback statuses, which the circuit breaker never counts, as either failure or success. Replaces the default set. | **ZMQ workers**: see [ZMQ Workers](../../getting-started/zmq-workers.md). | Field | Type | Default | Description | |-------|------|---------|-------------| | `dp_size` | integer | none | On a ZMQ worker, the number of data-parallel engines that dial into its one socket set. A value above 1 makes a grouped worker, and the gateway spreads requests across the group's engines. On other workers the gateway sets this field itself during data-parallel discovery. | | `zmq_handshake_address` | string | derived | `tcp://` address the gateway binds for an `ipc://` worker's engine handshake. The default is `tcp://127.0.0.1:`, with the port (20000 to 29999) derived from the `ipc://` path. Set it for an engine that dials a fixed address, such as `tcp://127.0.0.1:30500`, TokenSpeed's default. A `tcp://` worker's URL is its handshake address and takes no override. | Registration of a ZMQ worker fails when the runtime is not `vllm`, `tokenspeed`, or `sglang`, when `worker_type` is not `regular` and the runtime is not `tokenspeed` (only TokenSpeed engines serve the prefill, decode, and encode roles over the ZMQ wire; the other runtimes need gRPC for disaggregation), when no model ID is available, when `dp_size` is above 1 on a gateway running with `--dp-aware`, when an `ipc://` worker's handshake address is not `tcp://` or is already bound by another ZMQ worker, or when a `tcp://` worker names a wildcard host or sets `zmq_handshake_address`. Setting `zmq_handshake_address` on a non-ZMQ worker also fails registration. A disaggregated TokenSpeed ZMQ worker additionally needs an engine whose handshake declares the matching role — and, for `prefill` or `encode`, its bootstrap address — or the engine's connection fails when it dials in. Health checks stay on for ZMQ workers, because the probe is what reconnects a restarted engine. **PD disaggregation and KV transfer**: see [PD Disaggregation](../../concepts/routing/pd-disaggregation.md). | Field | Type | Default | Description | |-------|------|---------|-------------| | `bootstrap_port` | integer | none | KV bootstrap port of a `prefill` or `encode` worker. The gRPC and vLLM Mooncake paths use `8998` when it is unset. | | `kv_connector` | string | engine-reported | KV connector, such as `MooncakeConnector` or `NixlConnector`. Overrides the value a vLLM gRPC engine reports. | | `kv_engine_id` | string | engine-reported | vLLM `kv_transfer_config.engine_id`, used for Mooncake PD. For gRPC prefill and decode workers, the gateway reads it from the engine again when the worker recovers from `failed` or `not_ready` (a restarted engine gets a new ID), and the engine's value wins. | A worker keeps the spec it was registered with: a nested block you set (`health`, `overload`, `http_pool`, `resilience`) is echoed back by `GET /workers` as sent, and one you left empty is omitted. The other spec fields apply as follows: - `provider`: checked against the provider routers compiled into the build (see `PROVIDER_NOT_COMPILED` under [Worker Errors](#worker-errors)) and kept on the worker's spec. For a provider behind a host the gateway does not recognize, set `runtime_type` to `external`. - `kv_role`: the field wins over a `kv_role` label. vLLM gRPC engines report it themselves. - `pairing_protocol`: the field wins over a `pairing_protocol` label (which `SMG_PAIRING_PROTOCOL` in a gRPC engine's environment also sets). - `multimodal_tensor_transport` and `multimodal_shm_min_bytes`: consulted for the worker that runs the vision encoder (the encode worker in EPD); the gateway-wide `--multimodal-tensor-transport` and `--multimodal-shm-min-bytes` are the defaults (see [Multimodal](../../concepts/architecture/multimodal.md)). - `load_monitor_interval_secs`: a worker group polls at the smallest value its workers set, `--load-monitor-interval` when none does, never under one second. - `kv_block_size`: seeds the model's block size until the engine's KV event stream names it. - The `resilience` retry fields (`max_retries`, `initial_backoff_ms`, `max_backoff_ms`, `backoff_multiplier`, `jitter_factor`, `disable_retry`) and `disable_circuit_breaker`: they build the worker's retry and circuit-breaker settings at registration, over the gateway-wide `--retry-*`, `--disable-retries`, and `--disable-circuit-breaker` flags. Three fields are accepted but not read: - `connection_mode`: the URL scheme decides the transport, and responses report the result (`http`, `grpc`, or `zmq`). - `max_connection_attempts`: not read; registration keeps probing until `--worker-startup-timeout-secs`. A worker from the command line is registered with ten times `--health-success-threshold` (`20` by default) and a worker from service discovery with twenty times (`40` by default), which responses show; a worker registered through this API echoes the value it was given. - `dp_base_url`, `dp_rank`, and `dp_size` (except `dp_size` on ZMQ workers): the gateway sets them on data-parallel workers under `--dp-aware`. --- ### Worker Errors Worker endpoints report errors as `{"error": "", "code": ""}`, not in the shape described under [Error Responses](#error-responses): ```json { "error": "Invalid value for field 'worker_url': 10.0.0.5:8000 - URL must start with a lowercase http://, https://, grpc://, grpcs://, ipc://, or tcp:// scheme", "code": "BAD_REQUEST" } ``` | Status | `code` | When | |--------|--------|------| | `400` | `BAD_REQUEST` | `worker_id` is not a UUID. The body does not parse: `Content-Type: application/json` is missing, the JSON is malformed, or a field has a wrong type, an unknown enum value such as `"worker_type": "prefil"`, or is missing (`url`). The `url` is empty, longer than 2048 bytes, does not start with a lowercase `http://`, `https://`, `grpc://`, `grpcs://`, `ipc://`, or `tcp://` scheme, is not a valid URL or has no host, or is an `ipc://` URL without a path. The labels break the bounds under [Worker Spec](#worker-spec), carry a `smg.ai/discovery-` key, or set `max_running_requests`, `max_num_seqs`, or `dp_size` to anything but a positive count or `supports_vision` or `realtime` to anything but `"true"`/`"false"`. `cost` is not finite and above 0, `kv_block_size` or `dp_size` is 0, `bootstrap_port` is 0, or `dp_rank` comes without `dp_size` or is not below it. A `PUT` changes the URL or is sent to a gateway running with `--dp-aware`. | | `400` | `PROVIDER_NOT_COMPILED` | The spec targets a provider whose router is not compiled into this build. | | `404` | `WORKER_NOT_FOUND` | No worker has this ID. | | `409` | `WORKER_ALREADY_EXISTS` | `POST` for a URL that is already registered. The message names the existing ID. | | `409` | `WORKER_CREATE_IN_PROGRESS` | `POST` for a URL whose registration is still running. The message names the ID to poll. | | `409` | `WORKER_DISCOVERY_OWNED` | A `PUT`, or a `PATCH` that carries `labels`, of a worker a [discovery provider](../../getting-started/service-discovery.md) registered. The message names the provider; change the worker at its discovery source instead. | | `413` | `PAYLOAD_TOO_LARGE` | The body exceeds the gateway's payload limit (`--max-payload-size`). | | `500` | `INTERNAL_SERVER_ERROR` | The job queue is unavailable or full. | Other problems surface in the background job, after the `202`: - `overload.waiting_requests` of `0`, or `overload.token_usage` outside (0.0, 1.0] - the ZMQ constraints listed under [Worker Spec](#worker-spec) - a `tool_parser` or `reasoning_parser` the configured `--parser` cannot read with - a `constrained_decoding` label that names neither `requested` nor `always` - a worker that never answers, or `http_pool.http2: true` for a worker that only speaks HTTP/1.1 --- ### List Workers ``` GET /workers GET /workers?model={model_id} ``` Returns every registered worker. `model` keeps only workers that serve that model ID or alias; workers without a model list match any model. The `stats` counts cover the returned workers (`encode` workers are counted in `total` only). Workers whose registration job is still running are not listed. **Response:** `200 OK` (abridged: discovered labels and card fields vary by engine) ```json { "workers": [ { "id": "01997a4e-3c2b-7f1d-9a8e-2b6c4d5e7f80", "model_id": "meta-llama/Llama-3.1-8B-Instruct", "url": "grpc://gpu2:50051", "models": [ { "id": "meta-llama/Llama-3.1-8B-Instruct", "aliases": ["llama-3.1-8b"], "model_type": ["chat", "completions", "responses", "tools"], "context_length": 131072, "tool_parser": "llama" } ], "worker_type": "regular", "connection_mode": "grpc", "runtime_type": "sglang", "labels": { "model_path": "meta-llama/Llama-3.1-8B-Instruct", "tp_size": "1" }, "priority": 50, "cost": 1.0, "max_connection_attempts": 20, "is_healthy": true, "status": "ready", "load": 2, "http2": false } ], "total": 1, "stats": { "prefill_count": 0, "decode_count": 0, "regular_count": 1 } } ``` Each worker object carries its spec fields (never `api_key`) plus: | Field | Description | |-------|-------------| | `id` | Worker UUID. | | `model_id` | ID of the first model card. Absent for workers without a model list. | | `is_healthy` | `true` when `status` is `ready`. | | `status` | `pending` (registered, not yet proven healthy), `ready`, `not_ready` (failing probes, may recover), `failed`, or `draining` (being removed). Only `ready` workers receive traffic. | | `stalled` | Why the gateway's liveness tracker keeps the worker out of routing (`unreachable` or `wedged`) while its health status stands. Absent while it is routable. | | `load` | Requests the gateway has in flight to this worker. | | `http2` | Whether the gateway speaks HTTP/2 to this worker. | | `pd_pairing` | Prefill and decode workers only: the key used to pair them for KV transfer, either the explicit `pairing_protocol` or `runtime/transport/layout`. | | `engine_load` | The engine's last polled load report (per-DP-rank queue and KV-cache figures). Absent until the load monitor has polled the worker. | --- ### Get Worker ``` GET /workers/{worker_id} ``` Returns one worker in the same shape as a `GET /workers` entry. While a job for the worker is queued or running, or for about five minutes after one fails, the response also carries `job_status`. A job that succeeds clears it. While `POST /workers` is still registering the worker, the response is a placeholder: only `id`, `url`, `status`, and `job_status` are meaningful, and the other fields hold defaults. **Response:** `200 OK` ```json { "id": "01997a4e-3c2b-7f1d-9a8e-2b6c4d5e7f80", "url": "grpc://gpu2:50051", "worker_type": "regular", "connection_mode": "http", "runtime_type": "unspecified", "priority": 50, "cost": 1.0, "max_connection_attempts": 20, "is_healthy": false, "status": "pending", "load": 0, "http2": false, "job_status": { "job_type": "AddWorker", "worker_url": "grpc://gpu2:50051", "status": "processing", "message": null, "timestamp": 1790250060 } } ``` | `job_status` field | Description | |--------------------|-------------| | `job_type` | `AddWorker` (create or replace), `UpdateWorker`, or `RemoveWorker`. | | `worker_url` | URL of the worker the job targets. | | `status` | `pending` (queued), `processing`, or `failed`. | | `message` | Error text when `status` is `failed`; otherwise `null`. | | `timestamp` | Unix time, in seconds, of the last status change. | Registration keeps probing a worker that does not answer, every `--worker-startup-check-interval` seconds (default 30), until `--worker-startup-timeout-secs` (default 1800) runs out. If a `POST /workers` job fails, the gateway releases the reserved ID: `GET /workers/{worker_id}` returns `404`, and the reason appears only in the gateway log (`Failed job: type=AddWorker ...`). A failed `PATCH`, `PUT`, or `DELETE` job leaves the worker in place with a `failed` `job_status`. --- ### Update Worker (partial) ``` PATCH /workers/{worker_id} ``` Changes a few fields in place. Omitted fields keep their current values. The worker keeps its status and connections, and nothing is probed again. **Request Body:** ```json { "priority": 75, "labels": {"tier": "gold"}, "api_key": "new-api-key", "health": {"check_interval_secs": 15, "drain_settle_secs": 60} } ``` | Field | Type | Description | |-------|------|-------------| | `priority` | integer | New priority. | | `cost` | number | New cost. | | `labels` | object | Merged into the current labels: listed keys are added or overwritten, and the rest are kept, so `PATCH` cannot remove a label — except that setting a running-window label (`max_running_requests` or `max_num_seqs`) drops the window under the other name, making yours the worker's only window. A merge that would grow the worker past 128 labels is refused with `400`. | | `api_key` | string | New worker API key, for key rotation. | | `health` | object | Partial health overrides, merged into the worker's current settings: `timeout_secs`, `check_interval_secs`, `success_threshold`, `failure_threshold`, `disable_health_check`, `drain_settle_secs`. Setting `disable_health_check` to `true` makes the worker routable immediately. | Any other field, such as `overload`, `http_pool`, `resilience`, or `models`, is ignored. Use `PUT` to change those. A `PATCH` keeps the worker's circuit-breaker configuration and state: an open circuit stays open. For a worker a [discovery provider](../../getting-started/service-discovery.md) registered, a `PATCH` that carries `labels` is refused with `409` (`WORKER_DISCOVERY_OWNED`): its labels come from its discovery source. A `PATCH` of the fields no provider publishes — `priority`, `cost`, `api_key`, `health` — is taken. **Response:** `202 Accepted` ```json { "status": "accepted", "worker_id": "01997a4e-3c2b-7f1d-9a8e-2b6c4d5e7f80", "message": "Worker update queued for background processing" } ``` --- ### Replace Worker (full) ``` PUT /workers/{worker_id} ``` Replaces the worker's spec and re-runs the full registration workflow (probe, engine detection, metadata discovery) under the same ID. The body is a complete [worker spec](#worker-spec). Fields you omit take their defaults rather than keeping their current values, so resend any `labels`, `overload`, or other settings you want to keep. The `url` must equal the worker's current URL; to move a worker, `DELETE` it and `POST` the new URL. `PUT` is refused with `409` (`WORKER_DISCOVERY_OWNED`) for a worker a [discovery provider](../../getting-started/service-discovery.md) registered — its spec comes from its discovery source — and with `400` when the gateway runs with `--dp-aware`, because one spec expands to one worker per data-parallel rank; use `DELETE` and `POST` there too. If the worker is deleted or replaced again before the job runs, the job fails rather than overwrite the newer state. **Response:** `202 Accepted`, with the same body as `PATCH`. --- ### Delete Worker ``` DELETE /workers/{worker_id} ``` Removes a worker. A `ready` worker first moves to `draining`: it receives no new requests, and the gateway waits `health.drain_settle_secs` (default `--drain-settle-secs`, 5 seconds) so in-flight requests can finish before it removes the worker. A worker in any other status is removed without waiting. A discovered worker can be deleted, but its provider registers it again while its discovery source still publishes it. **Response:** `202 Accepted` ```json { "status": "accepted", "worker_id": "01997a4e-3c2b-7f1d-9a8e-2b6c4d5e7f80", "message": "Worker removal queued for background processing" } ``` --- ## Cache Management Flush the engines' KV caches, and read the engine load the gateway has polled from its workers. ### Flush Cache ```text POST /flush_cache ``` Asks every registered worker whose transport carries a cache flush to drop its KV prefix cache. The calls go out in parallel, and the response reports the outcome per worker. Requires admin [authentication](#authentication). | Worker | How it is flushed | |--------|-------------------| | HTTP | `POST {worker_url}/flush_cache`, with the worker's API key (if it has one) as a bearer token and a 45-second timeout. Any non-2xx answer counts as a failure, so the engine must serve this route (SGLang's HTTP server does). | | gRPC SGLang, TokenSpeed | `FlushCache` RPC. A TokenSpeed [EPD](../../concepts/routing/pd-disaggregation.md#epd-encode-prefill-decode) encode worker runs the vision tower only, with no KV cache to flush: it refuses the call and is reported under `failed` (`UNIMPLEMENTED`). | | gRPC vLLM | `FlushCache` RPC. Needs `smg-grpc-servicer` 0.12.0 or later; older servicers answer `UNIMPLEMENTED`. Resets vLLM's local prefix cache without preempting running requests or clearing connector caches. The gateway makes a single attempt, so a reset that vLLM refuses while requests are in flight is reported under `failed`. | | gRPC TRT-LLM, MLX | Not supported: reported under `failed` (`UNIMPLEMENTED`). | | ZMQ SGLang | Flushed over the engine's own wire, which carries the control calls. | | ZMQ TokenSpeed | Flushed when every connected engine declared the control reply at its handshake; otherwise skipped. | | ZMQ vLLM | Skipped: the engine-core wire carries no cache-flush call. | Skipped ZMQ workers are counted in `total_zmq_workers_skipped`; ZMQ engines that take the flush are counted under `total_grpc_workers`, since they ride the gRPC pipeline's admin path. The fan-out does not otherwise filter by runtime: external-provider workers registered over HTTP receive the `POST` too, and appear under `failed` if they reject it. The endpoint only calls the workers; it does not reset the gateway's own cache-aware routing state. **Response:** `200 OK` when every worker that was called succeeds (or there was nothing to call), `206 Partial Content` when at least one fails, including when all of them fail. ```json { "status": "success", "message": "Successfully flushed cache on all 3 workers", "workers_flushed": 3, "total_http_workers": 2, "total_grpc_workers": 1, "total_zmq_workers_skipped": 0, "total_workers": 3 } ``` On a partial failure, `status` is `"partial_success"` and two more fields list the outcome, `successful` (worker URLs) and `failed` (`{worker, error}` entries): ```json { "status": "partial_success", "message": "Cache flush: 2 succeeded, 1 failed (1 ZMQ workers skipped: no cache-flush RPC)", "workers_flushed": 2, "total_http_workers": 2, "total_grpc_workers": 1, "total_zmq_workers_skipped": 1, "total_workers": 4, "successful": ["http://gpu1:8000", "grpc://gpu3:50051"], "failed": [ { "worker": "http://gpu2:8000", "error": "flush_cache failed for worker http://gpu2:8000/flush_cache: HTTP 500 Internal Server Error" } ] } ``` | Field | Description | |-------|-------------| | `status` | `success` or `partial_success` | | `message` | Human-readable summary | | `workers_flushed` | Workers that confirmed the flush | | `total_http_workers`, `total_grpc_workers` | Registered HTTP workers; gRPC workers plus the ZMQ engines that take the flush | | `total_zmq_workers_skipped` | ZMQ workers that were not called (their wire carries no cache-flush call) | | `total_workers` | All registered workers (the three counts sum to it) | | `successful`, `failed` | Present only when at least one worker failed | --- ### Get Loads ```text GET /loads GET /get_loads ``` Returns the engine load the gateway's load monitor last collected: one entry per worker per DP rank, plus a fleet-wide `aggregate`. The body uses the same schema engines report on `/v1/loads`, with every entry tagged by `worker` and `worker_type`. The response is built from the monitor's cached snapshot, the same numbers the routing policies act on, so no request reaches a worker. Values can be up to one poll interval old (`--load-monitor-interval`, 10 seconds by default). Serving from the cache is also what makes the route safe to poll from another SMG gateway that registers this one as a worker. - **Auth**: `/loads` is a public route, served without auth like `/health`. `/get_loads` is a deprecated alias that stays on the control-plane tier and requires [authentication](#authentication). - **Filter**: `?model=` limits the response to workers serving that model. - **Missing workers**: workers without a current report (not `Ready` yet, no load source, or a failed last poll) are left out rather than reported as idle. Before the first poll, `loads` is empty and `aggregate` is absent. **Response:** `200 OK` ```json { "timestamp": "2026-09-24T18:20:11.482913+00:00", "version": "smg-1.14.0", "dp_rank_count": 2, "loads": [ { "worker": "http://gpu1:8000", "worker_type": "regular", "dp_rank": 0, "num_running_reqs": 12, "num_waiting_reqs": 3, "num_waiting_uncached_tokens": 5120, "num_total_reqs": 15, "num_used_tokens": 48000, "max_total_num_tokens": 131072, "token_usage": 0.37, "gen_throughput": 1850.5, "cache_hit_rate": 0.62, "utilization": 0.41, "max_running_requests": 256 }, { "worker": "http://gpu2:8000", "worker_type": "regular", "dp_rank": 0, "num_running_reqs": 9, "num_waiting_reqs": 0, "num_waiting_uncached_tokens": 0, "num_total_reqs": 9, "num_used_tokens": 30000, "max_total_num_tokens": 131072, "token_usage": 0.23, "gen_throughput": 1420.0, "cache_hit_rate": 0.55, "utilization": 0.3, "max_running_requests": 256 } ], "aggregate": { "total_running_reqs": 21, "total_waiting_reqs": 3, "total_reqs": 24, "avg_token_usage": 0.3, "avg_throughput": 1635.25, "avg_utilization": 0.355 } } ``` | Field | Description | |-------|-------------| | `timestamp` | When the gateway built this response (RFC 3339) | | `version` | `smg-` followed by the gateway version | | `dp_rank_count` | Number of entries in `loads`, across the whole fleet | | `loads[].worker`, `loads[].worker_type` | Worker URL and type (`regular`, `prefill`, `decode`, or `encode`) | | `loads[].dp_rank` | DP rank within that worker | | `loads[].num_running_reqs`, `num_waiting_reqs`, `num_total_reqs` | Requests running, queued, and both | | `loads[].num_waiting_uncached_tokens` | Queued tokens not served from the prefix cache (`0` when the engine does not report it) | | `loads[].num_used_tokens`, `max_total_num_tokens` | KV tokens in use and KV capacity (`0` for workers read from Prometheus `/metrics`) | | `loads[].token_usage` | KV token usage ratio, 0.0 to 1.0 | | `loads[].active_token_usage` | Optional, present when the engine reports it: non-evictable KV usage ratio, 0.0 to 1.0, leaving out reclaimable prefix cache. Overload protection scores the worker on it instead of `token_usage` | | `loads[].gen_throughput`, `cache_hit_rate`, `utilization`, `max_running_requests` | As reported by the engine | | `loads[].memory`, `loads[].queues` | Optional sections, present when the engine reports them: `memory` (`weight_gb`, `kv_cache_gb`, `graph_gb`, `token_capacity`) and `queues` (`waiting`, `grammar`, `paused`, `retracted`) | | `loads[].kv_transfer_latency_ms`, `kv_transfer_speed_gb_s`, `prefill_queue_reqs`, `decode_queue_reqs`, `disagg_mode` | Optional prefill/decode disaggregation fields, present when the engine reports them | | `aggregate` | Fleet roll-up across all entries: `total_running_reqs`, `total_waiting_reqs`, and `total_reqs` are sums; `avg_token_usage`, `avg_throughput`, and `avg_utilization` are means | The same per-worker report appears as `engine_load` on `GET /workers` and `GET /workers/{worker_id}`. It is unrelated to their `load` field, which counts requests the gateway has in flight to the worker. [Overload Protection](../../concepts/reliability/overload-protection.md) explains where each backend's report comes from. --- ## Model Information Query model and server information. These are public routes and need no authentication. ### List Models ``` GET /v1/models ``` Returns the models that the registered self-hosted workers serve, read from the gateway's worker registry rather than from the workers; every alias of a served model is listed as an entry of its own. A caller that sends its own provider key gets the external providers' model list instead, fetched with that key; a gateway key (`--api-key` or `--tenant-api-key`) is never forwarded to a provider. A request carrying an `anthropic-version` header gets the list, and its errors, in the Anthropic Messages shape. See [List Models](openai.md#list-models) for the details. **Response:** `200 OK` ```json { "object": "list", "data": [ { "id": "meta-llama/Llama-3.1-8B-Instruct", "object": "model", "created": 0, "owned_by": "self_hosted" } ] } ``` `created` is always `0`. With no self-hosted model to list, the route answers `503` in the gateway's [error envelope](#error-responses) with the code `no_models_available`. --- ### Retrieve Model ``` GET /v1/models/{model} ``` Returns one model's card from the self-hosted inventory, in the same shape as a [List Models](#list-models) entry (the Anthropic model object under an `anthropic-version` header). `{model}` takes the whole rest of the path, so an ID with a slash in it works raw or percent-encoded, and an alias (a `--model-alias` name or a model card `aliases` entry) resolves to its canonical model. An unknown ID answers `404` `model_not_found` in the caller's error envelope. See [Retrieve Model](openai.md#retrieve-model) for the details. --- ### Get Model Info ``` GET /get_model_info ``` Forwards the request to the first healthy regular HTTP worker (the first healthy prefill worker in HTTP PD mode) and returns that worker's answer. SGLang serves this route, and its answer includes fields such as `model_path`, `tokenizer_path`, and `is_generation`. An engine without the route, such as vLLM, answers `404`. In regular mode the worker's status and body pass through unchanged; in HTTP PD mode an error status comes back in the gateway's [error envelope](#error-responses). With no healthy worker to ask, the route answers `503`. !!! note "gRPC and ZMQ workers" The gRPC router, which also serves ZMQ workers, does not implement this route and answers `501 Not Implemented`. For gRPC workers, the gateway calls the backend's `GetModelInfo` RPC once at worker registration and surfaces the result as worker labels (`model_path`, `served_model_name`, `vocab_size`, `max_context_length`, ...) in [`GET /workers`](#list-workers). --- ### Get Server Info ``` GET /get_server_info ``` Forwards the request the same way as [Get Model Info](#get-model-info) and returns the worker's answer. SGLang returns its server arguments together with scheduler state and its `version`. !!! note "gRPC and ZMQ workers" The gRPC router answers `501 Not Implemented`. For gRPC workers, the gateway calls the backend's `GetServerInfo` RPC once at worker registration and surfaces a curated subset of `server_args` (`tp_size`, `dp_size`, `max_total_tokens`, `version`, ...) as worker labels in [`GET /workers`](#list-workers). Live scheduler state (running and waiting requests, KV utilization) is served by [`GET /loads`](#get-loads) instead. --- ## WASM Module Management Manage WebAssembly plugins. Modules are registered from files on the gateway host; the request body contains descriptors with paths, not binary payloads. A module's `file_path` must be an absolute path ending in `.wasm`, with no `.` or `..` components. Paths under `/etc/`, `/proc/`, `/sys/`, `/dev/`, `/boot/`, `/root/`, `/var/log/`, or `/var/run/` are refused, and after symlinks are resolved the target is checked again: it must be a `.wasm` file outside those directories. See [WASM Plugins](../../concepts/extensibility/wasm-plugins.md#security-model) for the full security model. These routes need `--enable-wasm`. Without it, each answers `500 Internal Server Error` with an empty body. ### Add WASM Module ``` POST /wasm ``` Registers one or more WASM modules. The gateway computes each file's SHA-256 hash at registration and refuses a module whose hash is already registered. **Request Body:** JSON `WasmModuleAddRequest` ```json { "modules": [ { "name": "custom-middleware", "file_path": "/opt/smg/wasm/custom-middleware.wasm", "module_type": "Middleware", "attach_points": [ {"Middleware": "OnRequest"}, {"Middleware": "OnResponse"} ] } ] } ``` The only supported `module_type` is `Middleware`. Valid `Middleware` attach points are `OnRequest`, `OnResponse`, and `OnError`, but `OnError` modules never run: the middleware calls only `OnRequest` and `OnResponse` modules, and skips `OnResponse` for streaming responses and for responses over the WASM body limit. **Response:** `200 OK` on full success, `400 Bad Request` if any module failed to register. The response body echoes every requested module with an `add_result` field indicating success (carrying the assigned UUID) or failure (carrying the error message). ```json { "modules": [ { "name": "custom-middleware", "file_path": "/opt/smg/wasm/custom-middleware.wasm", "module_type": "Middleware", "attach_points": [ {"Middleware": "OnRequest"}, {"Middleware": "OnResponse"} ], "add_result": { "Success": "550e8400-e29b-41d4-a716-446655440000" } } ] } ``` --- ### List WASM Modules ``` GET /wasm ``` Returns all registered WASM modules together with aggregate execution metrics. **Response:** `200 OK` ```json { "modules": [ { "module_uuid": "550e8400-e29b-41d4-a716-446655440000", "module_meta": { "name": "custom-middleware", "file_path": "/opt/smg/wasm/custom-middleware.wasm", "sha256_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "size_bytes": 65536, "created_at": "2024-01-15T12:00:00.000000000Z", "last_accessed_at": "2024-01-15T12:05:00.000000000Z", "access_count": 40, "attach_points": [ {"Middleware": "OnRequest"} ] } } ], "metrics": { "total_executions": 40, "successful_executions": 40, "failed_executions": 0, "total_execution_time_ms": 120, "max_execution_time_ms": 8, "average_execution_time_ms": 3.0 } } ``` `metrics` covers every module. `average_execution_time_ms` is `total_execution_time_ms` divided by `total_executions`, and is left out until a module has run. --- ### Remove WASM Module ``` DELETE /wasm/{module_uuid} ``` Removes a WASM module. The body is a plain text status message, not JSON. **Response:** `200 OK` ``` Module removed successfully ``` On failure returns `400 Bad Request` with the error text as the body. A `module_uuid` that is not a UUID gets `400` with an empty body. --- ## Diagnostics Two control-plane routes read the gateway process itself rather than its workers; see [Authentication](#authentication). ### In-flight Requests ```text GET /inflight ``` Lists the HTTP requests this gateway process is serving at the moment of the call: every request that has entered the metrics layer and whose response body has not ended yet, so a streamed answer counts until its last byte, and the `/inflight` call itself is one of them. An upgraded connection (the realtime WebSocket) leaves the list once its upgrade response is sent; those connections are drained separately. The ids are the process's own sequence numbers, not the `x-request-id` values of the request-id layer; `instance_id` is a UUIDv7 generated at startup, so two gateways behind one load balancer answer with different ids. **Response:** `200 OK` ```json { "instance_id": "01999c2e-9c3a-7c4e-8d5f-0f1a2b3c4d5e", "request_ids": [1842, 1845, 1846] } ``` | Field | Description | |-------|-------------| | `instance_id` | This process's id, generated at startup | | `request_ids` | The sequence numbers of the requests in flight, in no particular order | The count behind this list is the `smg_http_connections_active` gauge; the ages of the requests in it feed `smg_http_inflight_request_age_count`. --- ### Heap Profile ```text POST /heap_profile ``` Asks this process's allocator for a heap profile (jemalloc `prof.dump`) written under the directory `--jemalloc-prof-dir` names, and answers the file's path. It needs a gateway built with the `jemalloc-profiling` feature and started with profiling on (`_RJEM_MALLOC_CONF=prof:true,prof_active:true,lg_prof_sample:19`); the file reads with `jeprof`. The route exists on every build, so a client learns why a dump is unavailable rather than meeting a `404` for the route. **Response:** `200 OK` with the file written. The name carries the UTC second, the process id and a per-process sequence number, so two dumps in one second are two files. ```json { "path": "/var/lib/smg/heap/smg-heap-20261024T191500Z-12345-0.heap", "prof_active": true } ``` `prof_active` is jemalloc's sampling state at the dump; a profile taken while sampling is off holds nothing. Anything else answers with the gateway [error envelope](#error-responses), `code` saying which precondition is missing: | Status | `code` | When | |--------|--------|------| | `501` | `heap_profile_not_built` | The binary was built without the `jemalloc-profiling` feature | | `404` | `heap_profile_dir_not_configured` | `--jemalloc-prof-dir` is not set | | `409` | `heap_profile_inactive` | The process runs without `prof:true` in `_RJEM_MALLOC_CONF` | | `500` | `heap_profile_dump_failed` | The directory could not be created, or jemalloc refused the dump | A written profile increments `smg_allocator_prof_dumps_total{outcome="ok"}`; the `409` and `500` refusals increment it with `outcome="error"`. The `501` and `404` refusals record nothing. --- ## Error Responses The admin endpoints do not share one error format. The body depends on the part of SMG that rejects the request: | Source | Status | Body | |--------|--------|------| | Control-plane auth | `401`, `403` | Plain text, such as `Missing or invalid Authorization header`, `Invalid authentication token`, or `Admin role required for this control plane operation`. A `401` carries `WWW-Authenticate: Bearer realm="control-plane"` | | `--api-key` auth, when control-plane auth is not configured | `401` | The gateway error envelope below (without the `X-SMG-Error-Code` header), with the code `invalid_api_key`, and a `WWW-Authenticate: Bearer realm="data-plane"` challenge | | A tokenizer or WASM request body that does not parse | `400`, `415`, `422` | Plain text from the JSON extractor | | [Worker endpoints](#worker-errors), their unparsable bodies included | `400`, `404`, `409`, `413`, `500` | `{"error": "", "code": ""}` | | [Get Tokenizer](#get-tokenizer), [Get Tokenizer Status](#get-tokenizer-status) | `404` | `{"error": {"message": "", "type": "tokenizer_not_found"}}`, with `"type": "not_found"` from Get Tokenizer Status | | [Add Tokenizer](#add-tokenizer) | `409`, `503` | The `202` body shape with `"status": "failed"` | | [Remove Tokenizer](#remove-tokenizer) | `404` | `{"success": false, "message": ""}` | | `/parse/function_call`, `/parse/reasoning` | `400`, `503` | `{"error": "", "success": false}` | | [Add WASM Module](#add-wasm-module) | `400` | The module list, with `"add_result": {"Error": ""}` on each module that failed | | [Remove WASM Module](#remove-wasm-module), and every WASM route without `--enable-wasm` | `400`, `500` | Plain text or empty | | The gateway, on forwarded routes such as `/get_model_info` | `503` and others | The gateway error envelope below | Errors that the gateway generates itself, for example when no worker is available, use this envelope and repeat `code` in the `X-SMG-Error-Code` response header: ```json { "error": { "type": "server_error", "code": "no_workers", "message": "No workers are available", "param": null } } ``` `type` is the public API's vocabulary (`invalid_request_error` for a 4xx, `permission_error` for a 403, `rate_limit_error` for a 429, `server_error` for a 5xx), and `param` is always `null`. The inference endpoints use the same envelope; see [Error Responses](openai.md#error-responses) in the OpenAI-compatible API reference. --- ## Authentication The tokenizer, worker, WASM, cache flush, diagnostics, and deprecated `/get_loads` endpoints on this page are control-plane routes. Send the credential as `Authorization: Bearer `: 1. **With [control-plane auth](../../getting-started/control-plane-auth.md) configured** (`--control-plane-api-keys`, or `--jwt-issuer` with `--jwt-audience`): a control-plane API key or JWT. The `admin` role may use every method; the `user` role only `GET` and `HEAD`, so it can list and inspect but not change anything. A missing or invalid credential gets `401`, and a `user`-role credential on a mutating method gets `403`. The shared `--api-key` is not accepted. 2. **Without it**: the shared `--api-key`, for every method. Per-tenant `--tenant-api-key` keys are rejected, so a gateway with tenant keys but no `--api-key` answers `401` to every call. With no keys configured at all, the routes are open. With `--admin-port` set, the control-plane routes are served on the [admin listener](../configuration.md#listeners) instead, and the data listener answers `404` for them; authentication applies there the same way. Public endpoints need no authentication: the health probes, `/loads`, `/engine_metrics`, and the [Model Information](#model-information) routes. See [Auth Model](extensions.md#auth-model) for every route's tier.