openapi: 3.0.3 info: title: vLLM Semantic Router Model Runtime version: 5.0.0 description: | Contract of the vllm-sr model runtime. A runtime process serves one or more models; every request may name its `model` (required when a process serves several). `POST /v1/systemone` and `POST /v1/systemone/batch` speak the process's System One schema (`vllm-srun serve --systemone-schema`): `vllm-sr/5.0` by default, the System One standard plus the vllm-sr extension set (config/schemas/systemone/vllm-sr-5.0.extensions.yaml), or `vllm-sr/4.0`, which accepts the request forms below and every 5.0 form. Either is read into the runtime's internal request form, described here (DecisionRequest): the 4.0 forms plus `roles`. `POST /v1/decisions` and `POST /v1/bundle` read the internal form as it is, whatever the schema. In the internal form, `POST /v1/systemone` accepts System One requests, including `assets` (as base64 or as multipart/form-data parts) and compressed bodies, and answers them with System One fields. It also accepts vLLM Semantic Router extensions: `images`, `videos`, batches of named requests (`states`), Set and Span questions, `choices` / `levels` / `labels`, Box `at` and `track` questions on videos, and `options`. A request is a standard request (`state` and `questions`) or a batch (`states` alone), never both. `POST /v1/decisions` takes the same requests and answers a failed question in its answer, where `POST /v1/systemone` answers the request as a whole, as System One does. The model runtime reference documents both. `POST /v1/classify`, `POST /v1/embeddings` (OpenAI-compatible) and `POST /v1/rerank` serve fixed heads, embeddings and pair scores. `POST /v1/bundle` carries the work of several surfaces and models in one call. The router's Go client is generated from this file. license: name: Apache-2.0 paths: /v1/decisions: post: operationId: createDecisions summary: Answer typed Choice, Noul, Score, Box, Set and Span questions about one or more states description: | The body may be compressed with one layer of `Content-Encoding` `gzip` (or `x-gzip`) or `zstd` (a window of at most 8 MiB); `limits.max_request_bytes` applies to the body as sent and again once decoded. Another coding, or more than one, is 415 unsupported_media_type (reason `encoding_unsupported`); a body that does not decode is 400 (reason `encoding_malformed`). A body may be multipart/form-data (decompressed first). requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DecisionRequest' multipart/form-data: schema: $ref: '#/components/schemas/DecisionMultipart' responses: '200': description: | One answer per question, in request order (in a batch, per entry); a failed question carries its own error. A request in which no question is valid, of a standard request or of any entry of a batch, is answered 400 invalid_request, with each question's reason in the message. A batch entry whose images, videos, assets or tokens are refused fails each of its questions with `invalid_input` (reason `input_malformed`) and the refusal's message, and the other entries are answered; a batch every entry of which is refused is answered with the first refusal. headers: Server-Timing: {$ref: '#/components/headers/ServerTiming'} content: application/json: schema: $ref: '#/components/schemas/DecisionResponse' '400': {$ref: '#/components/responses/Error'} '404': {$ref: '#/components/responses/Error'} '413': {$ref: '#/components/responses/Error'} '415': {$ref: '#/components/responses/Error'} '422': {$ref: '#/components/responses/Error'} '429': {$ref: '#/components/responses/Error'} '500': {$ref: '#/components/responses/Error'} '503': {$ref: '#/components/responses/Error'} /v1/systemone: post: operationId: createSystemOne summary: System One requests; a failed question fails the request description: | In the process's System One schema. Under `vllm-sr/5.0` (the default) a request is one standard request with the extension fields flat in their places, any other member refused with 400 naming it; an object state without `roles` gives no part a role; assets whose placeholders lead the state (images, then videos, then one other item) are read in front of that item, as `images` and `videos` are; errors are the standard `detail` (a list on a 422) with the flat `code`, `question`, `state`, `reason` and `limit`; and an overloaded process answers 529 with Retry-After. Under `vllm-sr/4.0` the requests of /v1/decisions below are read as they are, and an error adds `detail` and the flat fields beside `error`. In the internal form: the requests of /v1/decisions: a standard request (`state` and `questions`) or a batch of named standard requests (`states`), whose answers come back under the same names. A request any of whose questions (of a standard request or of an entry of a batch) cannot be answered is answered as a whole with that question's error instead of a 200: the first failed question in request order gives the status (`invalid_question` 422; `max_length_exceeded`, `scan_budget_exceeded` and `invalid_input` 400; `invalid_model_output` 500; `unavailable` 503; `deadline_exceeded` 504) and the body is `question_failed` with that question's `question_error`, `reason` and `question` ID (in a batch, with its entry's name as `state`), and the message lists every failed question. System One answers carry no per-question error, and the official SDKs validate each answer by its type. Bodies, compression and headers are read as /v1/decisions reads them. requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/DecisionRequest' multipart/form-data: schema: $ref: '#/components/schemas/DecisionMultipart' responses: '200': description: One answer per question, in request order (in a batch, per entry); every question answered. headers: Server-Timing: {$ref: '#/components/headers/ServerTiming'} content: application/json: schema: $ref: '#/components/schemas/DecisionResponse' '400': {$ref: '#/components/responses/Error'} '404': {$ref: '#/components/responses/Error'} '413': {$ref: '#/components/responses/Error'} '415': {$ref: '#/components/responses/Error'} '422': {$ref: '#/components/responses/Error'} '429': {$ref: '#/components/responses/Error'} '500': {$ref: '#/components/responses/Error'} '503': {$ref: '#/components/responses/Error'} '504': {$ref: '#/components/responses/Error'} /v1/systemone/batch: post: operationId: createSystemOneBatch summary: Named System One requests answered in one call; a failed question fails the batch description: | vllm-sr/5.0's batch endpoint, on either schema: `requests` holds named standard requests (each may repeat the batch's `model`), answered under the same names in `responses`, each as it would be alone on /v1/systemone. A failed question answers the batch as /v1/systemone answers a batch, with its request's name as the error's `state`. An object state without `roles` gives no part a role under `vllm-sr/5.0` and keeps 4.0's implicit roles under `vllm-sr/4.0`. requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/SystemOneBatchRequest' responses: '200': description: Every request's response under its name, and the usage of them all. headers: Server-Timing: {$ref: '#/components/headers/ServerTiming'} content: application/json: schema: $ref: '#/components/schemas/SystemOneBatchResponse' '400': {$ref: '#/components/responses/Error'} '404': {$ref: '#/components/responses/Error'} '413': {$ref: '#/components/responses/Error'} '422': {$ref: '#/components/responses/Error'} '500': {$ref: '#/components/responses/Error'} '503': {$ref: '#/components/responses/Error'} '504': {$ref: '#/components/responses/Error'} /v1/classify: post: operationId: createClassification summary: Run a fixed head (label distribution, label scores or token spans) over texts, pairs or grounded answers requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/ClassifyRequest' responses: '200': description: One result per input, in input order; a failed input carries its own error. headers: Server-Timing: {$ref: '#/components/headers/ServerTiming'} content: application/json: schema: $ref: '#/components/schemas/ClassifyResponse' '400': {$ref: '#/components/responses/Error'} '404': {$ref: '#/components/responses/Error'} '413': {$ref: '#/components/responses/Error'} '422': {$ref: '#/components/responses/Error'} '429': {$ref: '#/components/responses/Error'} '500': {$ref: '#/components/responses/Error'} '503': {$ref: '#/components/responses/Error'} /v1/embeddings: post: operationId: createEmbeddings summary: OpenAI-compatible embeddings of text, images or audio, with dimensions and layer exits requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/EmbeddingsRequest' responses: '200': description: One embedding per input, in input order. headers: Server-Timing: {$ref: '#/components/headers/ServerTiming'} content: application/json: schema: $ref: '#/components/schemas/EmbeddingsResponse' '400': {$ref: '#/components/responses/Error'} '404': {$ref: '#/components/responses/Error'} '413': {$ref: '#/components/responses/Error'} '422': {$ref: '#/components/responses/Error'} '429': {$ref: '#/components/responses/Error'} '500': {$ref: '#/components/responses/Error'} '503': {$ref: '#/components/responses/Error'} /v1/rerank: post: operationId: createRerank summary: Score documents against a query with a pair scorer requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/RerankRequest' responses: '200': description: Documents by descending relevance. headers: Server-Timing: {$ref: '#/components/headers/ServerTiming'} content: application/json: schema: $ref: '#/components/schemas/RerankResponse' '400': {$ref: '#/components/responses/Error'} '404': {$ref: '#/components/responses/Error'} '413': {$ref: '#/components/responses/Error'} '422': {$ref: '#/components/responses/Error'} '429': {$ref: '#/components/responses/Error'} '500': {$ref: '#/components/responses/Error'} '503': {$ref: '#/components/responses/Error'} /v1/bundle: post: operationId: createBundle summary: Serve several surface requests, for one or more models, in one call requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/BundleRequest' responses: '200': description: One result per task, in task order, each with the status it would have had alone. headers: Server-Timing: {$ref: '#/components/headers/ServerTiming'} content: application/json: schema: $ref: '#/components/schemas/BundleResponse' '400': {$ref: '#/components/responses/Error'} '413': {$ref: '#/components/responses/Error'} '500': {$ref: '#/components/responses/Error'} /v1/models: get: operationId: listModels summary: Describe the served models, their surfaces, heads, limits, placement and verification responses: '200': description: The served models. content: application/json: schema: $ref: '#/components/schemas/ModelList' /health: get: operationId: getHealth summary: Readiness; 200 only when every model is loaded, verified and passed its golden check responses: '200': description: Ready. content: application/json: schema: $ref: '#/components/schemas/Health' '503': description: Not ready. A model whose forward has run longer than `--forward-timeout-seconds` is `degraded` until it returns. content: application/json: schema: $ref: '#/components/schemas/Health' /health/live: get: operationId: getLiveness summary: Liveness; 200 while the process serves HTTP and no forward has run longer than `--forward-timeout-seconds` responses: '200': description: Alive. content: application/json: schema: $ref: '#/components/schemas/Liveness' '503': description: Stalled; a forward has run longer than `--forward-timeout-seconds` and has not returned. Restart the process. content: application/json: schema: $ref: '#/components/schemas/Liveness' /metrics: get: operationId: getMetrics summary: Prometheus metrics responses: '200': description: Prometheus text exposition. content: text/plain: schema: type: string components: headers: ServerTiming: description: | The runtime's own time for the request in milliseconds, as a W3C Server Timing list: `parse;dur=0.021, tokenize;dur=0.153, queue;dur=0.008, forward;dur=4.871, post;dur=0.034, serialize;dur=0.019, total;dur=5.141`. `parse` reads and decodes the body. `tokenize` validates, renders and tokenizes the request. `queue` is the time it waited for its model, before and between its forwards. `forward` runs its forwards, readout included. `post` assembles the answers. `serialize` encodes the response. `total` runs from the handler's start until the response is ready to send; what it holds beyond the phases is the server's own overhead. A bundle whose tasks go to several models reports the queue, forward and post of the model answered last. A client's time for the call minus `total` is the transport: its own encoding and decoding, the connection and the HTTP exchange. schema: type: string responses: Error: description: A request-level error. headers: Server-Timing: {$ref: '#/components/headers/ServerTiming'} content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' schemas: DecisionRequest: type: object additionalProperties: false description: | A standard request or a batch; the two forms are mutually exclusive. A standard request is one context and its questions: `state` and `questions` (both required), and the `images` and `videos` every question sees. A batch is `states` alone: named standard requests, each answered under its name. A batch has no top-level `state`, `questions`, `images` or `videos`; a body with `states` and any of them is answered 400 invalid_request. `model`, `options` and `assets` belong to the request, and in a batch to every entry. properties: model: type: string description: Served model ID. Optional while a runtime serves one model. state: x-go-type-skip-optional-pointer: true description: | A standard request's context, which its questions are about; text, an object or an array of JSON values. Models with typed parts read an object's `request` / `user` / `prompt` as the user part, `answer` / `response` as the answer part and every other field as context. questions: type: object x-go-type-skip-optional-pointer: true description: A standard request's named questions; answers keep this order. minProperties: 1 additionalProperties: $ref: '#/components/schemas/Question' images: $ref: '#/components/schemas/DecisionImages' videos: $ref: '#/components/schemas/DecisionVideos' assets: $ref: '#/components/schemas/DecisionAssets' roles: $ref: '#/components/schemas/DecisionRoles' states: type: object minProperties: 1 description: | A batch: named standard requests answered in one call, so a client asks every question it has for one model at once. Questions about one context go in one entry, and each other context in an entry of its own. Each entry is read exactly as a standard request with its state, questions, images and videos and the batch's model, options and assets would be, and its answers come back under the same name in the response's `states`. Question IDs are unique within an entry. additionalProperties: $ref: '#/components/schemas/DecisionState' options: $ref: '#/components/schemas/RequestOptions' DecisionState: type: object additionalProperties: false required: [state, questions] description: One entry of a batch, a standard request without the batch's shared fields. properties: state: description: The context this entry's questions are about, as a standard request's `state`. questions: type: object description: Named questions about this state; answers keep this order. minProperties: 1 additionalProperties: $ref: '#/components/schemas/Question' images: $ref: '#/components/schemas/DecisionImages' videos: $ref: '#/components/schemas/DecisionVideos' roles: $ref: '#/components/schemas/DecisionRoles' DecisionRoles: type: object nullable: true # A client that leaves roles unset must omit the member: null means no roles. x-omitempty: true description: | What each key of an object state holds for a role-aware family (vela2, vela3, vela3_d3): the `user`'s content, `context`, or the `answer` under check; a key without one is context. Omitted, the keys name their parts as `state` says (vllm-sr/4.0's implicit roles). Null gives no part a role: the whole state is the user's content, as a vllm-sr/5.0 request without roles has it. Other families read the state as they render it, whatever the roles. additionalProperties: type: string enum: [user, context, answer] SystemOneBatchRequest: type: object required: [model, requests] additionalProperties: false properties: model: type: string requests: type: object minProperties: 1 description: Named vllm-sr/5.0 requests without options; each may repeat the batch's model. additionalProperties: type: object options: $ref: '#/components/schemas/RequestOptions' SystemOneBatchResponse: type: object required: [model, responses, usage] properties: model: type: string responses: type: object additionalProperties: $ref: '#/components/schemas/DecisionResponse' usage: $ref: '#/components/schemas/Usage' meta: $ref: '#/components/schemas/ResponseMeta' DecisionMultipart: type: object required: [request] description: | A request as multipart/form-data, as the System One SDKs send one with images: the first part is named `request` (application/json) and holds the request; every other part holds one image's bytes (named uniquely, typed image/png, image/jpeg, image/jpg, application/octet-stream or not at all) and is claimed by exactly one asset's `part`. Anything else is 400 invalid_request (reason `multipart_malformed`). properties: request: $ref: '#/components/schemas/DecisionRequest' additionalProperties: type: string format: binary DecisionAssets: type: object minProperties: 1 description: | Images by reference (System One `assets`), for models whose card lists the `image` modality. Each reference is 1 to 1,024 bytes and not blank, all of them together at most 65,536 bytes, and every one must be used (reason `asset_unreferenced`). A reference is replaced wherever it occurs in a string of the state, of a question's `instructions` or of its options (criteria values, `choices` and `labels` descriptions, `levels`), in a standard request or in every entry of a batch: a string equal to a reference is that image, a longer one is split at each occurrence (left to right, the longest reference first at one position), and object keys are never replaced. Each occurrence is one use, read where it sits and counted again, toward the token limits and `usage.input_tokens`; the bytes of all uses together are at most `limits.max_request_bytes` of the process (reason `asset_uses`). An image in the state is part of the state. An image in a question's instructions or options belongs to that question only, on models whose `image_placements` list `instructions` or `criteria`; another model answers that question with invalid_question, reason `question_images_unsupported`. One rejected image fails the request. additionalProperties: $ref: '#/components/schemas/Asset' Asset: type: object additionalProperties: false properties: type: type: string enum: [image] format: type: string enum: [png, jpg, jpeg] description: Optional; the format is read from the bytes (PNG, JPEG or WebP; animated images are refused). data_base64: type: string description: The image file, base64; at most `limits.image_max_bytes` decoded. part: type: string description: In a multipart request, the part holding the image file (data_base64 is then ignored). DecisionImages: type: array description: | Images every question of the state sees, for models whose card lists the `image` modality: base64 PNG, JPEG or WebP data URLs (`data:image/png;base64,...`), each at most `limits.image_max_bytes` bytes and `limits.image_source_max_pixels` pixels. The model reads each at up to `limits.image_max_pixels` pixels, in front of the text, in this order. A model without the modality answers a non-empty list with invalid_request, as it does a malformed image. items: type: string DecisionVideos: type: array description: | Videos every question of the state sees, after its images, for models whose card lists the `video` modality: base64 MP4, WebM, QuickTime or Matroska data URLs (`data:video/mp4;base64,...`), each at most `limits.video_max_bytes` bytes, `limits.video_max_seconds` seconds and `limits.video_source_max_pixels` pixels per frame. The model reads `limits.video_fps` frames per second, at least 4 and at most `limits.video_max_frames` spread over the whole video, each at up to `limits.video_max_pixels` pixels; every pair of frames is one group of input tokens after its timestamp, and the videos of one request take at most `limits.video_max_tokens` of them. A model without the modality answers a non-empty list with invalid_request, as it does a malformed or over-limit video. items: type: string Question: type: object additionalProperties: false description: | One question. Choice and Score may omit `instructions` (the model then chooses the best matching option); a Noul may when its criteria describe an outcome; Box and track need them; a `preset` names a question the model defines (see `presets` in /v1/models). Every family validates System One fields the same way: a question takes only its type's fields (`choices` for Choice and Noul, `levels` for Score, `labels` for Set and Span, `at` for Box, `rate` for track, `over`, `head` and `threshold` where the model reads them), and any other field is answered with invalid_question. `choices`, `levels` and `labels` are the Router config's ordered forms of `criteria`; a question takes one or the other. A Box question locates what its instructions describe in its target image: the image its instructions reference when there is one, else the state's only image (reasons `box_needs_image`, `box_target_ambiguous`); with `at`, at that moment of the state's only video; a track question locates it at every sampled moment of that video, `rate` times per second (reasons `video_needed`, `video_target_ambiguous`, `box_video_too_long` for a clip longer than `limits.box_video_max_seconds`, `at_out_of_range`). properties: type: type: string description: | choice, noul or score; box, track, set and span where the model declares them (`question_types` in /v1/models). Unknown types are answered with invalid_question (reason `type_unsupported`); an undeclared box or track with reason `box_unsupported` or `track_unsupported`. instructions: description: The question; non-blank text, an object or an array. at: type: number format: double minimum: 0 description: Box questions on a video; seconds from its start (the sampled moment nearest to it is read). rate: type: integer enum: [1, 2] default: 1 description: | Track questions: updates per second over the whole video, 1 (16-second windows at 2 frames per second) or 2 (8-second windows at 4); any other value is answered with invalid_question. criteria: description: | System One criteria. Choice: an object of 2..255 non-blank option keys to descriptions (descriptions may be null). Noul: an optional object with only false and true (a null description takes the model's default). Score: an ordered list of 2..10 level descriptions. Set and Span: an object of 1..255 labels to descriptions. Descriptions are non-blank text, an object or an array. choices: type: array description: Ordered Choice or Noul options, an alternative to a criteria object. items: $ref: '#/components/schemas/ChoiceOption' levels: type: array description: Ordered Score level descriptions, an alternative to a criteria list. items: {} labels: type: array description: Ordered Set or Span labels, an alternative to a criteria object. items: $ref: '#/components/schemas/ChoiceOption' over: description: | The state field a question reads (typed-part models): a field name, or a list of field names it reads together in state order; default the whole state, or for a span the answer if present, else the request. A span reads one field. require_full_input: type: boolean default: false description: | Require every supplied state part to be read completely, including context outside `over`. Clipping fails this question with max_length_exceeded unless complete window coverage is guaranteed. Cannot be combined with overflow truncate. Success includes input_coverage complete; absence of that proof must not be treated as full coverage by clients. Omitted or false retains the model's ordinary input-fitting behavior. overflow: type: string enum: [window, truncate] description: | Models with a scan budget (their card's `limits.max_scan_tokens`): how the question reads a field longer than one input. `window` (default): whole, in windows up to the request's `max_tokens`, failing with `scan_budget_exceeded` past it. `truncate`: its first `limits.truncate_tokens` tokens only (a Vela 3.0 evidence span question: its route's 32,768-token window). Questions of both kinds share their model inputs unless one has to be read in windows. head: type: string enum: [router, broad] description: | Span questions on models with a broad span head: the head that answers (default: the model's routing, reported per question in `span_heads`). Models without one answer `broad` with invalid_question. threshold: type: number format: double description: Set or Span decision threshold; the applied threshold is reported in `thresholds`. preset: type: string description: A question the model defines; the model fills in its type, instructions and criteria. decision_adapter: type: boolean default: false description: | Vela 3.0 d3 sizes (family `vela3_d3`) whose package declares `decision_opt_in`: true answers a Choice, Noul or Score question with the opt-in decision adapter (`decision_optin`, else the router adapter), as the package's runtime does. A router question keeps the router adapter; one the package routes to a task adapter takes the opt-in adapter instead. Only true turns it on; any other value, a package without opt-in, and a request with images, videos, assets or Box / track questions (which runs every adapter off) answer it as false. Other models answer the field with invalid_question. ChoiceOption: type: object additionalProperties: false required: [key] properties: key: type: string minLength: 1 description: nullable: true description: Text, an object, an array or null. RequestOptions: type: object additionalProperties: false properties: deadline_ms: type: number format: double exclusiveMinimum: true minimum: 0 description: Work not started by the deadline is not run; its items return deadline_exceeded. return_meta: type: boolean description: Include meta in the response (default false). long_context: type: boolean description: | Long context (default false): each question may have up to the model's `limits.max_input_tokens` instead of its `limits.validated_input_tokens`, and a state-once model's request up to `limits.max_request_tokens`. Slower and less accurate than the validated length. max_tokens: type: integer minimum: 1 description: | Scan budget: the most tokens of one state part a question reads whole (default its card's `limits.max_scan_tokens`; a part that fits one model input is always read). A longer part fails the questions that read it whole with `scan_budget_exceeded`. Only a model whose card has `max_scan_tokens` takes it. DecisionResponse: type: object required: [model, usage] description: | The answers to a standard request (`answers`, and `spans`, `sets`, `thresholds` and `span_heads` for the questions that have them) or to a batch (`states`: each entry's answers under its name). A batch's `usage` counts every entry, and its `meta` (with `return_meta`) is the one every entry carries. properties: model: type: string answers: type: object x-go-type-skip-optional-pointer: true description: | A standard request's answers, one per question; a Set question adds a Noul answer per label under `.