--- title: Sticky Sessions and Routing Keys --- # Sticky Sessions and Routing Keys A multi-turn conversation runs fastest when each turn lands on the worker that served the turn before it, because that worker still holds the conversation's prefix in its KV cache. Sticky sessions give SMG that property: each request carries a **routing key**, taken from the request body's `rid` or from a header, and SMG pins the key to one worker for as long as the key stays in use. SMG provides stickiness in two forms: - The **`manual` policy** pins every keyed request in its own key-to-worker map. - The **routing-key override** (`--routing-key-override`, alias `--sticky-sessions`) adds the same pinning on top of any other policy. Keyed requests stick; requests without a key, and by default the first request for each new key, are still placed by the configured policy, such as `cache_aware`. --- ## Overview
### :material-source-branch: Conversation Lineage The key comes from the body `rid` with turn and retry suffixes stripped, so `conv_t1`, `conv_t2`, and `conv_t2_r1` all pin as `conv`.
### :material-tag-outline: Routing-Key Headers SMG reads the first valid value from an ordered list of headers, `x-smg-routing-key` by default; the body `rid` is the fallback when no header carries one.
### :material-routes: Any Policy With the override, keyed requests stick on every policy, and by default the configured policy still chooses where a new key starts.
### :material-gauge: Bounded and Observable Idle pins expire, a key's requests beyond two in flight are placed again instead of going straight to its pinned worker, and inference responses name the worker that served them.
--- ## What a Routing Key Guarantees A routing key is a promise with bounds. While the key's pinned worker is available, a request that carries the key goes to it, up to the override's in-flight threshold described below; the moment the worker is not available, the request is placed elsewhere, with one bound: when every worker the request could use is vetoed at once, the gateway places it on the least-loaded of them (the pin included) rather than refuse it, and counts the fallback; under `--worker-overload-shed` an all-overloaded pool is refused with `503 worker_overload_protection_shed` instead, while a liveness veto still steers. In full: - **Same key, same worker.** Every request that resolves to the key (the first valid routing-key header, `x-smg-routing-key` by default, or, under the override, the body `rid`; see [Key Precedence](#key-precedence)) goes to the key's pinned worker, on every policy, with one exception: `consistent_hashing` honours a valid `X-SMG-Target-Worker` header (a healthy worker's index) over the key, as the policy table below states. Under `manual` and the override, each key remembers a primary worker and one failover candidate ([Reuse and Failover](#reuse-and-failover)); `consistent_hashing` keeps no map and recomputes the key's place on its ring, which moves about one key in N when a worker joins or leaves. - **Available means routable.** A pinned worker takes the request only while it is healthy (`Ready`), its circuit breaker is not open, and no veto excludes it. Two vetoes are on by default: [overload protection](../reliability/overload-protection.md) takes a worker out of routing while its last load report shows `8` or more waiting requests or KV-cache usage at or above `0.8` (`--worker-overload-waiting-requests`, `--worker-overload-token-usage`; `--disable-worker-overload-protection` switches it off), and the [liveness vetoes](../reliability/overload-protection.md#when-a-worker-dies-or-loses-its-link) exclude a worker whose transport failed and has gone silent, or whose engine has stopped producing. When none of a key's remembered workers is available, the request is placed again: in the override's default `delegate` mode the new worker becomes the primary and the old one its failover candidate; in the other modes the earlier worker stays first and the key returns to it once it is available (`occupied_miss` in the [metrics](#metrics)). When every candidate is vetoed at once there is no elsewhere: the request goes to the least-loaded worker that is routable but for its veto, overloaded ones before stalled ones, counted in `smg_worker_overload_fallback_total` or `smg_worker_liveness_fallback_total` (`stage="selection"`); under `--worker-overload-shed` an all-overloaded pool is refused with `503 worker_overload_protection_shed` instead, while a liveness veto still steers. - **The spill is bounded.** Under the override, once a key has `2` of its requests in flight on its pinned worker, the next concurrent request for the key is placed again (`cap_respill`) rather than queued behind the pin, so one busy conversation cannot stack every turn on one worker ([In-Flight Threshold](#in-flight-threshold)). The threshold triggers a new placement; it is not a cap on the worker: the placement may choose the pinned worker again when the policy prefers it, for example because it holds the conversation's prefix, so more than `2` of a key's requests can run there. - **A failure does not stay put — outside PD mode.** A request that fails on the pinned worker with a retryable answer is retried on another worker while an available one remains; only when nothing else is available does the retry return to the worker that just failed. In PD mode, a retry does not skip the workers it already tried: it selects its prefill/decode pair from all available workers again, so a keyed retry can return to the pinned legs even while other workers are free ([Retries](../reliability/retries.md)). - **Pins expire.** Under `manual` and the override, when eviction is enabled (`--max-idle-secs` and `--eviction-interval` both above zero), a key idle for longer than `--max-idle-secs` (default `14400`, four hours) is dropped by the next eviction sweep, and its next request starts over as a new key; `0` on either flag disables eviction ([Idle Eviction](#idle-eviction)). `consistent_hashing` keeps no pins to evict. What a routing key does not guarantee: - **One pin across gateway replicas.** Each replica keeps its own map ([Scoping](#scoping)). Send a conversation to one replica, or use `consistent_hashing`, which computes the same placement on every replica that sees the same workers. - **A warm cache.** The pin keeps a conversation on the worker that served it; whether that worker still holds the conversation's prefix is the engine's cache policy. For requests without a key, cache-aware routing follows the prefix instead. - **Order between concurrent turns.** Two requests for one key in flight at once are served in whichever order the worker schedules them, and, under the override, a third may run elsewhere (the bounded spill above). A minimal configuration, and what each flag changes: ```bash # Keyed requests stick on top of cache-aware placement; a new key starts where # cache-aware puts it, and the default overload veto and the two-in-flight spill stay on smg launch --worker-urls http://w1:8000 http://w2:8000 http://w3:8000 \ --policy cache_aware --routing-key-override ``` | What you set | What you get | |--------------|--------------| | `--routing-key-override` alone | Keys from `x-smg-routing-key` or, when no header carries a valid value, the body `rid` (lineage stripped, at most 128 bytes); the first request of a key is placed by the configured policy, then pinned | | `--routing-key-headers a b` | The first of `a`, `b` carrying a valid value (non-empty UTF-8, at most 128 bytes) is the key; under the override, a body `rid` that yields a key (at most 128 bytes after suffix removal) is the fallback when none does | | `--disable-worker-overload-protection` | A pinned worker stays the pin however deep its queue; the two-in-flight spill still applies | | `--max-idle-secs N` | A key idle for `N` seconds is dropped by the next eviction sweep, which runs every `--eviction-interval` seconds; `0` on either flag disables eviction | --- ## Deriving the Routing Key ### Key Precedence For each request, SMG uses the first of these that yields a key: 1. **Routing-key headers**: the first header named in `--routing-key-headers` that carries a valid value (non-empty UTF-8, at most 128 bytes); the override's pin key is that value with its lineage suffixes stripped (`manual` and `consistent_hashing` place on the raw value, see the warning below). `manual` and `consistent_hashing` also try `X-SMG-Routing-Key` when no configured header carries a valid value; the override map reads the configured names only. 2. **Body `rid`** (override enabled only): the `rid` with its lineage suffixes stripped, when the result is at most 128 bytes. 3. **No key**: the configured policy places the request, and nothing is pinned. A valid routing-key header wins even when the body carries a `rid`: an explicit key outranks request-ID lineage, so the client or proxy that sets the header chooses the pin, and a proxy must not stamp a per-request value into a routing-key header. `manual` and `consistent_hashing` apply this order themselves, with the same validity rules and the extra `X-SMG-Routing-Key` step. !!! note "Request IDs are not routing keys" `X-Request-ID` and the other request-ID headers identify a request in logs, traces, and backend request IDs, but never route it. Only the body `rid` field and the configured routing-key headers feed sticky routing. ### Body `rid` Lineage `rid` is an SGLang-style request ID accepted in the body of chat completions, completions, messages, `/generate`, embeddings, classify, and rerank requests (for rerank, the first ID of a list). The Responses API has no `rid`, so Responses requests can pin only through a header. SMG reads `rid` for routing only when `--routing-key-override` is enabled. SMG removes one trailing retry suffix, then one trailing turn suffix: ```text [_t][_r] -> ``` - A suffix is an underscore, a lowercase `t` (turn) or `r` (retry), and one or more ASCII digits, at the very end of the `rid`. - The retry suffix is removed first, so it has to come last: `conv_t2_r1` becomes `conv`, while `conv_r1_t2` becomes `conv_r1`. - Each suffix is removed at most once. - If nothing would be left, the whole `rid` is the key. - If the resulting key is longer than 128 bytes, SMG ignores the `rid`; a request whose headers carried no valid key is then placed without a key. | `rid` | Routing key | |-------|-------------| | `conv` | `conv` | | `conv_t2` | `conv` | | `conv_t2_r1` | `conv` | | `conv_r1` | `conv` | | `conv_t1_t2` | `conv_t1` | | `conv_r1_t2` | `conv_r1` | | `conv_t`, `conv_tx1`, `conv_T2` | Unchanged | | `under_scored_id` | `under_scored_id` | | `_t1` | `_t1` | Stripping only computes the key; the request keeps its full `rid`. On gRPC workers outside PD mode, SMG sends the `rid` as the engine request ID, so the suffixes let every turn and retry carry a distinct ID while sharing one key. ### Routing-Key Headers `--routing-key-headers` takes an ordered list of header names; the default is `x-smg-routing-key`. Each name must be a valid HTTP header name, or SMG refuses to start, and names match case-insensitively. For each request, SMG walks the list and takes the first header whose value is: - non-empty, - valid UTF-8, and - at most 128 bytes. A value that fails these checks is skipped without an error, and SMG tries the next name. With the override enabled, header keys get the same lineage stripping as a `rid`, so a proxy that forwards `conv_t2` in a header pins the same entry as a body `rid` of `conv_t2`. ```bash # Read an upstream proxy's header first, and keep the SMG header as a fallback smg launch --worker-urls http://w1:8000 http://w2:8000 \ --routing-key-override \ --routing-key-headers x-routing-key x-smg-routing-key ``` !!! warning "`manual` and `consistent_hashing` read headers their own way" These two policies apply the header-first order themselves, even with the override enabled, with the same validity rules as every policy (non-empty UTF-8, at most 128 bytes): the first valid header of the `--routing-key-headers` list, then `X-SMG-Routing-Key`, then the body `rid`. Their own placement key is the raw header value: neither strips lineage suffixes from a header (under the override, the pin map's key is the stripped value). A body `rid` is stripped and is the fallback on both when no header carries a key and the stripped result is at most 128 bytes. --- ## Configuration Enable the override on top of any policy, here `cache_aware`: ```bash smg launch --worker-urls http://w1:8000 http://w2:8000 http://w3:8000 \ --policy cache_aware \ --routing-key-override ``` Then give each turn of a conversation a `rid` that shares a base: ```bash curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-3.1-8B-Instruct", "rid": "chat-7f3a_t1", "messages": [{"role": "user", "content": "Plan a three-day trip to Kyoto."}] }' ``` The next turn sends `"rid": "chat-7f3a_t2"`, and a retry of that turn sends `"rid": "chat-7f3a_t2_r1"`. All three pin under the key `chat-7f3a`. A client that cannot set `rid` sends `-H "x-smg-routing-key: chat-7f3a"` instead. | Flag | Default | Description | |------|---------|-------------| | `--routing-key-override` | Off | Pin keyed requests on any policy. Alias: `--sticky-sessions` | | `--routing-key-headers` | `x-smg-routing-key` | Ordered header names to read the key from; the first valid value wins | | `--assignment-mode` | `delegate` for the override, `random` for `--policy manual` | How a new key picks its worker: `random`, `min_load`, `min_group`, or `delegate` | | `--max-idle-secs` | `14400` | Seconds a pin can go unused before it is evicted. Alias: `--sticky-key-idle-secs` | | `--eviction-interval` | `120` (Python launcher: `60`) | Seconds between eviction sweeps. Also sets the cache-aware tree eviction interval | `--assignment-mode`, `--max-idle-secs`, and `--eviction-interval` apply to both the `manual` policy and the override. See the [Configuration Reference](../../reference/configuration.md#manual-policy-and-sticky-sessions) for every routing option. ### Manual Policy or Override? | Aspect | `--policy manual` | `--routing-key-override` | |--------|-------------------|--------------------------| | First worker for a new key | `--assignment-mode` (default `random`) | The configured policy (`delegate`, the default) | | Requests without a key | Placed by `--assignment-mode`, not pinned | Placed by the configured policy, unchanged | | Key sources | The first valid header of `--routing-key-headers`, then `X-SMG-Routing-Key`, then the body `rid` when the override is also enabled (lineage stripped, at most 128 bytes) | `--routing-key-headers`, then the body `rid` (lineage stripped, at most 128 bytes) | | Lineage stripping of header keys | No | Yes | | Pins scoped by model | No (by PD leg only) | Yes (by model and PD leg) | | Per-key in-flight threshold | No | Yes | Choose the override to keep another policy, usually `cache_aware`, in charge of where conversations start. Choose `manual` when placement should depend only on the assignment mode. --- ## How the Override Composes with Each Policy | Configured policy | With `--routing-key-override`, a keyed request is | Without the override, a routing key is | |-------------------|---------------------------------------------------|----------------------------------------| | `cache_aware`, `round_robin`, `random`, `power_of_two`, `least_load`, `bucket`, `passthrough` | Pinned in the override's sticky map; under `delegate`, the policy chooses a new key's first worker | Ignored | | `prefix_hash` | Pinned in the override's sticky map; under `delegate`, the policy chooses a new key's first worker | Hashed onto the ring in place of the prompt's token IDs or text, which `prefix_hash` otherwise hashes | | `manual` | Pinned in the manual policy's own map, from the first valid routing-key header (`--routing-key-headers`, then `X-SMG-Routing-Key`) and the body `rid` as the fallback (at most 128 bytes after stripping) | Pinned from the first valid routing-key header | | `consistent_hashing` | Hashed onto the ring, from the routing-key header first and the body `rid` as the fallback (at most 128 bytes after stripping); `X-SMG-Target-Worker` still wins | Hashed onto the ring | Requests without a key go to the configured policy unchanged. Under `delegate` assignment, the override's default, the first request of a conversation is placed exactly as it would be without the override (by prefix match under `cache_aware`, for example), and later turns follow it. Pinning keeps the worker's cached prefix reachable on every turn; it does not make the engine keep that cache any longer. In PD mode, the prefill and decode legs are pinned independently. --- ## Assignment Modes and Eviction ### Assignment Modes `--assignment-mode` decides where a key goes the first time SMG sees it, and where it moves when none of its remembered workers is available. | Mode | A new key goes to | |------|-------------------| | `random` | A random available worker | | `min_load` | The available worker with the fewest in-flight requests | | `min_group` | The available worker with the fewest active routing keys (distinct keys with requests in flight on it) | | `delegate` | The worker the configured policy selects. `--policy manual` has nothing to delegate to, so there it behaves like `min_load` | `min_load` and `min_group` break ties at random, and each gateway replica counts only the requests it dispatched. The default is `random` for `--policy manual` and `delegate` for the override. One flag sets both, so leave `--assignment-mode` unset to keep delegation under the override. ### Reuse and Failover Each key remembers up to two workers: a primary and one failover candidate. A request goes to the first remembered worker that is **available**: healthy, with its circuit breaker not open, and not vetoed by [overload protection](../reliability/overload-protection.md). Adding workers never moves a key. When no remembered worker is available, SMG places the request again and records the result: - Under the override in `delegate` mode, the configured policy picks the worker, which becomes the new primary; the previous primary is kept as the failover candidate. - In the other modes, and in every mode under `--policy manual`, the assignment mode picks the worker and adds it behind the existing candidate, dropping the older one when two are already remembered. The earlier worker stays first, so the key returns to it as soon as it is available again. ### In-Flight Threshold Under the override, when a key already has **2** requests in flight on its pinned worker, SMG places the next concurrent request for that key again: through the configured policy in `delegate` mode, otherwise through the assignment mode. The request goes wherever that placement lands. The threshold triggers a new placement; it is not a hard limit. The placement can pick the pinned worker again, for example when `cache_aware` finds the conversation's prefix there, so a key can have more than 2 requests in flight on one worker. The pin moves only if the placement picks a different worker that has fewer than 2 of the key's requests in flight, so fleet-wide pressure cannot walk a conversation away from the worker that holds its prefix. Other traffic on the worker never triggers a new placement, and each gateway replica counts its own requests. The `manual` policy has no such threshold. ### Idle Eviction Every `--eviction-interval` seconds (default `120`, or `60` under the Python launcher), SMG removes pins that have not been used for `--max-idle-secs` (default `14400`, four hours). Each request that reuses a pin refreshes it. A key whose pin was evicted is placed like a new key on its next request. The map has no size limit: it holds one entry per key used within the idle window (per model and PD leg under the override), and each entry remembers at most two workers. Setting `--max-idle-secs` to `0` disables eviction, and the map then grows without bound. `--eviction-interval 0` also disables it, but SMG refuses to start with that value when the policy is `cache_aware`, the default. --- ## Scoping - **By model (override):** a key used with several models keeps an independent pin for each, so a conversation that switches models cannot evict its own pins. The `manual` policy's map is not scoped by model. - **By PD leg:** prefill and decode pins are independent, under both `manual` and the override. - **By gateway replica:** pins live in each replica's memory. They are not shared through [mesh HA](../architecture/high-availability.md), and a restart clears them. With several replicas, send each conversation to the same replica upstream, or use `consistent_hashing`, which computes the same placement on every replica that sees the same workers as available. - **Self-hosted workers only:** requests to external providers and to the Realtime API are placed by SMG's least-load selector and are never pinned. --- ## Request Body Streaming SMG can stream a large request body straight to an HTTP worker instead of buffering it, when nothing in the gateway needs to read the body; see [Request Streaming](../performance/request-streaming.md). Sticky routing changes that decision: - **Override enabled:** SMG buffers every request body so it can read the `rid`, even when the request carries routing hints. These requests are counted in `smg_router_request_body_path_total{path="buffered", reason="routing_key_override"}`. - **Override disabled:** a valid `x-smg-routing-tokens` header lets `cache_aware` and `bucket` place the request without its body, so a large body can stream. A routing-key header does not lift that requirement. The hint header format is in the [Request Headers reference](../../reference/api/openai.md#routing-hint-headers). --- ## Observability ### Routed Worker Header Inference responses from HTTP workers carry `x-smg-routed-worker-id`: the URL of the worker that served the request, as registered with the gateway, including the `@` suffix for data-parallel workers. For a prefill/decode pair it names the decode worker. Compare it across turns to confirm that a conversation sticks. Responses served by gRPC and ZMQ workers carry the header too, on streaming and non-streaming requests alike, with the same decode-worker rule. The value keeps the worker's registered scheme: `grpc://` for a gRPC worker, `ipc://` or `tcp://` for a ZMQ worker. From these workers, only successful responses carry the header; error responses and Responses API responses do not. ```bash curl -s -D - -o /dev/null http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -H "x-smg-routing-key: user-1234" \ -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hi"}]}' \ | grep -i x-smg-routed-worker-id # x-smg-routed-worker-id: http://w2:8000 ``` ### Metrics | Metric | Type | Labels | Description | |--------|------|--------|-------------| | `smg_manual_policy_branch_total` | Counter | `branch` | Sticky decisions by outcome: `occupied_hit` (pin reused), `occupied_miss` (remembered workers unavailable, key placed again), `vacant` (new key), `cap_respill` (in-flight threshold reached, request placed again), `no_routing_id` (`manual` request without a key), `no_healthy_workers` | | `smg_manual_policy_cache_entries` | Gauge | — | Entries in the sticky map, updated on each sticky decision | | `smg_routing_key_source_total` | Counter | `source` | Where the key came from for each keyed request the override routes: `rid` or `header` | | `smg_worker_routing_keys_active` | Gauge | `worker` | Distinct routing keys with requests in flight on each worker; the value `min_group` balances | | `smg_consistent_hashing_policy_branch_total` | Counter | `branch` | `consistent_hashing` outcomes: `target_worker_hit`, `target_worker_miss`, `routing_key_hit`, `implicit_key_hit` (no routing key; the request's `x-forwarded-for` or `cookie` header hashed instead), `random_fallback`, `no_healthy_workers` | | `smg_router_request_body_path_total` | Counter | `path`, `reason` | `reason="routing_key_override"` counts bodies buffered because the override is enabled | The `manual` policy and the override both report through the `smg_manual_policy_*` metrics. The full list is in the [Metrics Reference](../../reference/metrics.md). ### Debug Logs At `--log-level debug`, every decision the override makes logs one `Sticky routing decision` line with the key source (`rid` or `header`), the key (prefixed internally with the model and PD leg), the branch, the chosen worker, and the model ID. An eviction sweep that removes pins logs `ManualPolicy TTL eviction` at `info`. --- ## Troubleshooting ??? question "Turns of one conversation land on different workers" Check these causes in order: - **The override is off.** Without `--routing-key-override`, SMG ignores the body `rid`, and every policy except `manual`, `consistent_hashing`, and `prefix_hash` ignores routing-key headers. - **The `rid` does not match the suffix grammar.** `conv-t2`, `conv_turn2`, and `conv_T2` are each a key of their own. A high `vacant` rate in `smg_manual_policy_branch_total` under steady traffic points here. - **The key is invalid.** A key longer than 128 bytes, or an empty or non-UTF-8 header value, is ignored, on every policy. - **The header name is not configured.** The routing-key headers are the names in `--routing-key-headers` (`x-smg-routing-key` by default) on every policy; `manual` and `consistent_hashing` also read `X-SMG-Routing-Key` when none of the configured names carries a valid value. - **The pinned worker became unavailable.** Look for `occupied_miss`, then check worker health, circuit breakers, and overload vetoes. - **A key has more than two requests in flight.** The extra requests are placed again; look for `cap_respill`. - **The pin expired.** A key idle for longer than `--max-idle-secs` starts over. - **Several gateway replicas.** Each replica pins keys independently. ??? question "Load is uneven across workers" A pinned key stays on its worker while that worker is available; only new keys and keys placed again see current load. Choose `min_load` or `min_group`, or keep `delegate` with a load-aware policy such as `cache_aware`. ??? question "Large request bodies are always buffered" With the override enabled, SMG buffers every body to read the `rid`; check `smg_router_request_body_path_total{reason="routing_key_override"}`. To stream large bodies, disable the override and route with `x-smg-routing-tokens` instead. ??? question "The sticky map keeps growing" Each distinct key (per model and PD leg under the override) keeps an entry until it has been idle for `--max-idle-secs`. Watch `smg_manual_policy_cache_entries`, make sure neither `--eviction-interval` nor `--max-idle-secs` is `0`, and lower `--max-idle-secs` if conversations are short-lived. --- ## What's Next?
### :material-scale-balance: Load Balancing Every routing policy, including `manual` and `consistent_hashing`. [Load Balancing →](load-balancing.md)
### :material-cached: Cache-Aware Routing The policy that usually places a conversation's first turn. [Cache-Aware Routing →](cache-aware.md)
### :material-transfer-right: Request Streaming When SMG buffers a request body and when it streams it. [Request Streaming →](../performance/request-streaming.md)
### :material-format-list-bulleted: Request Headers Every header SMG reads or adds, including the routing hints. [Request Headers →](../../reference/api/openai.md#request-headers)
### :material-cog: Configuration Reference All routing policy options. [Configuration Reference →](../../reference/configuration.md#manual-policy-and-sticky-sessions)
### :material-chart-box: Metrics Reference Sticky routing, consistent hashing, and body-path metrics. [Metrics Reference →](../../reference/metrics.md)