` balances the memory-resident decode work instead: a worker's score is the resident context tokens (prompt plus output generated so far) its load report carries, plus the prompts the gateway has sent it since that report, plus `request_tokens` for every request it holds; in-flight requests that neither the report nor those prompts account for are estimated at the worker's mean context size. `request_tokens` is the per-request price in tokens and must be a non-negative integer — a bare `resident_tokens` is rejected at startup.
Engines whose load records carry the resident-token count report it; others leave it absent, and a worker's count is read only when every one of its DP ranks reports it. A worker without the count takes the best reporting worker's score plus a token charge for each of its own in-flight requests, so it neither attracts every request nor gets starved; when no candidate reports the count, the fewest in-flight requests decides.
`least_requests` and `resident_tokens` give up the policy's O(1) selection: they sample no pair and score every available worker. Whatever the metric, a single available worker is selected directly.
**Use when:** Large or heterogeneous fleets where cache locality doesn't matter and each pick should read only two workers' load.
---
## Least Load
Scores every available worker by its **expected wait**, meaning how long the work already queued on it will take to drain, plus a penalty that rises steeply as its KV cache fills. It routes to the lowest score. Work sent since a worker's last load report counts too, so a burst that arrives between two polls spreads across workers instead of landing on the one that looked idlest.
```bash
smg launch \
--policy least_load \
--worker-urls grpc://worker1:50051 grpc://worker2:50052 \
--model-path meta-llama/Llama-3.1-8B-Instruct
```
#### :material-check-circle: Advantages
- Weighs requests by token work, not request count
- Steers away from workers near KV-cache exhaustion
- In-flight credit prevents herding between load polls
#### :material-close-circle: Limitations
- No cache locality
- Scores every available worker on each request
- Estimates HTTP requests without token IDs at a mean size, so it is intended for gRPC workers
### Expected-Wait Score
```text
score = (queued_tokens + running_tokens + in_flight_tokens) / throughput
+ kv_pressure_weight * k / (1 - k)
```
| Term | Meaning |
|------|---------|
| `queued_tokens` | Uncached tokens waiting in the worker's queue, from its load report (summed across DP ranks). An engine that reports a queue depth but no token count, such as one scraped from Prometheus `/metrics`, is estimated at waiting requests × `--least-load-mean-prefill-tokens` |
| `running_tokens` | The report's running requests, each estimated at `--least-load-mean-prefill-tokens`; reports carry no remaining-work token count |
| `in_flight_tokens` | Work this gateway has sent the worker since its last load report: each request's token count when its token IDs are known (gRPC workers, pre-tokenized requests), otherwise `--least-load-mean-prefill-tokens`. A report releases the credit for the dispatches it already reflects |
| `throughput` | The worker's live generation rate: its reported generation throughput in tokens/s, summed over the DP ranks that have requests running. Dividing by it turns token work into seconds, so faster workers take proportionally more. Prefill workers and workers without a live rate are priced as described below |
| `k` | KV-cache usage from 0 to 1 (mean across DP ranks). `k / (1 - k)` stays small until the cache is nearly full, then climbs steeply |
Both terms are in seconds, so they add directly. The gateway's own in-flight count floors the estimate: requests it still holds on the worker beyond what the report and the credit cover are priced at the mean, so a stale report can never make a worker look emptier than the gateway's outstanding requests on it. Scores within a microsecond of the minimum count as a tie, broken uniformly at random, so reports that differ only in a trailing digit do not hand every request to the first worker.
For prefill workers, `throughput` is read as input tokens/s: prefill work consumes input tokens, which a generation rate cannot price. A prefill worker whose load report carries a **measured prefill capacity** on every DP rank — the uncached prompt tokens per second of active prefill its servicer measured on that rank — is priced at the sum of those rates. A worker whose report lacks the measurement — a rank without a valid rate, or a report that does not cover every rank — is priced at the mean measured rate of the candidates that have one; `--least-load-default-throughput`, still read as input tokens/s, stands in only when no candidate reports a measured rate.
The measured capacity is an observed service rate, not a hardware constant, and it ages out: each rank's samples expire after 60 seconds, and a rank reports a rate only while at least two valid samples remain, so a worker that has prefilled nothing for a minute returns to the fallback pricing above until fresh prefills rebuild its rate.
The measurement comes from the worker's servicer. The [Rust vLLM servicer](../../getting-started/grpc-workers.md#rust-vllm-servicer) times each request's prefill from the engine's own scheduling events, which vLLM emits only with its scheduler stats (`log_stats`) enabled: uncached prompt tokens per second of active prefill, counting overlapping prefills' time once and skipping preempted requests and pure cache hits. The Rust [SGLang](../../getting-started/grpc-workers.md#rust-sglang-servicer), [TokenSpeed](../../getting-started/grpc-workers.md#rust-tokenspeed-servicer) and TensorRT-LLM servicers measure on their own clock instead, timing every request from hand-off to first output; seconds that several requests' intervals share — a wait in queue behind another prefill — count once. Their interval includes wire hops and, on a PD prefill worker, the KV transfer, so the reading errs low, never high. A worker whose reports never carry a rate — the Python servicers do not measure one — is always priced by the fallback chain above.
Missing signals degrade gracefully:
- A DP rank with no requests running contributes no rate — its throughput gauge holds whatever its last decode interval left behind, which is not a live rate. A worker with no live rate on any rank is priced at the mean live rate of the candidates that have one, so an idle worker beside busy peers reads as fast as they are, not slower; `--least-load-default-throughput` stands in only when no candidate reports a live rate. A prefill worker without a complete measured capacity falls back the same way: the candidates' mean, then the configured default.
- A worker that has not reported a load yet, such as one that just joined, scores as the best-scoring reporting worker plus its own in-flight requests, so it neither attracts every request nor gets starved.
- When no worker reports at all (a cold start, or engines that never report), the policy picks the worker with the fewest in-flight requests.
`least_load` has no waiting-queue cap of its own. The per-worker ceiling is [Overload Protection](../reliability/overload-protection.md)'s veto, which skips a worker whose reported waiting requests reach `--worker-overload-waiting-requests` — under `least_load` and every other policy, apart from the two placements exempt from the veto ([Excluding a Worker](../reliability/overload-protection.md#excluding-a-worker)). A policy configuration that still sets the retired `max_waiting_requests` key is refused at startup, naming the overload knobs that replaced it.
### Tuning Flags
| Flag | Default | Description |
|------|---------|-------------|
| `--least-load-kv-pressure-weight` | `0.15` | Weight of the KV-pressure term, in seconds. Raise it to steer harder away from nearly full KV caches; `0` removes the term |
| `--least-load-default-throughput` | `2000` | Fallback throughput (tokens/s) when no candidate reports a live rate (for prefill workers the live rate is a measured prefill capacity; for the others, a generation rate). Set it to your per-replica input rate for prefill, output rate otherwise; it co-tunes with the KV-pressure weight |
| `--least-load-mean-prefill-tokens` | `1024` | Tokens assumed per request when the real count is unknown |
These flags tune only `least_load`; `power_of_two` and `cache_aware` use the same score with its default tuning. Loads are polled every `--load-monitor-interval` seconds (default `10`), and the in-flight term covers the gap between polls. See [Least Load Policy Options](../../reference/configuration.md#least-load-policy-options).
**Use when:** gRPC workers serve prompts of mixed sizes, cache locality doesn't matter, and you want each request on the worker with the shortest expected wait.
---
## Consistent Hashing
Routes each request by its routing key on a consistent hash ring with 150 virtual nodes per worker. Minimizes redistribution when workers scale—only ~1/N keys move when adding/removing workers. Each model has its own ring; requests that name no model, such as `/generate`, use a ring that spans every worker.
```bash
smg --policy consistent_hashing --worker-urls http://w1:8000 http://w2:8000
```
#### :material-check-circle: Advantages
- Minimal redistribution on scaling
- Automatic failover to next healthy worker
- O(log n) lookup time
#### :material-close-circle: Limitations
- No load awareness
- No cache locality
- Keyless requests go to a random worker
### Routing Headers
| Header | Description |
|--------|-------------|
| `X-SMG-Target-Worker` | Route to a worker by 0-based index into the model's currently available workers, so indices shift when a worker becomes unavailable. An index past the end of that list, or a value that is not a number, fails the request with `503` instead of falling back. When overload protection vetoes every worker of the pool, the all-vetoed fallback places the request without reading the header |
| `X-SMG-Routing-Key` | Hash this key onto the ring for session affinity. Names listed in `--routing-key-headers` are read first. Values must be non-empty UTF-8 of at most 128 bytes |
**Priority order:** `X-SMG-Target-Worker` → routing-key header → body `rid` (with `--routing-key-override`) → implicit keys (`X-Forwarded-For`, `Cookie`) → random fallback
On regular (not prefill/decode) workers, a request placed by routing key — a routing-key header, or the override's `rid`-derived key, with no `X-SMG-Target-Worker` — keeps its ring owner even while the owner is over the [overload protection](../reliability/overload-protection.md#excluding-a-worker) thresholds, so a session is not moved off the worker holding its cached prefix each time that worker crosses them. The request moves only when the owner is unhealthy, behind an open circuit breaker, or excluded by a liveness veto; `--worker-overload-shed` restores the overload veto. Implicit keys identify a client rather than a session and do not keep a vetoed owner.
See [Sticky Sessions and Routing Keys](sticky-sessions.md) for how routing keys are derived and validated.
**Use when:** Session affinity needed, user-to-worker pinning, or consistent routing for stateful applications.
---
## Prefix Hash
A lightweight alternative to full cache-aware routing. Hashes the head of each request onto a consistent hash ring, so requests that start the same way land on the same worker, and moves a request off that worker only when it is clearly overloaded.
```bash
smg --policy prefix_hash --prefix-token-count 256 --worker-urls http://w1:8000 http://w2:8000
```
#### :material-check-circle: Advantages
- Predictable O(log n) performance
- Lower memory than cache_aware
- Groups requests that share a head
#### :material-close-circle: Limitations
- Prefix grouping, not exact matching
- Less precise than cache_aware
- Load check sees only this gateway's in-flight requests
### Hash Key
Each request is hashed on the first of these it carries:
1. A valid routing-key header (`X-SMG-Routing-Key` by default), hashed as is
2. Token IDs (gRPC workers, pre-tokenized requests, or an `x-smg-routing-tokens` header): the first `--prefix-token-count` tokens
3. Prompt text: the first 4 × `--prefix-token-count` characters
A request with none of these goes to the least-loaded worker. On HTTP workers the prompt is available only when the gateway parses the request body. A request it [streams to the worker unparsed](../performance/request-streaming.md), which can happen when router retries are disabled or the body exceeds `--max-buffered-request-bytes`, is hashed on one of the headers above or sent to the least-loaded worker.
With `--cache-boundaries` set (ascending token positions, for example `2048,8192`), the key is the request's head up to the deepest boundary it reaches, counting 4 characters per token for text. A short turn and its longer follow-up that share a boundary-aligned head then hash together. Requests shorter than the smallest boundary keep the `--prefix-token-count` key.
### Load Check
A ring worker counts as overloaded when its in-flight requests exceed both `--prefix-hash-load-factor` times the fleet average and the average plus `--prefix-hash-balance-abs-threshold` (default `10`). The absolute margin keeps the check from firing on noise when each gateway replica sees only part of the traffic; `0` makes the check purely relative. For an overloaded target the policy first tries each shallower boundary (with `--cache-boundaries`), then falls back to the least-loaded worker that is not overloaded, or to the ring target anyway if every worker is overloaded. The fallback is counted in the `load_balance_walk` branch of `smg_prefix_hash_policy_branch_total`.
### Comparison with Cache-Aware
| Aspect | prefix_hash | cache_aware |
|--------|-------------|-------------|
| Lookup | O(log n) | O(prefix_len) |
| Memory | O(workers × virtual_nodes) | O(total_tokens) |
| Precision | Same head (fixed length or boundary) | Longest matching prefix |
**Use when:** Need some cache locality with predictable performance and lower memory footprint.
---
## Manual
Pins each routing key to a worker in an explicit key-to-worker map. Keys come from the `X-SMG-Routing-Key` header, or from the request body's `rid` when `--routing-key-override` is also enabled. Unlike consistent hashing, adding workers never moves an existing key; a key moves only when its worker becomes unavailable or the key goes unused for `--max-idle-secs`.
```bash
smg launch --policy manual --assignment-mode min_load --worker-urls http://w1:8000 http://w2:8000
```
#### :material-check-circle: Advantages
- Strong session stickiness
- Failover to a second worker, and back once the original recovers
- Keys idle longer than `--max-idle-secs` are evicted
#### :material-close-circle: Limitations
- No load balancing for keys already pinned
- Requests without a key are spread by the assignment mode and never pinned
- Pins live in each gateway replica's memory
### Assignment Modes
| Mode | Description |
|------|-------------|
| `random` | Randomly select from available workers (default) |
| `min_load` | Select the worker with the fewest in-flight requests |
| `min_group` | Select the worker with the fewest active routing keys |
| `delegate` | Same as `min_load` under this policy |
To pin conversations while another policy such as `cache_aware` decides where each one starts, use `--routing-key-override` instead. See [Sticky Sessions and Routing Keys](sticky-sessions.md).
**Use when:** Stateful chat sessions where context is stored on workers, or when session continuity is critical.
---
## Round Robin
Rotates through the available workers in order. The rotation is kept per candidate set, the exact list of workers a selection chooses from, so when one policy instance serves several sets (such as the decode partners of each PD prefill), each set is covered evenly. A worker that becomes unavailable changes the set, which then starts its own rotation.
```bash
smg --policy round_robin --worker-urls http://w1:8000 http://w2:8000
```
#### :material-check-circle: Advantages
- Even distribution within each candidate set
- Predictable routing pattern
- Small state (one counter per candidate set, at most 4,096 sets)
#### :material-close-circle: Limitations
- No load awareness
- No cache locality
- Ignores request characteristics
**Use when:** All workers have equal capacity and you want predictable, even distribution.
---
## Random
The simplest policy—each available worker has equal probability of selection. Zero state overhead.
```bash
smg --policy random --worker-urls http://w1:8000 http://w2:8000
```
#### :material-check-circle: Advantages
- Zero state overhead
- O(1) selection time
- Completely stateless
#### :material-close-circle: Limitations
- No load awareness
- No cache locality
- Can create hot spots
**Use when:** Testing environments or completely homogeneous workloads where simplicity is preferred.
---
## Passthrough
Forwards every request to the first available worker. It does no balancing and never reads worker load or cache state, and because only `cache_aware` subscribes to KV-cache events, it adds no event subscription either. It is built for a gateway in front of a single worker, where a balancing policy has nothing to choose between.
```bash
smg launch --policy passthrough --worker-urls http://w1:8000
```
#### :material-check-circle: Advantages
- No scoring or per-request state
- No load or cache state to maintain
- No KV-event subscription on gRPC workers
#### :material-close-circle: Limitations
- With several workers, all traffic goes to the first available one
- Other workers take traffic only while the first is unavailable
- No load awareness or cache locality
If more than one worker is registered, the gateway logs a warning once and keeps sending everything to the first available worker. `smg serve` switches to `passthrough` on its own when it launches a single worker (`--data-parallel-size 1`, the default), overriding `--router-policy`. By default the gateway still polls worker loads, for engine metrics and overload protection; with `--disable-load-monitoring` it polls only when something else, such as `--engine-metrics` or overload protection, needs the data.
**Use when:** The gateway fronts exactly one worker.
---
## Choosing a Policy
### Decision Guide
| Requirement | Recommended Policy |
|-------------|-------------------|
| Production LLM inference | `cache_aware` |
| Load-aware routing without cache affinity | `least_load` (gRPC workers) or `power_of_two` |
| Session affinity (sticky sessions) | `manual` or `consistent_hashing`, or [sticky sessions](sticky-sessions.md) on any policy |
| PD disaggregation | `cache_aware` or `bucket` for prefill, `power_of_two` for decode |
| Lightweight cache locality | `prefix_hash` |
| Even distribution | `round_robin` |
| A single worker | `passthrough` |
| Testing/development | `random` |
### Scenario Guide
#### :material-message-text: Conversational AI
**Recommended:** `cache_aware`
Maximizes KV cache reuse for multi-turn conversations with shared system prompts.
#### :material-file-search: RAG Applications
**Recommended:** `cache_aware`
Exploits common document prefixes for faster Time to First Token.
#### :material-account-group: Multi-Tenant Platform
**Recommended:** `consistent_hashing` or `manual`
User-to-worker affinity for tenant isolation or stateful sessions.
#### :material-server-network: PD Disaggregation
**Recommended:** `cache_aware` or `bucket` (prefill) + `power_of_two` (decode)
Prefix affinity or length buckets for prefill, load-based selection for decode workers.
#### :material-chart-bar: Mixed Prompt Sizes
**Recommended:** `least_load`
Scores token work rather than request counts, so long prompts don't pile onto one worker, and steers away from nearly full KV caches.
#### :material-server: Single Worker
**Recommended:** `passthrough`
Nothing to balance, so skip the selection and cache bookkeeping entirely.
---
## Policy Scope
### Per-Model Policies
`--policy` applies to every model unless a worker names another policy. A worker registered with a `policy` label, for example `"labels": {"policy": "least_load"}` in its [worker spec](../../getting-started/multiple-workers.md), sets the policy for its model. The first worker registered for a model decides, and the model keeps that policy until its last worker is removed.
A label that names the same policy as `--policy` gets a per-model instance built from your flags. A label naming any other policy gets that policy's built-in defaults, which can differ from the CLI defaults; its flags are not applied. In PD and EPD mode, the legs always use the leg policies below, whatever the labels say.
### PD and EPD Legs
In PD mode, `--prefill-policy` and `--decode-policy` set each leg's policy and default to `--policy`. Both accept `least_load`. `--decode-policy bucket` is rejected at startup (in PD mode outside IGW, the Rust binary skips that check when service discovery is configured; see [Bucket](#bucket)), and the Rust binary also rejects `passthrough` on either leg. Outside IGW mode, and without service discovery or `--dp-aware` (under which each URL registers one worker per DP rank), a leg given `power_of_two` through `--prefill-policy` or `--decode-policy` needs at least two workers on that leg, or startup fails. In EPD mode, `--encode-policy` accepts `random`, `round_robin` or `consistent_hashing` and defaults to `consistent_hashing`. See [PD Disaggregation](pd-disaggregation.md).
---
## Observability
- `smg_worker_selection_total` counts every policy selection by `worker_type`, `connection_mode`, `model` and `policy`.
- `smg_cache_aware_policy_branch_total`, `smg_prefix_hash_policy_branch_total`, `smg_consistent_hashing_policy_branch_total` and `smg_manual_policy_branch_total` count which branch each decision took (label `branch`), for example a `prefix_hash` request moved off its ring worker (`load_balance_walk`). The cache-aware counter covers tree-mode and KV-event decisions; hash-mode selections and decisions made while KV-cache imbalance suspends affinity record no branch.
- `smg_worker_requests_active` is the per-worker in-flight count that load-aware policies compare.
- At `--log-level debug`, `cache_aware`, `prefix_hash`, `least_load`, `power_of_two` and sticky-session selections log each routing decision with the chosen worker. See [Logging](../../getting-started/logging.md).
See the [Metrics Reference](../../reference/metrics.md) for the full list.
---
## What's Next?
### :material-cached: Cache-Aware Routing
Deep dive into the radix tree architecture and routing algorithm.
[Cache-Aware Routing →](cache-aware.md)
### :material-shield: Circuit Breakers
How SMG handles worker failures gracefully.
[Circuit Breakers →](../reliability/circuit-breakers.md)