--- title: High Availability --- # High Availability Run several SMG routers as one cluster. With `--enable-mesh`, routers form a peer-to-peer mesh: they track each other's membership with a SWIM-style gossip protocol, replicate the workers each router registers, and share cache-aware routing updates, so every router routes with the cluster's workers and cache affinity. The mesh does not steer client traffic. Put the routers behind a load balancer or a Kubernetes Service; that is what moves clients to the remaining routers when one fails. --- ## Overview
### :material-shield-check: Fault Tolerance When a router fails, your load balancer sends its clients to the others, which keep the cache-aware routing state the failed router shared. Workers that only the failed router registered are removed shortly after it is marked `DOWN`, so register each worker on more than one router ([Worker Registry Sync](#worker-registry-sync)).
### :material-arrow-expand-all: Scalability Add routers without restarting the others. A new router joins through a running peer or Kubernetes router discovery and picks up the cluster's workers.
### :material-sync: State Synchronization Workers registered on any router and cache-aware routing tree updates reach every peer. Rate limits, sticky sessions, and other policy state stay local to each router.
### :material-rocket-launch: Rolling Updates Replace routers one at a time: each announces its leave to the mesh and drains in-flight requests on shutdown. With Kubernetes router discovery, a replaced pod rejoins as soon as it is Ready.
--- ## Mesh Architecture ```text clients │ load balancer or Kubernetes Service ┌────────────────┼────────────────┐ ▼ ▼ ▼ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ router A │◀───▶│ router B │◀───▶│ router C │ mesh: gRPC on --mesh-port └────┬─────┘ └────┬─────┘ └────┬─────┘ └────────────────┼────────────────┘ ▼ workers ``` Every router is equal: there is no leader and no quorum. Each pair of routers (A and C included) keeps one sync stream over the mesh port, and state converges without coordination.
### :material-connection: Gossip Membership SWIM-style membership and failure detection. - One probe per second to a random peer - Indirect probes through up to three peers - Status changes broadcast to live peers
### :material-swap-horizontal: Sync Streams One bidirectional gRPC stream per pair of routers. - Opened by the router whose name sorts first - Each side sends pending updates every second - Kept alive by a heartbeat every 15 seconds - Closed after 60 seconds without a frame
### :material-database-sync: CRDT Worker State Conflict-free replicated keys for worker state. - No locks or coordination - Resent until each peer acknowledges - Relayed through intermediate peers
### :material-share-variant: Cache-Tree Updates Best-effort broadcasts for cache-aware routing. - Batched per model every gossip round - Unknown prefixes trigger a repair from a peer - Dropped updates are not resent
--- ## Configuration ### Quick Start Start the first router, then point the next one at it with `--mesh-peer-urls`: ```bash # Router 1 (bootstrap) smg launch --enable-mesh --mesh-advertise-host 10.0.0.1 --mesh-port 39527 \ --worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000 # Router 2, joining router 1 smg launch --enable-mesh --mesh-advertise-host 10.0.0.2 --mesh-port 39527 \ --mesh-peer-urls 10.0.0.1:39527 ``` Router 2 has no workers of its own; it imports router 1's workers through the mesh. Imported workers last only while the router that registered them stays in the cluster — if router 1 goes down for good, router 2 removes them after a short grace (see [Worker Registry Sync](#worker-registry-sync)). To keep workers served through a router failure, register them on every router, as under [Basic Configuration](#basic-configuration). ### Command Line Options | Flag | Default | Description | |------|---------|-------------| | `--enable-mesh` | `false` | Start the mesh listener and join the cluster | | `--mesh-server-name` | `Mesh_` plus 4 random alphanumeric characters | Node name, unique within the cluster. Must be non-empty and must not contain `:`. With router discovery, set it to the pod name | | `--mesh-host` | `0.0.0.0` | Bind address for the mesh listener (an IP address) | | `--mesh-advertise-host` | `--mesh-host` | IP address that peers dial. Required when `--mesh-host` is an unspecified address such as `0.0.0.0` | | `--mesh-port` | `39527` | Mesh port, used for both binding and advertising. `0` is rejected | | `--mesh-peer-urls` | (none) | Bootstrap peers as `IP:port`, space- or comma-separated. While the membership table holds no other member, the entries are dialed in turn until one answers | | `--mesh-tls-ca-cert` | (none) | CA certificate (PEM) that mesh peer certificates must chain to. Set together with `--mesh-tls-cert` and `--mesh-tls-key` to enable [mesh mTLS](#mesh-mtls) | | `--mesh-tls-cert` | (none) | This node's certificate (PEM), with the node's IP as a SAN | | `--mesh-tls-key` | (none) | This node's private key (PKCS#8 PEM) | | `--router-selector` | (none) | Label selector for [Kubernetes router discovery](#kubernetes-router-discovery). Takes effect only when Kubernetes worker discovery is selected, with `--service-discovery` or `--discovery-provider kubernetes` | !!! note "Mesh addresses must be IP addresses" `--mesh-host`, `--mesh-advertise-host`, and `--mesh-peer-urls` are parsed as socket addresses and never resolved through DNS. A hostname such as `smg-0.smg-mesh:39527` fails at startup: the Python CLI reports `Invalid mesh peer URL`, the Rust binary `Invalid value for field 'mesh_peer_urls'`. With the default `--mesh-host 0.0.0.0`, the router also refuses to start until `--mesh-advertise-host` names a routable IP, so peers never try to dial an unspecified address. On Kubernetes, use [router discovery](#kubernetes-router-discovery) instead of peer URLs. !!! warning "The mesh port is plaintext by default" Without the `--mesh-tls-*` flags, mesh traffic is plaintext gRPC and peers are not authenticated: anything that can reach the mesh port can join the cluster and publish workers to every router. A router started with a plaintext mesh says so in its first log lines, with a `SECURITY POSTURE` warning that names the mesh listener and the flags that close it. Enable [mesh mTLS](#mesh-mtls) so only routers holding a certificate from your CA can join, and expose the port only to other routers either way (for example with a firewall rule or a Kubernetes NetworkPolicy). ### Mesh mTLS `--mesh-tls-ca-cert`, `--mesh-tls-cert`, and `--mesh-tls-key` encrypt the mesh port and authenticate both ends of every mesh connection. The three flags come together: a router started with only one or two of them refuses to start, and the error names the missing flags. ```bash smg launch --enable-mesh --mesh-advertise-host 10.0.0.1 \ --mesh-tls-ca-cert /etc/smg/mesh/ca.pem \ --mesh-tls-cert /etc/smg/mesh/node.pem \ --mesh-tls-key /etc/smg/mesh/node-key.pem ``` With the flags set, the mesh listener serves TLS with the node's certificate and accepts only peers that present a certificate chaining to the CA. Every dial to a peer presents this node's certificate and verifies the peer's against the CA. Issue every router a certificate from the same CA that carries the node's IP — the address its peers dial, so its `--mesh-advertise-host` — as an IP SAN. A dial verifies the peer's certificate against the address it dials, and mesh addresses are always IPs (see the note above), so a certificate with only a hostname never matches. An IPv6 node's SAN is the unbracketed address: `fd00::7`, not `[fd00::7]`. The private key must be PKCS#8 PEM. Each of the three files must exist and be readable at startup, or the router fails to start with an error naming the flag whose file it cannot read; the router also warns when the key file is readable or writable by its group or by others. The mesh listener reads the certificate and key once at startup and does not reload them; to rotate certificates, restart the router. Configure every router in the cluster the same way: a router with the flags serves and dials only TLS, a router without them only plaintext, so a mixed cluster's routers cannot reach each other. ### Python Entrypoint The Python CLI (`smg launch`) and the Docker image's entrypoint (`python3 -m smg.launch_router`) accept the same mesh flags with the same defaults and address checks, so container `args` can pass them directly: ```bash smg launch --enable-mesh --mesh-host 0.0.0.0 --mesh-advertise-host 10.0.0.11 --mesh-port 39527 ``` `smg serve` takes router flags with a `--router-` prefix, for example `--router-enable-mesh` and `--router-mesh-advertise-host`. ### Basic Configuration
**Node 1** (Bootstrap) ```bash smg launch --enable-mesh \ --mesh-server-name node1 \ --mesh-host 0.0.0.0 \ --mesh-advertise-host 10.0.0.11 \ --mesh-port 39527 \ --worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000 ```
**Node 2** (Join) ```bash smg launch --enable-mesh \ --mesh-server-name node2 \ --mesh-host 0.0.0.0 \ --mesh-advertise-host 10.0.0.12 \ --mesh-port 39527 \ --mesh-peer-urls 10.0.0.11:39527 \ --worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000 ```
**Node 3** (Join) ```bash smg launch --enable-mesh \ --mesh-server-name node3 \ --mesh-host 0.0.0.0 \ --mesh-advertise-host 10.0.0.13 \ --mesh-port 39527 \ --mesh-peer-urls 10.0.0.11:39527 \ --worker-urls http://10.0.1.1:8000 http://10.0.1.2:8000 ```
!!! tip "Start order does not matter" A router dials its `--mesh-peer-urls` peers as long as its membership table holds no other member to probe — at startup, and again if every other member is gone. The entries are taken in turn, one per round, each retried under the same backoff as a failed probe (1 second, doubling up to 60 seconds), until one answers; the rest of the cluster is learned from that member through gossip. Node 2 and node 3 can therefore start before node 1 and join as soon as it comes up. --- ## Gossip Protocol ### Membership and Failure Detection 1. **Probe**: Every second, each router pings one random peer that is not `DOWN` or `LEAVING`. The ping carries the sender's membership table, and the receiver adopts any entry with a higher version. 2. **Indirect probe**: If the ping fails, the router asks up to three other `ALIVE` peers to ping the target (ping-req). 3. **Suspect, then down**: If no probe reaches the target, it moves from `ALIVE` to `SUSPECTED`, or from `SUSPECTED` to `DOWN`, and the new status is broadcast to every `ALIVE` peer. 4. **Sync stream**: After a successful direct probe of an `ALIVE` peer, the router whose name sorts first opens the pair's sync stream if none is open. A probe that only a relay answered opens none: the pair's own link has just failed. ### Node Status States | Status | Description | |--------|-------------| | `ALIVE` | Reachable. Every router starts in this state | | `SUSPECTED` | A probe failed: neither the direct ping nor any indirect ping reached the router. Still probed. Router discovery also sets it for a known pod that is not Ready, unless that router is already `DOWN` | | `DOWN` | A probe failed again while `SUSPECTED`, or the router's pod was deleted or is terminating. Peers stop probing it | | `LEAVING` | The router announced a graceful leave, as every router does when its [graceful shutdown](../reliability/graceful-shutdown.md) begins. Peers stop probing it | | `INIT` | Defined in the protocol but not used | !!! warning "A `DOWN` or `LEAVING` router is not probed again" Gossip cannot clear a `DOWN` or `LEAVING` mark: a router that restarts under the same name announces itself with a lower version than the old entry, and its peers no longer probe it. This applies after a graceful shutdown (which leaves a `LEAVING` entry), after a crash, and after a network partition long enough for routers to mark each other `DOWN`. Router discovery re-admits a router whenever its watch reports the pod as Ready, as it does when a restarted pod becomes Ready. With static peers, restart the isolated router under a new `--mesh-server-name` (omitting the flag picks a random one) and point `--mesh-peer-urls` at a running router. ### Failure Detection Timing | Phase | Value | |-------|-------| | Gossip round (probe, send, and batching cadence) | 1 second | | Indirect probe fan-out | Up to 3 `ALIVE` peers | | Probe connect / request timeout | 5 seconds / 10 seconds | | Redial backoff after a failed probe of the same peer | 1 second, doubling up to 60 seconds | | Sync stream heartbeat (sent by the dialing router, answered by the accepting one) | 15 seconds | | Sync stream idle timeout (no frame received) | 60 seconds | These values are fixed; there are no flags to tune them. --- ## State Synchronization ### Synchronized State Types
### :material-server: Worker Registry Workers each router registers, shared with every peer. - URL, model, health, and load - Worker spec, without its API key - ZMQ workers (`ipc://` and `tcp://`) excluded
### :material-tree: Routing Trees Cache-aware routing state shared across routers. - HTTP string trees and gRPC token trees - Tree inserts, sent as per-model deltas - Repair from a peer for unknown prefixes
### :material-account-group: Membership The cluster's node table. - Node names, addresses, and status - Carried on every gossip ping
### :material-lan-disconnect: Stays Local Each router keeps its own: - Rate limits and admission queues - Sticky-session (`manual` policy) assignments - Health checks and circuit-breaker state - Routing policy and its settings
Because each router enforces [rate limits](../reliability/rate-limiting.md) on its own, the cluster-wide limit is roughly the per-router limit times the number of routers. [Sticky-session](../routing/sticky-sessions.md) assignments are also per router, so the same routing key can map to different workers on different routers. ### Worker Registry Sync Each router publishes the workers it registered itself, whether from `--worker-urls`, the worker API, or service discovery, whenever one is added, changes status, or is removed. Peers import them into their own registries: - **Imports are health-checked locally.** A peer rebuilds the worker from its published spec and probes it itself. The owner's health flag sets the import's initial state and afterwards only nudges it: healthy promotes a pending or not-ready import to ready, and unhealthy demotes a ready one. When the owner replaces a worker in place — a new model or spec at the same URL — peers rebuild their imports from the republished spec. - **Workers are matched by URL.** A router that registered a URL itself keeps its own worker and ignores peers' copies. Give a worker the same URL on every router, and never a loopback address: a peer that imports `http://127.0.0.1:8000` routes to its own host. - **Only the owner removes a worker.** Removing a worker on the router that registered it removes it everywhere. Deleting an imported worker through another router's API lasts at most until that router's next reconcile pass (every 30 seconds) imports it again. - **A removed worker is not re-imported from peers.** Once a worker is removed on the router that registered it, that router stops importing the endpoint from the mesh: a peer's copy — a twin registration of the same URL still published by another router, or a lagging peer's stale state — is held back instead of imported again. For a worker registered by service discovery, the hold is tied to the removed instance: a different instance published at the same address (for example, a new pod reusing the IP and port) imports as usual, but a peer copy that names no instance is held back. A removed worker that had no instance identity — one you registered yourself — holds back every peer copy of its URL. Registering a worker at that URL on that router re-admits the endpoint. The first copy held back per endpoint is logged, and the hold lives in memory — up to 4096 removed endpoints, oldest dropped first — so a router restart clears it. - **A departed router's workers are removed.** From the moment the cluster state shows a router `DOWN` or `LEAVING`, peers import nothing new from it, so a router that joins afterwards never picks up its workers. Once the router has been marked gone for 20 seconds, peers also remove the workers they already imported from it at their next reconcile pass (every 30 seconds). A shorter blip removes nothing, and a router that returns has its workers imported again. Workers a peer registered itself — the same URL given to several routers — are its own, not imports, and stay. - **API keys stay local.** The published spec leaves out the worker's `api_key`. !!! note "ZMQ workers stay with their router" Workers reached over ZMQ — `ipc://` and `tcp://` URLs alike — are never published, and peers ignore a ZMQ worker that an older router publishes. An `ipc://` socket is reachable only on the router's own machine; a `tcp://` engine may sit on another host but still belongs to the router that registered it. Each router keeps and routes to its own ZMQ workers; see [ZMQ Workers](../../getting-started/zmq-workers.md). ### Cache-Aware State Sync The `cache_aware` policy keeps an approximate prefix tree per model: a string tree for HTTP requests and a token tree for gRPC requests. With the mesh on, routing decisions made through these trees are shared: 1. **Publish**: After choosing a worker, the router records it in its own tree and queues a delta: a hash of the request's prefix path plus the worker URL. Once per gossip round, each model's queued deltas go to every connected peer as one batch. Picks made by the [KV-pressure fallback](../routing/cache-aware.md#expected-wait-selection) land in the router's own tree but are not broadcast. 2. **Apply**: A peer that already knows that prefix path adds the worker to it in its own tree. 3. **Repair**: A peer that does not know the path, such as a router that just started, asks a random `ALIVE` peer for its whole tree for that model and tree type and replays it. The tree arrives in pages of up to about 2 MiB; a repair that makes no progress for 5 seconds is retried, preferably with another peer, up to 3 times. Unknown prefixes for the same model and tree type that arrive meanwhile are folded into the repair in flight. The default policy, per-model cache-aware policies, and the prefill, decode, and encode policies of disaggregated mode all take part, with no flag beyond `--enable-mesh`. What consistency to expect: - **Approximate.** Deltas and repairs move the trees toward each other, but they are not guaranteed to match: a dropped delta is not resent, and a repair starts only when a later delta names a path the peer does not know. Each router still evicts its own tree on its own schedule (`--eviction-interval`, `--max-tree-size`); evictions are not shared. - **Best effort.** A delta batch that a peer's stream cannot accept is dropped, not resent. The peer catches up through repair the next time it sees a prefix it does not know. - **Survives a router failure.** Routing decisions that a failed router already shared stay in its peers' trees, so traffic that moves to them keeps that cache affinity. Not synchronized: event-driven cache-aware routing (KV events), which each router builds from the workers' KV event streams, and the hash placement index (`--cache-index hash`). ### How State Is Replicated | Key prefix | Channel | Delivery | Carries | |------------|---------|----------|---------| | `worker:` | CRDT | Last-writer-wins by Lamport timestamp, then replica. Resent until each peer acknowledges; relayed through peers | One key per worker, written only by the router that owns it | | `rl:` | CRDT | Epoch-max-wins | Rate-limit counter shards. Registered, but nothing writes to it | | `td:` | Stream, broadcast | Best effort, once per gossip round, to directly connected peers | Cache-tree deltas, one entry per model per round | | `tree:req:`, `tree:page:` | Stream, targeted | Retried after 5 seconds without progress, up to 3 times | Cache-tree repair requests and pages | Worker-state merges order updates with Lamport clocks rather than wall-clock time, so clock skew between routers does not decide which update wins. --- ## Deployment Patterns
### :material-server-network: Static Peers For VMs and bare metal. Give each router a unique name, its own IP as `--mesh-advertise-host`, and another router's address in `--mesh-peer-urls`. - Start routers in any order; the peer is dialed until it joins - Restart an isolated router under a new name [Basic Configuration →](#basic-configuration)
### :material-kubernetes: Kubernetes Router Discovery Routers find each other by pod label, with no bootstrap peer to configure. - Pods join when Ready and are marked `DOWN` when deleted - Restarted pods rejoin under their pod name [Kubernetes Deployment →](#kubernetes-deployment)
### Cluster Size Any number of routers works. There is no leader election or quorum, so odd counts are not required, and each router keeps serving with the state it has. Every pair of routers keeps one sync stream, so N routers hold N×(N−1)/2 streams and each router sends its updates to N−1 peers. --- ## Kubernetes Deployment ### StatefulSet Configuration ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: smg namespace: inference spec: serviceName: smg-mesh replicas: 3 selector: matchLabels: app: smg template: metadata: labels: app: smg annotations: sglang.ai/mesh-port: "39527" spec: serviceAccountName: smg containers: - name: smg image: ghcr.io/smg-project/smg:latest args: - --service-discovery - --service-discovery-namespace=inference - --selector=app=sglang-worker - --service-discovery-port=8000 - --router-selector=app=smg - --enable-mesh - --mesh-server-name=$(POD_NAME) - --mesh-host=0.0.0.0 - --mesh-advertise-host=$(POD_IP) - --mesh-port=39527 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP ports: - containerPort: 30000 name: http - containerPort: 39527 name: mesh ``` Each router discovers the `sglang-worker` pods itself and finds its peers through the `app=smg` label. The stable pod names of a StatefulSet double as mesh names, and the `smg` service account needs the same Pod permissions as [worker discovery](service-discovery.md). !!! tip "Engine images" For all-in-one deployments where each pod runs both gateway and engine, use an engine image tag (for example `ghcr.io/smg-project/smg:1.15.0-vllm-v0.31.0`, following `{smg_version}-{engine}-{engine_version}`). See [Getting Started](../../getting-started/index.md#install) for available tags. Workers are shared by URL, so a co-located engine reaches other routers only through an address other pods can dial: ZMQ workers (`ipc://` and `tcp://`) stay with their router, and a loopback URL names each router's own host. ### Headless Service ```yaml apiVersion: v1 kind: Service metadata: name: smg-mesh namespace: inference spec: clusterIP: None selector: app: smg ports: - port: 39527 name: mesh ``` This is the StatefulSet's governing Service. Router discovery dials Pod IPs, so the mesh does not use these DNS names; expose the HTTP port to clients through a regular Service or load balancer. ### Kubernetes Router Discovery Router discovery watches Pods that match `--router-selector` and writes them straight into the mesh membership table, replacing `--mesh-peer-urls`. It runs as its own task alongside worker discovery. ```bash smg launch --enable-mesh \ --mesh-server-name "$POD_NAME" \ --mesh-advertise-host "$POD_IP" \ --service-discovery \ --service-discovery-namespace inference \ --selector app=sglang-worker \ --router-selector app=smg ``` | Setting | Requirement | |---------|-------------| | `--enable-mesh` | Required. Without it the router logs `Router selector configured but mesh is not enabled` and skips router discovery | | `--service-discovery` or `--discovery-provider kubernetes` | One of the two is required; both select the same Kubernetes provider. Router discovery is configured through service discovery, so `--router-selector` alone does nothing, and the `file`, `slurm`, and `consul` providers do not run it. Selecting the provider also starts worker discovery; set `--selector` for your worker pods | | `--router-selector` | Required. Space-separated `key=value` labels, all of which a Pod must carry. Without a selector, router discovery does not start | | `--service-discovery-namespace` | Namespace watched for router Pods, shared with worker discovery. Unset watches all namespaces | | `--mesh-server-name` | Must equal the pod name, because discovery registers each peer under its pod name | | `sglang.ai/mesh-port` annotation | Mesh port to dial for that Pod. If it is missing or invalid (invalid values log a warning), the discovering router dials its own `--mesh-port` | | RBAC | `get`, `list`, and `watch` on Pods, as for [worker discovery](service-discovery.md) | How Pods map to node status: - **Running and Ready**: `ALIVE` at the Pod IP and mesh port - **Not Ready** (already known and not `DOWN`): `SUSPECTED` - **Deleted or terminating**: `DOWN` !!! tip "Label selectors" Keep `--router-selector` disjoint from `--selector`: a router Pod that matched the worker selector would also be registered as a worker. --- ## Monitoring !!! note "No HTTP API for mesh state" The router has no HTTP endpoint for mesh membership or sync state (the `/ha/*` routes of earlier releases were removed). Use the metrics below, the router logs, and each router's `GET /workers` list. ### Mesh Metrics Mesh metrics are served with the other gateway metrics on the Prometheus port (default `29000`). | Metric | Type | Labels | Description | |--------|------|--------|-------------| | `router_mesh_peer_connections` | Gauge | `peer` | Open sync streams accepted from `peer`: normally `1` while the stream is open and `0` after it closes. A redial that overlaps the old stream's close briefly counts both | | `router_mesh_peer_reconnects_total` | Counter | `peer` | Sync streams from `peer` into this router that ended | | `router_mesh_sync_round_duration_seconds` | Histogram | `peer` | Time to queue one round of updates on the stream to `peer` | Each sync stream is opened by the router whose name sorts first and counted by the router that accepts it, under the name the dialer's first frame carries, so a router reports `router_mesh_peer_connections` only for peers whose names sort before its own, and `router_mesh_sync_round_duration_seconds` only for peers whose names sort after it. Scrape every router and sum: a fully connected mesh of N routers has N×(N−1)/2 streams. After you remove a router for good, its series stays at `0` on the routers that accepted its streams until they restart. ### Alerting Rules ```yaml groups: - name: smg-mesh rules: # A fully connected mesh of N routers has N*(N-1)/2 sync streams. # 3 routers -> 3 streams. Scrape every router. - alert: SMGMeshStreamsMissing expr: (count(router_mesh_peer_connections > 0) or vector(0)) < 3 for: 2m labels: severity: warning annotations: summary: "SMG mesh has fewer sync streams than expected" - alert: SMGMeshStreamDown expr: router_mesh_peer_connections == 0 for: 2m labels: severity: warning annotations: summary: "Mesh sync stream from router {{ $labels.peer }} is down" ``` --- ## Best Practices
### :material-account-multiple-check: No Quorum Needed Any router count works. There is no leader or quorum, so odd counts are not required.
### :material-earth: Availability Zones Spread routers across zones so that a zone outage leaves routers holding the shared cache state, and register each worker on routers in more than one zone to keep it served.
### :material-lock: Protect the Mesh Port Enable [mesh mTLS](#mesh-mtls) so only routers with a certificate from your CA can join, and expose the port only to other routers.
### :material-monitor: Monitoring Alert when `count(router_mesh_peer_connections > 0)` drops below N×(N−1)/2 for N routers.
--- ## Troubleshooting ### Common Issues | Symptom | Cause | Solution | |---------|-------|----------| | Startup fails with `Invalid mesh peer URL` (Python CLI) or `Invalid value for field 'mesh_peer_urls'` (Rust binary) | A peer address is a hostname | Pass `IP:port`; on Kubernetes, use router discovery | | Startup fails with `mesh advertise address cannot be unspecified` | `--mesh-host` is `0.0.0.0` and `--mesh-advertise-host` is unset | Set `--mesh-advertise-host` to the node's IP | | Startup fails with `mesh port cannot be 0` | `--mesh-port 0` | Use a fixed port | | A router stays alone in the mesh, logging `No peer address available to connect` once per round at debug level | No `--mesh-peer-urls` is set and no peer has joined its membership table (for example, router discovery found none) | Set `--mesh-peer-urls` to another router's `IP:port`, or configure router discovery | | A joining router logs `Error connecting to peer init_peer` with growing retry delays | Its `--mesh-peer-urls` peers are not reachable yet. The entries are named `init_peer`, `init_peer_1`, … by position until an answer names them | The dials are retried with backoff and the routers join once a peer is up; if none ever connects, check the addresses, ports, and network path | | A router stays out of the mesh after a restart or partition | Its peers hold it as `DOWN` or `LEAVING` and no longer probe it | Restart it under a new name, or use router discovery; see [Node Status States](#node-status-states) | | Routers appear under both their pod name and another name in the membership log lines (`Member ... is ...`) | `--mesh-server-name` differs from the pod name under router discovery | Set `--mesh-server-name=$(POD_NAME)` | | Log shows `Router selector configured but mesh is not enabled` | `--router-selector` without `--enable-mesh` | Add `--enable-mesh` | | No `Router node discovery enabled` log line | `--router-selector` without Kubernetes worker discovery (the `file`, `slurm`, and `consul` providers do not run router discovery) | Add `--service-discovery` or `--discovery-provider kubernetes` | | A router Pod never joins | Pod not Running and Ready, labels do not match, or RBAC is missing | Check readiness, labels, and the Role | | ZMQ workers are missing on peers | ZMQ workers (`ipc://` and `tcp://`) are never published to the mesh | Expected: each router keeps its own ZMQ workers | ### Debug Logging ```bash RUST_LOG=warn,smg=info,smg_mesh=debug,smg::mesh=debug \ smg launch --enable-mesh ... ``` Gossip, sync streams, and CRDT merges log under `smg_mesh`; the worker and cache-tree sync adapters and router discovery under `smg::mesh`. `RUST_LOG` replaces the filter built from `--log-level`, so keep a base level in it. ### Verify Cluster Health ```bash # Membership: every status change logs one INFO line, # "Member is at version ()" kubectl -n inference logs smg-0 | grep 'Member ' | tail -n 5 # Worker sync: every router lists the workers its peers registered curl -s http://:30000/workers | jq -r '.workers[] | "\(.url) healthy=\(.is_healthy)"' ``` At `--log-level trace`, each router also logs its whole node table once per gossip round as `Round Status:{...}`. ```promql # Sync streams across all routers: N*(N-1)/2 when fully connected count(router_mesh_peer_connections > 0) ``` --- ## What's Next?
### :material-kubernetes: Service Discovery Discover workers in Kubernetes and set up the Role that router discovery shares. [Service Discovery →](service-discovery.md)
### :material-lightning-bolt: Cache-Aware Routing How the prefix trees that the mesh shares drive routing. [Cache-Aware Routing →](../routing/cache-aware.md)
### :material-power: Graceful Shutdown Allow in-flight requests to complete during shutdown. [Graceful Shutdown →](../reliability/graceful-shutdown.md)
### :material-chart-box: Metrics Reference Gateway metrics to scrape alongside the mesh metrics. [Metrics Reference →](../../reference/metrics.md)