# switchyard-server

`switchyard-server` exposes libsy algorithms through OpenAI Chat Completions, OpenAI Responses,
and Anthropic Messages endpoints. A TOML file explicitly defines the LLM clients, targets, and
algorithm routes served by the process.

```toml
# routes.toml
schema_version = 1

[llm_clients.example]
format = "openai_chat"
base_url = "https://example.com/v1"
api_key_env = "API_KEY"
max_retries = 2

[targets.model_a]
id = "model/a"
llm_client = "example"
extra_body = { service_tier = "priority" }

[targets.model_b]
id = "model/b"
llm_client = "example"

[routes.general]
id = "switchyard/general"
type = "random"
targets = ["model_a", "model_b"]
weights = [1, 3]
seed = 42

[routes.classified]
id = "switchyard/classified"
type = "llm_classifier"
mode = "capability"
classifier_target = "model_a"
strong_target = "model_a"
weak_target = "model_b"
base_threshold = 0.5

[routes.passthrough]
id = "switchyard/passthrough"
type = "passthrough"
target = "model_a"

[routes.stage]
id = "switchyard/stage"
type = "stage_router"
capable_target = "model_a"
efficient_target = "model_b"
picker = "efficient_first"
confidence_threshold = 0.5
```

```bash
export API_KEY="..."
cargo install --locked switchyard-server
switchyard-server --config routes.toml
```

Ctrl+C and Unix `SIGTERM` stop new connections and allow active requests to drain for up to
`--shutdown-timeout` (30 seconds by default) before they are terminated.

The server logs exactly one structured terminal event per LLM request: successful responses at
`INFO`, 4xx responses at `WARN`, and 5xx responses at `ERROR`. Set
`RUST_LOG=switchyard_server=debug,libsy=debug` to include routing decisions and nested failure
details. A streaming failure is logged separately because it can occur after the response starts.

Target and route table names are local references. A target's `id` is the exact model ID sent
upstream, and a route's `id` is the model clients send to select that algorithm.

Each target references an entry under `llm_clients`. All configured clients use
`TranslatingLlmClient`; supported formats are `openai_chat`, `openai_responses`, and
`anthropic_messages`. Supported algorithms are `noop`, `random`, `passthrough`,
`llm_classifier`, and `stage_router`. An `api_key_env` value names an environment variable; the TOML
never contains the secret itself. If omitted, the client sends no authentication.
Target-level `extra_body` values are shallow-merged into the upstream request when
the request does not already contain that key.
`max_retries` defaults to `2` and applies to transport failures, timeouts, HTTP 408/429, and 5xx
responses.

Random-route `weights` are relative, follow target order, and do not need to sum to one. Omit them
for equal weighting. The optional `seed` reproduces the selection sequence for the same call order.

## Session routing log

Pass `--routing-log-file PATH` to append one JSON record after each completed routed response.
Streaming responses are recorded after the stream drains. When enabled,
`GET /v1/routing/session-stats?session_id=ID` rescans the durable log and returns call and token
totals for that exact `proxy_x_session_id`, grouped by served model. The endpoint returns `404` when
the session has no records and is not registered when routing logging is disabled.

An `llm_classifier` route sends each task to `classifier_target` for a capability verdict, then
routes to `weak_target` or `strong_target`. Beyond the three targets it accepts these keys; only
`base_threshold` is required, and anything the judge cannot decide routes to `strong_target`:

| Key | Default | Meaning |
|---|---|---|
| `base_threshold` | *required* | Lowest solve probability that routes a task to `weak_target`. Raise it to send less traffic to the weak model. |
| `threshold_step` | `0.0` | Finite, non-negative amount added once for uncertain or unmatched verdicts and twice for unsupported verdicts. `base_threshold + 2 * threshold_step` must be at most `1`. |
| `session_affinity` | `false` | Reuses a session's first routing decision on later turns, so the judge is called once per session rather than once per turn. |
| `message_hash_fallback` | `false` | Extends affinity to clients that send no session header, keying on the first user message. Requires `session_affinity = true`. |

Session affinity retains a decision for the process lifetime, including a `strong_target`
fallback produced while the judge was unreachable. `message_hash_fallback` keys on request
content rather than a session id, so unrelated callers sending identical text share one
assignment.

A `stage_router` route scores tool-result and agent-progress signals from recent turns to pick a
tier per turn, without an extra classifier call on every turn. `capable_target`,
`efficient_target`, `picker` (`efficient_first` or `capable_first`), and `confidence_threshold`
are required. Optional handoff notes, per-tier system prompts, and a capability-judge fallback are
documented in [Stage-Router Routing](../../docs/routing_algorithms/stage_router_routing.md).

## Endpoints

| Method | Path | Purpose |
|---|---|---|
| `POST` | `/v1/chat/completions` | OpenAI Chat Completions |
| `POST` | `/v1/messages` | Anthropic Messages |
| `POST` | `/v1/responses` | OpenAI Responses |
| `POST` | `/v1/messages/count_tokens` | Token count from a route's Anthropic target |
| `GET` | `/v1/models` | Routes served by this deployment |
| `GET` | `/v1/stats` | Per-model usage plus curated algorithm stats |
| `POST` | `/v1/stats/reset` | Clear accumulated stats |
| `GET` | `/metrics` | Prometheus text, see [Metrics](#metrics) |
| `GET` | `/health` | Liveness |

Requests name a route by its `id`, so `POST /v1/chat/completions` with `"model": "switchyard/general"`
routes through the `[routes.general]` entry above. Any of the three request formats can address any
route, and the server translates between them.

For `stage_router`, `algorithm_stats.stage_router` groups routing decisions by source and semantic
target and summarizes its score, confidence, and input-dimension histograms. These values reset
with `/v1/stats/reset`; the process-lifetime counters on `/metrics` remain cumulative.

Token counting selects an Anthropic-format completion target, preferring target names or model IDs
containing `opus`, `sonnet`, then `haiku`. Other ties preserve the route's target order.

## Metrics

`GET /metrics` exposes Prometheus text from the server's process-wide OpenTelemetry provider.
Routed-call compatibility metrics are:

| Metric | Type | Labels | Meaning |
|---|---|---|---|
| `switchyard_build_info` | gauge | `version` | Constant `1` for this server version |
| `switchyard_total_requests` | gauge | none | Successful and failed final routed calls |
| `switchyard_total_errors` | gauge | none | Failed final routed calls |
| `switchyard_requests_total` | counter | `model` | Successful final routed calls |
| `switchyard_errors_total` | counter | `model` | Failed final routed calls |
| `switchyard_model_call_latency_ms` | histogram | `model` | Successful final routed-call latency |
| `switchyard_prompt_tokens_total` | counter | `model` | Input tokens, including cached and cache-creation tokens |
| `switchyard_completion_tokens_total` | counter | `model` | Output tokens |
| `switchyard_cached_tokens_total` | counter | `model` | Cached input tokens |
| `switchyard_cache_creation_tokens_total` | counter | `model` | Cache-creation input tokens |
| `switchyard_reasoning_tokens_total` | counter | `model` | Reasoning output tokens |
| `switchyard_total_latency_ms` | histogram | `model` | Full-turn latency for successful routed responses |
| `switchyard_routing_overhead_ms` | histogram | `algorithm` | Algorithm run time minus the call that served it |
| `switchyard_classifier_fail_open_total` | counter | `judge_model`, `reason` | Judge failures that made a classifier route without a verdict |
| `switchyard_client_responses_total` | counter | `outcome` | Final LLM-route responses |
| `switchyard_upstream_attempts_total` | counter | `outcome`, `code` | Actual upstream HTTP attempts |
| `switchyard_router_retry_recovered_total` | counter | none | Retry recoveries (currently always zero) |

`switchyard_classifier_fail_open_total` counts requests that still reached a target after the
judge call failed. `judge_model` names the configured judge target, and `reason` is one of eight
fixed error categories.

`switchyard_total_latency_ms` observes an aggregate when it becomes available or a stream when it
ends cleanly. Its clock starts in a router-wide middleware, before the request body is read and
decoded, so it measures request ingress through response completion. It still excludes connection
accept and TLS handshake, which hyper completes before the server sees the request.

`switchyard_routing_overhead_ms` is what routing cost on top of the model call: the algorithm's run
time minus the call that served the request. Classifier calls are not subtracted, so an
LLM-classifier route reports its classification time here while `passthrough` and `random` report
the sub-millisecond cost of picking a target. It carries only `algorithm`, since the number
describes the router and not the target it chose, and a run that served nothing records nothing. Its
buckets start at 0.1 ms via a view in the server; the SDK defaults start at 5 ms.

Both clocks stop when the routed call resolves, which for a streamed response is when the stream
handle arrives rather than when the stream ends, so SSE relay time is in neither term. The Python
summary of the same name measures its total through stream completion, making its streaming values
mostly generation time.

See [CONFIGURATION.md](CONFIGURATION.md) to add an LLM client, target, or algorithm.