# Backend reference

<!-- This file is generated by docs/tools/gen_backend_boilerplate.py from the C++ backend
descriptors. Do not edit the regions between the GENERATED markers by hand; run
the generator instead. Prose outside the markers is preserved. -->

## Backends

<!-- BEGIN GENERATED: backends-overview -->
| Recipe | Name | Selectable backend | Uses ctx_size | Backends |
|--------|------|--------------------|---------------|----------|
| `acestep` | ACE-Step | yes | no | cuda, rocm, vulkan |
| `ds4` | DwarfStar4 | no | yes | rocm |
| `flm` | FastFlowLM NPU | no | yes | npu |
| `gufo` | Gufo | no | yes | rocm |
| `halogen` | Halogen Flash | no | yes | rocm |
| `kokoro` | Kokoro | no | no | cpu, metal |
| `llamacpp` | Llama.cpp GPU | yes | yes | cpu, cuda, halobox, metal, nathanw, rocm, system, vulkan |
| `llamacpp-hrx` | HRX GPU | no | yes | hrx |
| `moonshine` | Moonshine | no | no | cpu |
| `onnxruntime` | ONNX Runtime | no | no | cpu |
| `openmoss` | OpenMOSS TTS | yes | no | cuda, vulkan |
| `rocmfpx` | ROCm FPX | no | yes | rocm |
| `ryzenai-llm` | Ryzen AI LLM | no | yes | npu |
| `sd-cpp` | StableDiffusion.cpp | yes | no | cpu, cuda, metal, rocm, vulkan |
| `thenoise` | TheNoise | yes | no | cuda, rocm |
| `thinksound` | ThinkSound | yes | no | cuda, rocm, vulkan |
| `trellis` | TRELLIS.2 | yes | no | cuda, cuda12, rocm, vulkan |
| `vllm` | vLLM ROCm | yes | yes | rocm |
| `whispercpp` | Whisper.cpp | yes | no | cpu, metal, npu, rocm, vulkan |
<!-- END GENERATED: backends-overview -->

## Support matrix

Each backend has a tier and a format; see [Backend Tiers and Formats](../api/lemonade.md#backend-tiers-and-formats).

<!-- BEGIN GENERATED: backends-matrix -->
| Recipe | Backend | Tier | Format | OS | Device families |
|--------|---------|------|--------|----|-----------------|
| `acestep` | cuda | experimental | native | linux, windows | nvidia_gpu |
| `acestep` | vulkan | experimental | native | linux, windows | amd_gpu; cpu (x86_64); nvidia_gpu |
| `acestep` | rocm | experimental | native | linux, windows | amd_gpu (gfx103X, gfx110X, gfx1150, gfx1151, gfx1152, gfx120X) |
| `ds4` | rocm | guest | container | linux | amd_gpu (gfx1151) |
| `flm` | npu | core | native | linux, windows | amd_npu (XDNA2) |
| `gufo` | rocm | guest | container | linux | amd_gpu (gfx1151) |
| `halogen` | rocm | guest | container | linux | amd_gpu (gfx1151) |
| `kokoro` | metal | core | native | macos | metal |
| `kokoro` | cpu | core | native | linux, windows | cpu (x86_64) |
| `llamacpp` | system | core | native | linux | cpu (arm64, x86_64) |
| `llamacpp` | metal | core | native | macos | metal |
| `llamacpp` | cuda | core | native | linux, windows | nvidia_gpu (sm_100, sm_120, sm_121, sm_75, sm_80, sm_86, sm_89, sm_90) |
| `llamacpp` | vulkan | core | native | linux, windows | amd_gpu; cpu (arm64, x86_64) |
| `llamacpp` | rocm | core | native | linux, windows | amd_gpu (gfx103X, gfx110X, gfx1150, gfx1151, gfx1152, gfx120X, gfx908, gfx90a, gfx942, gfx950) |
| `llamacpp` | cpu | core | native | linux, windows | cpu (arm64, x86_64) |
| `llamacpp` | nathanw | guest | container | linux | amd_gpu (gfx1151) |
| `llamacpp` | halobox | guest | container | linux | amd_gpu (gfx1151) |
| `llamacpp-hrx` | hrx | experimental | native | linux | amd_gpu (gfx1100, gfx1151) |
| `moonshine` | cpu | core | python | windows | cpu (x86_64) |
| `moonshine` | cpu | core | python | linux | cpu (arm64, x86_64) |
| `moonshine` | cpu | core | python | macos | cpu (arm64) |
| `onnxruntime` | cpu | experimental | native | windows | cpu (x86_64) |
| `onnxruntime` | cpu | experimental | native | linux | cpu (arm64, x86_64) |
| `onnxruntime` | cpu | experimental | native | macos | cpu (arm64) |
| `openmoss` | cuda | experimental | native | linux, windows | nvidia_gpu |
| `openmoss` | vulkan | experimental | native | linux, windows | amd_gpu; cpu (x86_64); nvidia_gpu |
| `rocmfpx` | rocm | guest | container | linux | amd_gpu (gfx1151) |
| `ryzenai-llm` | npu | core | native | windows | amd_npu (XDNA2) |
| `sd-cpp` | metal | core | native | macos | metal |
| `sd-cpp` | cuda | core | native | linux, windows | nvidia_gpu (sm_100, sm_120, sm_121, sm_75, sm_80, sm_86, sm_89, sm_90) |
| `sd-cpp` | vulkan | core | native | linux, windows | amd_gpu; cpu (x86_64); nvidia_gpu |
| `sd-cpp` | rocm | core | native | linux, windows | amd_gpu (gfx103X, gfx110X, gfx1150, gfx1151, gfx1152, gfx120X) |
| `sd-cpp` | cpu | core | native | linux, windows | cpu (x86_64) |
| `thenoise` | rocm | core | python | linux, windows | amd_gpu (gfx103X, gfx110X, gfx1150, gfx1151, gfx1152, gfx120X) |
| `thenoise` | cuda | experimental | python | linux | nvidia_gpu |
| `thinksound` | cuda | experimental | native | linux, windows | nvidia_gpu |
| `thinksound` | vulkan | experimental | native | linux, windows | amd_gpu; cpu (x86_64); nvidia_gpu |
| `thinksound` | rocm | experimental | native | linux, windows | amd_gpu (gfx103X, gfx110X, gfx1150, gfx1151, gfx1152, gfx120X) |
| `trellis` | cuda12 | experimental | native | linux, windows | nvidia_gpu (sm_60, sm_61, sm_62, sm_70, sm_72) |
| `trellis` | cuda | experimental | native | linux, windows | nvidia_gpu |
| `trellis` | vulkan | experimental | native | linux, windows | amd_gpu; cpu (x86_64); nvidia_gpu |
| `trellis` | rocm | experimental | native | linux, windows | amd_gpu (gfx103X, gfx110X, gfx1150, gfx1151, gfx1152, gfx120X) |
| `vllm` | rocm | experimental | python | linux | amd_gpu (gfx110X, gfx1150, gfx1151, gfx120X) |
| `whispercpp` | npu | core | native | windows | amd_npu (XDNA2) |
| `whispercpp` | metal | core | native | macos | metal |
| `whispercpp` | vulkan | core | native | linux, windows | amd_gpu; cpu (x86_64) |
| `whispercpp` | rocm | core | native | linux, windows | amd_gpu (gfx110X, gfx1150, gfx1151, gfx120X) |
| `whispercpp` | cpu | core | native | linux, windows | cpu (x86_64) |
<!-- END GENERATED: backends-matrix -->

> **Note:** The `llamacpp` `rocm` row lists `linux, windows` for the family as a
> whole, but MI350X (`gfx950`) is currently gated to **Linux + stable channel
> only** — the Windows TheRock distribution and the ROCm nightly build for
> gfx950 are not yet published, so `gfx950` installs are rejected on Windows and
> on the nightly channel. The OS column reflects the row's overall reach; the
> per-architecture restriction is enforced by the backend's install gate.

## Recipe options

<!-- BEGIN GENERATED: backend-options -->
#### `acestep` — ACE-Step

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `acestep_backend` | `--acestep` | BACKEND | "" | ACE-Step backend to use |

#### `ds4` — DwarfStar4

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `ctx_size` | `--ctx-size` | SIZE | -1 | Context size for the model |
| `ds4_args` | `--ds4-args` | ARGS | "" | Custom arguments to pass to ds4-server |

#### `flm` — FastFlowLM NPU

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `ctx_size` | `--ctx-size` | SIZE | -1 | Context size for the model |
| `flm_args` | `--flm-args` | ARGS | "" | Safe flm serve tuning args: --pmode, --prefill-chunk-len, --img-pre-resize, --socket, --q-len, --preemption |

#### `gufo` — Gufo

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `ctx_size` | `--ctx-size` | SIZE | -1 | Context size for the model |
| `gufo_args` | `--gufo-args` | ARGS | "" | Custom arguments to pass to gufo serve llm |

#### `llamacpp` — Llama.cpp GPU

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `ctx_size` | `--ctx-size` | SIZE | -1 | Context size for the model |
| `llamacpp_backend` | `--llamacpp` | BACKEND | "" | LlamaCpp backend to use |
| `llamacpp_device` | `--llamacpp-device` | DEVICES | "" | Comma-separated list of accelerator devices to use (e.g. Vulkan0) |
| `llamacpp_args` | `--llamacpp-args` | ARGS | "" | Custom arguments to pass to llama-server |

#### `llamacpp-hrx` — HRX GPU

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `ctx_size` | `--ctx-size` | SIZE | -1 | Context size for the model |
| `hrx_args` | `--hrx-args` | ARGS | "" | Custom arguments to pass to the HRX llama-server |

#### `moonshine` — Moonshine

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `moonshine_args` | `--moonshine-args` | ARGS | "" | Custom arguments to pass to moonshine-server |

#### `onnxruntime` — ONNX Runtime

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `onnxruntime_args` | `--onnxruntime-args` | ARGS | "" | Custom arguments to pass to ort-server |

#### `openmoss` — OpenMOSS TTS

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `openmoss_backend` | `--openmoss` | BACKEND | "" | OpenMOSS TTS backend to use |
| `pcm_sample_rate` | — | SIZE | 0 | Sample rate of the PCM this model emits, reported as X-MOSS-Sample-Rate (0 omits the header) |
| `pcm_channels` | — | SIZE | 0 | Channel count of the PCM this model emits, reported as X-MOSS-Channels (0 omits the header) |

#### `rocmfpx` — ROCm FPX

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `ctx_size` | `--ctx-size` | SIZE | -1 | Context size for the model |
| `rocmfpx_args` | `--rocmfpx-args` | ARGS | "" | Custom arguments to pass to the ROCm FPX llama-server |

#### `sd-cpp` — StableDiffusion.cpp

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `sd-cpp_backend` | `--sdcpp` | BACKEND | "" | SD.cpp backend to use |
| `sdcpp_args` | `--sdcpp-args` | ARGS | "" | Custom arguments to pass to sd-server (must not conflict with managed args) |
| `steps` | — | SIZE | 20 | Number of diffusion steps |
| `cfg_scale` | — | SIZE | 7.0 | Classifier-free guidance scale |
| `width` | — | SIZE | 512 | Output image width |
| `height` | — | SIZE | 512 | Output image height |
| `sampling_method` | — | ARGS | "" | Sampling method |
| `flow_shift` | — | SIZE | 0.0 | Flow shift |
| `soft_pixel_cap` | — | SIZE | 0 | Largest output area, in pixels, that images/edits infers from the first reference image when size is omitted (0 = server default) |

#### `thenoise` — TheNoise

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `thenoise_backend` | `--thenoise` | BACKEND | "" | TheNoise backend to use |
| `steps` | — | SIZE | 20 | Number of denoising steps |
| `cfg_scale` | — | SIZE | 7.0 | CFG scale (<= 1.0 disables CFG) |
| `width` | — | SIZE | 512 | Output image width |
| `height` | — | SIZE | 512 | Output image height |
| `sampler` | — | ARGS | "" | Denoising solver (euler \| er_sde) |
| `negative_prompt` | — | ARGS | "" | Negative prompt |
| `qwen_vae_enhance` | — | BOOL | false | Nyquist notch post-filter (removes 2px grid artifacts) |
| `film_grain` | — | SIZE | 0.0 | Film grain strength (0.0-10.0) |
| `sharpening` | — | SIZE | 0.0 | RCAS sharpening strength (0.0-1.0) |
| `lora_specs` | — | ARGS | "" | Comma-separated LoRA specs, e.g. "style:0.8,sub/detail:0.5" |
| `soft_pixel_cap` | — | SIZE | 0 | Largest output area, in pixels, that images/edits infers from the first reference image when size is omitted (0 = server default) |

#### `thinksound` — ThinkSound

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `thinksound_backend` | `--thinksound` | BACKEND | "" | ThinkSound backend to use |

#### `trellis` — TRELLIS.2

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `trellis_backend` | `--trellis` | BACKEND | "" | Trellis backend to use |
| `trellis_args` | `--trellis-args` | ARGS | "" | Custom arguments to pass to trellis-server |

#### `vllm` — vLLM ROCm

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `ctx_size` | `--ctx-size` | SIZE | -1 | Context size for the model |
| `vllm_backend` | `--vllm` | BACKEND | "" | vLLM backend to use |
| `vllm_args` | `--vllm-args` | ARGS | "" | Custom arguments to pass to vllm-server |

#### `whispercpp` — Whisper.cpp

| Option | CLI flag | Type | Default | Description |
|--------|----------|------|---------|-------------|
| `whispercpp_backend` | `--whispercpp` | BACKEND | "" | WhisperCpp backend to use |
| `whispercpp_args` | `--whispercpp-args` | ARGS | "" | Custom arguments to pass to whisper-server |
<!-- END GENERATED: backend-options -->


## Implementation notes

### ACE-Step (`acestep`)

`ace-server` exposes an asynchronous job API: `POST /lm` or `POST /synth` returns a job id immediately, `GET /job?id=N` polls the status, and `GET /job?id=N&result=1` fetches the finished result. `AceStepServer::run_job` wraps this submit/poll/fetch cycle with a ceiling of roughly 20 minutes per stage at a 1-second poll cadence. Synth results arrive as `multipart/mixed` (an audio part plus a latent part); Lemonade extracts the first audio part.

Vocals are a two-stage pipeline. `POST /lm` with `lm_mode: "generate"` runs the ACE-Step language model, which turns the caption and lyrics into audio codes plus LM-filled metadata, returned as a JSON array of enriched requests. That array is accepted by `POST /synth` verbatim, so Lemonade feeds it through unchanged. `POST /synth` on its own is the DiT-only instrumental path — it has no language model and cannot sing, so a `lyrics` value (other than the sentinel below) is what routes a request through `/lm` first.

`[Instrumental]` — any case, surrounding whitespace ignored — is ACE-Step's sentinel for the no-vocals path, matching the Python reference implementation. Instrumental requests send the sentinel explicitly rather than an empty string because the synth stage also feeds the lyrics text into its conditioning.

Errors from `audio_generations` are written into the response sink as a JSON error payload; the endpoint handler turns that into an HTTP error instead of shipping it as audio.

The model download fetches the DiT checkpoint variant plus three companions when present in the repo: the language model (`acestep-5Hz-lm-4B-Q8_0.gguf`, required for vocals and auto-lyrics), the Qwen3 text encoder, and the VAE. The checkpoint path handed to `--models` is the directory of GGUFs; `ace-server` scans it by architecture, and `--keep-loaded` keeps models resident across requests.

### OpenMOSS (`openmoss`)

`moss-tts-server` hosts exactly one `--model` per process, so the voice generator that ships as a component of the speech model cannot be served by the same process. `OpenMossServer` runs the cascade one model at a time: the speech process is stopped, a transient process is spawned on the voice-generator checkpoint to render a single reference sample, and the speech process is brought back. Holding both would require a card that fits the pair, a strictly harder requirement than running the model that was actually asked for. For the same reason, `server_models.json` keeps `size` as the peak resident requirement of the main speech model rather than adding the non-resident VoiceGen download size. Rendered samples are cached per description for the life of the load, so repeating a description costs nothing. The speech process is restarted even when design fails, so an unsuccessful design cannot leave a loaded model with no process behind it.

The legacy `MOSS-VoiceGen` registry entry remains as a compatibility model for existing clients; integrated voice design on `OpenMOSS-TTS` and `MOSS-TTS-Local` does not depend on that standalone entry.

`request_mutex_` serialises a process-changing voice-design request against normal OpenMOSS inference, `load()`, and `unload()`. Speech and SFX hold it shared; voice design and lifecycle changes hold it exclusively. During the intentional speech -> VoiceGenerator -> speech swap, `process_swap_in_progress_` keeps `is_backend_alive()` true so the router does not mistake the temporarily empty speech-process handle for a crash. A request arriving in that window is accepted and then waits on `request_mutex_` until speech is restored. When the speech process fails its readiness wait, `start_server()` stops it, so the voice-design path does not call `unload()`, which would take the same exclusive lock again.

The transient design process is a local `NativeProcess` in `render_reference_sample()` rather than a `start_server()` call, because it does not own `port_`. Its readiness loop polls its own `/health` and checks `designer.running()` each round, so a subprocess that dies during model loading fails the request at once instead of polling to the deadline, and destroying `designer` stops the process and logs its exit code. `server_command()` picks the port with `find_free_port()` instead of `choose_port()` for the same reason: `choose_port()` assigns `port_`, and `start_server()` points `port_` at the speech process only.

Reference-conditioned TTS can exceed OpenMOSS's default 8192-token context. Lemonade intentionally keeps the upstream context default instead of forcing a larger allocation, preserving compatibility with memory-constrained GPUs. OpenMOSS v0.3 chunks prefill batches, but a complete prompt still has to fit `n_ctx`; unusually long references may therefore be rejected by upstream with guidance to shorten the reference or raise the context size.

Lemonade does not inject `max_audio_frames` into OpenMOSS speech requests. OpenMOSS v0.3 owns this policy: its VoiceGenerator derives a text-based duration window when needed, while MOSS-TTS/MOSS-TTSD keep their reference-aware/native stop behavior. Keeping that distinction in the backend avoids truncating conditioned or multi-speaker speech.

Voice design is opt-in through the `voice_design_description` extension and is never inferred from `voice`, which keeps its OpenAI-compatible meaning and is forwarded as an instruction. A client sending `"voice": "default"` gets speech rather than a design run for a voice literally named "default". The field is ignored when the request already carries `reference_wav_b64`. Request fields are read with a type-checking accessor rather than `json::value()`, which throws on a type mismatch instead of falling back to the default and would turn a client's wrong-typed field into a 500.

`MOSS-SoundEffect` uses the same recipe but is an audio-generation model: `audio_generations()` forwards to the backend's `/sfx` endpoint, accepting `duration`/`cfg` as aliases for `seconds`/`cfg_scale`.

### llama.cpp (`llamacpp`)

Lemonade launches `llama-server` with `--parallel 1` and leaves the upstream `--cache-ram` host prompt cache at its default. `LlamaCppServer` does not override `downsize()`: slot `erase` frees no device memory (the KV buffer is allocated once at model load) and discards the slot's KV state without going through the host-cache save path, so soft idle leaves the slot resident and a resumed conversation reuses its cached prefix.

### Backend auto-selection

When a recipe's backend is not pinned in `config.json` (the `backend` key is absent or `"auto"`), the default backend reported in `system-info` — and used by `RecipeOptions` when resolving `*_backend` options — is chosen as follows: the first supported backend in `RECIPE_DEFS` preference order wins, unless a later supported backend is already locally installed (state `installed`, `update_available`, or `update_required`) while the earlier candidates are merely installable. In that case the first installed one wins. This makes explicitly installing a variant (e.g. the Vulkan build of a GPU backend) an effective override: auto-selection uses what is on disk instead of downloading the preference-order winner. An explicit `backend` value in `config.json` always takes precedence over both rules. The llamacpp `system` variant is never auto-selected unless `prefer_system` is set.