# Architecture
> Full technical reference for minimaxmusic.cpp. For a quick start guide, see [README.md](../README.md).
# minimaxmusic.cpp
Portable C++17 implementation of MiniMax Music 3 song generation using GGML.
Structured caption + lyrics in, stereo 44.1kHz MP3 or WAV out. Runs on CPU,
CUDA, Vulkan.
Source of truth: the HF diffusers layout of `MiniMaxAI/MiniMax-Music3`
(`modular_model_index.json`) and the reference implementation merged in
diffusers (PR 14456), expected as a sibling clone at `../diffusers` for the
parity suite. The `qwen_7B/` subfolder of the checkpoint is the fused SGLang
serving model and `dav.pth` / `flowmatching_vae.pth` are training
checkpoints: none of them is used by this port.
## Build
```bash
git submodule update --init
mkdir build && cd build
# macOS (Metal + Accelerate BLAS auto-enabled)
cmake ..
# Linux with NVIDIA GPU
cmake .. -DGGML_CUDA=ON
# Linux with Vulkan
cmake .. -DGGML_VULKAN=ON
cmake --build . --config Release -j$(nproc)
```
### Windows
Install [Visual C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)
(select "Desktop development with C++" workload) and optionally the
[CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) and/or the
[Vulkan SDK](https://vulkan.lunarg.com/sdk/home).
```cmd
git submodule update --init
call "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat"
mkdir build
cd build
rem NVIDIA GPU
cmake .. -DGGML_CUDA=ON
rem AMD/Intel GPU (Vulkan)
cmake .. -DGGML_VULKAN=ON
rem all backends (CUDA + Vulkan + CPU, runtime loading)
cmake .. -DGGML_CPU_ALL_VARIANTS=ON -DGGML_CUDA=ON -DGGML_VULKAN=ON -DGGML_BACKEND_DL=ON
cmake --build . --config Release -j %NUMBER_OF_PROCESSORS%
```
Builds six binaries: `mm-lm` (autoregressive stage CLI), `mm-synth`
(full pipeline CLI), `mm-server` (HTTP server with embedded WebUI),
`neural-codec` (flow VAE latent decoder), `mp3-codec` (MP3
encoder/decoder) and `quantize` (GGUF requantizer).
A single `build/` directory serves every backend combination. With
`buildall`, the backend is picked at runtime: the `GGML_BACKEND`
environment variable forces a specific device (`CUDA0`, `Vulkan0`, `CPU`),
unset picks the best available one.
## Models
Five GGUF files, one per pipeline component, named after the official
checkpoint subfolders:
| GGUF | Component | Native dtype | Size |
|------|-----------|--------------|------|
| MiniMax-Music3-language_model-BF16.gguf | global LM 8B (Qwen3) | BF16 | 17.2 GB |
| MiniMax-Music3-rvq_depth_decoder-BF16.gguf | RVQ depth decoder 0.6B | BF16 | 1.3 GB |
| MiniMax-Music3-condition_encoder-F32.gguf | condition encoder | F32 | 101 MB |
| MiniMax-Music3-transformer-F32.gguf | flow matching DiT 2.4B | F32 | 9.7 GB |
| MiniMax-Music3-vocoder-F32.gguf | flow VAE encoder + decoder | F32 | 306 MB |
The converter always produces the native dtype of the safetensors source,
byte for byte: BF16 tensors pass through untouched, F32 tensors stay F32.
No dtype exists in a GGUF that does not exist in the checkpoint. The only
transformation is the VAE weight norm folding (`w = g * v / ||v||`,
axis 0), which is the inference form of the same weights.
The vocoder GGUF carries a second source: the `encoder.*` tensors come
from the `dav.pth` training checkpoint at the repository root, whose
decoder half is bit-identical to the published `vocoder/` subfolder, so
its encoder is the exact companion of the decoder. Only the encoder and
the posterior mean projection are extracted; the training-side flow and
variance projection stay in `dav.pth`.
Quantized variants are generated from the natives:
| Component | Quants |
|-----------|--------|
| language_model (BF16) | Q5_K_M 6.3 GB, Q6_K 7.0 GB, Q8_0 9.1 GB |
| transformer (F32) | Q4_K_M 1.4 GB, Q5_K_M 1.7 GB, Q6_K 2.0 GB, Q8_0 2.6 GB |
| rvq_depth_decoder (BF16) | Q8_0 0.7 GB |
| condition_encoder, vocoder | never quantized |
The mapping follows the audio model quantization experience of acestep.cpp:
no Q4 on the code-generating LM (audio code LMs degrade below Q5), small
models only get Q8_0, and the bandwidth-bound frontends (condition encoder,
vocoder) stay native. Norm tensors are kept in F32 inside the quants.
The LM GGUF is self-contained: it embeds the Qwen2 BPE tokenizer (151643
base vocab + 32 added tokens) and the model `config.json`, so no external
file is needed at runtime.
Building GGUFs from source (checkpoints + convert)
```bash
pip install hf gguf numpy
hf download MiniMaxAI/MiniMax-Music3 --local-dir checkpoints
./convert.py # convert each missing native GGUF from checkpoints/
./quantize.sh # generate each missing quant from the natives
```
Both scripts take zero arguments and are idempotent: existing outputs are
skipped (`skip X: Y exists` / `[Skip]`), so a partial download or an
interrupted run resumes where it left off. `convert.py` reads the sharded
or single safetensors of each component subfolder and writes to `models/`.
## VRAM and model routing
Module loads go through a `ModelStore` (`src/model-store.h`), the single
owner of the GGML module instances. The pipeline borrows modules through
refcounted RAII handles (`ModelHandle`), stage by stage, and the store
decides what stays in VRAM following its eviction policy:
- `EVICT_STRICT` (default, hardcoded in the CLIs): at most one
coexistence group resident at a time. The AR group `{ LM, depth }` is
interleaved per frame, the synthesis group `{ cond, DiT, VAE }` per
window and per song, so eviction operates on groups: when the
synthesis group is required, the AR group has been released and is
unloaded. The LM weights and the DiT weights never coexist, peak VRAM
is the AR group.
- `EVICT_NEVER` (`mm-server --keep-loaded`): nothing is ever evicted,
modules accumulate. Swapping a quant under this policy keeps both
instances resident; the user opting in declares the budget for it.
A require of an already resident key is a cache hit on the same
instance. A conflicting require while a module of another group is still
held aborts: the strict invariant is enforced, not documented.
| Combo | LM | Depth | DiT | STRICT peak | Resident (`--keep-loaded`) |
|-------|----|-------|-----|-------------|----------------------------|
| Full native | BF16 | BF16 | F32 | ~19 GB | ~29 GB |
| Q8_0 | Q8_0 | Q8_0 | Q8_0 | ~10 GB | ~13 GB |
| Light | Q5_K_M | Q8_0 | Q4_K_M | ~7 GB | ~9 GB |
`pipeline_configure()` records the resolved GGUF path of each component;
the loads happen inside `pipeline_generate()` at stage boundaries,
through the store. Under `--keep-loaded`, switching the DiT quant in the
UI loads the new DiT and hits the cache for the other four modules; under
STRICT every job reloads each stage as it reaches it. A failed load
surfaces as a failed job and the next job retries it.
Model routing is a property of the request, not of the command line. The
`MM3Request` carries one optional GGUF filename per component (`lm_model`,
`depth_model`, `cond_model`, `dit_model`, `vae_model`), resolved against
the registry scanned from `--models ` at startup. The registry
classifies each GGUF by its embedded `general.architecture` metadata into
the five buckets, so file names are free. An empty field keeps the
previously requested model, or falls to the first bucket entry
(alphabetical: the native variant with the official names) on the first
job, so a request without model fields never forces a swap. Unknown
names get a 400 from the server and a FATAL from the CLI.
## Pipeline
```
caption + lyrics (ChatML, Qwen2 BPE, 200k music vocab)
v
global LM 8B (Qwen3 dense) semantic codebook 16384, 25 Hz
v hidden states
RVQ depth decoder 0.6B 7 acoustic codebooks x 1024 per frame
v fused hidden states
condition encoder 8 state mix, 4096 -> 2048, 25 -> 86.13 Hz
v condition track
flow matching DiT 2.4B latent 128 ch at 86.13 Hz
v
flow VAE decoder 123M 2 x 64 ch tracks -> 44.1 kHz stereo
```
Frame rates: LM 24000 / 960 = 25 Hz, VAE latent 44100 / 512 = 86.13 Hz.
## Components
### Global LM (`language_model/`, Qwen3ForCausalLM)
Stock Qwen3 dense: 36 layers, hidden 4096, GQA 32/8, head_dim 128, SwiGLU
12288, RMSNorm eps 1e-6, q/k norm, RoPE theta 1e6, ctx 10240, untied
lm_head, vocab 200000 (text + semantic audio tokens). Predicts the first
RVQ codebook frame by frame at 25 Hz.
### RVQ depth decoder (`rvq_depth_decoder/`)
Intra frame causal transformer over codebook positions, run once per
25 Hz frame:
```
projection [4096, 4096] global LM hidden -> position 0
pos_embedding [16, 4096] learned, added after projection
audio_embeddings [7168, 4096] 7 codebooks x 1024 entries
layers.{0..3} to_{q,k,v,out} 4096, 16 heads x 256,
SwiGLU gate/up/down 6144, RMSNorm pair eps 1e-6
norm [4096]
audio_heads.{0..6} [1024, 4096]
```
Causal attention without RoPE, scale 1/16. A sequence of length S predicts
codebook S-1 through `audio_heads[S-2]`.
### Condition encoder (`condition_encoder/`)
```
layer_weight_logits [8] softmax mix over 8 hidden states
layer_scale [1]
proj conv1d 4096 -> 2048, k=3, pad 1
```
The 8 fused states per frame are the global LM last hidden state plus the
conditional hidden state of each of the 7 depth decoder steps, concatenated
layer-major. `softmax(layer_weight_logits) * layer_scale` is constant at
inference and folds into 8 fixed mix weights at load time. After the
projection, the track is resampled 25 Hz -> 86.13 Hz by nearest neighbor
(torch convention: `src = floor(i * T_in / T_out)`), with
`latent_length = int(n_frames * 44100 / 24000 * 960 / 512)`.
### Flow matching DiT (`transformer/`, MiniMaxMusic3Transformer1DModel)
36 self attention blocks, model dim 2048 (32 heads x 64), no cross
attention, no adaLN. Conditioning enters by channel concat: the input is
`[latent 128, zeros 128, condition 2048]` = 2304 channels through
`preprocess_conv` (k=1, residual: `conv(x) + x`), and `postprocess_conv`
(also residual) maps back to 128 latent channels.
The timestep is a prefix token: Fourier features `2*pi*t*w` ->
`cat(cos, sin)` 256 -> linear -> SiLU -> linear -> 2048, prepended to the
sequence and removed after the blocks. RoPE is NeoX-style partial
(32 of 64 dims per head, theta 10000) with positions that include the
timestep token. Attention is bidirectional, scale 1/8. The feed forward is
a chunked SwiGLU: `ff_in` 2048 -> 16384 splits into (value, gate) and
`out = ff_out(value * silu(gate))`. LayerNorm with bias, eps 1e-5.
The zeros channel slot is structurally a context latent input (the
architecture reserves `2 * in_channels + cond_dim` concat channels), but
the published model hardcodes it to zeros: no audio input, repaint or
continuation is exposed.
```
preprocess_conv, proj_in, time_proj, time_embed.linear_{1,2}
transformer_blocks.N.{norm1, attn.to_{q,k,v,out}, norm2, ff_in, ff_out}
proj_out, postprocess_conv
```
### Flow VAE (`vocoder/` + `dav.pth`, MiniMaxMusic3Vocoder)
DAC style stack. All convs are weight normalized in the checkpoint
(`weight_g` / `weight_v`), folded at conversion. The snake activation
uses the alpha parameter directly:
`y = x + sin^2(a * x) / (a + 1e-9)`.
The encoder (`src/vae-enc.h`, used by `neural-codec --encode`) is the
mirror: per side, conv_in (1 -> 64, k=7), 4 blocks of 3 res_units
(dilations 1, 3, 9) then snake and a strided conv (strides 2/4/8/8,
k = 2s, pad s/2), snake_out, conv_out (1024 -> 1024, k=3), and the
posterior mean projection (1024 -> 64, k=1). Downsample 512x, the exact
inverse of the decoder hop.
```
dec_in_proj conv1d 64 -> 1024, k=1 (not weight normalized)
conv_in conv1d 1024 -> 1536, k=7
blocks.{0..3} snake1 -> conv_t1 (stride s, k = 2s, pad s/2)
-> 3 x res_unit (dilations 1, 3, 9)
strides [8, 8, 4, 2], dims 1536 -> 768 -> 384 -> 192 -> 96
res_unit skip -> snake1 -> conv1 k=7 -> snake2 -> conv2 k=1 -> + skip
snake_out, conv_out conv1d 96 -> 1, k=7 (with bias) -> tanh
```
The 128 latent channels are two independent 64 channel tracks, channels
0..63 for the left side. The decoder runs once per side and emits mono at
x512, 44.1 kHz.
GGML lowering: transposed convs as GEMM + `col2im_1d`, snake activations
fused by graph pattern recognition (same custom ops as acestep.cpp and
pocket-tts, see [Patched GGML fork](#patched-ggml-fork)). Conv weights are
stored F16 on device with F32 activations.
## Inference recipe
Pinned against the diffusers modular pipeline (`../diffusers`,
`src/diffusers/{models,modular_pipelines,schedulers}/*minimax_music3*`).
### Prompt assembly
```
ids = [151644 im_start, 151671 caption_start]
+ bpe(clean_caption)
+ [151672 caption_end, 151673 lyrics_start]
+ bpe(normalize_lyrics)
+ [151674 lyrics_end, 151645 im_end, 151669 audio_start]
```
The unconditional CFG stream is the same sequence with indices [1, n-3]
replaced by the audio CFG token 151654: only `im_start` and the trailing
`[lyrics_end, im_end, audio_start]` survive.
`_clean_caption` strips markdown and rewrites `<|x y|>` tags as "x is y";
any `Global Metadata` header text is part of the trained prompt format and
passes through. `_normalize_lyrics` keeps leading `[tag]` markers alone on
their line, lowercases the tag interior, and prepends `[start]\n`; lines in
parentheses pass through verbatim (the official demo prompts mix both
conventions freely). Exact ports live in `src/prompt.h`. BPE encoding adds
no EOS.
### Autoregressive stage
Semantic tokens occupy LM vocab ids [151675, 151675 + 16384), the end of
audio token is 151670. The LM runs the conditional and unconditional
streams as one batch of 2 over two KV cache sets. The first decode step
only advances past `audio_start` and emits no frame. Budget: 9000 frames
(6 minutes), also the cap applied to `duration`.
Semantic sampling per frame: logits masked to the semantic range plus the
end token, CFG on logits `guided = uncond + (cond - uncond) * lm_cfg`
restricted to the conditional branch's top `lm_top_k`, then softmax
sampled (mt19937 seeded by `lm_seed`).
Depth decoding per frame is a single fused graph: the 7 acoustic codebook
steps run as batch 2 (cond, uncond) with sampling inside the graph
(top-k by `argsort`, guided softmax, cumsum, CDF crossing against host
pre-drawn uniforms, `get_rows` on the device embedding table feeds the
next step). The sequence grows as
`[proj(LM last hidden), proj(LM_embed(semantic + 151675)),
proj(audio_emb(c_i + (i-1) * 1024)), ...]` with the learned positional
embedding added after projection. The conditional hidden state of each
step is collected before sampling; those 7 states plus the LM last hidden
are the 8 states fused by the condition encoder.
LM frame feedback: the next LM input embedding is
`(LM_embed(semantic + 151675) + sum of the 7 audio embeddings) * 8^-0.5`,
injected through the input_embeds path of the batched forward.
`--no-batch-cfg` splits both stages back into two separate forwards
(reference sequential path; outputs differ from the batched path at
epsilon level, which the sampling then amplifies, as with acestep).
### Denoising stage
The condition track is processed in windows of 200 LM frames with hop 100.
Per window, the DiT runs `steps` Euler iterations at T = latent window
length (689 for a full window):
- Sigmas: `1 - linspace(1, 1/N, N)`, ascending, with a final 1.0 appended
(scheduler config: shift 1.0, `invert_sigmas`, num_train_timesteps 1).
The DiT consumes the current sigma directly as flow time (0 = noise).
Euler update: `x += (sigma_next - sigma) * v`.
- CFG on the velocity: `v = v_uncond + (v_cond - v_uncond) * dit_cfg`,
the unconditional branch conditions on zeros. Both branches run as one
batch of 2 under batch CFG.
- Overlap blending, applied at every step on the first 172 latent frames:
`lat[:172] = (1 - (1 - 1e-6) * t) * noise_prompt + t * prev_latent`,
restored after the step. The carry for the next window is latent range
[L - 344, L - 172).
- Initial noise: `philox_normal4` seeded by `seed`, drawn as one
continuous stream across windows.
### Decode and stitch
The VAE decodes each window; the first window keeps its left edge, later
windows crop 86 latents (x512 samples) on the left, and all but the last
crop 258 on the right. Segments are stitched and clamped.
### Post-processing
The pipeline outputs planar stereo float `[L: T][R: T]` at 44100 Hz, full
range. Normalization and encoding belong to the output stage (server
worker or CLI): percentile peak normalization targeting the
`1 - peak_clip / 1e6` percentile (`peak_clip = 0` is plain peak
normalization; WAV32 skips normalization entirely and writes raw IEEE
float), then MP3 (bitrate `mp3_bitrate`) or WAV 16/24/32 encoding in
memory.
## Request JSON reference
Every field has a default. Omitting a field is strictly equivalent to
sending it with its default value. Only `caption` and `lyrics` are
required: the server rejects requests missing either.
```json
{
"caption": "",
"lyrics": "",
"duration": 60.0,
"steps": 30,
"seed": -1,
"lm_seed": -1,
"lm_cfg": 1.5,
"lm_top_k": 50,
"lm_batch_size": 1,
"synth_batch_size": 1,
"dit_cfg": 1.7,
"peak_clip": 10,
"output_format": "mp3",
"mp3_bitrate": 128,
"lm_model": "",
"depth_model": "",
"cond_model": "",
"dit_model": "",
"vae_model": ""
}
```
The defaults mirror the diffusers pipeline: `lm_cfg`, `lm_top_k` and
`dit_cfg` are hardcoded constants in the reference and exposed as
parameters here with the reference values as defaults.
**`caption`** (string, required)
Music description fed to the global LM. The trained prompt format is the
Structured Caption of the official demos (`Global Metadata` header, bpm,
key, scale, per-section emotional progression); a plain natural language
description also works. See `tools/webui/example/` for the 61 official
demo prompts.
**`lyrics`** (string, required)
Song lyrics with structural tags. `[tag]` lines are normalized (kept alone
on their line, lowercased); parenthesized lines pass through verbatim
(section titles or backing vocals, interpreted in context by the model).
Instrumental tracks still carry lyrics in the official demos.
**`duration`** (float seconds, default `60.0`)
Target audio duration, capped by the 9000 frame LM budget (6 minutes).
The LM can end the song earlier with the end of audio token.
**`steps`** (int, default `30`)
Euler steps per DiT window. Minimum 2.
**`seed`** (int64, default `-1` = random)
DiT noise seed (Philox, low 32 bits consumed). Same seed, same noise.
**`lm_seed`** (int64, default `-1` = random)
Autoregressive sampling seed (mt19937, low 32 bits consumed). The song
structure, melody and length come from this one.
**`lm_cfg`** (float, default `1.5`)
CFG scale applied on LM and depth decoder logits.
**`lm_top_k`** (int, default `50`)
Top-K restriction, ranked on the conditional branch before guidance.
**`lm_batch_size`** (int, default `1`)
Number of songs generated from the prompt in one batched autoregressive
pass. Song i samples with its own stream seeded `lm_seed + i`
(consecutive internal seeds), so a song is bit-identical whether
generated alone or in a batch. Limited by the server's `--max-batch`.
**`synth_batch_size`** (int, default `1`)
Number of flow matching variations per song, batched in the DiT on the
shared condition track with consecutive noise seeds (`seed + j`).
Between 1 and 9. The job returns `lm_batch_size * synth_batch_size`
tracks in song-major order.
**`audio_codes`** (string, default `""`)
Explicit code stream, flat comma separated, 8 values per frame (the
semantic code then the 7 acoustic codebooks). Non-empty replaces the
autoregressive sampling: the hidden states are re-derived teacher-forced
from the codes (about an order of magnitude faster than sampling) and
the song renders deterministically, so the synthesis side (models,
steps, seed, CFG) can be iterated without re-rolling the LM. Produced by
`mm-lm`, written by `mm-synth` next to every rendered track, and
returned by the server as the JSON part paired with each audio track;
`lm_batch_size` is ignored when codes are present.
**`dit_cfg`** (float, default `1.7`)
CFG scale on the DiT velocity field.
**`peak_clip`** (int, default `10`)
Output normalization percentile control: the normalization peak is the
`1 - peak_clip / 1e6` percentile of the absolute signal. `0` normalizes to
the true peak with no clipping. Ignored by `wav32`.
**`output_format`** (string, default `"mp3"`)
Audio encoder: `"mp3"`, `"wav16"`, `"wav24"`, `"wav32"`. `wav32` writes
raw IEEE float without normalization.
**`mp3_bitrate`** (int, default `128`)
MP3 encoder bitrate in kbps. WAV outputs ignore it.
**`lm_model`**, **`depth_model`**, **`cond_model`**, **`dit_model`**,
**`vae_model`** (string, default `""`)
GGUF filename per component, resolved against the `--models` registry.
Empty keeps the previously requested model, or falls to the first
registry entry. Unknown names get a 400 from the server, a FATAL from the CLI. See
[VRAM and model routing](#vram-and-model-routing).
## mm-lm reference
Runs the autoregressive stage alone (global LM + depth decoder) and
writes one replayable request JSON per song, the sampled code stream in
`audio_codes`. The expensive stochastic stage runs once; feeding the
output back to `mm-synth --request` or `POST /synth` re-renders the same
song with any synthesis parameters.
```
Usage: ./mm-lm --models --request [options]
./mm-lm --models --caption --lyrics [options]
Required:
--models Directory of GGUF model files
--request Input request JSON (carries model routing)
Optional:
--caption Caption (instead of --request)
--lyrics Lyrics (instead of --request)
--out Output request JSON (default: request.json)
--duration Target duration in seconds
--lm-seed Autoregressive sampling seed
Output is numbered for batches: request.json -> request0.json ...
Debug:
--max-seq LM KV cache size (default: model context)
--no-fa Disable flash attention
--no-batch-cfg Split CFG into two separate forwards (LM + DiT)
--clamp-fp16 Clamp hidden states to FP16 range
--dump-tokens Dump prompt token IDs (CSV)
```
Only the LM and the depth decoder load (about 18.5 GB native, 7 GB
quantized). The written requests carry the input request fields plus the
song's codes and its traceable seed (`lm_seed + i`).
The replay re-derives the hiddens without sampling: the LM runs the
whole feedback sequence as one forward (conditional stream only) and the
depth decoder runs one causal S=8 forward per frame, so the CFG batch
shape differs from generation and the replayed hiddens sit at the usual
activation epsilon from the sampled run (measured rel rms ~2e-3, final
audio cosine ~0.99999: the same song).
## mm-synth reference
```
Usage: ./mm-synth --models --request [options]
./mm-synth --models --caption --lyrics [options]
Required:
--models Directory of GGUF model files
--request Input request JSON (carries model routing)
Optional:
--caption Caption (instead of --request)
--lyrics Lyrics (instead of --request)
--out Output audio path (default: out.mp3)
--duration Target duration in seconds
--steps Euler steps per DiT window
--seed DiT noise seed
--lm-seed Autoregressive sampling seed
Debug:
--max-seq LM KV cache size (default: model context)
--no-fa Disable flash attention
--no-batch-cfg Split CFG into two separate forwards
--clamp-fp16 Clamp hidden states to FP16 range
--dump Dump intermediate tensors
```
`--request` takes the same MM3Request JSON as the server; CLI flags
override nothing, they are the lightweight alternative for one-shot runs.
The output container follows `output_format` in the request (`--out` names
the file). A default run and the server produce bit-identical audio for
the same resolved request.
Every rendered track gets its replay request written next to it, the
audio extension swapped to `.json`: the base request with `audio_codes`
and the exact seed of the track. Feeding it back re-renders the track
deterministically. Batches number the base with song then variation
index (`song.mp3` -> `song00.mp3` + `song00.json` ...).
`--dump ` writes the intermediate tensors consumed by the cosine
similarity harness: per-frame fused hidden states, per-window condition
and latent tracks, per-step DiT velocities and states of the first window,
and the decoded audio, as flat binary f32 with a ndims + shape header.
The `quantize` tool regenerates any GGUF at another type:
```
./build/quantize
```
`quantize.sh` drives it with the validated component mapping.
## mm-server reference
HTTP server exposing the pipeline behind an asynchronous job queue, with
the WebUI embedded (gzipped single page app, served at `/`).
```
Usage: ./mm-server --models [options]
Required:
--models Directory of GGUF model files
Server:
--host Listen address (default: 127.0.0.1)
--port Listen port (default: 8086)
--max-batch LM batch limit (default: 1)
--max-seq LM KV cache size (default: model context)
--keep-loaded Keep every model resident in VRAM (default: evict between stages)
Debug:
--no-fa Disable flash attention
--no-batch-cfg Split CFG into two separate forwards (LM + DiT)
--clamp-fp16 Clamp hidden states to FP16 range
--dump Dump intermediate tensors
```
`--max-batch` sizes the LM KV cache at load time (2N sets of about
1.5 GB each) and bounds `lm_batch_size`; requests above the limit get
a 400.
The debug flags are global to the process and applied to the components as
they load (the graph caches bake them in), so they are boot options, not
request fields.
Models are loaded lazily on the first job: startup touches no GPU. A load
failure is treated as permanent and fails subsequent jobs fast.
### Endpoints
```
POST /synth Submit a generation job, returns job ID
body: application/json MM3Request (the Content-Type header is
required, urlencoded bodies are capped at 8 KB by the HTTP server)
response: {"id":"1a2b..."}
400 on malformed JSON, missing caption or lyrics, duration <= 0,
steps < 2, prompt over the 5000 token budget, invalid output_format,
unknown model name
GET /job?id=N Poll job status
response: {"status":"queued|running|done|failed|cancelled"}
GET /job?id=N&result=1 Fetch job result
multipart/mixed, boundary mm3-batch-boundary: one application/json
replay request part (the request with audio_codes and the exact seed
of the track) then one audio/mpeg or audio/wav part per track
(output_format of the request), song-major order
404 while the result is not ready
POST /job?id=N&cancel=1 Cancel a specific job
response: {"status":"cancelled"}
GET /health Server health check
response: {"status":"ok"}
GET /props Version, models, default request
response: application/json
GET /logs SSE stream of server stderr
response: text/event-stream
GET / Embedded WebUI (gzipped HTML)
```
Request bodies are limited to 8 MB (caption + lyrics JSON only). Error
responses are JSON: `{"error":"message"}`.
**GET /props** returns the model buckets and the full default request
(source of truth for the WebUI selects and placeholders):
```json
{
"version": "...",
"models": {
"lm": ["MiniMax-Music3-language_model-BF16.gguf", "..."],
"depth": ["..."],
"cond": ["..."],
"dit": ["..."],
"vae": ["..."]
},
"defaults": { "caption": "", "duration": 60.0, ... }
}
```
### Concurrency
One worker thread consumes jobs from a FIFO queue and runs them serially
through the shared pipeline. The HTTP handler enqueues and returns a job
id immediately; the client polls. Completed jobs sit in memory and are
evicted FIFO past 32 entries, so a disconnected client can still fetch its
result after reconnecting.
Each job has a cancel flag polled by the pipeline between AR frames,
between DiT steps and between VAE windows, and passed down to the MP3
encoder. Shutdown (SIGINT/SIGTERM) cancels the active job through the same
flag: Ctrl+C lands in about 100 ms even mid-generation.
## neural-codec reference
GGML-native codec for the flow VAE latent space. The decoder weights come
from the published `vocoder/`; the encoder comes from the `dav.pth`
training checkpoint, whose decoder half is bit-identical to the published
vocoder, so the encoder is the exact companion of the decoder. Both live
in the vocoder GGUF (`encoder.*` tensors). The encode is deterministic:
the posterior mean, no sampling and no flow.
```
Usage: ./neural-codec --vae --encode|--decode -i [-o