package autogen // Launch-command rendering: turns a sized profile (ctx, offload split, KV // types) into the llama-server / sd-server / tts-server argv lines that land in // the generated config. RenderSoloCmd is the same path with a ${PORT} // placeholder, used by the UI's cogwheel preview and ad-hoc commands. import ( "fmt" "strings" ) // isVLModel reports whether a model is a dedicated vision-language model (its // core identity is image chat: Qwen2/3-VL, InternVL, CogVLM), as opposed to a // text LLM that merely ships an mmproj sidecar for optional vision. Only the // former gets its vision twin's ctx capped to VisionCtx. VL archs carry "vl" in // their gguf arch (qwen2vl/qwen3vl/qwen3vlmoe) and such models are conventionally // named "*-VL"; a plain LLM-with-vision (gemma3, mllama, mistral) matches neither. func isVLModel(name, arch string) bool { return strings.Contains(strings.ToLower(name), "vl") || strings.Contains(strings.ToLower(arch), "vl") } // cmdPath renders a filesystem path as ONE cmd argument. NOT `%q`: the emitted // cmd is re-split by shlex (Windows rules on Windows), where a backslash inside // double quotes is a literal separator — so Go's escaping turned D:\Models\x // into D:\\Models\\x and llama-server failed to open it. Separators are // normalized to "/" (accepted by Windows too, and what slotKvPath already does), // and quotes are kept for paths with spaces. func cmdPath(p string) string { p = strings.ReplaceAll(strings.TrimSpace(p), "\\", "/") return `"` + strings.ReplaceAll(p, `"`, "") + `"` // a `"` can't appear in a real path } // corsOriginsFlag locks a spawned llama-server down to localhost origins. // // llama-server defaults to --cors-origins '*' WITH credentials, which means it // echoes back whatever Origin it is given: any page you happen to visit can read // http://127.0.0.1:/props (model path + full launch args) and // /slots (live prompt and KV contents), and can drive generation. Binding to // 127.0.0.1 does not help — the browser is already inside the loopback. It is // also what llama-server warns about at init ("CORS is set to allow all origins // ('*') and no API key is set"). // // Nothing in a browser is supposed to reach an upstream directly: the UI and // every API client go through quartermaster's own listener, which proxies // server-side and has no Origin to speak of. So the lockdown costs nothing. // 'localhost' is llama-server's special value for "reflect the Origin only if it // is localhost". Emitted early, so a user's extraArgs can still override it. const corsOriginsFlag = "--cors-origins localhost" // loadMode renders llama-server's --load-mode, the single flag that replaced // --mmap/--no-mmap, --mlock and -dio (all three now warn "DEPRECATED in favor of // `--load-mode`" at startup). Returns "" or a flag with ONE trailing space, so // it can be interpolated into the flag run without leaving a double space. // // Modes are an enum, not a set, so the three old booleans do not all survive // contact: there is no dio+mlock. DirectIO wins that one — it is a streaming // load path that bypasses the page cache, which is the opposite of what pinning // pages in RAM is for, so the pair was already self-defeating. // // mmap alone stays UNSET on purpose. The default is 'auto' ("mmap, unless a // device does not support it"), which is what we emitted before by omission; // forcing 'mmap' would take llama.cpp's device check out of the loop. // // Note this makes a generated config require a llama-server new enough to know // --load-mode. Every backend the installer fetches has it; a hand-pointed // binary from before the flag landed would fail to spawn on an unknown arg. func loadMode(mmap, mlock, dio bool) string { switch { case dio: return "--load-mode dio " case mmap && mlock: return "--load-mode mmap+mlock " case mlock: return "--load-mode mlock " case mmap: return "" default: return "--load-mode none " } } // defaultSamplerFor returns the arch-derived server-side sampler baseline for a // model: the values an override leaves nil fall back to. nil/nil => emit nothing // and take llama-server's own defaults. // // Only top-k and min-p are seeded, and deliberately so. Neither has an OpenAI-API // field, so a client physically cannot set them through /v1/chat/completions — // llama-server's defaults (top-k 40, min-p 0.05) are what every request gets, for // every model, forever, unless the launch command says otherwise. Temperature and // top-p are the opposite: near enough every client sends them, so a launch-flag // value would be overwritten on arrival and would only ever act as a floor. Those // stay opt-in per model. // // The Qwen3 family (3.x dense and MoE alike, all reporting "qwen3*" — llama.cpp // has not split out a separate arch per Qwen minor, so 3.5, 3.6 and 3.8 are // indistinguishable here) documents // top_k 20 / min_p 0.0 across every model card, in both thinking and instruct // mode, so it is safe to apply without knowing which mode a request wants // (checked on the 3.6-27B and 3.8-27B cards; 3.6 additionally lists a coding // preset, which likewise moves only temperature). The // mode-dependent values (temperature, top-p, presence-penalty) are NOT seeded: // one process serves both modes here, so there is no single correct choice, and // they are the client-settable ones anyway. // // Muse Glimmer is the only other family found to pin a top-k. It asks for 64 — // wider than llama's 40, and constant across all four reasoning-strength levels — // but says nothing about min-p, so min-p stays nil rather than being invented. // That asymmetry is exactly why this returns two independent pointers instead of // a struct that would tempt a caller to fill both. // // A survey of ~19 current local models (Llama 3.x/4, Mistral, Magistral, Gemma 3, // Phi-4, DeepSeek R1/V3.1, GLM-4.6, gpt-oss, Kimi-K2, Nemotron Nano) turned up no // other top-k or min-p recommendation at all: the values those cards publish are // temperature and top-p, which clients send anyway. Absence here is a finding, // not a gap — for those models llama's own defaults are the documented state of // the art, so do not add speculative entries. func defaultSamplerFor(meta Metadata) (topK *int, minP *float64) { arch := strings.ToLower(meta.Architecture) switch { case strings.HasPrefix(arch, "qwen3"): k, p := 20, 0.0 return &k, &p // Prefix, not equality, to match the convention llama.cpp uses for family // members (qwen3 -> qwen3moe / qwen35moe): a future muse-glimmer MoE or minor // would land on "muse-glimmer*" and should inherit the same top-k. case strings.HasPrefix(arch, "muse-glimmer"): k := 64 return &k, nil } return nil, nil } // samplerLines renders the sampler-default flags for a resolved override, layered // over the arch baseline. An override value always wins, including an explicit 0 // (hence the pointers — 0 is a real, non-default value for each of these). func samplerLines(meta Metadata, ov *Override) []string { topK, minP := defaultSamplerFor(meta) var temp, topP, presence *float64 if ov != nil { if ov.TopK != nil { topK = ov.TopK } if ov.MinP != nil { minP = ov.MinP } temp, topP, presence = ov.Temp, ov.TopP, ov.PresencePenalty } var out []string if temp != nil { out = append(out, fmt.Sprintf("--temp %g", *temp)) } if topK != nil { out = append(out, fmt.Sprintf("--top-k %d", *topK)) } if topP != nil { out = append(out, fmt.Sprintf("--top-p %g", *topP)) } if minP != nil { out = append(out, fmt.Sprintf("--min-p %g", *minP)) } if presence != nil { out = append(out, fmt.Sprintf("--presence-penalty %g", *presence)) } return out } // smallCardVramGB is the budget below which a high-ctx profile falls back to // ub=512. The ub=1024 compute buffer (~0.5 GB larger) only fails to fit on a // genuinely small card at 64k+; bigger cards keep ub=1024 everywhere (bench: // ub=512 costs ~52% prefill vs 1024). const smallCardVramGB = 12.0 // MaxParallelSlots caps --parallel. Slots are cheap in code but not in VRAM: // every extra slot multiplies the KV pool sizing (see buildCmdLines), and the // slot cache keeps per-slot bookkeeping. 8 is far past any local multi-agent // setup; a typo of 400 would otherwise size a pool nothing can hold. const MaxParallelSlots = 8 // profileParallel resolves the slot count for one profile the same way emit // does: the variant's own parallel wins over the model-wide override. It exists // so sizing can charge the slot multiplier BEFORE effOv is assembled. func profileParallel(prof profile, ov Override) int { if v := prof.Variant; v != nil && v.Parallel > 0 { return EffectiveParallel(&Override{Parallel: v.Parallel}) } return EffectiveParallel(&ov) } // EffectiveParallel resolves llama-server's --parallel (server slot count) for a // model: the per-model/variant override, else 1, clamped to MaxParallelSlots. // Sizing (generate.go) and command emission (buildCmdLines) MUST agree on this // number — one charges N x KV per token, the other multiplies -c by it. func EffectiveParallel(ov *Override) int { if ov == nil || ov.Parallel <= 0 { return 1 } return min(ov.Parallel, MaxParallelSlots) } // effectiveUb picks the physical batch. 512 for dense models, 1024 for MoE. // Drop to 512 at high ctx ONLY when the VRAM budget is small (< smallCardVramGB) // — the auto-sized solo profile (Ctx=0) learns its real ctx after sizing, so // ctx>=longCtxThreshold is checked alongside prof.IsLong. budgetGB<=0 (unknown) // keeps the class default. An explicit Ub override always wins. // // The dense/MoE split is benched, and the two classes want opposite things: // // - Dense, fully GPU-resident: a bigger physical batch buys nothing. Measured // on a 7900 XTX with Qwen3.8-27B-UD-Q4_K_XL (b=2048 held fixed), pp2048 was // 682 / 679 / 667 t/s at ub 512 / 1024 / 2048 — flat-to-inverted, with 512 // ~2% ahead at every depth tested (d0 682 vs 667, d16384 543 vs 532, d65536 // 335 vs 327). Since computeBufferGB caps its vocab-scaled logits term at // exactly computeLogitsTokens (1024), halving ub to 512 halves that term as // well as activations — ~0.37GB of VRAM back on a 27B/151k-vocab model, for // free-to-slightly-positive throughput. That headroom is worth more as ctx. // - MoE with CPU expert offload: the opposite. A larger ub amortises the // per-micro-batch PCIe expert fetch over more tokens — benched 380 -> 647 // t/s going ub 512 -> 1024 at b=2048 on the 8GB/32GB rig. Keep 1024. // // Keyed on meta.IsMoE rather than on whether offload was actually planned: a // fully-resident MoE has no fetch to amortise and would likely behave like the // dense case, but that is untested, so MoE keeps the conservative 1024. func effectiveUb(meta Metadata, prof profile, ov *Override, ctx int, budgetGB float64) int { ub := 512 if meta.IsMoE { ub = 1024 } if (prof.IsLong || ctx >= longCtxThreshold) && budgetGB > 0 && budgetGB < smallCardVramGB { ub = 512 } if ov != nil && ov.Ub > 0 { ub = ov.Ub } if prof.Ub > 0 { ub = prof.Ub } return ub } // buildCmdLines returns the generated launch command as a list of flag lines // (no leading indentation), shared by emitProfile (which writes them as a YAML // `cmd: >` block) and RenderSoloCmd (which joins them for the editor preview). // The user's custom launch arguments are NOT applied here: callers pass the // result through ComposeCmd, which drops the generated flags a custom knob owns // and appends the user's text last. // pinsOwnSplit reports whether the user hand-wrote the --tensor-split vector for // this model, in which case we must NOT emit --device. // // A hand-written ratio is positional against whatever list the backend would // have enumerated on its own, which is what it meant when the user typed it. // --device replaces that list with ours, in main-last order, so the same string // would silently start addressing a different pair of cards. The override is // appended after the generated flags and wins on the last-flag-parsed rule, so // it survives either way: the choice here is only whether it lands on the cards // the user meant. Same refusal contract as everywhere else in this path, a split // that cannot be mapped is not rearranged. func pinsOwnSplit(ov *Override) bool { return ov != nil && strings.TrimSpace(ov.TensorSplit) != "" } func buildCmdLines(s Settings, meta Metadata, row GgufRow, prof profile, ctx, ngl, ncpuMoe int, kvK, kvV string, kvInRam bool, ov *Override) []string { cpuMoeFlag := "" if ncpuMoe > 0 { cpuMoeFlag = fmt.Sprintf(" --n-cpu-moe %d", ncpuMoe) } // mmap only helps when some weights live on the CPU: mmap gives lazy // demand-paging + natural OS page-caching of the on-CPU tensors. With NO CPU // offload the weights are copied straight into VRAM, so mmap only leaves a // redundant file-backed copy in host page cache — waste. So default to // mmap off and only keep it when CPU offload is actually happening: // - --n-cpu-moe > 0 (experts on CPU), or // - partial layer offload (ngl <= blocks => some layers on CPU). // Unknown block count (parse miss) assumes GPU-resident => mmap off. // Explicit Mmap:"on"/"off" always wins over this default. cpuOffload := ncpuMoe > 0 || (meta.BlockCount > 0 && int64(ngl) <= meta.BlockCount) mmapOn := cpuOffload if ov != nil && ov.Mmap == "off" { mmapOn = false } else if ov != nil && ov.Mmap == "on" { mmapOn = true } loadModeFlag := loadMode(mmapOn, ov != nil && ov.Mlock, ov != nil && ov.DirectIo) kvoFlag := "" if kvInRam { kvoFlag = "--no-kv-offload " } fa := "on" if ov != nil && ov.FlashAttn != "" { fa = ov.FlashAttn } // One slot by default; a per-model override raises it for multi-agent harnesses // (Qwen Code = main + memory subagent, an IDE plugin + a CLI on the same model) // that would otherwise thrash one slot's preamble. The slot cache pins each // conversation to a slot (id_slot) and saves/restores that slot, so >1 is a // supported configuration, not a foot-gun — see internal/server/slotcache.md. parallel := EffectiveParallel(ov) threads := s.Threads if ov != nil && ov.Threads > 0 { threads = ov.Threads } ub := effectiveUb(meta, prof, ov, ctx, s.TargetVramGB) // -b (logical batch) decoupled from -ub (physical/compute tile). A larger logical // batch pipelines more ub-micro-batches per decode(), overlapping CPU expert-fetch // with GPU compute on MoE offload — measured +38% pp at ub1024, +20% at ub512, for // ZERO extra VRAM (the compute buffer is sized by ub, not b; b only sizes the // token/pos arrays). 2048 is the plateau (bench: b past 2048 flat). Clamp >=ub and // <=ctx. ponytail: fixed 2048, expose a per-model b override only if a model wants // less pipelining depth. bTok := 2048 if bTok < ub { bTok = ub } if bTok > ctx { bTok = ctx } spec := effectiveSpec(meta, ov, row.DraftKind) if prof.Spec != "" { spec = prof.Spec } reason := prof.ReasoningFmt if reason == "" && ov != nil { reason = ov.ReasoningFmt } rfmt := "auto" reasoningFlag := "" switch { case reason == "off": rfmt = "none" reasoningFlag = " --reasoning off" case reason != "": rfmt = reason } modelPath := strings.ReplaceAll(row.FullPath, "\\", "/") lines := []string{ s.ServerExe, fmt.Sprintf("-m %s", modelPath), "--port ${PORT}", "--host 127.0.0.1", corsOriginsFlag, fmt.Sprintf("-ngl %d", ngl), // -c is the TOTAL KV pool: --kv-unified makes all slots share one buffer, so // N slots each keeping the sized per-slot context needs N x that pool. ctx is // already the PER-SLOT window (sizing charged N x the per-token KV cost and // clamped to VRAM), so multiplying here is what actually reserves it. fmt.Sprintf("-c %d", ctx*parallel), fmt.Sprintf("-ub %d -b %d", ub, bTok), fmt.Sprintf("-fa %s -ctk %s -ctv %s", fa, kvK, kvV), fmt.Sprintf("--parallel %d %s%s--kv-unified --no-warmup --no-ui --metrics --props", parallel, loadModeFlag, kvoFlag), } if ov.CpuDevice() { // -ngl 0 alone still opens the GPU: a compute buffer and runtime context // stay resident and op offload borrows it for large batches. --device none // is what makes a CPU pin hold no VRAM, which the cpu pool assumes. lines = append(lines, "--device none") } // Multi-GPU: split the layers (and, under -sm layer, their KV with them) // across every eligible card, pinning the buffers that do NOT split to the // device with the most free memory. Emitted only when the sizer actually // resolved a split, so a single-GPU box and a box with no telemetry both // produce byte-identical output to before. // // -sm layer, not row: row-split needs the interconnect to carry activations // per layer and is a loss on consumer boards with no NVLink, which is what a // mismatched desktop pair is. if len(prof.TensorSplit) > 1 && ngl > 0 { // --main-gpu is INERT under -sm layer. Measured, not inferred: at a fixed // --device list, --main-gpu 0 and --main-gpu 1 produced byte-identical // model, KV and compute buffer lines. It is kept because it costs nothing // and is the correct pin for anyone who puts -sm none or -sm row in // extraArgs, but nothing here should be read as controlling placement. // // What DOES control placement under layer split is the ORDER of the device // list: llama.cpp puts the non-splittable output weight on the device // listed last. MainLastOrder puts the plan's main device there, and // PermuteSplit moves --tensor-split with it so both flags still address the // same cards. That is what makes the main device's larger fixed budget // (mainFixedGB, see splitBy) land on the card that was charged for it. // // Naming the devices is also what makes the positions mean anything: // without --device they are positions in the BACKEND's list, which counts // adapters we filtered out and need not be in our order, and Vulkan/ROCm // have no CUDA_DEVICE_ORDER to pin it with. When the list cannot be // resolved the flags stay as they were, which is what shipped and what // issue #4 was tested on. split := fmt.Sprintf("-sm layer --main-gpu %d", prof.MainGpu) ts := prof.TensorSplit if devs, order := DeviceFlagFor(s.ServerExe, s.GpuSetOrEmpty(), prof.MainGpu); devs != "" && !pinsOwnSplit(ov) { ts = PermuteSplit(ts, order) split = fmt.Sprintf("--device %s -sm layer --main-gpu %d", devs, len(order)-1) } lines = append(lines, split, fmt.Sprintf("--tensor-split %s", FormatSplit(ts)), ) } // Vision twin loads the projector for image input. --no-mmproj-offload keeps // the CLIP tower on the CPU: no VRAM for the projector (the sizer already // priced the twin that way), slower image encode, same token throughput. // ov is the EFFECTIVE override here (model-wide with the vision variant's // knobs merged in), so an explicit mmprojFile - from either level - is // already resolved by the time the argv is rendered. mmprojFileOv := "" if ov != nil { mmprojFileOv = ov.MmprojFile } mmprojPath, _ := mmprojFor(row, mmprojFileOv) if prof.Vision && mmprojPath != "" { lines = append(lines, fmt.Sprintf("--mmproj %s", strings.ReplaceAll(mmprojPath, "\\", "/"))) if prof.CpuMmproj { lines = append(lines, "--no-mmproj-offload") } } // spec is a "+"-joined list of backends (draft-mtp / draft-dflash / ngram-map-k4v // / ngram-mod / none); llama-server accepts them chained, so emit one --spec-type // each and the sub-knobs for whichever backends are present. for _, st := range strings.Split(spec, "+") { lines = append(lines, fmt.Sprintf("--spec-type %s", st)) } // draft-mtp and draft-dflash both drive off a draft model checked each step // against the main model, so share the -md/-ngld wiring; only the sane // default draft length differs. DFlash proposes a whole diffusion block per // pass so it tolerates a longer chain than single-token MTP, but 5 is the // measured sweet spot on Qwen3.6-35B-A3B (own n-max sweep, 3/4/5/6): reasoning // tg jumps ~15% at n=5 vs n=3/4 and ties n=6, while n=5 also edges out n=6 on // creative tg — higher (6+, 12/15) over-drafts, accept rate falls faster than // length rises, so the wasted GPU verify compute makes TG worse. if isDraftSpec := specHas(spec, "draft-mtp") || specHas(spec, "draft-dflash"); isDraftSpec { nmax := 2 if specHas(spec, "draft-dflash") { nmax = 5 } if ov != nil && ov.SpecDraftNMax > 0 { nmax = ov.SpecDraftNMax } lines = append(lines, fmt.Sprintf("--spec-draft-n-max %d", nmax)) // A separate draft file (MTP sidecar like Gemma-4, or any DFlash drafter — // DFlash has no baked-in variant, it's always separate): baked-in MTP // models need no -md. Only attach the file when its kind matches the active // draft backend — a draft-dflash gguf must NOT be loaded as a draft-mtp // draft (arch mismatch: the sidecar is arch=dflash) and vice versa; a // baked-in MTP layer (DraftKind=="") correctly emits no -md. Pin the draft // fully to VRAM (-ngld 99): spec decode is serial (draft proposes, main // verifies each step), so a CPU-resident draft stalls the GPU main model // every round. DraftSizeGB charges this via draftOverheadGB. mdMatches := (specHas(spec, "draft-mtp") && row.DraftKind == "mtp") || (specHas(spec, "draft-dflash") && row.DraftKind == "dflash") if row.DraftPath != "" && mdMatches { lines = append(lines, fmt.Sprintf("-md %s", strings.ReplaceAll(row.DraftPath, "\\", "/"))) lines = append(lines, "-ngld 99") } // Draft KV quant (-ctkd/-ctvd), defaulted to the model's OWN -ctk/-ctv. // This is not "draft models only": a baked-in MTP head has no -md and // still gets a full second context whose cache type comes from this same // field, and llama's default for it is f16 no matter what -ctk says. On // Qwen3.8-27B at 160k that lone nextn layer was paying 0.61 GB of f16 KV // while the main cache ran q8_0; matching them halves it for free. Flash // attention is global, so the same fa that licenses a quantized main KV // covers the draft's. An override still wins on either side. ovDraftK, ovDraftV := "", "" if ov != nil { ovDraftK, ovDraftV = ov.KvKDraft, ov.KvVDraft } dKvK, dKvV := draftKvPair(ovDraftK, ovDraftV, kvK, kvV) lines = append(lines, fmt.Sprintf("-ctkd %s -ctvd %s", dKvK, dKvV)) } if specHas(spec, "ngram-map-k4v") && ov != nil { if ov.SpecDefault { lines = append(lines, "--spec-default") } if ov.SpecNgramSizeN > 0 { lines = append(lines, fmt.Sprintf("--spec-ngram-map-k4v-size-n %d", ov.SpecNgramSizeN)) } if ov.SpecNgramSizeM > 0 { lines = append(lines, fmt.Sprintf("--spec-ngram-map-k4v-size-m %d", ov.SpecNgramSizeM)) } if ov.SpecNgramMinHits > 0 { lines = append(lines, fmt.Sprintf("--spec-ngram-map-k4v-min-hits %d", ov.SpecNgramMinHits)) } } lines = append(lines, fmt.Sprintf("--jinja --reasoning-format %s%s", rfmt, reasoningFlag)) // Cap thinking tokens when a budget is set and reasoning is on. if ov != nil && ov.ReasoningBudget > 0 && reason != "off" { lines = append(lines, fmt.Sprintf("--reasoning-budget %d", ov.ReasoningBudget)) } // Keep prior-turn in history (Qwen3.6+); pointless when thinking is // off. --reasoning-preserve is llama.cpp's own knob for it — it sets the // template's preserve_thinking variable itself, and llama-server logs a nag at // init when a template supports preservation and the flag is absent. Verified // equivalent to the old --chat-template-kwargs '{"preserve_thinking":true}' on // both template wordings (3.8's preserve-by-default and 3.6's opt-in), and // safe on a template that does not support it at all: llama-server warns // "--reasoning-preserve has no effect" and carries on. // Default ON (nil), so a model with no override still keeps its thinking; only // an explicit false strips. See Override.PreserveThinking. if reason != "off" && (ov == nil || ov.PreserveThinking == nil || *ov.PreserveThinking) { lines = append(lines, "--reasoning-preserve") } // Always emit so runtime matches our reserve (else llama-server defaults to 32). ckptConstGB := GetKvCostModel(meta, kvK, kvV).ConstGB ckptRecurrent := meta.FullAttnInterval > 0 ckpts := effectiveCtxCheckpoints(prof, defaultCtxCheckpoints(ckptConstGB, ckptRecurrent)) lines = append(lines, fmt.Sprintf("--ctx-checkpoints %d", ckpts)) // Spacing is charged per checkpoint by checkpointReserveGB, so emit whatever it // assumed — ALWAYS, never "only when it differs from a default". llama-server's // own default is 8192 (LlamaDefaultCheckpointMinStep), so omitting the flag // launched every recurrent and plain-attention model 32x wider than the reserve // charged for. An ov-only pin (a preview profile built without the field) still // wins over the arch default. if ckpts > 0 { stepProf := prof if stepProf.CheckpointMinStep == 0 && ov != nil { stepProf.CheckpointMinStep = ov.CheckpointMinStep } lines = append(lines, fmt.Sprintf("-cms %d", effectiveCheckpointMinStep(stepProf, ckptConstGB, ckptRecurrent))) } // DRY sampler: defaults to settings.DryDefault (nil => off) and a per-model // ov.Dry wins (nil => the default, false => off, true => on). Values default // to 0.8 / 1.75 / 3; a non-zero override replaces each independently. dryOn := s.DryDefault != nil && *s.DryDefault if ov != nil && ov.Dry != nil { dryOn = *ov.Dry } if dryOn { mult, base, allow := 0.8, 1.75, 3 if ov != nil { if ov.DryMultiplier != 0 { mult = ov.DryMultiplier } if ov.DryBase != 0 { base = ov.DryBase } if ov.DryAllowedLength != 0 { allow = ov.DryAllowedLength } } lines = append(lines, fmt.Sprintf("--dry-multiplier %g --dry-base %g --dry-allowed-length %d", mult, base, allow)) } // Server-side sampler defaults (arch baseline + per-model override). These // apply only to requests that omit the field, so a client stays in control of // anything it actually sends. lines = append(lines, samplerLines(meta, ov)...) lines = append(lines, fmt.Sprintf("-t %d%s", threads, cpuMoeFlag)) // Slot KV persistence: expose llama-server's save/restore slot endpoints. Path // is quoted (it lives under a per-user dir that may contain spaces) and matches // the emitted slotCache.path the server's LRU uses. Per-model SlotCache is // OPT-IN (nil => off): the master switch only arms the feature, and a model // joins with slotCache:true from its config editor. if s.SlotCache.Enable && ov != nil && ov.SlotCache != nil && *ov.SlotCache { lines = append(lines, fmt.Sprintf("--slot-save-path %q", slotKvPath(s.SlotCache))) } // Chat template: ONLY what the user asked for. A model's baked-in gguf // template is otherwise left alone: chat templates are the user's to manage, // and substituting one silently drops whatever the baked template supports // that the replacement does not. Quoted: user paths routinely contain spaces. if ov != nil && strings.TrimSpace(ov.ChatTemplateFile) != "" { lines = append(lines, "--chat-template-file "+cmdPath(ov.ChatTemplateFile)) } // Advanced / power-user knobs. Each is gated on a set override field (zero/ // empty => omit), so a model with none of them set emits exactly as before. if ov != nil { if ov.ThreadsBatch > 0 { lines = append(lines, fmt.Sprintf("-tb %d", ov.ThreadsBatch)) } if ov.Prio > 0 { lines = append(lines, fmt.Sprintf("--prio %d", ov.Prio)) } if ov.NoOpOffload { lines = append(lines, "--no-op-offload") } if ov.NoRepack { lines = append(lines, "--no-repack") } if ov.CacheReuse > 0 { lines = append(lines, fmt.Sprintf("--cache-reuse %d", ov.CacheReuse)) } if ov.CacheRamMB > 0 { lines = append(lines, fmt.Sprintf("-cram %d", ov.CacheRamMB)) } if ov.LogVerbosity > 0 { lines = append(lines, fmt.Sprintf("-lv %d", ov.LogVerbosity)) } switch ov.CacheIdleSlots { case "on": lines = append(lines, "--cache-idle-slots") case "off": lines = append(lines, "--no-cache-idle-slots") } if ov.SwaFull { lines = append(lines, "--swa-full") } switch ov.ContextShift { case "on": lines = append(lines, "--context-shift") case "off": lines = append(lines, "--no-context-shift") } if ov.SpecDraftNMin > 0 { lines = append(lines, fmt.Sprintf("--spec-draft-n-min %d", ov.SpecDraftNMin)) } if ov.SlotPromptSimilarity > 0 { lines = append(lines, fmt.Sprintf("-sps %g", ov.SlotPromptSimilarity)) } if ov.RopeScaling != "" { lines = append(lines, fmt.Sprintf("--rope-scaling %s", ov.RopeScaling)) } if ov.RopeScale > 0 { lines = append(lines, fmt.Sprintf("--rope-scale %g", ov.RopeScale)) } else if ropeExtends(ov.RopeScaling) { // Derive the factor from the ctx actually chosen. Emitting the type // alone leaves llama.cpp on the model's own factor (1.0 unless the // publisher fine-tuned for extension), i.e. a longer window over // untrained positions. if f := ropeFactor(meta, ctx); f > 0 { lines = append(lines, fmt.Sprintf("--rope-scale %g", f)) } } if ov.RopeFreqBase > 0 { lines = append(lines, fmt.Sprintf("--rope-freq-base %g", ov.RopeFreqBase)) } if ov.YarnOrigCtx > 0 { lines = append(lines, fmt.Sprintf("--yarn-orig-ctx %d", ov.YarnOrigCtx)) } if ov.SplitMode != "" { lines = append(lines, fmt.Sprintf("-sm %s", ov.SplitMode)) } if ov.TensorSplit != "" { lines = append(lines, fmt.Sprintf("-ts %s", ov.TensorSplit)) } if ov.MainGpu > 0 { lines = append(lines, fmt.Sprintf("-mg %d", ov.MainGpu)) } if ov.OverrideTensor != "" { lines = append(lines, fmt.Sprintf("-ot %s", ov.OverrideTensor)) } // LoRA adapters, one flag each. Last so a user's extraArgs (appended // after this whole block) can still add more: --lora is Additive in the // flag table, so a custom occurrence does NOT drop these. lines = append(lines, llmLoraLines(s, ov, row.FullPath)...) } return lines } // RenderSoloCmd previews the full launch command for a candidate override, // reusing the solo-profile sizer so the editor's launch-parameters box matches // what a save would emit. Returns the command on one line (with `${PORT}` intact). func RenderSoloCmd(s Settings, meta Metadata, row GgufRow, ov Override) (string, error) { cc, err := RenderSoloCmdLayers(s, meta, row, ov) if err != nil { return "", err } return cc.Effective, nil } // plainCmd wraps a rendered command that has no composed layers: the non-llama // classes keep their emitter's own custom-text handling until their forms get // the same treatment (see ui-svelte/launch-args.md, non-goals). func plainCmd(cmd string) (ComposedCmd, error) { return ComposedCmd{Generated: cmd, Effective: cmd}, nil } // RenderSoloCmdLayers is RenderSoloCmd with the layered view the config editor // needs: the generated command, the effective one a save would produce, and // per-token provenance for rendering it. func RenderSoloCmdLayers(s Settings, meta Metadata, row GgufRow, ov Override) (ComposedCmd, error) { // SAM models render a sam3_server command (no metadata; matched by IsSam). if row.IsSam { return plainCmd(strings.Join(samCmdLines(s, row, &ov), " ")) } // TRELLIS.2 packages render a trellis2-server command (a directory, no // metadata, matched by IsTrellis). if row.IsTrellis { lines, err := trellisCmdLines(s, row, &ov) if err != nil { return ComposedCmd{}, err } return plainCmd(strings.Join(lines, " ")) } // HF folders render a vllm command or nothing: the same refusal emitHFModel // makes when no vllm backend is registered. meta is the caller's // ReadHFMetadata, since the folder has no gguf header. if row.IsHF { be := resolveBackendPreferring(s, &ov, "llm", "vllm") if !strings.EqualFold(be.Kind, "vllm") { return ComposedCmd{}, fmt.Errorf("%s is a safetensors model folder and needs a vllm backend; llama.cpp loads gguf only", row.FileName) } return plainCmd(strings.Join(vllmCmdLines(s, row, &ov, row.FullPath, be, meta), " ")) } // Video DiTs render the same sd-server command shape as an image model, but // have to be tested FIRST: their arch (when they declare one at all) still // reads as an image arch, and the video branch is what adds --audio-vae and // the frames/fps profile. if vid := videoInfoFrom(meta); vid.is() { // The sd path composes inside imageCmdLines, so the layered view comes // back already carrying custom/generated/effective and per-token // provenance: the editor's panes light up for a diffusion model exactly // as they do for llama-server, instead of falling back to plainCmd's // bare string, which owns no knobs and badges nothing on the form. cmd, _, _, _, _ := imageCmdLines(s, row, &ov, effectiveImageArch(meta), row.FullPath, meta.CondHidden, vid) cmd.Issues = validateComposedWith(SdFlags, cmd, imageExe(s, &ov)) return cmd, nil } // Diffusion models render an sd-server command, not a llama-server one. if imgArch := effectiveImageArch(meta); isImageArch(imgArch) { cmd, _, _, _, _ := imageCmdLines(s, row, &ov, imgArch, row.FullPath, meta.CondHidden, videoInfo{}) cmd.Issues = validateComposedWith(SdFlags, cmd, imageExe(s, &ov)) return cmd, nil } // Decision models render their /v1/systemone command, ahead of the embedder // check for the same reason as in emitModel. if IsDecisionModel(meta) { return plainCmd(strings.Join(decisionCmdLines(s, row, &ov, meta), " ")) } // Embedders render a minimal --embeddings command (no KV/spec sizing). if IsEmbeddingModel(meta) { return plainCmd(strings.Join(embeddingCmdLines(s, row, &ov, meta), " ")) } // audio.cpp models render an audiocpp_server command. Tested ahead of the // two legacy speech engines: an audio.cpp gguf names its own family in the // header, so the match is exact and neither of the others could read it. if IsAudioCppModel(meta) { return plainCmd(strings.Join(audioCppCmdLines(s, row, &ov, audioCppClass(meta.AudioFamily)), " ")) } // Speech models render a tts-server command: qwentts (talker + paired codec) // or TTS.cpp (--model-path), whichever engine the model resolves to. if IsTTSModel(meta, row.FileName) { return plainCmd(strings.Join(ttsCmdLines(s, row, &ov, meta), " ")) } // Parakeet ASR models render a parakeet-server command (model + port only). if IsASRModel(meta, row.FileName) { return plainCmd(strings.Join(asrCmdLines(s, row, &ov), " ")) } // vllm-backed LLMs render a vllm command; a chosen llama build swaps the exe. be := resolveBackend(s, &ov, "llm") if strings.EqualFold(be.Kind, "vllm") { // Same refusal the emitter makes: vllm loads only the shard it is given, // so there is no command to preview for a split set. if isSplitGguf(row) { return ComposedCmd{}, fmt.Errorf("vllm cannot load split gguf shards; merge %s with llama-gguf-split --merge or pick a llama backend", row.FileName) } return plainCmd(strings.Join(vllmCmdLines(s, row, &ov, row.FullPath, be, meta), " ")) } if be.Exe != "" { s.ServerExe = be.Exe } // Custom launch arguments size the preview, not just its argv (pins.go): a // pinned -c has to show up in the sized window and the memory numbers, or the // box and the panel would disagree about the same command. ov is a local copy. if pins := PinsFromArgs(ov.CustomArgsText()); !pins.Empty() { ov = pins.ApplyToOverride(ov) } // Same default as emitModel — this used to hardcode q8_0 with no MoE branch, // so the editor previewed a KV type the emitted config never used. soloTarget := s.TargetVramGB if ov.VramTargetGB > 0 { soloTarget = ov.VramTargetGB } soloSpec := effectiveSpec(meta, &ov, row.DraftKind) soloDraftGB := matchedDraftSizeGB(soloSpec, row.DraftKind, row.DraftSizeGB) kvDef := defaultKvQuant(s, meta, soloTarget, s.VramOverheadGB+draftOverheadGB(soloSpec, soloDraftGB)) kvK, kvV := kvDef, kvDef if ov.KvK != "" { kvK = ov.KvK } if ov.KvV != "" { kvV = ov.KvV } if !ValidKvPair(kvK, kvV) { kvK, kvV = kvDef, kvDef } perTokGB, kvConstGB := 0.0, 0.0 if m := GetKvCostModel(meta, kvK, kvV); m.OK { perTokGB, kvConstGB = m.SlopeGB, m.ConstGB } // Rope scaling lifts the trained-length ceiling; without it this is nativeCtx. modelMax := ropeCeiling(meta, ov.RopeScaling, ov.Ctx) target := s.TargetVramGB if ov.VramTargetGB > 0 { target = ov.VramTargetGB } specOh := draftOverheadGB(soloSpec, soloDraftGB) prof := profile{ Name: "preview", Target: target, Overhead: s.VramOverheadGB + specOh, Ctx: ov.Ctx, CpuOffload: ov.CpuOffload, KvK: kvK, KvV: kvV, IsLong: ov.Ctx >= longCtxThreshold, CtxCheckpoints: ov.CtxCheckpoints, CheckpointMinStep: ov.CheckpointMinStep, } // The pin's ctx/placement belong on the profile (per-slot window, exact // rounding): the override copy above only carried the model-wide scalars. pinsForProfile(&prof, ov, meta) soloUb := effectiveUb(meta, prof, &ov, prof.Ctx, s.TargetVramGB) // ...plus the second llama_context a baked-in MTP drafter runs, which // allocates a graph of its own at that same ub (see mtpDraftComputeGB). prof.Overhead += computeBufferGB(meta, soloUb, s.ComputeBufFactor) + mtpDraftComputeGB(meta, soloSpec, soloDraftGB, soloUb, s.ComputeBufFactor) // The baked-in MTP drafter's own KV scales with the window, so it belongs in // the slope the sizer solves ctx against (see mtpDraftSlopeFor). sdKvK, sdKvV := draftKvPair(ov.KvKDraft, ov.KvVDraft, kvK, kvV) prof.DraftSlopeGB = mtpDraftSlopeFor(meta, soloSpec, sdKvK, sdKvV, soloDraftGB) // Multi-slot: every slot carries its own window over the one shared -c pool, // so the sizer solves the per-slot ctx against N x the KV — same as emitModel. // Sizing one slot here previewed a command that over-committed the pool. if slots := profileParallel(prof, ov); slots > 1 { perTokGB *= float64(slots) kvConstGB *= float64(slots) prof.DraftSlopeGB *= float64(slots) } ctx, plan, kvReserve, _, err := sizeProfile(meta, s, prof, perTokGB, kvConstGB, modelMax, ov.KvInRam) if err != nil { return ComposedCmd{}, err } ngl, ncpuMoe := forceLowActiveMoE(meta, plan, prof, kvReserve) if ov.CpuOffload > 0 || prof.CpuOffloadSet { ngl, ncpuMoe = applyForcedOffload(meta, prof.CpuOffload) } lines := buildCmdLines(s, meta, row, prof, ctx, ngl, ncpuMoe, kvK, kvV, ov.KvInRam, &ov) cc, err := ComposeCmd(lines, ov.CustomArgsText()) if err != nil { return cc, err } cc.Issues = validateComposed(cc, be.Exe) return cc, nil } // specHas reports whether a "+"-joined spec list contains backend b. func specHas(spec, b string) bool { for _, s := range strings.Split(spec, "+") { if s == b { return true } } return false } // formatCtxTag renders a short ctx tag: 8192->"8k", 131072->"128k", 1048576->"1m". // effectiveSpec resolves the spec-type: MTP-capable models (baked-in nextn // layer or a paired mtp-*.gguf sidecar) default to draft-mtp+ngram-mod (the // chain beats mtp alone), everything else to ngram-mod. A paired DFlash drafter // is never auto-selected — see below. // An explicit override spec wins (set spec: "ngram-mod" to force ngram on a // drafted model, or spec: "draft-dflash" to opt into DFlash). // // A DFlash sidecar sitting in the model's dir is a deliberate choice (the user // downloaded it) but is NOT auto-selected: it wins a short flat-prompt bench // (GPU-bound: +17% vs mtp) but its resident draft weights + own full-context KV // crowd an already-tight VRAM budget over a long real session, cratering to a // GPU-mem-oversubscription cliff mtp never hits (mtp is a baked-in head with no // separate weights or KV). Confirmed in real multi-turn agent use on // Qwen3.6-35B-A3B-100k: switching qwen3.6-35b-a3b-ud-q4_k_s-100k off dflash back // to mtp fixed a production slowdown. An explicit `spec: draft-dflash` override // still works — this only removes it as the automatic pick. func effectiveSpec(meta Metadata, ov *Override, draftKind string) string { // Explicit override always wins. if ov != nil && ov.Spec != "" { return ov.Spec } switch { case meta.IsMTP || draftKind == "mtp": // draft-mtp CHAINED with model-less ngram-mod: benched better than mtp // alone. The MTP head drafts short high-confidence continuations; ngram-mod // fills the gaps mtp misses (verbatim repeats — code, quoted context, tool // echoes) at zero extra VRAM (no draft weights/KV). llama-server verifies // both proposals each step, so the union lifts accept rate over either solo. return "draft-mtp+ngram-mod" } return "ngram-mod" }