# Local LLM GPU-backed local LLM serving for the homelab. The **default (`overlays/okd`) backend is llama-swap** — chat models run on **vLLM XPU** with MTP speculative decoding, embedding models run on **llama.cpp SYCL** — on the Intel Arc Pro B70 (2× GPUs, 64 GB VRAM pooled). The host (`gpu-1`) is a 6-core/12-thread Ryzen 5 3600 with 82 GB RAM — see [Performance](#performance-backend-history-and-current-state) for measured throughput and the Vulkan → SYCL → vLLM backend history. For GPU provisioning, see [intel-device-plugins](../intel-device-plugins/README.md). ## Table of Contents - [Local LLM](#local-llm) - [Table of Contents](#table-of-contents) - [Component READMEs](#component-readmes) - [Backends / Overlays](#backends--overlays) - [GPU Monitoring](#gpu-monitoring) - [Metrics](#metrics) - [Prometheus metric names](#prometheus-metric-names) - [llama.cpp metrics via the metrics-exporter sidecar](#llamacpp-metrics-via-the-metrics-exporter-sidecar) - [vLLM metrics (chat models, via metrics-exporter)](#vllm-metrics-chat-models-via-metrics-exporter) - [Example PromQL queries](#example-promql-queries) - [Grafana dashboard](#grafana-dashboard) - [Performance: backend history and current state](#performance-backend-history-and-current-state) - [Current: vLLM XPU + MTP3 (2026-09-10)](#current-vllm-xpu--mtp3-2026-09-10) - [Backend history](#backend-history) - [Historical: Vulkan-era CPU-dispatch-bound analysis](#historical-vulkan-era-cpu-dispatch-bound-analysis) - [Scaling](#scaling) - [Layout](#layout) - [Roadmap: connector auto-sync (Onyx)](#roadmap-connector-auto-sync-onyx) - [Future Work](#future-work) - [REF](#ref) ## Component READMEs | Component | Description | | --------------------------------------------- | -------------------------------------------------------------------------- | | [llama-swap](components/llama-swap/README.md) | LLM orchestrator: model matrix, B70 tuning, SYCL, metrics-exporter sidecar | | [LiteLLM](components/litellm/README.md) | API gateway & GPU-aware routing plugin | | [Open WebUI](components/open-webui/README.md) | Chat front-end: OIDC, RAG, embeddings, storage | ## Backends / Overlays | Overlay | Backend | Hardware | Notes | | -------------- | ------------------------------------------------------------------- | ---------------------- | ---------------------------------------------- | | `overlays/okd` | **llama-swap** (vLLM XPU chat + llama.cpp SYCL embeddings, default) | Intel Arc Pro B70 (2×) | One model per GPU, data-parallel + spread sets | ## GPU Monitoring Live Intel GPU telemetry tools are packaged in `containers/intel-gpu-monitor/` and can be run on-demand with `oc debug` — no persistent deployment needed. GPU power limits are managed by the `gpu-power-manager` DaemonSet (160W TDP per GPU) — see [GPU Power Tuning Notes](../intel-device-plugins/GPU_POWER_TUNING.md) for rationale, expected temperatures, and performance impact. ```bash export KUBECONFIG=$HOME/.kube/okd # On-demand debug pod with gpu.intel.com/xe request: oc run intel-gpu-debug --image=registry.arthurvardevanyan.com/homelab/intel-gpu-monitor \ -n llm --rm -i --restart=Never --overrides=' { "spec": { "containers": [{ "name": "debug", "image": "registry.arthurvardevanyan.com/homelab/intel-gpu-monitor:not_latest", "args": ["sleep", "300"], "resources": { "requests": {"gpu.intel.com/xe": "2"}, "limits": {"gpu.intel.com/xe": "2"} }, "stdin": true, "tty": true, "volumeMounts": [{ "name": "dri", "mountPath": "/dev/dri" }] }], "volumes": [{ "name": "dri", "hostPath": { "path": "/dev/dri" } }] } } # Once inside the debug pod: xpu-smi discovery xpu-smi stats -d 0 xpu-smi dump -d 0 -m 0,1,2,3,5 intel_gpu_top -l clinfo | grep -i "Device Name" ``` Workload-level metrics (llama-swap exposes Prometheus metrics via its `/metrics` endpoint on port 8080; real `llamacpp:*` token-throughput metrics are on the `metrics-exporter` sidecar's port 9100 — see [Metrics](#metrics)): ```bash oc -n llm exec deploy/llama-swap -c llama-swap -- curl -sS "http://localhost:8080/metrics" oc -n llm exec deploy/llama-swap -c metrics-exporter -- curl -sS "http://localhost:9100/metrics" ``` ## Metrics llama-swap exposes **proxy-level** Prometheus metrics on `/metrics` (port 8080). Real llama.cpp (`llamacpp:*`) token-throughput/decode metrics are collected by a **metrics-exporter sidecar** (port 9100) — see [llama.cpp metrics via the metrics-exporter sidecar](#llamacpp-metrics-via-the-metrics-exporter-sidecar) below. ### Prometheus metric names This cluster runs two Prometheus replicas; the same scrape jobs are defined identically in both `kubernetes/prometheus/components/prometheus/config-map.yaml` and `prometheus-nas/config-map.yaml`. LLM-stack targets (intel-gpu, LiteLLM, and both llama-swap endpoints) share a single combined `llm` job, with `app` distinguishing each target via per-target `static_configs` labels: ```yaml - job_name: "llm" scrape_interval: 30s static_configs: - targets: ["xpumd.intel-device-plugins-operator.svc.cluster.local.:8080"] labels: { app: intel-gpu } - targets: ["litellm-svc.llm.svc.cluster.local.:4000"] labels: { app: litellm } - targets: ["llama-swap-svc.llm.svc.cluster.local.:8080"] labels: { app: llama-swap } - targets: ["llama-swap-svc.llm.svc.cluster.local.:9100"] labels: { app: llama-swap } ``` > **Known double-scrape (accepted, not yet fixed):** the static job above > and `ServiceMonitor/llama-swap` (`components/llama-swap/service-monitor.yaml`) > both scrape `llama-swap-svc:8080/metrics` **and**, since the > metrics-exporter sidecar landed, `llama-swap-svc:9100/metrics` too. > Harmless beyond a small amount of duplicate Prometheus series/storage; > deduplicating is low priority. Key metric families (llama-swap proxy only, no llama-server sub-scrapes): | Metric family | Type | Description | | ------------------------------- | ------- | -------------------------------------------------------- | | `llamaswap_cpu_util_percent` | Gauge | CPU utilization per core (0–100) | | `llamaswap_memory_total_bytes` | Gauge | Total system memory (bytes) | | `llamaswap_memory_used_bytes` | Gauge | Used system memory (bytes) | | `llamaswap_memory_free_bytes` | Gauge | Free system memory (bytes) | | `llamaswap_swap_total_bytes` | Gauge | Total swap capacity (bytes) | | `llamaswap_swap_used_bytes` | Gauge | Used swap (bytes) | | `llamaswap_load_average` | Gauge | Load average (labels: 1m, 5m, 15m) | | `llamaswap_network_bytes_total` | Counter | Network bytes transferred (labels: interface, direction) | ### llama.cpp metrics via the metrics-exporter sidecar Real `llamacpp:*` token-throughput/decode metrics require both: 1. llama-server started with `--metrics` — now set in `cmd_base` (`components/llama-swap/llama-swap.yaml`). 2. Solving llama-swap's **dynamic per-model port assignment**: each llama-server child gets a random `${PORT}` at runtime, so Prometheus can't scrape it directly with a static target. Both are solved by the `metrics-exporter` sidecar container (`containers/llama-swap-metrics-exporter/`, added to `deployment.yaml`, built with `ko` — no Containerfile), exposing aggregated metrics on port 9100: 1. **Discovery** (every 10s): `GET http://localhost:8080/running` on llama-swap's own API returns the currently active model IDs and their assigned ports. 2. **Scrape**: for each active model, `GET http://localhost:/metrics?model=` against the llama-server child directly (`?model=` is required in llama-swap's router mode). 3. **Re-export**: scraped series are re-exposed on the sidecar's own `/metrics` (port 9100) with a `model=""` label added, giving Prometheus one static target regardless of how many models are loaded or which ports they're on. The `Service`/`ServiceMonitor` (`service.yaml`, `service-monitor.yaml`) expose/scrape this as the named `metrics` port; the static Prometheus configs (`kubernetes/prometheus/components/prometheus{,-nas}/config-map.yaml`) scrape the same port 9100 as one of the targets in the combined `llm` job (`app="llama-swap"`). Key `llamacpp:*` metrics (see `notes/llama-swap-metrics.md` for the full design writeup): | Metric | Type | Description | | ----------------------------------------- | ------- | ----------------------------------- | | `llamacpp:prompt_tokens_total` | Counter | Prompt tokens processed | | `llamacpp:prompt_seconds_total` | Counter | Prompt process time | | `llamacpp:prompt_tokens_seconds` | Gauge | Average prompt throughput (t/s) | | `llamacpp:tokens_predicted_total` | Counter | Generation tokens processed | | `llamacpp:tokens_predicted_seconds_total` | Counter | Predict process time | | `llamacpp:predicted_tokens_seconds` | Gauge | Average generation throughput (t/s) | | `llamacpp:requests_processing` | Gauge | Requests currently processing | | `llamacpp:requests_deferred` | Gauge | Requests deferred (queued) | | `llamacpp:n_tokens_max` | Counter | High-watermark context size seen | | `llamacpp:n_decode_total` | Counter | Total `llama_decode()` calls | | `llamacpp:n_busy_slots_per_decode` | Gauge | Average busy slots per decode | ### vLLM metrics (chat models, via metrics-exporter) | Metric | Type | Description | | ---------------------------------------------------- | --------- | ----------------------------------------------- | | `vllm:num_requests_running` | Gauge | Concurrent decode tasks | | `vllm:num_requests_waiting` | Gauge | Requests in the scheduling queue | | `vllm:prompt_tokens_total` | Counter | Prompt tokens processed | | `vllm:generation_tokens_total` | Counter | Decoded tokens generated | | `vllm:kv_cache_usage_perc` | Gauge | PagedAttention KV cache block utilization (0-1) | | `vllm:prefix_cache_hits_total` | Counter | KV cache prefix matching hits | | `vllm:prefix_cache_queries_total` | Counter | KV cache prefix matching queries | | `vllm:time_to_first_token_seconds_bucket` | Histogram | Time from request to first output token | | `vllm:request_time_per_output_token_seconds_bucket` | Histogram | Decode time per output token | | `vllm:e2e_request_latency_seconds_bucket` | Histogram | End-to-end request latency | | `vllm:spec_decode_num_draft_tokens_total` | Counter | Speculative decoding draft tokens | | `vllm:spec_decode_num_accepted_tokens_total` | Counter | Accepted speculative tokens | | `vllm:spec_decode_num_drafts_total` | Counter | Speculative verification steps | | `vllm:spec_decode_num_accepted_tokens_per_pos_total` | Counter | Accepted tokens per draft position | All metrics carry the exporter-added `model` label (e.g. `35b-gpu0`, `27b-gpu1`), so per-model dashboards/alerts use `{model="..."}` or `{model=~"$model"}` selectors. Verify from inside the pod: ```bash export KUBECONFIG=$HOME/.kube/okd # llama.cpp metrics (embed-spread) oc -n llm exec deploy/llama-swap -c metrics-exporter -- curl -sS localhost:9100/metrics | grep '^llamacpp:' | head -5 # vLLM metrics (chat models) oc -n llm exec deploy/llama-swap -c metrics-exporter -- curl -sS localhost:9100/metrics | grep '^vllm:' | head -5 ``` > **Known limitation: transient scrape gaps under heavy load.** llama-server's > `/metrics` endpoint shares the same small HTTP thread pool > (`--threads-http 4`, set via `cmd_base_llama` in `llama-swap.yaml`) as > the rest of its API. Observed directly: a 125K-token prompt on `35b-gpu0` > blocked its `/metrics` endpoint for several minutes (`context deadline exceeded` in the exporter logs) while the manual `curl` against the same > endpoint succeeded instantly once the request finished — i.e. this is > llama-server being genuinely busy, not an exporter bug. The exporter > mitigates this by: > > - a 15s per-model scrape timeout (`defaultScrapeTimeout` in `main.go`), > generous enough for most transient contention without hanging a whole > scrape cycle, > - serving the **last successfully-scraped value** for a model when a > scrape fails, for up to 5 minutes (`maxStaleness`) — avoids a full data > gap in Grafana during exactly the busiest, most-interesting moments, > - `exporter_model_last_success_timestamp_seconds{model="..."}` — alert on > `time() - exporter_model_last_success_timestamp_seconds > N` if a model > goes stale for longer than expected, > - throttled error logging (first failure, then every 6th) instead of one > log line per 10s discovery cycle. ### Example PromQL queries ```promql # --- llama-swap proxy metrics (host-level) --- # Average CPU utilization across all cores avg(llamaswap_cpu_util_percent) # Memory usage percentage llamaswap_memory_used_bytes / llamaswap_memory_total_bytes * 100 # Load average llamaswap_load_average # Network throughput (bytes/sec) rate(llamaswap_network_bytes_total[5m]) # --- llamacpp metrics (per-model, via metrics-exporter) --- # Generation throughput per model llamacpp:predicted_tokens_seconds{model=~"$model"} # Active/deferred requests per model llamacpp:requests_processing{model=~"$model"} llamacpp:requests_deferred{model=~"$model"} # Prompt latency (ms/token), rate-derived 1000 * rate(llamacpp:prompt_seconds_total{model=~"$model"}[5m]) / rate(llamacpp:prompt_tokens_total{model=~"$model"}[5m]) ``` ### Grafana dashboard `kubernetes/grafana/base/dashboards/llama-swap.json` contains both the host-level llama-swap panels (Memory/CPU/Network, from the proxy metrics above), the per-model `llamacpp:*` panels (Overview/Performance/Concurrency/ Efficiency/Diagnostics rows, re-scoped to `{model="embed-spread"}`), and the per-model `vllm:*` panels (new "vLLM Chat Models" and "Speculative Decoding (MTP)" rows), templated on a `$model` variable (`label_values(exporter_model_last_success_timestamp_seconds, model)`, multi-select, default `All`). ## Performance: backend history and current state ### Current: vLLM XPU + MTP3 (2026-09-10) Chat models run on vLLM XPU with MTP (Multi-Token Prediction) speculative decoding: 27B dense achieves **~40–44 tok/s** clean decode, 35B-A3B MoE achieves **~55–72 tok/s** (peak 112.2). See the llama-swap README's [Measured Performance (2026-09-10)](components/llama-swap/README.md#measured-performance-2026-09-10) for the full session-level breakdown, MTP acceptance-rate analysis, and known gaps (position-4 acceptance tuning, long-running decay test, concurrency, etc.). > **Sleep mode disabled.** With the matrix commented out, all models remain > resident 24/7. The `--enable-sleep-mode`, `cmdStop`, and `unloadTimeout` > settings on 27B models have been disabled. See > [Scaling](components/llama-swap/README.md#scaling) for the current operating > mode. ### Backend history | Era | Backend | Config | Key numbers | | ---------------------- | -------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------- | | Aug 12–14, 2026 | llama.cpp Vulkan | f16 KV, dual GPU, ngram-simple | decode p50 20.6, p95 45.0, p99 70.2, max 76.7 t/s (n=847) | | Aug 14, 2026 | + `--load-mode none` | | working set 32 GiB → 4.7 GiB; cold load 82.3 s (still in effect on the embed model via `cmd_base_llama`) | | Aug 18, 2026 | llama.cpp SYCL | f16 KV, dual GPU | 35B 1087–1230 t/s, 27B 862–1052 t/s (llama-swap Activity "Gen Speed" column) | | Sep 10, 2026 (current) | vLLM XPU + MTP3 | GPTQ-Int4, fp8 KV cache | 27B ~40–44 tok/s, 35B ~55–72 tok/s (vLLM engine 10s-window log means) | > **Open question — SYCL vs vLLM measurement gap:** the Aug 18 llama.cpp SYCL > figures (862–1230 t/s) come from llama-swap's Activity page "Gen Speed" > column; the Sep 10 vLLM figures (~40–72 tok/s) are 10-second-window means > from the vLLM engine log. These are different measurement methodologies > and are **not directly comparable** — the apparent ~20× gap has not been > reconciled and should not be read as a confirmed regression. ### Historical: Vulkan-era CPU-dispatch-bound analysis The Vulkan-era data (decode p50 20.6 t/s against a 76.7 t/s ceiling, n=847 requests) showed the median request running at roughly a quarter of the observed maximum, while the GPU used only ~11% of its 608 GiB/s memory bandwidth under dual-slot load — pointing at the decode loop being **CPU-dispatch-bound, not GPU-bandwidth- or GPU-compute-bound**: 2 llama-server processes each defaulting to `-1` (auto-detect-all-cores) threads, on a 6-core host also running LiteLLM, Open WebUI, Dragonfly, CNPG, and (opportunistically) Tekton CI builds, inside a 5-CPU pod quota. Mitigations applied 2026-08-14 (`--spec-type ngram-simple`, `--poll 0`, thread tuning, a `1`-core CPU request/VPA floor) are captured in the llama-swap README's [vLLM XPU Tuning Rationale](components/llama-swap/README.md#vllm-xpu-tuning-rationale) (the historical llama.cpp flags now live in `cmd_base_llama`, used only by the embedding model). This analysis no longer applies to the **chat model path**: vLLM + MTP operates differently, and [Measured Performance](components/llama-swap/README.md#measured-performance-2026-09-10) shows MTP acceptance rate — not CPU dispatch or GPU bandwidth — is the dominant factor in decode throughput. ## Scaling > **Matrix disabled.** With the `matrix:` section commented out, llama-swap > operates with **fixed model assignments** — all chat and embedding models > boot via `hooks.on_startup.preload` and remain resident on their assigned > GPUs 24/7 (`ttl: 0`). No dynamic switching or evictions occur. > > The scaling options below are preserved for reference and can be re-enabled > by restoring the `matrix:` section in `llama-swap.yaml`. For higher throughput (when matrix is re-enabled): - **Data parallel** (`dual_35b`, `dual_27b`): llama-swap runs the same model on both GPUs. Each slot gets a full 22 GB weight copy, but concurrent requests get full throughput on both cards, both GPUs always required. - **Mixed** (`dual_35b0-27b1`, `dual_35b0d-27b1`, `dual_27b0-35b1`, `dual_27b0-35b1d`): 35B on one GPU + 27B on the other. The solver picks the mixed set when a cross-family request arrives and the other GPU already has a model, avoiding unnecessary eviction. - **Spread**: one model spanning both GPUs via `--split-mode layer --tensor-split 1,1`. - **Multiple llama-swap replicas** with a LoadBalancer: add replicas in `overlays/okd/llama-swap.yaml` and expose via a LoadBalancer service. llama-swap's config matrix handles the shared hardware — no external orchestrator needed for GPU-aware scheduling. - **Horizontal Pod Autoscaler** (HPA): not yet configured. With data parallel mode and ~2 slots per GPU, the current setup handles concurrent requests well. Add HPA once load patterns are measured. ## Layout > **Correction:** an earlier version of this diagram described > `base/llama-swap.yaml`, `base/llama-swap-configmap.yaml`, and a top-level > `config/llama-swap.yaml` — none of these paths exist. The actual layout is > below. ```text kubernetes/llm/ ├── base/ │ ├── namespace.yaml # llm Namespace (PSS restricted, GPU toleration) │ └── network-policy.yaml ├── components/ # one directory per app, each a Kustomize Component │ ├── llama-swap/ # orchestrator: Deployment, Service, PVCs, │ │ │ # llama-swap.yaml (model matrix, mounted via │ │ │ # configMapGenerator — no separate config/ dir) │ │ │ # also runs the metrics-exporter sidecar │ │ │ # (containers/llama-swap-metrics-exporter/) │ │ └── llama-swap.yaml # the actual model matrix config │ ├── litellm/ # gateway + llama_swap_affinity routing plugin │ ├── open-webui/ # chat front-end │ ├── searxng/ # web-search backend for Open WebUI │ ├── model-downloader/ # CronJob (suspended) to (re)fetch GGUFs │ ├── dragonfly-litellm/, dragonfly-open-webui/ # Redis-compatible caches │ ├── cnpg-litellm/, cnpg-open-webui/ # CloudNativePG Postgres │ └── *-gateway/ # Gateway API HTTPRoute + Certificate per app ├── overlays/ │ └── okd/ │ ├── kustomization.yaml # composes base + all components for this cluster │ └── egress-firewall.yaml └── README.md # this file ``` ## Roadmap: connector auto-sync (Onyx) Planned: automated document ingestion connectors (GitHub repos, web scraping, email) that continuously sync documents into the vector store. This would eliminate manual document uploads and keep RAG data fresh. See [Onyx](https://github.com/onyx-dot-app/onyx) (or equivalent project) for reference implementations. Integration into the ArgoCD application topology is the next step once the connector architecture is decided. ## Future Work Items deliberately not implemented yet: - **Model storage on node-local storage instead of `rook-ceph-block-ci`.** `gpu-1` has ~900 GB free on its node filesystem; a local PV would cut the ~20 GB cold-load time (82.3s observed off RBD; page-cache-warm second load was 7.3s — readahead loss from `--load-mode none` is a real cost, though the ~10× working-set reduction is the primary trade-off). Blocked on installing a local-storage provisioner (no LVM Storage / Local Storage Operator CSI driver is currently installed — only `rook-ceph.rbd`, `rook-ceph.cephfs`, and `csi.spiffe.io` exist on this cluster). Out of scope for a config-only tuning pass. - **MTP `num_speculative_tokens: 3` vs `4` experiment.** Position-4 MTP draft-token acceptance is often <50% (see [Known Gaps](components/llama-swap/README.md#known-gaps)); testing 3-token drafting is a low-cost tuning lever. - **30+ minute long-running decode-decay test.** Sessions measured so far are ~4–5 minutes; a long session near KV cache exhaustion is needed to characterize decode decay under vLLM + MTP. - **Multi-stream concurrency measurement.** Only single-stream tested so far (`--max-num-seqs 4`, 1 running request per model). - **Prometheus percentiles for token latency.** Current data is 10-second window averages from vLLM engine logs; true p50/p99/p999 inter-token latency requires querying the vLLM `/metrics` endpoint directly. - **Speculative output quality validation.** MTP uses a smaller draft model; confirm output quality matches the base model (human eval or benchmark comparison). - **Reconcile the SYCL-era vs vLLM throughput measurement gap** — see [Backend history](#backend-history). - **Open WebUI `TASK_MODEL` offload** — see [Task-model offload (deferred)](components/open-webui/README.md#task-model-offload-deferred). ## REF - [llama.cpp](https://github.com/ggerganov/llama.cpp) - [Intel Level Zero](https://github.com/oneapi-src/level-zero) - [Qwen3.6 model family](https://qwenlm.github.io) - [llama-swap](https://github.com/llama-swap/llama-swap) - [Open WebUI](https://github.com/open-webui/open-webui) - [Intel Arc Pro B70 (Battlemage) architecture](https://www.intel.com) - [75 t/s on a single B70 (Reddit)](https://www.reddit.com/r/IntelArc/comments/1u3l4zx/qwen3635ba3b_at_75_tokens_per_second_on_a_single/) - [Intel Arc B70 context decay: the KV cache setting that fixes it](https://jonathanmann.tech/blog/intel-arc-b70-context-decay-kv-cache/)