--- name: quark-torch-llm-eval description: >- End-to-end LLM accuracy evaluation on AMD ROCm (ROCm-only) — container setup, vLLM/SGLang/ATOM serving, lm-eval / lighteval / evalscope benchmarks. Use when the user wants to evaluate, benchmark, or compare an LLM's accuracy. Trigger for "evaluate this model", "run gsm8k/mmlu/mmlu_pro/aime/gpqa/hellaswag/arc", "test accuracy", "measure perplexity", "compare quantized model accuracy", "does this mxfp4 model lose accuracy". For evaluating Quark Agent Skills themselves, use quark-torch-eval-runner instead. layer: l1-atomic primary_artifact: eval_report.md --- # LLM Evaluation on AMD ROCm End-to-end LLM accuracy evaluation on AMD ROCm — from Docker container to final accuracy table. ## When to use Trigger this skill when the user asks to: - Evaluate, benchmark, or test a model's accuracy - Run gsm8k / mmlu / mmlu_pro / aime / math_500 / gpqa / hellaswag / arc - Deploy a model with vLLM, SGLang, or ATOM and measure accuracy - Compare quantized model quality against a reference - Verify a newly quantized model ("does this mxfp4 model lose accuracy?") ## Inputs | Parameter | Required | Description | |-----------|----------|-------------| | `model_path` or `hf_id` | **Yes** | Local path to model weights, or HuggingFace model ID | | `benchmark` | No | Benchmark to run (default: `gsm8k`) | | `backend` | No | Serving backend: `vLLM` (default), `SGLang`, or `ATOM` | | `image` | No | Docker image (default: `vllm/vllm-openai-rocm:latest`; overridden by model card / Recipes / user). Ignored when `runtime=host`. | | `runtime` | No | `docker` (default) or `host`. Auto-detected in Phase 0.5: `docker` when the daemon is reachable, else `host` when a native `vllm` / `sglang` is importable. ATOM is docker-only. Stop and report if neither path is available. | ## Outputs `$EVAL_STATE_DIR/eval_report.md` — see `templates.md` §2 for the full skeleton, formatting rules, and the example used as the canonical shape. ## Design Principles 1. **Trust the model card** — launch commands from model card / Recipes / Cookbook are author-verified. Use them as-is, only adjusting TP and `HIP_VISIBLE_DEVICES`. Do not preemptively stack flags. 2. **Reactive error handling** — `troubleshooting.md` is a diagnostic guide, not a prevention checklist. Launch first, consult on failure. ## Interaction Flow ### Unified confirmation gate After Phase 0 + 0.5 + 1 finish collecting all information, present a **single confirmation summary** to the user before any container or server operations begin. Use the template in `templates.md` §1 — it defines all required fields, command format, and display rules. Wait for explicit user approval before executing any of these commands. This avoids piecemeal confirmations and lets the user review the full pipeline at once. **After approval**, the following are pre-approved by the unified gate — execute without re-asking: - `docker run` / `docker exec` / `docker pull` of the planned container/image - The exact `vllm serve` / `python -m sglang.launch_server` / `python -m atom.entrypoints.openai_server` command shown in the plan (with the planned GPU/TP/port) - The exact `lm_eval` / `lighteval` / `evalscope` command shown in the plan - `pip install` of frameworks listed in `eval-frameworks.md` §Installation — **docker runtime only** (inside the planned container). In host runtime, pip install is never pre-approved; see Phase 0.5 host-mode constraints. Each of the following **still requires per-command confirmation**, even after the unified gate: - Any edit to model `config.json` or backend source code (show full diff first) - `kill -9` / `sudo` / `rm -rf` / `mv` outside `$EVAL_STATE_DIR` and outputs - Any `vllm serve` / eval command that **deviates** from the plan (different flags, different port, retry with new args) - `docker stop` / `docker rm` (Phase 7 cleanup — always ask) Read-only probes (`ls`, `cat`, `du`, `rocm-smi`, `docker ps`, `docker images`, `curl -sI`, `curl -fsSL`, Read) do **not** need confirmation. ### Never kill other users' processes - Never use `pkill -f ` or `ps | grep | awk | kill` broad matching - Verify any PID is alive and belongs to the expected backend via `kill -0 ` + `ps -fp ` (check `USER` and `CMD` columns) before sending a signal — PIDs are recycled - Send `SIGTERM` first, wait up to 30 s, then `SIGKILL` only if still alive ### HuggingFace fetch rules All HuggingFace HTTP requests follow `model-inspection.md` §2 — `curl -fsSL`, `${HF_ENDPOINT:-https://huggingface.co}`, token from `~/.hf_token` then `$HF_TOKEN`. Read that section before the first HF request in any phase. ### Phase 0 — Auto-discover **Prerequisite: user must provide `model_path` (local path) or `hf_id` (HuggingFace ID).** - **local path** → verify it exists (`ls /config.json`) → proceed - **hf_id only** → fetch `config.json` and `tokenizer_config.json` from HF API (use the HuggingFace fetch rules above), then continue to Phase 0.5 → 1. Before Phase 2, ask: **A** "I have a local copy" or **B** "Download from HuggingFace" - **model name only** (e.g., "llama 7b") → search HuggingFace, list candidates, let user choose → handle as hf_id - **nothing** → ask for model_path or hf_id Once a local path is confirmed, read `model-inspection.md` and follow its rules (infer hf_id, parse config.json, detect model type, estimate size, compute TP candidates). Write findings to `$EVAL_STATE_DIR/current-eval.yaml`. ### Phase 0.5 — Environment preflight (fail-fast) Before Phase 1, run a 60-second environment probe to fail fast: ```bash which curl python3 # Probe both runtimes; do NOT fail if docker is missing — host mode covers that. docker --version >/dev/null 2>&1 && docker info >/dev/null 2>&1 \ && echo "docker_available=true" || echo "docker_available=false" python3 -c "import vllm; print('vllm', vllm.__version__)" 2>/dev/null \ && echo "host_vllm_available=true" || echo "host_vllm_available=false" python3 -c "import sglang; print('sglang', sglang.__version__)" 2>/dev/null \ && echo "host_sglang_available=true" || echo "host_sglang_available=false" ``` **Runtime selection (record `runtime:` in `current-eval.yaml`):** 1. If the caller passed `runtime` explicitly (`docker` or `host`), honour it; validate the matching tool exists. 2. Else if `docker_available=true`, use `docker` (default — production path). 3. Else if `backend=vLLM` and `host_vllm_available=true`, or `backend=SGLang` and `host_sglang_available=true`, fall back to `runtime=host` and **report the fallback in the unified confirmation**. 4. Else stop and report: docker is not available, and the requested backend is not importable on the host either. List which of (docker, vllm, sglang) is missing and instruct the user to install one. **ATOM is docker-only** — if `backend=ATOM` and docker is missing, stop here regardless of host imports. Host-mode constraints (announce in the unified confirmation): - `image` is ignored; the host's currently-installed framework version is what runs. - All later phases that show `docker exec ${CONTAINER} ` become `` directly on the host (the "Host runtime variant" call-out under each phase below). - `pip install` of frameworks is NOT performed — the host environment is used as-is. Missing a benchmark framework (e.g., `lm_eval`) → stop and ask the user to install it explicitly; do not pip install on the host without consent. **Session path setup (first thing in this phase):** generate a unique, per-session state directory to avoid collisions between concurrent evals. `bash` shell state does NOT persist across separate `Bash` tool calls — `EVAL_USER` / `EVAL_TS` exported in one call are gone in the next. The path must therefore be **resolved once and persisted to disk**, not re-exported each phase. ```bash EVAL_USER=$(whoami) EVAL_TS=$(date +%Y%m%d_%H%M%S) EVAL_STATE_DIR="/tmp/eval-state/${EVAL_USER}/${EVAL_TS}" mkdir -p "${EVAL_STATE_DIR}" echo "eval_state_dir: ${EVAL_STATE_DIR}" > "${EVAL_STATE_DIR}/current-eval.yaml" echo "${EVAL_STATE_DIR}" > /tmp/eval-state/.last-session-path-${EVAL_USER} ``` All subsequent state files live under `$EVAL_STATE_DIR`: - `$EVAL_STATE_DIR/current-eval.yaml` — phase progress (machine-readable YAML) - `$EVAL_STATE_DIR/server-runtime.md` — server state - `$EVAL_STATE_DIR/vllm-server.log` — server log (redirected here) - `$EVAL_STATE_DIR/eval-output.log` — eval framework output - `$EVAL_STATE_DIR/monitor.pid` — background monitor task IDs (Phase 3) **Recovering `$EVAL_STATE_DIR` in later phases**: every later Bash call starts with ```bash EVAL_STATE_DIR=$(cat /tmp/eval-state/.last-session-path-$(whoami)) ``` or reads the value from the first line of the most recent `current-eval.yaml`. Do not call `date +%Y%m%d_%H%M%S` again — it produces a different timestamp and a different (empty) directory. Record results in the `env_preflight` section of the state file (capture all probe outputs plus the chosen `runtime`). If neither `docker run` nor a native backend is usable per the Runtime selection rules above → **stop and report**, listing what is missing. ### Phase 1 — Info collection (fully autonomous) **Do not ask the user any questions during Phase 0, 0.5, or 1.** Collect all information autonomously, apply sensible defaults for anything missing, and present everything in a single unified confirmation at the end. The user makes one decision — approve or adjust. Defaults for missing info: - `image`: no model card / recipe specifies one → default by backend: vLLM `vllm/vllm-openai-rocm:latest`; ATOM `rocm/atom-dev:latest`; SGLang `lmsysorg/sglang:v0.5.11-rocm720-mi35x` - `benchmark`: unspecified and no model card recommendation → default to `gsm8k` - `backend`: unspecified → default to `vLLM` **Image availability check**: run `docker images --format '{{.Repository}}:{{.Tag}}'` and check if the chosen image already exists locally. Report the result in the unified confirmation (e.g., "Image: vllm/vllm-openai-rocm:latest (already pulled)" or "Image: vllm/vllm-openai-rocm:latest (needs pull)"). Fetch model card and look for: recommended Docker image, serving / launch command, evaluation command, recommended benchmark. **Model card fetch**: always use `curl` to get the raw README.md directly — returns raw markdown without needing HTML parsing or JS rendering. Apply the HuggingFace fetch rules above: ```bash curl -fsSL "${HF_BASE}//raw/main/README.md" \ ${HF_TOKEN:+-H "Authorization: Bearer $HF_TOKEN"} ``` **Fetch strategy — per-item priority chain with two-stage model card lookup:** Five pieces of info are collected: `image`, `launch_cmd`, `eval_cmd`, `eval_settings`, and `reference_scores`. Each one independently walks the priority chain below. Finding one item at a given level does NOT skip that level for other items — only skip lower levels for the specific item already found. `eval_settings` includes: `num_fewshot`, evaluation mode (`base`/`chat`, i.e. `local-completions`/`local-chat-completions`), and prompting strategy (`CoT`/`direct`). These settings determine how the eval command is constructed and are critical for reproducing published results. Example: model card provides `launch_cmd` but not `eval_cmd` → `launch_cmd` is settled. For `eval_cmd`, continue down the chain until found. Priority levels: 1. **Quantized model's own model card** — `curl -sL https://huggingface.co//raw/main/README.md` (e.g., `amd/MiniMax-M2.7-mxfp4`). Highest trust: quant-specific commands verified by the quantized model's author. 2. **Sibling quant variant's model card** — if the quantized model has no HF page or no useful commands, search for sibling quant variants under the same org: ```bash curl -fsSL "${HF_BASE}/api/models?search=/&author=" \ ${HF_TOKEN:+-H "Authorization: Bearer $HF_TOKEN"} ``` If a variant with a different quant strategy is found (e.g., `amd/Model-fp8` when evaluating `amd/Model-mxfp4`), fetch its model card. Commands from a sibling variant share the same architecture and only differ in quantization flags. Record `launch_cmd_source: similar_quant_card` and `reference_model: `. 3. **Base model's model card** — `curl -sL https://huggingface.co//raw/main/README.md` (e.g., `MiniMaxAI/MiniMax-M2.7`). Commands need adaptation: replace model path, may need quantization-specific adjustments. Record `launch_cmd_source: base_model_card`. 4. **vLLM Recipes** — use `base_hf_id` (base model org/name, not quantized variant). Recipes pages are JS-rendered; try `curl -sL https://recipes.vllm.ai//` and extract from `__NEXT_DATA__` or HTML. If not accessible, skip to next level. 5. **ATOM Recipes** / **SGLang Cookbook** — backend-specific sources 6. **Framework repos** / **InferenceX** — eval commands only 7. **Paper / technical report** — for `eval_cmd` and `eval_settings` only. When levels 1–6 do not provide an explicit eval command or eval settings, extract settings from the model's arxiv paper. See `eval-frameworks.md` §Paper Eval Settings Extraction for details. Record `eval_cmd_source: paper` and `eval_settings_source: paper`. 8. **Generic templates** from `eval-frameworks.md` — last resort. Record `launch_cmd_source: template`. Record each item and its source in the state file (e.g., `launch_cmd_source: quant_model_card`, `eval_cmd_source: template`). **Reference scores collection**: while fetching model cards and papers in the priority chain above, also collect benchmark reference scores for the target benchmark(s). Walk the lookup chain in `eval-frameworks.md` §Accuracy Comparison — it covers model card, base model card, paper/technical report, Open LLM Leaderboard, and similar-scale fallback. Record the score, source, and **eval setting** (num_fewshot, CoT/direct, base/chat) for each benchmark. This enables the unified confirmation to show expected accuracy and flag setting mismatches before the eval runs. **Note:** The local path may be a quantized variant (e.g., `/shareddata/amd/Model-mxfp4`), but Recipes uses the base model ID (`base_hf_id`). In Phase 3, replace the base model ID in the command with the local path. ### Phase 2 — Container **Host runtime variant:** if `runtime=host` (set in Phase 0.5), **skip this phase entirely**. Set `CONTAINER=""` (sentinel) in the state file so Phase 3+ branches detect host mode. Jump straight to Phase 3. **Docker runtime (default):** Read `backends.md` for the full `docker run` template. Name the container `eval---` (username from `whoami`). ### Phase 3 — Pick & adapt launch command **Starting point — use the highest-priority source found in Phase 1:** 1. Model card command → use as-is (highest trust) 2. Recipes / Cookbook command → use if model card had none 3. No command from any source → minimal template by backend: - **vLLM**: `vllm serve --tensor-parallel-size ` - **ATOM**: `python -m atom.entrypoints.openai_server --model --kv_cache_dtype fp8 -tp ` - **SGLang**: `python -m sglang.launch_server --model-path --tp ` **Model card command rule**: when a model card provides a launch command, preserve ALL flags from that command. The allowed/forbidden modifications table below only applies to changes YOU make on top of the model card command — do not strip flags that the model card author included (e.g., `--enforce-eager`, `--reasoning-parser`, `--mm-encoder-tp-mode`, env vars like `VLLM_ROCM_USE_AITER=1`). **Adaptation rules**: see `backends.md` §Cross-Backend Launch Adaptation Rules — applies to all three backends. Only the changes in that table are allowed on top of the source command; reactive flags (`--enforce-eager` etc.) come from `troubleshooting.md` only when an error matches. **TP priority**: see `model-inspection.md` §5 — recipe TP takes precedence over auto-inferred. **ATOM-specific notes:** ATOM uses `--kv_cache_dtype fp8` by default for memory efficiency. ATOM auto-detects quantization configs from HuggingFace model configs (FP8, MXFP4, INT8, INT4). For MTP speculative decoding, add `--method mtp --num-speculative-tokens 3`. ATOM also supports running as a vLLM plugin — if `atom` is installed alongside `vllm`, `vllm serve` automatically picks up ATOM's optimized kernels. **Pre-launch checks (immediately before launching):** 1. GPU index mapping: follow `model-inspection.md` §4 — build the rocm-smi → HIP index mapping via PCIe bus addresses. `rocm-smi` indices and `HIP_VISIBLE_DEVICES` indices are NOT the same. 2. GPU availability: use `rocm-smi --showmeminfo vram` for exact free VRAM, `--showuse` for GPU%, `--showpids` for processes. Identify idle GPUs by rocm-smi index, then convert to HIP indices using the mapping. 3. Select final TP: smallest value from Phase 0 TP candidates that fits the available idle GPU count and whose aggregate free VRAM ≥ model size × 1.3 4. Port selection: default to **8080** (not 8000 — commonly occupied by other services). If model card specifies a port, use that instead. Check with `lsof -i :` — if occupied, try 8081, 8082, etc. Add `--port ` to the launch command. 5. Write chosen GPUs (both rocm-smi and HIP indices), TP, and port to state file **Launch template — server log path is mandatory.** Phase 4/5/7 monitoring all assume the log is reachable at `${EVAL_STATE_DIR}/vllm-server.log`; do not deviate. Both runtimes write the server log directly to `${EVAL_STATE_DIR}/vllm-server.log` — for docker, this works because the `docker run` template in `backends.md` bind-mounts `${EVAL_STATE_DIR}` into the container at the same path. **Docker runtime (default):** ```bash docker exec -d ${CONTAINER} bash -c \ "HIP_VISIBLE_DEVICES=${GPUS} ${LAUNCH_CMD} > ${EVAL_STATE_DIR}/vllm-server.log 2>&1" ``` **Host runtime variant** (when `runtime=host`): ```bash # Launch directly on the host; PID captured into the state dir for later signals. ( HIP_VISIBLE_DEVICES=${GPUS} ${LAUNCH_CMD} > "${EVAL_STATE_DIR}/vllm-server.log" 2>&1 ) & echo "$!" > "${EVAL_STATE_DIR}/server.pid" ``` In host mode, every later step that says `docker exec ${CONTAINER} ` becomes just `` on the host; every "tail the in-container log" becomes `tail -F "${EVAL_STATE_DIR}/vllm-server.log"` (same path works in docker mode too, since the log file is bind-mounted). Write `$EVAL_STATE_DIR/server-runtime.md` immediately after launch (PID, port, container, image, full launch command). **Readiness wait (up to 300s):** use the poll template in `backends.md` §Readiness Poll Template — polls health every 5s with process liveness check, breaks immediately on process death instead of waiting 300s. **Background server monitor**: after the server is ready, launch the monitor via `Bash` with `run_in_background: true` and persist the returned `task_id`: ```bash # Bash, run_in_background: true. Capture task_id, then: echo "server_monitor=${task_id}" >> "${EVAL_STATE_DIR}/monitor.pid" # Both runtimes — log is bind-mounted into the container, so tail from host either way: tail -F "${EVAL_STATE_DIR}/vllm-server.log" \ | grep -E --line-buffered "Avg prompt throughput|Avg generation throughput|ERROR|error|OOM|Killed|CUDA|torch.OutOfMemoryError|engine is dead" ``` `TaskStop` every ID in `monitor.pid` at every exit path: Phase 5 retry (before each new launch), Phase 6 early-exit, Phase 7 cleanup. Skipping this leaks a background `tail -F` per session. ### Phase 4 — Smoke test + quality gate Follow `backends.md` §Smoke Test Templates — verify model name via `/v1/models`, then send a simple inference request (chat or base). Both HTTP 200 and reasonable content are required. Either gate fails → enter Phase 5. ### Phase 5 — Fix if needed Entered on Phase 3 launch failure or Phase 4 gate failure. Mode comes from `launch_cmd_source` in `current-eval.yaml`. **Reproduce mode** (`launch_cmd_source ∈ {quant_model_card, similar_quant_card}`) — the command is verified for this arch+quant combo, so do **not** auto-fix: 1. `TaskStop` every ID in `$EVAL_STATE_DIR/monitor.pid` 2. Report: (a) core error line, (b) analysis from `troubleshooting.md`, (c) suggested fix 3. Wait for the user to choose explicitly: **`fall through to bringup`** (re-enter Phase 5 in Bringup mode) or **`abort`** (stop, offer Phase 7 cleanup). Do not silently fall through. Do not retry without instruction. **Bringup mode** (`launch_cmd_source ∈ {base_model_card, recipes, template}`) — run the fix cycle below. **Three rules (bringup only):** one fix at a time (no flag stacking); rerun the exact failing step (Phase 3 or 4, no skipping); max 3 attempts then stop and report the full log. **Fix cycle:** 1. Get server log or traceback, locate the **core error line** 2. Match error signature against `troubleshooting.md` 3. Apply **one** minimal fix: - env var / launch flag → apply directly - source code / `config.json` → show full diff, wait for user approval - no match → search `github.com/vllm-project/vllm/issues`; same approval rule for any source mod - kernel / hardware-unsupported → stop and inform user 4. Rerun the exact failing step 5. Record `{attempt, error, fix, result}` in `current-eval.yaml` ### Phase 6 — Eval + report Three steps, in order. Do not present the final report until all three are done. **Step 1 — Run eval.** Read `eval-frameworks.md` for command templates, task type classification, and `num_fewshot` lookup. Format the raw eval output as shown in the Outputs section above. **Step 2 — Accuracy comparison.** Use the `reference_scores` already collected in Phase 1. If Phase 1 found a reference, use it directly. If not, do a fallback lookup per `eval-frameworks.md` §Accuracy Comparison (paper → Open LLM Leaderboard → similar-scale model). Build the comparison table with the eval setting from the reference source. If the reference's eval setting differs from what was actually used, note the difference explicitly. No benchmark may be silently omitted — if no reference exists, write "N/A" with reason. **Step 3 — Anomaly detection.** Flag any delta > 5% absolute. See `eval-frameworks.md` §Anomaly detection for the diagnostic table. ### Phase 7 — Post-eval cleanup After the evaluation report is produced, ask the user how to handle the running resources: 0. **Stop background monitors** — for every line in `$EVAL_STATE_DIR/monitor.pid`, call `TaskStop` on the recorded `task_id`. Truncate the file after. 1. **Stop server?** - Docker runtime: kill the inference process inside the container (`docker exec ${CONTAINER} pkill -f "vllm serve|sglang.launch_server|atom.entrypoints"`), then verify the port is released via `lsof -i :${PORT}`. - Host runtime: stop the PID in `${EVAL_STATE_DIR}/server.pid` following the "Never kill other users' processes" rule above (L72-76) — verify the PID is alive and still our backend, then SIGTERM → wait → SIGKILL only if needed. Verify port with `lsof -i :${PORT}`. 2. **Remove container?** — `docker stop` + `docker rm`. Skip in host runtime. 3. **Rollback file edits?** — if `source_edits` is non-empty in `$EVAL_STATE_DIR/current-eval.yaml`, offer to restore each `.bak.` file: `cp .bak. ` 4. **Keep everything** — user may want to re-run or debug Always ask — never auto-kill. After cleanup (or skip), update `$EVAL_STATE_DIR/server-runtime.md`. ## State Files `$EVAL_STATE_DIR` is set in Phase 0.5 to `/tmp/eval-state///` — unique per session and user. Every later phase **reads** the path from `/tmp/eval-state/.last-session-path-$(whoami)` rather than re-deriving from `whoami` + timestamp. Primary: `$EVAL_STATE_DIR/current-eval.yaml` — updated after each Phase. Server: `$EVAL_STATE_DIR/server-runtime.md` — written at Phase 3 launch, kept current. **Fallback**: if `/tmp/eval-state/` is not writable, use `./.eval-state//` in the current working directory. Phase 0.5 preflight probes this and writes the resolved path as the first field of `current-eval.yaml` and into the `.last-session-path-` pointer. ## References | File | When to read | |------|-------------| | `model-inspection.md` | Phase 0, for config parsing / GPU probing / TP math; §2 for all HF HTTP requests | | `backends.md` | Phase 2-3, for docker setup + serving commands (all backends) | | `troubleshooting.md` | Phase 5, on launch failure or smoke test failure | | `eval-frameworks.md` | Phase 6, for eval commands, accuracy comparison lookup, and anomaly detection | | `templates.md` | Phase 1 (pre-run confirmation) and Phase 6 (assembling `eval_report.md`) |