--- name: test-macafm description: Run the maclocal-api (AFM/MLX) test suite — automated assertions and smart analysis. Use when asked to test, validate, regression-check, or benchmark AFM before release, after code changes, or for model onboarding. --- # test-macafm Run the maclocal-api test suite: automated pass/fail assertions and smart analysis (the smart suite's AI judge is opt-in — default off; ask the user before enabling it). ## Triggers Use this skill when the user asks to: - **Test** or **validate** the server (e.g., "run the tests", "test AFM", "validate the build") - **Regression check** after code changes - **Onboard a new model** (verify it works correctly with the server) - **Release check** before tagging or pushing - **Benchmark** or **profile** model performance ## First Questions to Ask 1. **Model** — Which model to test? (Ask if not specified. Default: whatever's loaded.) 2. **Tier** — smoke / standard / full? (Suggest based on context.) 3. **Binary path** — Default `.build/release/afm`. Ask if user has a custom build location. 4. **Port** — Default 9998. Ask if user's server is on a different port. 5. **Server running?** — Is the server already running, or should tests start it? ## Tier Decision Tree | Tier | Time | When to use | What runs | |------|------|-------------|-----------| | **smoke** | ~2 min | Quick sanity check, any small model, CI | `test-assertions.sh --tier smoke` | | **standard** | ~15 min | After feature changes, mid-size model | `test-assertions.sh --tier standard` | | **full** | ~60 min | Release validation, production model | `test-assertions.sh --tier full` + `mlx-model-test.sh` (smart suite; AI judge opt-in — ask the user) with `test-llm-comprehensive.txt` + promptfoo agentic evals | **Quick guide:** - "Just run a quick test" → smoke - "Test before merging" → standard - "Full release validation" or "onboard new model" → full - User doesn't specify → suggest standard ## Execution Workflow ### 1. Build Check ```bash # Ensure release build is current swift build -c release ``` ### 2. Start Server (if not running) ```bash MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache \ .build/release/afm mlx -m MODEL --port 9998 \ --tool-call-parser afm_adaptive_xml \ --enable-prefix-caching \ --enable-grammar-constraints & # Wait for server to be ready until curl -sf http://127.0.0.1:9998/v1/models >/dev/null 2>&1; do sleep 1; done ``` **Recommended flags for testing:** - `--tool-call-parser afm_adaptive_xml` — best tool call parser with JSON-in-XML fallback - `--enable-prefix-caching` — 67-79% prompt token savings on repeated requests - `--enable-grammar-constraints` — EBNF constrained decoding forces valid XML tool calls, improving success from 60% to 100% on realistic workloads ### 3. Run Automated Assertions ```bash ./Scripts/test-assertions.sh --tier TIER --model MODEL --port 9998 ``` **Interpret results immediately.** If any FAIL, investigate before proceeding. ### 4. Run Smart Analysis (full tier only) The smart analysis harness manages its own server (port 9877) — do NOT pass `--port`. It uses `test-llm-comprehensive.txt` which has an `[all]` baseline prompt and `[@ label]` template sections. **The AI judge is OPT-IN — default OFF.** By default, run the smart suite WITHOUT an AI judge: it executes every prompt and records the model's raw outputs to the report for manual review, with no `claude`/`codex` scoring. Before running, **ask the user** (e.g. via AskUserQuestion) whether to enable the AI judge — it adds latency/cost and invokes an external CLI: > "Run the smart suite with an AI judge (claude) scoring each response, or without it > (just record outputs for manual review)? Default: without." Default — **no AI judge** (records outputs only; omit `--smart` entirely): ```bash AFM_BIN=.build/release/afm ./Scripts/mlx-model-test.sh \ --model MODEL \ --prompts Scripts/test-llm-comprehensive.txt ``` Only if the user opts in — append `--smart 1:claude` (the `--smart` flag accepts a batch mode prefix and tool list): ```bash AFM_BIN=.build/release/afm ./Scripts/mlx-model-test.sh \ --model MODEL \ --prompts Scripts/test-llm-comprehensive.txt \ --smart 1:claude ``` **Smart analysis options (only when the AI judge is enabled):** - `--smart claude` or `--smart codex` — batch mode 0 (one big swoop, may fail on large test suites) - `--smart 1:claude` or `--smart 1:codex` — batch mode 1 (test-by-test, more reliable) - `--smart 1:claude,codex` — run multiple AI judges - `--tests 1,5,10` — run only specific test numbers (1-indexed) **Note:** The `[all]` prompt runs for every test variant. With high `max_tokens` (e.g., 32768 on code tests), thinking models may generate very long reasoning for the baseline prompt. Total run time for full suite: ~45-90 min depending on model speed. ### 5. Run GPU Shader Profile (full tier, or when investigating perf) Generates an interactive HTML report with measured DRAM bandwidth, GPU utilization/power timelines, and per-kernel Metal shader names from xctrace Shader Timeline. **One-time setup** (creates custom Instruments template with Shader Timeline enabled): ```bash python3 Scripts/create-shader-template.py ``` **Run the profile** (no server needed — uses single-prompt mode): ```bash python3 Scripts/gpu-profile-report.py MODEL [max_tokens] [prompt] # Default: 4096 tokens, built-in GPU analysis prompt # Example: python3 Scripts/gpu-profile-report.py mlx-community/Qwen3.5-35B-A3B-4bit ``` This does everything automatically: 1. Warms up mactop (bandwidth monitor, no sudo) 2. Runs inference with `--gpu-profile --gpu-trace 15` 3. Collects 300ms bandwidth/GPU/power samples via PTY during inference 4. Extracts shader kernel names from the xctrace trace 5. Generates `/tmp/afm-gpu-profile.html` and opens in browser **Or use individual flags** on any AFM invocation: ```bash afm mlx -m MODEL --gpu-profile -s "prompt" # Zero-overhead stats afm mlx -m MODEL --gpu-profile-bw -s "prompt" # + mactop bandwidth (~5s) afm mlx -m MODEL --gpu-trace 10 -s "prompt" # xctrace shader trace ``` **Live bandwidth monitor** (run in separate terminal during server requests): ```bash ./Scripts/gpu-profile.sh bandwidth ``` **What the report shows:** - Device info (chip, memory, architecture) - Prefill/decode tok/s with exact timing - Memory breakdown (model weights vs KV cache) - DRAM bandwidth timeline chart (measured via mactop) - GPU utilization & power timeline chart - Per-kernel Metal shader names (from Shader Timeline) - Exact command line for reproducibility **What to look for:** - GPU utilization <100% during decode → CPU-GPU pipeline bubbles - Bandwidth utilization >80% → memory-bound, kernel optimization won't help - Bandwidth utilization <20% with MoE model → normal (only active experts read) - Key kernels: `affine_qmv_fast` (decode bottleneck), `steel_gemm_fused` (prefill), `sdpa_vector` (attention) ### 5b. API-Based GPU Profiling (per-request, no CLI flags needed) Clients can request GPU profiling data via the `X-AFM-Profile` HTTP header. No server flags required — works on any running AFM server. **Two levels:** ```bash # Summary: GPU power, memory, bandwidth, tok/s curl http://127.0.0.1:9999/v1/chat/completions \ -H "X-AFM-Profile: true" \ -d '{"model":"m","messages":[{"role":"user","content":"Hi"}]}' # Extended: summary + 300ms time-series samples (for charts/dashboards) curl http://127.0.0.1:9999/v1/chat/completions \ -H "X-AFM-Profile: extended" \ -d '{"model":"m","messages":[{"role":"user","content":"Hi"}]}' ``` **Response fields (`afm_profile`):** - `gpu_power_avg_w` / `gpu_power_peak_w` — GPU power via native IOReport (no mactop) - `memory_weights_gib` / `memory_kv_gib` / `memory_peak_gib` — memory breakdown in GiB - `prefill_tok_s` / `decode_tok_s` — throughput - `est_bandwidth_gbs` — DRAM bandwidth from IOReport power (calibrated at startup via MLX GPU stress) - `chip` / `theoretical_bw_gbs` — hardware context - `gpu_samples` — number of 300ms readings taken **Extended adds (`afm_profile_extended`):** - `summary` — same as `afm_profile` - `samples[]` — per-300ms readings: `{t, bw_gbs, gpu_pct, gpu_power_w, dram_power_w}` **How it works internally:** - IOReport `Energy Model` + `GPU Stats` channels sampled every 300ms via DispatchSource timer - DRAM bandwidth derived from DRAM power using chip-specific calibration constant - Calibration runs once at startup: 1 GiB MLX GPU stress test (~2s, async, non-blocking) - Per-request isolation: concurrent profiled requests are guarded (second request skips gracefully) - Zero overhead when header not sent (one string lookup per request) - Works for both streaming (SSE event before `[DONE]`) and non-streaming **What to look for:** - `gpu_power_peak_w` ~28W during decode on M3 Ultra (matches mactop) - `est_bandwidth_gbs` ~170-180 GB/s for Qwen3.5-35B-A3B-4bit (21% of 800 GB/s theoretical) - Short requests (<300ms): at least 1 sample (timer first-fires at 100ms) - `afm_profile` absent from response when header not sent (no null pollution) ### 6. Run Promptfoo Agentic Evals (full tier, or when validating tool calling / structured output) The promptfoo agentic eval suite tests AFM's tool-calling and structured-output across multiple server configurations and real-world agent framework schemas. It manages its own server lifecycle. **Prerequisites:** `promptfoo` CLI must be installed (`npm install -g promptfoo`). **Run the full suite:** ```bash AFM_MODEL=MODEL \ AFM_BINARY=.build/arm64-apple-macosx/release/afm \ MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache \ ./Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh all ``` **Run individual suites:** ```bash # Just structured output tests ./Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh structured # Just tool calling (all 3 parser profiles) ./Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh toolcall # Just grammar constraint validation (8 server phases) ./Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh grammar-constraints # Just one agent framework ./Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh opencode ``` **Available modes:** `all`, `structured`, `structured-stress`, `toolcall`, `toolcall-quality`, `grammar-constraints`, `agentic`, `frameworks`, `opencode`, `pi`, `openclaw`, `hermes`, `default`, `adaptive-xml`, `adaptive-xml-grammar` #### Suite Coverage (~137 test cases across 16 configs) | Suite | Tests | Profiles | What it validates | |-------|-------|----------|-------------------| | **structured** | 6 | 1 (api json_schema) | `response_format=json_schema` strict compliance | | **structured-stress** | 4 | 1 | Nested arrays, enums, nullable types in schema | | **toolcall** | 7 | 3 (default, adaptive-xml, grammar) | Basic tool call parsing: weather, time, multi-tool | | **toolcall-quality** | 6 | 3 | BFCL-inspired when-to-call decisions (should model use a tool?) | | **grammar-constraints** | 17 | 8 server phases | Schema + tool enforcement across: no-grammar, grammar-enabled, adaptive-xml, concurrent, prefix-cache, mixed-strict, header downgrade/enforce | | **agentic** | 4 | 3 | Multi-turn coding workflow tool chains | | **frameworks** | 8 | 3 | Agent framework tool shapes (OpenCode, Pi, OpenClaw, Hermes) | | **opencode** | 37 | 3 | OpenCode built-in tools (primary-source derived) | | **pi** | 20 | 3 | Pi coding-agent tools | | **openclaw** | 12 | 3 | OpenClaw tool coverage | | **hermes** | 12 | 3 | Hermes agentic framework tools | #### Server Profiles (managed automatically by the script) | Profile | AFM flags | Purpose | |---------|-----------|---------| | `default` | (none) | Baseline: auto-detected tool call format | | `adaptive-xml` | `--tool-call-parser afm_adaptive_xml` | Adaptive XML with JSON-in-XML fallback | | `adaptive-xml-grammar` | `--tool-call-parser afm_adaptive_xml --enable-grammar-constraints` | Adaptive XML + EBNF grammar enforcement | | `grammar-enabled` | `--enable-grammar-constraints` | Grammar without adaptive XML | | `grammar-enabled-adaptive-xml` | Both flags | Regression guard: grammar + adaptive XML | | `grammar-enabled-concurrent` | `--enable-grammar-constraints --concurrent 2` | Grammar under concurrency | | `grammar-enabled-prefix-cache` | `--enable-grammar-constraints --enable-prefix-caching` | Grammar + prefix caching interaction | | `grammar-enabled-concurrent-cache` | All three flags | Full feature stack | #### Custom Provider & Judges - **`providers/afm_provider.mjs`** — Custom promptfoo provider with two transports: `api` (OpenAI-compatible HTTP) and `cli-guided-json` (direct binary invocation). Supports extract modes: `content`, `tool_calls`, `normalized_message`, `full_response`. Captures `responseHeaders` for grammar header assertions. - **`judges/assert-grammar-header.mjs`** — Validates `X-Grammar-Constraints` response header: expects `"downgraded"` when grammar not available, absent when grammar active. - **`judges/classify-failures.mjs`** — Post-run AI-based failure classifier: categorizes each failure as `afm_bug` (server/protocol), `model_quality` (wrong tool/args), or `harness_bug` (false negative). #### Environment Variables | Variable | Default | Purpose | |----------|---------|---------| | `AFM_MODEL` | `mlx-community/Qwen3.5-35B-A3B-4bit` | Model to test | | `AFM_BINARY` | `.build/arm64-apple-macosx/release/afm` | Binary path | | `AFM_PROMPTFOO_OUT_DIR` | `/Volumes/edata/promptfoo/data/maclocal-api/current` | Report output dir | | `AFM_PROMPTFOO_PORT` | `9999` | Server port | | `MACAFM_MLX_MODEL_CACHE` | (none) | Model cache dir | #### Output JSON reports per suite+profile in `$AFM_PROMPTFOO_OUT_DIR`: - `structured-MODEL_SLUG.json` - `toolcall-{default,adaptive-xml,adaptive-xml-grammar}-MODEL_SLUG.json` - `grammar-{schema,tools}-{no-grammar,grammar-enabled,adaptive-xml,concurrent,prefix-cache}-MODEL_SLUG.json` - `{agentic,frameworks,opencode,pi,openclaw,hermes}-{default,adaptive-xml,adaptive-xml-grammar}-MODEL_SLUG.json` ### 7. Review Reports - Assertion report: `test-reports/assertions-report-*.html` - Smart analysis: `test-reports/smart-analysis-{tool}-*.md` - HTML report: `test-reports/mlx-model-report-*.html` - GPU profile: `/tmp/afm-gpu-profile.html` (+ `/tmp/afm-metal.trace` for Instruments) - JSONL data: `test-reports/assertions-report-*.jsonl`, `test-reports/mlx-model-report-*.jsonl` - Promptfoo evals: `$AFM_PROMPTFOO_OUT_DIR/{suite}-{profile}-MODEL_SLUG.json` (default: `/Volumes/edata/promptfoo/data/maclocal-api/current/`) ### 7. Stop Server (if we started it) ```bash kill %1 # or whatever the background job is ``` ## Interpreting Results ### Assertion Test Failures | Group | Common failures | What to check | |-------|----------------|---------------| | **Stop** | Stop string found in output | Check `MLXModelService.swift` stop buffer logic, streaming vs non-streaming paths | | **Logprobs** | Schema invalid, logprob > 0 | Check `resolveLogprobs()` and `buildChoiceLogprobs()` | | **Think** | `` tags in content | Check `extractThinkContent()` and `extractThinkTags()` | | **Tools** | No tool_calls, invalid JSON args | Check `extractToolCallsFallback()`, model's tool call format | | **Cache** | cached_tokens always 0 | Check `enablePrefixCaching`, `findPrefixLength()`, `PromptCacheBox` | | **Concurrent** | Non-200 responses | Check `SerialAccessContainer` locking, request queuing | | **Error** | Wrong HTTP status codes | Check controller validation logic | | **Kwargs** | Thinking not disabled by `enable_thinking: false` | Check `chat_template_kwargs` merging into `additionalContext` in `MLXModelService.swift` | | **Perf** | Low tok/s, high TTFT | Check model quantization, Metal kernel performance | | **OpenAI-compat** | Stream usage chunk missing, logprobs absent | Check `StreamingUsageChunk` encoding, empty choices on final chunk | | **Guided JSON** | Schema validation failure, invalid JSON | Check `--guided-json` / `response_format` pipeline, grammar constraints | | **Batch** | Garbage output, wrong answers at B>1 | Check BatchScheduler, KV cache isolation, mask generation | ### Smart Analysis False Positives Known patterns where AI judges score incorrectly (see `references/interpreting-scores.md`): - Stop sequences truncating output scored as "low quality" — truncation IS the expected behavior - Empty content when stop fires on first visible token — correct behavior - JSON mode not constraining thinking models — prompt injection, not grammar-constrained - "Missing reasoning" when model doesn't support `` — correct, not a bug - Thinking model consuming entire `max_tokens` budget on reasoning with empty visible content — model behavior, not a server bug - `[all]` baseline prompt scored low when it runs with a code/math test's high `max_tokens` and system prompt — irrelevant context for the baseline prompt ### Promptfoo Eval Failures | Category | Typical pass rate | What failures mean | |----------|-------------------|-------------------| | **structured, structured-stress** | 100% | Server bug in `response_format` pipeline — investigate immediately | | **toolcall** (all profiles) | 100% | Server bug in tool call parsing — investigate immediately | | **toolcall-quality** | ~80% | Model chose wrong tool or missed when-to-call — model quality, not server | | **grammar-schema / grammar-tools** (non-concurrent) | 100% | Grammar constraint enforcement broken — server bug | | **grammar-schema / grammar-tools** (concurrent) | ~50-70% | Known race condition in `--concurrent 2` grammar path — not release blocker | | **grammar-header / grammar-mixed** | 100% | `X-Grammar-Constraints` header or mixed-strict wiring broken — server bug | | **agentic** | ~75-100% | Multi-turn failures are usually model quality; 0% pass = server bug | | **frameworks** | 100% | Framework tool shapes must parse correctly — server bug if failing | | **opencode** | ~70-80% | Complex 37-tool scenarios; model can't always pick correct tool — model quality | | **pi** | ~80-90% | Model prompt injection resistance varies — model quality | | **openclaw** | ~80-85% | Model quality on OpenClaw-specific schemas | | **hermes** | ~90-100% | Hermes format failures on adaptive-xml profiles = parser difference, not bug | **Key rule:** `structured`, `toolcall`, `grammar-*` (non-concurrent), `frameworks` suites should be **100% pass**. Any failure there is a server bug. Everything else has model-quality variance. **Post-run failure classification** (optional): Run `judges/classify-failures.mjs` on any result JSON to get AI-based `afm_bug` vs `model_quality` vs `harness_bug` classification. ### When to Escalate - **SDPA regression**: NaN or garbage in long-context tests → check MLX version, see MEMORY.md - **Tool call format mismatch**: Unknown format → check `ToolCallFormat.infer()` and model's config.json - **Build failure**: Vendor patch conflict → run `Scripts/apply-mlx-patches.sh --check` ## Concurrency Benchmark Full-harness concurrency sweep that starts the server, runs warmup, tests all concurrency levels, collects GPU metrics via mactop, saves JSON results, and generates a comparison chart. ### Script `Scripts/benchmarks/benchmark_afm_vs_mlxlm.py` ### Usage ```bash # AFM-only concurrency sweep (recommended for quick benchmarks) python3 Scripts/benchmarks/benchmark_afm_vs_mlxlm.py --afm-only # Full AFM vs mlx-lm comparison (both servers, fair A/B) python3 Scripts/benchmarks/benchmark_afm_vs_mlxlm.py # Re-generate graph from existing results python3 Scripts/benchmarks/benchmark_afm_vs_mlxlm.py --graph python3 Scripts/benchmarks/benchmark_afm_vs_mlxlm.py --graph Scripts/benchmark-results/FILE.json ``` ### What it does 1. Detects hardware (chip, memory) 2. Starts server(s) with `--concurrent N` 3. 60s GPU settle + multi-round warmup (JIT kernel compilation) 4. Sweeps concurrency levels: `[1, 2, 4, 8, 12, 16, 20, 24, 32, 40, 50]` 5. At each level: fires N simultaneous streaming 4096-token requests, measures aggregate tok/s, per-request tok/s, GPU power/temp/usage via mactop 6. Saves JSON to `Scripts/benchmark-results/concurrency-benchmark-TIMESTAMP.json` 7. Generates PNG chart to `Scripts/benchmark-results/concurrency-benchmark-TIMESTAMP.png` ### Configuration (top of script) | Variable | Default | Purpose | |----------|---------|---------| | `MODEL_ID` | `mlx-community/Qwen3.5-35B-A3B-4bit` | Model to benchmark | | `MAX_TOKENS` | 4096 | Tokens per request (forces long decode) | | `MAX_CONCURRENT` | 50 | `--concurrent` flag value (must be >= max level) | | `LEVELS` | `[1,2,4,8,12,16,20,24,32,40,50]` | Concurrency levels to test | | `AFM_PORT` | 9999 | Port for AFM server | ### Reference results (March 18, v0.9.7, M3 Ultra 512GB, --concurrent 28) ``` B Agg t/s Per-req Wall GPU% GPU W 1 118.7 118.7 34.5s 94% 28.5W 2 193.9 97.0 42.2s 93% 41.6W 4 298.4 74.6 54.9s 97% 62.7W 8 407.3 50.9 80.5s 96% 75.5W 12 493.4 41.1 99.6s 98% 83.4W 16 573.9 35.9 114.2s 99% 88.2W 20 581.6 29.1 140.8s 98% 79.1W 24 629.6 27.4 149.6s 99% 83.2W ``` ### Additional batch validation scripts | Script | Purpose | |--------|---------| | `Scripts/feature-mlx-concurrent-batch/batch_stress_mactop.py` | Quick stress test at arbitrary concurrency (client-only, needs running server on port 9876) | | `Scripts/feature-mlx-concurrent-batch/batch_stress_ioreg.py` | Same but uses ioreg for GPU stats (less accurate) | | `Scripts/feature-mlx-concurrent-batch/validate_responses.py` | Known-answer correctness at B={1,2,4,8} | | `Scripts/feature-mlx-concurrent-batch/validate_mixed_workload.py` | Mixed short+long workload batch validation | | `Scripts/feature-mlx-concurrent-batch/validate_multiturn_prefix.py` | Multi-turn prefix cache under concurrency | ## Key File Reference | File | Purpose | |------|---------| | `Scripts/benchmarks/benchmark_afm_vs_mlxlm.py` | Full concurrency benchmark harness (server lifecycle, warmup, sweep, GPU metrics, chart generation) | | `Scripts/test-assertions.sh` | Automated pass/fail assertion tests (unit/smoke/standard/full tiers, includes `swift test`) | | `Scripts/test-llm-comprehensive.txt` | Comprehensive smart analysis test suite (model-generic, `[@ label]` template mode, has `[all]` baseline) | | `Scripts/test-Qwen3.5-35B-A3B-4bit.txt` | Model-specific test suite for Qwen3.5-35B-A3B-4bit (same tests as comprehensive, hardcoded model) | | `Scripts/test-edge-cases.txt` | Legacy smart analysis test prompts (smaller set) | | `Scripts/test-sampling-params.sh` | Sampling parameter tests (seed, temp, top_p, etc.) | | `Scripts/test-structured-outputs.sh` | JSON schema / structured output tests | | `Scripts/test-tool-call-parsers.py` | Unit tests for tool call parsing | | `Scripts/mlx-model-test.sh` | Test harness: runs prompts, collects results, generates reports | | `Scripts/test-chat-template-kwargs.sh` | Standalone chat_template_kwargs tests (includes --no-think CLI + precedence) | | `Scripts/regression-test.sh` | Quick regression smoke test | | `Scripts/feature-codex-optimize-api/test-openai-compat-evals.py` | OpenAI-python SDK compatibility evals (non-stream, stream, logprobs, vllm bench) | | `Scripts/feature-codex-optimize-api/test-guided-json-evals.py` | Guided JSON / structured output evals (API, streaming, CLI, SDK parse, edge cases) | | `Scripts/feature-mlx-concurrent-batch/validate_responses.py` | Batched generation correctness: known-answer questions at B={1,2,4,8} | | `Scripts/feature-mlx-concurrent-batch/validate_mixed_workload.py` | Mixed short+long workload batch validation with GPU metrics | | `Scripts/feature-mlx-concurrent-batch/validate_multiturn_prefix.py` | Multi-turn prefix cache validation under concurrency | | `Scripts/gpu-profile-report.py` | Full GPU shader profiling harness: mactop BW + --gpu-profile + --gpu-trace + HTML report | | `Scripts/gpu-profile.sh` | GPU profiling helpers: bandwidth monitor, capture, trace, power | | `Scripts/create-shader-template.py` | One-time: patches Metal System Trace template for per-kernel shader names | | `Tests/MacLocalAPITests/StreamingUsageChunkTests.swift` | Unit tests: streaming usage chunks, finish reasons, Foundation commonPrefixLength | | `Tests/MacLocalAPITests/ConcurrentBatchTests.swift` | Unit tests: RequestSlot, StreamChunk, BatchScheduler internals | | `Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh` | Promptfoo agentic eval orchestrator: 11 modes, 8 server profiles, 16 configs | | `Scripts/feature-promptfoo-agentic/providers/afm_provider.mjs` | Custom promptfoo provider: api + cli-guided-json transports, 4 extract modes | | `Scripts/feature-promptfoo-agentic/judges/assert-grammar-header.mjs` | Custom assertion: validates X-Grammar-Constraints response header | | `Scripts/feature-promptfoo-agentic/judges/classify-failures.mjs` | AI-based failure classifier: afm_bug vs model_quality vs harness_bug | | `Scripts/feature-promptfoo-agentic/promptfooconfig.*.yaml` | 16 promptfoo config files (~137 test cases total) | | `Scripts/feature-promptfoo-agentic/datasets/` | 16 YAML dataset files across structured, toolcall, grammar, agentic directories | ## Validation Checklist ### Smoke Tier - [ ] Server reachable, model loaded - [ ] Basic completion returns content - [ ] Stop sequences work (absent from output, correct finish_reason) - [ ] Logprobs schema valid - [ ] Think extraction works (if model supports it) - [ ] Basic tool call works - [ ] Error handling (empty messages, malformed JSON) ### Standard Tier (adds) - [ ] All smoke checks - [ ] Streaming stop sequence parity - [ ] Streaming logprobs - [ ] Prompt cache: cached_tokens=0 first, >0 second - [ ] Concurrent requests (2 and 3 simultaneous) - [ ] Multi-tool calls - [ ] Additional stop edge cases - [ ] chat_template_kwargs: `enable_thinking=false` disables thinking (if model supports it) - [ ] chat_template_kwargs: streaming parity - [ ] chat_template_kwargs: default behavior unaffected - [ ] OpenAI-compat evals: `test-openai-compat-evals.py` (non-stream, stream, logprobs, usage chunk) - [ ] Guided JSON evals: `test-guided-json-evals.py` (API schema, streaming schema, SDK parse) ### Full Tier (adds) - [ ] All standard checks - [ ] Performance: TTFT < 5s, tok/s > 1 - [ ] Long context (2K, 4K tokens) no crash/NaN - [ ] Smart analysis: test-llm-comprehensive.txt (AI judge opt-in — default off, ask the user; enable with `--smart 1:claude` / `--smart 1:codex`) - [ ] Streaming parity (assembled content matches non-streaming) - [ ] Cache timing improvement visible - [ ] Batch correctness: `validate_responses.py` at B={1,2,4,8} - [ ] Batch mixed workload: `validate_mixed_workload.py` (short+long decode, GPU metrics) - [ ] Batch prefix cache: `validate_multiturn_prefix.py` (multi-turn conversations under concurrency) - [ ] GPU shader profile: `gpu-profile-report.py` (bandwidth, power, kernel names, HTML report) - [ ] API profile: `X-AFM-Profile: true` returns `afm_profile` with GPU power + bandwidth - [ ] API profile: `X-AFM-Profile: extended` returns `afm_profile_extended` with samples array - [ ] API profile: no header → no `afm_profile` fields in response (no null pollution) - [ ] API profile: streaming → profile SSE event before `[DONE]` - [ ] API profile: concurrent profiled requests → second skips gracefully - [ ] Promptfoo structured: 100% pass (json_schema + stress) - [ ] Promptfoo toolcall: 100% pass (all 3 profiles) - [ ] Promptfoo grammar-constraints (non-concurrent): 100% pass - [ ] Promptfoo frameworks: 100% pass (all 3 profiles) - [ ] Promptfoo opencode/pi/openclaw/hermes: >70% pass (model quality variance expected) - [ ] Promptfoo grammar-header: downgrade/enforce headers correct