# InferenceX AgentX MVP Benchmark This page walks you through running the **AgentX MVP** benchmark in AIPerf. It's aimed at someone who hasn't worked with the scenario before — after a short orientation, you'll get a copy-pasteable command, then explanations of what it does and why. --- ## What Is AgentX MVP? AgentX MVP is a multi-turn, agentic-coding benchmark proposed by SemiAnalysis as part of their InferenceX effort. The idea: instead of measuring an inference server with synthetic 1-turn prompts, measure it with realistic *coding-agent sessions* — long conversations with KV-cache reuse and inter-turn think time. Sessions come from the public **Weka agentic-coding trace corpus** captured by Callan Fox ([kv-cache-tester](https://github.com/callanjfox/kv-cache-tester)), which records real Claude Code sessions byte-for-byte. AgentX MVP runs against the current **with-subagents** corpus, where parent coding sessions can spawn helper conversations. Most rejoin the parent before it resumes; a subagent with no following parent turn runs in the background and the parent does not wait for it. See [the Weka tutorial](weka-trace.md) for the source format and the SPAWN/JOIN mapping. AgentX MVP is essentially a *recipe* on top of those traces: a fixed set of replay rules so two different teams running on two different servers produce results you can compare. Things like "preserve each trace's original request timing", "never leave the whole system idle for more than 10 seconds", "the server must be allowed to generate full responses (no early stop)", "warm up the cache before measuring", and so on. AIPerf bundles every one of those rules into a single CLI flag: `--scenario inferencex-agentx-mvp`. When you pass that flag, AIPerf locks the relevant settings and rejects conflicting flags. It also stamps a `submission_valid` field onto the JSON outputs (per-run and aggregate) as a quick sanity check that the run followed the locked rules. --- ## Quick Start You'll need: - An **OpenAI-compatible inference server** running and reachable. `--url` takes a bare `host:port` (`http://` is prepended automatically) or a full scheme-prefixed URL; the endpoint path (e.g. `/v1/chat/completions`) is appended automatically based on `--endpoint-type`. If the server requires authentication, add `--api-key ` (sent as a `Bearer` token). - AIPerf installed: `pip install aiperf` (or run it ad hoc with `uvx aiperf`). - The model's **HuggingFace tokenizer** resolvable from `--model`: the Weka loaders rebuild every prompt through it. For a gated repo, export `HF_TOKEN`; if `--model` isn't a resolvable HF repo name (a local server alias, a private build), pass `--tokenizer ` explicitly. See [Tokenizer Auto-Detection](../reference/tokenizer-auto-detection.md). The trace corpus is fetched automatically from HuggingFace (`semianalysisai/cc-traces-weka-062126`, public, no auth) — no manual clone required; HF caches it locally. The `semianalysis_cc_traces_weka_with_subagents` value in the command below is a *rolling* alias tracking the current with-subagents corpus (presently the `062126` drop); the date-pinned `semianalysis_cc_traces_weka_062126` locks the corpus for reproducibility. Prefer the date-pinned alias for any run you intend to compare or submit — the rolling alias advances when a new corpus drops, and two runs on different drops are not comparable. Each corpus also has a `_256k` variant (e.g. `semianalysis_cc_traces_weka_with_subagents_256k`) that pre-drops individual requests whose input + output exceeds 256k tokens — pick it when your server's context window is ~256k. Then: ```bash aiperf profile \ --scenario inferencex-agentx-mvp \ --url localhost:8000 \ --model deepseek-ai/DeepSeek-V4-Pro \ --max-context-length 128000 \ --endpoint-type chat \ --public-dataset semianalysis_cc_traces_weka_with_subagents \ --concurrency 32 \ --use-server-token-count \ --streaming \ --extra-inputs ignore_eos:true \ --cache-bust first_turn_prefix \ --system-idle-gap-cap-seconds 10 \ --trajectory-start-min-ratio 0.0 \ --trajectory-start-max-ratio 1.0 \ --benchmark-duration 1800 \ --random-seed 20260707 \ --ui simple ``` The command mixes three kinds of flags — knowing which is which tells you what you must set, what you may tune, and what you shouldn't touch: **Flags you choose** — your server, model, and load: - **`--scenario inferencex-agentx-mvp`** is the only flag that's specific to this benchmark. Everything else is normal AIPerf. - `--model` is whatever you're serving — you don't have to match the model names baked into the trace corpus; AIPerf rewrites every trace request's `model` field. See [Per-Trace Model Rewriting](weka-trace.md#per-trace-model-rewriting) in the Weka tutorial for how multiple `--model` values map. - **`--max-context-length 128000`** drops, before replay, any trace whose peak *recorded* **input + output** exceeds 128k tokens. Input is the request's recorded `input_length` and output is its requested `max_tokens`, with the peak taken across parent and subagent requests. These are the trace's recorded counts, not a re-tokenization of the rebuilt prompt with your model's tokenizer, so leave some headroom. Filtering happens before `--num-dataset-entries` is applied, so the cap keeps the first N *eligible* traces. Set it to the maximum context your server accepts — i.e. the model's native maximum, since AgentX MVP expects the server to run at its default max length. Because it mirrors the server's real capacity, the run replays only traces the server can serve — which is why the scenario allows it while rejecting the arbitrary client-side cap `--synthesis-max-isl`. The flag is optional and the scenario doesn't check it: omit it and no client-side filtering happens, so over-length traces fail at the server and count toward the 1% context-overflow threshold instead. During warmup that is not merely counted: any root warmup failure, including a context-overflow rejection, aborts the run as `run_cancelled`. Setting `--max-context-length` to the server's real limit avoids that. - **`--use-legacy-max-tokens`**: each request's recorded output length is sent as `max_completion_tokens`. If your server only honors `max_tokens`, add this flag; otherwise, with `ignore_eos` locked on, responses may not be length-bounded. - **`--concurrency`** sets how many session trees stay live throughout the run, i.e. the sustained load. It must be a single integer under `--scenario`; comma-list sweeps are rejected. When concurrency exceeds the number of distinct loaded traces, the trace pool wraps — the scenario locks `--cache-bust first_turn_prefix`, and an active cache-bust target satisfies the wrap opt-in. Without cache-bust, wrapping still requires `--allow-dataset-wrap`. - `--url`, `--endpoint-type chat`, `--use-server-token-count`, and `--ui` round out the group ([`--use-server-token-count`](#tokenization-options---apply-chat-template-and---use-server-token-count) is explained below). **Flags the scenario hard-locks** — `--streaming`, `--extra-inputs ignore_eos:true`, `--cache-bust first_turn_prefix`, and `--system-idle-gap-cap-seconds 10`. You can omit all of them (AIPerf fills in exactly these values under `--scenario`), but writing them out makes the command document what runs. If you pass one of them with a *conflicting* value, AIPerf errors up front rather than silently producing an invalid result. The scenario forbids `--inter-turn-delay-cap-seconds`; `--trace-idle-gap-cap-seconds` remains an optional CLI control and is unset by default. Only the `ignore_eos` key of `--extra-inputs` is checked: other keys are merged into every request body after the trace's own fields, so `max_tokens`, `max_completion_tokens` or `stream` there would override the recorded output length or streaming without a lock error. Keep extra inputs to `ignore_eos` plus knobs that do not affect length or streaming. Likewise, don't add `--request-count` or `--num-sessions`: they are not rejected, but act as extra stop conditions (see [Troubleshooting](#troubleshooting)). **Flags the scenario only defaults** — auto-filled when omitted, but an explicit value is honored *silently*, with no error and no change to the `submission_valid` stamp: - **`--trajectory-start-min-ratio 0.0` / `--trajectory-start-max-ratio 1.0`** set the window where each lane's starting instant `t*` is sampled. The scenario never validates these — override them and the run still stamps `submission_valid: true` while replaying a materially different workload. Leave them at `0.0`/`1.0` for any run you intend to compare. - **`--dataset-sampling-strategy`**, **`--synthesis-max-osl`**, **`--burst-phase-starts`**: not validated. `random`/`shuffle` sampling changes lane assignment and recycle order; `--synthesis-max-osl` caps the requested `max_tokens`, so the server decodes fewer tokens than recorded; `--burst-phase-starts` also starts every lane's first profiling request immediately instead of keeping its recorded offset. The run still stamps `submission_valid: true`, so leave all three unset for comparable runs. - **`--benchmark-duration 1800`** (30 minutes) is the scenario default; 900 seconds (15 minutes) is the enforced minimum, and AIPerf rejects anything shorter. Longer values are accepted without complaint. - **`--random-seed`**: omitted, AIPerf picks a fresh random seed and logs it; passing your own (any non-negative integer) makes the run reproducible up front. Pinning a seed also keys the reconstructed-dataset disk cache: unseeded scenario runs draw a fresh seed each time and repay the full multi-minute corpus reconstruction on every run (see [Troubleshooting](#troubleshooting)). **Optional extras** (not in the command above): - **`--num-profile-runs N`** repeats the benchmark N times and adds an aggregate file with confidence intervals across the runs. See [Reading the Result](#reading-the-result-submission_valid) below for where the `submission_valid` stamp lands. Only the first run performs the optional cache-pressure warmup (`--agentic-cache-warmup-duration` / `--warmup-requests-per-lane`); later runs still send the trajectory primers but skip the extra stage. Pass `--no-profile-run-disable-warmup-after-first` to give every run the same warmup. - Every eligible trace loads by default: all 393 when `--max-context-length` is unset, otherwise every trace that fits under it. `--num-dataset-entries N` keeps the first N *eligible* traces after `--max-context-length` filtering (filter-then-cap). Reducing the corpus changes the replayed workload and is *not* caught by the scenario locks — the run still stamps `submission_valid: true` — so use it only for smoke tests, never for runs you intend to compare against other AgentX MVP results. If your smoke concurrency exceeds N, the locked cache-bust target lets the pool wrap; with cache-bust off you would need `--allow-dataset-wrap` or a lower concurrency. You don't need to touch the scheduling or warmup knobs: the scenario picks the agentic-replay scheduler and auto-fills the values shown above. Don't pass `--fixed-schedule`, `--request-rate`, or `--ignore-trace-delays` — they conflict with the locked scheduling mode and the run will error. ### Tokenization options: `--apply-chat-template` and `--use-server-token-count` **Optional: `--apply-chat-template` (only matters without `--use-server-token-count`).** When `--use-server-token-count` is set, as in the Quick Start, the reported ISL is the server's `usage.prompt_tokens` and this flag does not change it. Without server counts and with the flag off (the default), the reported ISL is the bare-text encode of the wire payload. With the flag on, the record processor re-tokenizes each request's wire payload through the tokenizer's own `apply_chat_template`, so the reported ISL also counts the chat-template wrapping (role headers and the generation prompt) on top of the wire text, which already includes the cache-bust marker either way, and is directly comparable to a server's `usage.prompt_tokens`. See [Input Sequence Length (ISL) Tokenization](../reference/isl-tokenization.md) for the full picture. **`--use-server-token-count` (included in the Quick Start command; fixes OSL mismatch).** Without it, AIPerf computes output sequence length (OSL) by re-tokenizing the server's response with the model's local tokenizer. If that tokenizer disagrees with the server's own — a different revision, different BPE merges, a different chat template — the reported OSL drifts from the server's actual emitted token count, and the per-run console shows an "Output Sequence Length Mismatch Warning" panel even though `ignore_eos=true` is locked and the server really did emit `max_tokens`. With the flag, AIPerf takes token counts from the server's `usage` block: ISL from `usage.prompt_tokens`, and OSL from `usage.completion_tokens` minus any `usage.reasoning_tokens`, so the mismatch goes away. If the server reports no usage, the token-count metrics are left empty. ### Benchmarking through a router (multiple replicas) Make the routing conversation-aware, or cross-replica scatter will destroy the prefix-cache reuse this benchmark exists to measure. Server side: SGLang Model Gateway `--policy cache_aware` (or `--policy manual`) / Dynamo `--router-mode kv` (add `--router-session-affinity-ttl-secs` on the frontend to honor the Dynamo session header below). Client side, AIPerf keeps a stable per-conversation ID and exposes it as an additive session-affinity header via an environment variable: `AIPERF_HTTP_X_SMG_ROUTING_KEY_FROM_CORRELATION_ID=1` (SGLang `manual`), `AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=1` (Dynamo session affinity; sends `X-Dynamo-Session-ID`, plus `X-Dynamo-Parent-Session-ID` on subagent requests), or `AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID=1` (any other router). For the server-side prefix-affinity setup, see [SGLang Model Gateway](https://docs.sglang.io/docs/advanced_features/sgl_model_gateway) and [Dynamo KV-aware routing](https://docs.nvidia.com/dynamo/dev/cli/kv-aware-routing/overview); the sticky-session headers above are AIPerf-side and are described in [Environment Variables](../environment-variables.md). ### What you should see A run moves through four externally visible stages; knowing them keeps you from killing a healthy run (a Ctrl+C stamps it `run_cancelled` / invalid): 1. **Dataset configuration.** On the first run this includes the corpus download from HuggingFace plus a CPU-heavy reconstruction of every trace into a tokenized, cache-structured dataset — expect several minutes of apparent silence, and see [Troubleshooting](#troubleshooting) if it exceeds the default configuration timeout. Later runs with the same corpus, settings, and a pinned `--random-seed` restore it from the on-disk cache in seconds. 2. **Warmup.** Each lane sends one primer request per live stream (the root plus any active subagents): that stream's final request before the lane's starting instant `t*`, which primes the server's KV cache. Every warmup request, primers and optional cache-pressure traffic alike, is limited to one output token. With deep histories in real coding traces this is a meaningful chunk of wall time on its own. Every 30 s the log prints a `Phase warmup progress` line with `returned`, `sent`, `in_flight`, `errors`, and `elapsed` counts (interval: `AIPERF_SERVICE_WARMUP_PROGRESS_LOG_INTERVAL`, 0 disables). If `returned` stops climbing while `in_flight` stays non-zero, a request is stuck; snapshot warmup waits indefinitely by default, so set `--agentic-warmup-grace-period` to bound it. 3. **Profiling** for `--benchmark-duration` (1800 s by default) — with `--ui simple`, per-phase progress and request counts tick along as traffic flows. 4. **Drain and export.** In-flight requests finish during a grace-period drain, then the console prints the metrics tables and the exact artifact paths (under `./artifacts/` unless you set `--artifact-dir`). The console tables do not show `submission_valid` or the context-overflow count: open `profile_export_aiperf.json` and check `metadata.submission_valid` before trusting a run. End to end, a cold first run therefore takes noticeably longer than the 30 minutes the duration flag suggests — reconstruction, warmup, and drain all sit on top of it. --- ## What `--scenario inferencex-agentx-mvp` Locks for You When you pass the scenario flag, AIPerf checks (and in some cases sets) the following settings before the run starts. If any of them conflict with what you asked for, the run errors immediately with a clear message naming the offending flag, unless you pass [`--unsafe-override`](#--unsafe-override), which downgrades conflicts to warnings and stamps `submission_valid: false`. Omitting a Weka dataset entirely is an error even then. | Locked setting | What it means | Why it matters | |---|---|---| | `timing_mode` is `agentic_replay` | Use the multi-turn agentic-replay scheduler (stamped onto the profiling phase by the scenario; not a user-selectable flag). Passing `--request-rate`, `--arrival-pattern`, `--user-centric-rate`, `--fixed-schedule`, or `--adaptive-scale` (or declaring a rate-shaped YAML phase) is a scenario violation | This is the scheduling discipline AgentX MVP requires (warmup → steady-state — the [profiling phase](#profiling-phase-faithful-replay-recycle-global-idle-guard) — with sampler-driven trace recycle and per-session-tree concurrency). | | `extra_inputs.ignore_eos = true` | Server is told to ignore its end-of-stream token and generate the full requested length | Without this, models stop early and you measure their decision to stop, not the server. | | `--streaming` is on | Responses stream token-by-token (auto-enabled when unset; explicitly setting streaming to false, e.g. `streaming: false` in YAML, errors) | The per-token latency metrics (TTFT, ITL) are core to this benchmark and need streaming responses. | | Replay delays are end-to-start (default; not validated) | Each turn's replay delay is the recorded idle gap from the previous response's *end* to the next request's *start*, not the start-to-start delta. This is the Weka loader default. The scenario does not lock it: `--use-think-time-only` switches delays to the recorded per-request `think_time` and is accepted silently, so leave it unset for comparable runs. | Replay dispatches each turn after the previous one completes, so start-to-start deltas would double-count the previous request's server time, making every session drift later turn by turn and overstating how many sessions overlap at once. | | `--ignore-trace-delays` is off | Trace-derived delays are preserved | The replay retains the captured agent pacing and KV-cache reuse intervals. | | Per-trace idle-gap cap is optional | `--trace-idle-gap-cap-seconds` is unset by default and accepts an explicit CLI value | Setting it bounds observed runtime idle across the root and every descendant stream without rewriting dataset timestamps or bypassing spawn/join dependencies. | | Per-turn cap is forbidden | `--inter-turn-delay-cap-seconds` must remain unset | Independently clamping parent and subagent delays can distort their relative timing. | | `--system-idle-gap-cap-seconds = 10` | When no request is active or ready, all pending replay timers shift uniformly so the next request arrives within 10 seconds | The benchmark avoids measuring long periods with no server work while preserving request order and relative spacing across every pending trajectory. | | `--cache-bust first_turn_prefix` | A unique per-trajectory-tree marker (shared by the root session and its subagents) is injected at the start of the first user turn of the request prefix for every play (each fresh dispatch of a trace, initial or recycled); a tree that continues from warmup into profiling keeps its marker | Without this, every time a trace is recycled the server's prefix cache would warm up further on identical content, and steady-state cache-hit rates would inflate the longer the run goes. The marker gives every recycled play a fresh prompt prefix. | | Loader is a pinned Weka with-subagents corpus | The dataset must be a with-subagents `--public-dataset` alias or `--hf-weka-dataset semianalysisai/cc-traces-weka-062126` (`weka_hf`). A local `weka_trace` directory is format-compatible but unpinned — the run refuses unless you pass `--unsafe-override` (which stamps `submission_valid: false`). The [Troubleshooting](#troubleshooting) entry for this lock lists the exact flag forms. | Submission validity requires a known public corpus identity; arbitrary local dirs are not hash-verifiable. | | `--benchmark-duration >= 900` (defaults to 1800 when unset) | The run lasts at least 15 minutes; if omitted, it runs for 30 minutes | Steady-state needs time to stabilize; short runs are noise. | | No client-side input truncation | `--synthesis-max-isl` (the synthesis ISL filter) is rejected for both `--public-dataset` / `--hf-weka-dataset` and file-based Weka inputs, because it drops traces whose input length exceeds the cap. `--synthesis-max-osl` is not checked (see [Quick Start](#quick-start)) | Truncating prompts on the client side would falsify the workload. | | `--random-seed` is set | If you didn't pass one, AIPerf picks a strong random one and logs it | Reproducibility — every replayed result can be regenerated. | If you forget to pass the `ignore_eos` extra-input, `--streaming`, `--cache-bust`, or `--random-seed`, AIPerf injects the locked value and tells you at INFO log level. The same goes for `--system-idle-gap-cap-seconds` and `--benchmark-duration` (1800s default) when you don't set them explicitly. If you pass one of these explicitly with a value that conflicts with the scenario, AIPerf errors with all the violations listed at once — you don't have to fix them one at a time. The trajectory-start ratios (`--trajectory-start-min-ratio` / `--trajectory-start-max-ratio`) are the soft exception: the scenario auto-fills `0.0`/`1.0` but never validates them, so an explicit different value is honored silently — no error, no change to `submission_valid` — even though it materially changes the workload. Treat them as fixed for any run you intend to compare (see the flag groups in [Quick Start](#quick-start)). --- ## Reading the Result: `submission_valid` All output files land under the artifact directory. By default that is an auto-named subdirectory of `./artifacts/` (relative to where you ran `aiperf`), named from the model, endpoint, and load settings, for example `artifacts/-openai-chat-concurrency10/`. Pass `--artifact-dir` to use an exact directory with no auto-named subdirectory. With multiple profile runs, each run's files go under `profile_runs//` inside that directory. AIPerf prints the exact output locations at the end of the run. When you use `--scenario`, AIPerf stamps a submission-validity flag onto the AIPerf metrics JSON outputs for the run. The per-run `profile_export_aiperf.json` carries it under its `metadata` block, and when you pass `--num-profile-runs >= 2` (and at least two runs complete successfully) the aggregate file (`aggregate/profile_export_aiperf_aggregate.json` under your artifact directory) carries it too: ```json { "metadata": { "scenario": "inferencex-agentx-mvp", "submission_valid": true, ... }, "request_throughput": { ... }, "request_latency": { ... }, ... } ``` (In the per-run file, metric results are top-level fields alongside `metadata`; the aggregate file nests them under a `metrics` key instead.) `metadata.dataset` records the corpus that actually ran: `loader` (your alias), `hf_dataset_name` (the resolved HF repo), and `num_dataset_entries` if you capped it. Use it to confirm two results used the same drop. Three possible states for `submission_valid`: - **`submission_valid: true`** — the run honored every scenario rule AIPerf checks, and finished cleanly. - **`submission_valid: false`** — something went wrong (or you forced something). The same metadata block also contains `submission_invalid_reasons`, a list of short tags explaining why. Possible values: - `"unsafe_override"` — you passed `--unsafe-override` along with one or more rule-breaking flags. See [`--unsafe-override`](#--unsafe-override) below. - `"context_overflow_rate_exceeded"` — more than 1% of the responses came back with a context-overflow error from the server (the threshold is `AIPERF_AGENTX_CONTEXT_OVERFLOW_RATE_LIMIT`, default `0.01`; exactly 1% is accepted), which means the server is rejecting prompts the benchmark requires it to handle. This usually means the server was started with a reduced max model length; AgentX MVP requires the model's default. A failed response counts as an overflow only if its body (or `error.message`) contains `context length`, `maximum context`, `context_length_exceeded`, or `prompt is too long` (case-insensitive). If your server words it differently, extend the list with `AIPERF_AGENTX_CONTEXT_OVERFLOW_SUBSTRINGS`; otherwise those rejections are ordinary errors and do not count toward this threshold (an empty list disables detection). When the server rejects a non-final turn as too long, AIPerf ends that trajectory there and the lane recycles once the tree drains. Overflowed requests are left out of the latency metrics, error counts and `profile_export.jsonl`; they appear only in `context_overflow_count` and `skipped_context_overflow_count` and in the 1% rate check. - `"run_cancelled"` — the run was cancelled early (Ctrl+C). On a single Ctrl+C AIPerf cancels gracefully and still writes the export files with whatever partial metrics it collected (a second Ctrl+C force-quits immediately, so the export files may be incomplete or missing); a cancelled run is always stamped invalid. - `"scenario_reresolve_failed"` — multi-run aggregate export could not re-apply the scenario lock on the base config (import/env breakage). Fail-closed: the aggregate stamps `submission_valid: false` rather than assuming the lock held. - `"insufficient_profile_metric_coverage"` — neither TTFT nor inter-token-latency observations reached 95% of the configured profiling duration; see [Troubleshooting](#troubleshooting). - `"profile_metric_coverage_validation_failed"` — AIPerf could not validate that coverage because required accumulator data or phase timestamps were unavailable. - `"scenario_with_sweep"` — a parameter sweep was run under `--scenario` with `--unsafe-override`. The tag appears in each variation's aggregate file (`aggregate//profile_export_aiperf_aggregate.json` or `/aggregate/profile_export_aiperf_aggregate.json`, depending on the sweep's iteration order), which is written whenever at least two runs succeed, even with `--num-profile-runs 1`. Individual per-run files do not carry this tag. - **Field absent** — you ran without `--scenario`. The submission-validity machinery is gated on the scenario flag. Also check the top-level `is_complete` field of the per-run file. When `false`, `incomplete_reason` says why. It is not part of `submission_valid`, so treat an incomplete run as unusable even if stamped `true`. If you see `submission_valid: false`, look at `submission_invalid_reasons` and the AIPerf log. The reasons correspond to a scenario rule you broke or a runtime check that failed or could not be validated. Treat the stamp as a guideline, not a certification. It reflects only the checks AIPerf can perform itself — the flag locks applied at startup plus the runtime thresholds it tracks. `true` means AIPerf detected no rule violation; it can't attest to anything outside its view (a server started with a reduced context window, for example, only shows up indirectly through the context-overflow rate). Whether two results are genuinely comparable still depends on the full setup on both sides. The stamp also ignores environment variables: the `AIPERF_DATASET_WEKA_*` reconstruction knobs from the [Weka tutorial](weka-trace.md) change session structure or wire shape yet leave `submission_valid: true`. Leave them at defaults for comparable runs and check none are set in your shell or pod environment. --- ## How a Run Executes ### Warmup Phase: Trajectories and `k_i` Before AIPerf measures anything, it runs a **warmup phase** that primes the server's KV cache. This isn't the generic AIPerf warmup — it's a trajectory-based warmup specific to the agentic-replay scheduler. The generic warmup flags (`--warmup-request-count`, `--warmup-num-sessions`, `--warmup-duration`, `--warmup-concurrency`, or a `warmup` phase in YAML) are accepted without a warning but have no effect: agentic replay replaces any declared warmup phase with its own. To add warmup load use `--agentic-cache-warmup-duration` or `--warmup-requests-per-lane`. Because the scheduler synthesizes this phase rather than reading it from your `phases:` list, it reports under the reserved name `agentic.warmup` in logs, progress output, and the Kubernetes `AIPerfJob` status. The `.` is what makes the name reserved: phase names you declare must match `^[A-Za-z_][A-Za-z0-9_-]*$`, so no phase of yours can ever collide with it. Here's the picture. You set `--concurrency 100`. The scheduler builds 100 active trajectory lanes, drawing traces from the dataset sampler. Filling more lanes than distinct loaded roots requires `--allow-dataset-wrap` or an active `--cache-bust` target (which the scenario locks on); without either, a bounded run (by duration, request count, or a session budget larger than the corpus) fails at startup with `concurrency N exceeds the M distinct loaded traces while dataset wrapping is disabled`. Separately, if too few traces are long enough to split, the pool is capped with a warning. When wrapping is enabled, the same trace can back multiple lanes, each with a deterministic per-lane start position. For each lane, it samples a random starting instant `t*` somewhere between 0% and 100% of that trace's recorded duration (the `--trajectory-start-min-ratio` / `--trajectory-start-max-ratio` window the scenario auto-fills, widening the generic 25%–75% default) and derives the "starting turn" `k_i` from it, always leaving at least one profiling turn after warmup. Then it dispatches the warmup turn(s) per lane: turn `k_i` for simple (non-subagent) trajectories, with the full prefix history (turns 0 through `k_i-1`) attached as message context. Lanes whose subagent branches are already live at `k_i` dispatch one warmup request per active stream. By default primers are dispatched on a ramp that aligns every trajectory's `t*` to a common instant: a primer that fired `lead` seconds before its `t*` is sent at `max_lead - lead`. Each lead is clamped to `--system-idle-gap-cap-seconds` (10 s under the scenario), so alignment is approximate and the ramp is at most about 10 s long; `--trace-idle-gap-cap-seconds` does not apply to warmup. With `--burst-phase-starts` all primers are sent at once. The point is that the server's prefix cache fills with a realistic mix of multi-turn coding contexts before any measurement starts. When the profiling phase begins, every trajectory resumes from `k_i + 1` — and the server's cache already holds the prefix. The `k_i` values are deterministic given the random seed: same dataset + same seed = same trajectories + same start points + same recycle order, on any machine. That's why the scenario insists on a seed. AIPerf aborts the run as soon as any **root-conversation** warmup request fails terminally — each warmup request gets exactly one attempt; there is no retry — and it does not wait for the rest of the warmup to drain. A single terminal failure means profiling would start on a degraded trajectory pool, so AIPerf cancels in-flight warmup immediately, logs the failing trace at `WARNING` ("aborting run early"), and shuts down as a cancelled run stamped `submission_valid: false` (reason `run_cancelled`). A *subagent* stream's warmup failure does not trigger the abort — only root (depth-0) conversations gate it. Slow-but-healthy warmups are also *not* aborted: the warmup grace period (`--agentic-warmup-grace-period`; under `--scenario`, `--warmup-grace-period` aliases onto it when the dedicated flag is unset — see the [Warmup Phase tutorial](warmup.md)) is unlimited by default for the snapshot warmup (including `--warmup-requests-per-lane`), so a merely slow warmup runs to completion. With `--agentic-cache-warmup-duration` the default drain is bounded to the larger of the benchmark grace period and `min(duration, 300s)` unless `--agentic-warmup-grace-period` is set explicitly. If warmup requests are still on the wire when that limit is reached, AIPerf cancels them and the run fails with `Accelerated warmup drain timed out before all wire requests returned`. Raise `--agentic-warmup-grace-period` (or pass `inf`) if your server is slow to drain deep one-token warmup requests. #### Optional cache-pressure warmup Set `--agentic-cache-warmup-duration SECONDS` to add a sustained cache-pressure stage after the initial trajectory warmup described above. AIPerf continues the same live session trees for that duration with recorded idle delays removed and every request limited to one output token. Because the delays are removed outright, the system-idle guard is not applied during this stage; the per-trace idle cap applies to profiling only, and during warmup the system-idle cap only clamps the baseline primer leads. When the duration expires, it stops issuing new requests, drains requests already on the wire, snapshots each live root, subagent, and unresolved join, and starts profiling from that exact state. Each stream's full next-turn delay is carried across the phase boundary; time spent waiting for the global warmup drain does not consume it. For repeatable warmup depth, use `--warmup-requests-per-lane REQUESTS` instead. After the mandatory snapshot primers complete, each concurrency lane sends exactly that many additional warmup wire requests. For example, `--concurrency 16` with `--warmup-requests-per-lane 10` adds 160 cache-pressure requests after the primers, with a strict 10-additional-request quota on every lane. At handoff, timestamped root, subagent, and background streams are restored to their next request's position on one shared per-trajectory dataset clock. The earliest pending request starts profiling immediately; the remaining streams retain their flattened cross-stream order and relative start spacing. For a timestamp-less dataset, AIPerf falls back to each stream's recorded end-to-start delay. `--agentic-cache-warmup-duration` and `--warmup-requests-per-lane` are mutually exclusive: choose a time-bounded warmup or a deterministic request-bounded warmup. These requests remain part of warmup, so they are excluded from exported request metrics. They are reported separately under the top-level `warmup_metrics` key of `profile_export_aiperf.json`. `profile_export.jsonl` still contains warmup rows, tagged `metadata.benchmark_phase: "warmup"`; filter them out when analyzing profiling records. ### Profiling Phase: Faithful Replay, Recycle, Global Idle Guard After warmup, the profiling phase opens. Now you're measuring. Each trajectory keeps replaying its conversation from turn `k_i + 1` onward, honoring the original recorded inter-turn gaps as end-to-start delays counted from the moment the previous turn completes. At the phase boundary, AIPerf subtracts one global minimum from the pending first-request offsets: the earliest request starts immediately, while every other trajectory keeps the same relative spacing. With `--trace-idle-gap-cap-seconds S`, a per-trajectory watchdog covers initial future work and restarts when the last in-flight request across the root and all descendant streams completes. If that tree remains idle for `S` seconds, AIPerf uniformly advances only its pending timers; dataset timestamps, relative timer spacing, request order, and spawn/join gates remain unchanged. If every request completes while future requests remain scheduled more than 10 seconds away, AIPerf shifts all pending request timers earlier by the same amount. This bounds true system-idle time without changing request order or the relative spacing among pending trajectories. A scheduled turn may become retained by a cross-stream replay barrier instead of reaching the wire; AIPerf re-evaluates the guard after that scheduler task drains so a nearby blocked turn cannot hide a much later dispatchable timer. The phase-end log reports how many global jumps occurred and how many seconds they skipped. Concurrency here is **per session tree**: each lane holds one slot for a whole tree — the root conversation plus every subagent worker stream it spawns. A lane recycles only once its **entire tree drains** — the root has sent its last turn *and* every subagent has finished — not merely when the root's final turn is acknowledged. Exactly `--concurrency` trees stay live at all times. Never more: a new root can't start until a tree fully drains. Never fewer: a lane whose root has finished but whose subagents are still draining keeps its slot. The shared tree id (`root_correlation_id`) is written to every record in `profile_export.jsonl` (in the artifact directory), so `aiperf analyze swim-lane` groups each tree under one lane and renders exactly `--concurrency` slots as a per-lane timeline. When a tree drains, the lane recycles by drawing the next root from the **dataset sampler** (the same sampler that built the initial trajectories, honoring the dataset's `sampling_strategy`). As long as the corpus is larger than the trajectory count, a sequential/shuffle sampler plays every trace at least once before repeating any. A few wrinkles worth knowing: - **Recycled traces start at turn 0**, not at a random `k_i`. The "start somewhere in the middle" rule applies only to the initial trajectories — the intent is to spread the *initial* state across the conversation length, not to keep injecting mid-conversation jumps forever. - **Each play of a trace gets a fresh cache-bust marker.** When a trace is first dispatched (or recycled), AIPerf prepends a unique short tag like `[rid:8a3f2c1b9e7d]\n\n` to the first user turn. The marker is generated once per play, reused on every turn of that play, and replaced when the trace recycles into a new play — that's what keeps the server's prefix cache from warming progressively on identical content as the run goes on. Markers also differ across runs, since each run's auto-generated benchmark ID feeds the marker digest. - **Warmup and profiling share the marker for a given play.** A trajectory's warmup turn `k_i` and its first profiling turn `k_i+1` carry the *same* `[rid:…]` — that's how the KV-cache prefix work done during warmup transfers into measurement instead of being thrown away. - **Concurrency may exceed the number of distinct loaded traces only with wrap or cache-bust enabled.** Traces too short to split into a warmup + profiling turn are skipped (the pool is capped with a warning when it cannot fill after skips). Filling more lanes than distinct loaded roots requires `--allow-dataset-wrap` or an active `--cache-bust` target. With wrap on, one source trace can back several lanes (each keeping its own start position and recycle behavior). An *empty* pool after filtering is still an error. - **Profiling ends** when `--benchmark-duration` elapses. Anything in flight finishes during a grace-period drain and is included in the metrics; nothing *new* starts after the duration ends. The drain is bounded by `--benchmark-grace-period` (default 30 s, also applied when the scenario fills in the default duration; `inf` waits indefinitely). Requests still in flight when it expires are cancelled (logged as `Phase profiling timed out, cancelling all credits`), so raise it if long responses are being cut off. - **A context-overflow error ends that trajectory.** If the server rejects a non-final turn as too long, AIPerf sends no further turns on that stream and logs `Terminating trajectory early at turn i/n: context-overflow error from server`. If it was the root, the tree drains and the lane recycles into a fresh trace. Other errors don't stop the trajectory. - **Other errors during profiling don't stop the run.** A failed request is recorded as an error and its stream moves on to the next recorded turn. A subagent that fails on its final turn still releases the parent's join. Set `AIPERF_DAG_FAIL_FAST=1` to abort the parent and its sibling subagents instead. ### Subagents The AgentX MVP corpus is the current **with-subagents** variant. Parent turns can spawn one or more helper conversations, and the first later parent turn recorded at or after a helper's end waits on the corresponding `SPAWN_JOIN` prerequisite before resuming (a long helper may run through several parent turns before it gates one). As covered in the [Profiling Phase](#profiling-phase-faithful-replay-recycle-global-idle-guard) section, `--concurrency` counts live session **trees**, with each slot held until the whole tree drains. The new wrinkle here is the *request* count: helper conversations run alongside their parent, so the instantaneous number of in-flight requests can rise above `--concurrency` at a fan-out point — the cap is on concurrent trees, not on individual in-flight requests. AIPerf constructs this topology from `WekaSubagentEntry` blocks in the trace: subagents spawn from the last parent turn before them and join the first later parent turn recorded at or after their end, becoming SPAWN/JOIN branches; subagents that no later parent turn reaches become background branches that do not block the parent; subagents sharing the same spawning and joining turns collapse into one multi-child branch; and subagents with no preceding parent turn are dropped. Within each subagent entry, AIPerf additionally detects nested context chains, so one recorded subagent may replay as several parallel child streams (helper sidecars, worker groups, solo workers). Two behaviors matter for the load: each child stream dispatches at its **recorded offset** from the spawn rather than bursting when the parent turn completes, so the in-subagent request schedule replays on the recorded timeline; and the parent's SPAWN_JOIN waits on *all* of a subagent's child streams. For the detection rules and the `::sa:`/`::aux:`/`::wg:`/`::fa:` stream-naming scheme, see the [Weka Traces tutorial](weka-trace.md). To check subagent behavior, read the top-level `branch_stats` block in `profile_export_aiperf.json` (`children_spawned`, `children_completed`, `children_errored`, `parents_suspended`, ...; see [DAG benchmarks](../benchmark-modes/dag.md)). A failed subagent is counted in `children_errored`, and the phase-end `BranchOrchestrator stats:` log line reports the same `errored=` counts. --- ## `--unsafe-override` Sometimes you intentionally want to break a scenario rule — to study the sensitivity of one variable, to run a 1-minute smoke test instead of a 15-minute proper run, to try a different cache-bust target for an ablation. For that: ```bash aiperf profile \ --scenario inferencex-agentx-mvp \ --unsafe-override \ --benchmark-duration 60 \ ... ``` What `--unsafe-override` does: - **Converts most scenario rule violations from an error into a warning.** The run starts. - **Does not bypass a missing weka loader.** If you omit `--public-dataset` / `--input-file`, the CLI defaults to synthetic; AgentX still refuses to start (continuing would look like a corpus load while producing 1-turn sessions that empty the trajectory pool). Pass an allowed weka loader explicitly. - **Stamps `submission_valid: false`** under `metadata` in `profile_export_aiperf.json` (and, when `--num-profile-runs >= 2`, in `aggregate/profile_export_aiperf_aggregate.json`), with `"unsafe_override"` in `submission_invalid_reasons`, but only when at least one per-run scenario rule was actually broken. Passing the flag without breaking any rule is a no-op. Exception: pairing `--scenario` with a parameter sweep is only downgraded to a logged warning by `--unsafe-override` and does not by itself stamp the outputs. Once the flag is on and a rule is broken, the stamp stays `false` in that run's outputs — there's no way to flip it after the fact, so the override is always visible in the result. The flag is a no-op without `--scenario` (since there are no scenario rules to override). Use this for development and ablation studies. Don't use it for anything you want to compare against other AgentX MVP runs. --- ## Troubleshooting **`UnknownScenarioError: Unknown scenario 'inferencex-agentx-mvp'. Valid scenarios: …`** The AIPerf you're running predates this scenario — scenarios ship with the package and are registered in code, not in a data file you can regenerate. Upgrade AIPerf (`pip install -U aiperf`) and re-run. **`EmptyTracePoolError: Loader produced 0 traces; trajectories cannot be built.`** The HF dataset download or row validation produced no usable traces. Check your network connectivity to `huggingface.co` and confirm the `--public-dataset` value is a with-subagents alias (e.g. `semianalysis_cc_traces_weka_with_subagents` or `semianalysis_cc_traces_weka_062126`), or `weka_hf` with `--hf-weka-dataset semianalysisai/cc-traces-weka-062126`. On an offline machine, replay a local Weka-format trace directory via `--custom-dataset-type weka_trace --input-file --unsafe-override` (offline smoke only — `submission_valid: false`) — see the [Weka Traces tutorial](weka-trace.md) for how to obtain or capture one, and note the model tokenizer must also be available offline (pre-populate the HF cache and set `HF_HUB_OFFLINE=1`, or point `--tokenizer` at a local path). **`ScenarioLockError` … `got synthetic` / `cannot be bypassed with --unsafe-override`** You set `--scenario inferencex-agentx-mvp` (and maybe `--unsafe-override` for a short smoke duration) but omitted a weka dataset source. The CLI then defaults to synthetic prompts (`session_000000`, 1 turn each). AgentX refuses that combination even under `--unsafe-override`. Add `--public-dataset semianalysis_cc_traces_weka_062126` (or another allowed alias / `--hf-weka-dataset semianalysisai/cc-traces-weka-062126`). **The tokenizer can't be resolved or downloaded** The Weka loaders rebuild every prompt through a HuggingFace tokenizer derived from `--model`, so the dataset build fails if that repo can't be reached. If the model repo is gated, export `HF_TOKEN`; if `--model` isn't a resolvable HF repo name (a local server alias, a private build), pass `--tokenizer ` explicitly. See [Tokenizer Auto-Detection](../reference/tokenizer-auto-detection.md). **Run aborts early: `aborting run early (requesting PROFILE_CANCEL)` / warmup failure** Your inference server rejected a warmup request. Each warmup request gets exactly one attempt — there is no retry — and AgentX MVP aborts on the **first** terminal root-conversation warmup failure rather than producing a partial result: the run cancels immediately and the failing trace is named in the `WARNING` log. Check the AIPerf and server logs — the underlying cause is usually a connection error (`Connection refused`: wrong `--url` or the server isn't running), an auth failure (`401`/`403`: pass `--api-key`), a model-name mismatch (`404` / `model not found`: `--model` doesn't match what the server is serving), or the server's `max-model-len` set lower than the trace's requested context. **Run completes but `submission_valid: false` with `"context_overflow_rate_exceeded"`** Your server is rejecting prompts as too long for more than 1% of requests (the default of `AIPERF_AGENTX_CONTEXT_OVERFLOW_RATE_LIMIT=0.01`; a rate exactly at the limit is accepted). The most common cause is starting the server with a reduced `--max-model-len` (or equivalent flag) — AgentX MVP requires the model's default. Restart the server without overriding the max length and try again. If the model's *native* window is itself around 256k (MiniMax-class models), the fix is a matching `_256k` corpus instead — e.g. `--public-dataset semianalysis_cc_traces_weka_with_subagents_256k`. That beats capping client-side with `--max-context-length`: the `_256k` build is pre-baked at the source and drops only the individual over-limit *requests*, so each trace stays multi-turn with its think-time pacing intact, whereas `--max-context-length` drops a *whole trace* as soon as one turn exceeds the limit. The number of requests that overflowed appears as the `context_overflow_count` metric (a raw count, not a rate), top-level in `profile_export_aiperf.json` and under `metrics` in the aggregate file — divide it by `request_count + error_request_count + skipped_context_overflow_count` (the same denominator the aggregate exporter uses for the 1% threshold) to see how close you were to the limit. If you see overflows in the server log but nothing in `error_summary` or `profile_export.jsonl`, that is expected: under the scenario an overflowed request is not counted as an error, does not count toward `--failed-request-threshold`, and shows up only in `context_overflow_count` and `skipped_context_overflow_count` in `profile_export_aiperf.json`. **Run exits non-zero with `ProfileMetricCoverageError`** The server stopped producing both TTFT and inter-token-latency observations before 95% of the configured profiling duration elapsed (the coverage ratio this scenario requires). Either signal proves global server activity, so a long streaming response remains valid even when no new request starts near the boundary. AIPerf retains the result artifact, marks it invalid with `insufficient_profile_metric_coverage`, and reports the observed coverage for both signals. Check the inference-server logs for a crash or stalled request processing. Warmup metrics and profiling phases shorter than the scenario's minimum valid duration are excluded. If you added `--request-count` or `--num-sessions`, the scenario does not reject them but they act as extra stop conditions: when one ends profiling before `--benchmark-duration`, the 95% coverage check fails with this error. Remove them. **"scenario `'inferencex-agentx-mvp'` requires loader=any of …" / cannot verify corpus identity for a local weka_trace directory** The AgentX MVP scenario stamps `submission_valid: true` only for a pinned public SemiAnalysis Weka corpus: one of the allow-listed with-subagents `--public-dataset` aliases (the rolling `semianalysis_cc_traces_weka_with_subagents` alias, currently `semianalysisai/cc-traces-weka-062126`; the date-pinned `_060226`/`_060526`/`_060826`/`_061326`/`_061526`/`_062126` aliases; or any of their `_256k` variants; the legacy no-subagents aliases are rejected), or the generic HuggingFace Weka loader (`weka_hf`), which is constrained to `semianalysisai/cc-traces-weka-062126`. Pass one of: - `--public-dataset semianalysis_cc_traces_weka_with_subagents` (zero-setup; rolling alias for the current corpus), - `--public-dataset semianalysis_cc_traces_weka_062126` (zero-setup; date-pinned), or - `--hf-weka-dataset semianalysisai/cc-traces-weka-062126` (auto-selects `weka_hf`). A local `--custom-dataset-type weka_trace --input-file ` (or a bare `--input-file ` that auto-detects as `weka_trace`) is accepted for offline smoke tests only with `--unsafe-override` — the result is marked `submission_valid: false`, because AIPerf cannot fingerprint an arbitrary local directory as the public corpus. The same override path applies if you intentionally replay a *different* Weka-format corpus under this scenario. **"scenario `'inferencex-agentx-mvp'` requires `cache_bust.target=first_turn_prefix`; got ``"** You explicitly passed `--cache-bust ` (e.g. `system_suffix` or `none`) alongside `--scenario`, and AIPerf refuses to silently override an explicit user choice. If you didn't pass `--cache-bust` at all, the validator auto-injects `first_turn_prefix` and you'll never see this error. If you genuinely need a different cache-bust target for an ablation study, pass `--unsafe-override` and accept the `submission_valid: false` stamp. **"scenario `'inferencex-agentx-mvp'` forbids client-side input truncation; `--synthesis-max-isl` …"** You passed `--synthesis-max-isl `, which drops any trace whose recorded input length exceeds `N` tokens. Under the scenario that's forbidden because it changes the replayed workload (a different subset of the corpus, with the hardest prompts removed). Either drop the flag (and let the server handle its own context-length errors, which the scenario tracks via the `context_overflow_rate_exceeded` reason), or pass `--unsafe-override` and accept `submission_valid: false`. **"My run finished but I can't find `submission_valid` anywhere"** You probably ran without `--scenario`. The validity stamp is gated on the scenario flag — re-run with `--scenario inferencex-agentx-mvp` and look in the per-run `profile_export_aiperf.json` (under `metadata`). If you passed `--num-profile-runs >= 2`, it appears in the aggregate file at `aggregate/profile_export_aiperf_aggregate.json` under the artifact directory too. **Configuration times out before any traffic starts** The first run on a machine reconstructs the whole corpus into a tokenized, cache-structured dataset before sending anything, and for the with-subagents corpora that reconstruction can exceed the default 300-second configuration timeout. Raise `AIPERF_DATASET_CONFIGURATION_TIMEOUT` and `AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT` (which must be at least as large) to ~1800 seconds for a cold run. The cost is one-time: the reconstructed dataset lands in an on-disk cache (default `~/.cache/aiperf/dataset_mmap`) whose key includes the random seed, so pin `--random-seed` and later runs restore it in seconds. **On Kubernetes**, each `AIPerfJob` starts with empty per-pod HuggingFace and dataset directories, so every job downloads and rebuilds the corpus even with a pinned `--random-seed`. Local `export`s don't reach the pods; pass raised timeouts and `HF_TOKEN` through the pod template (see [Kubernetes configuration](../kubernetes/configuration.md)). **Run is slower than I expected** On a cold first run, the corpus reconstruction described in the previous entry happens before any traffic — budget several extra minutes for it. After that, the warmup phase replays a full conversation prefix for every trajectory stream before profiling starts; with deep histories in real coding traces, that's a meaningful chunk of wall time on its own. Subagent fan-out can also create additional in-flight requests beyond the active parent trajectory count. If your server is concurrency-limited and you raised `--concurrency` above its limit, you'll also see queueing. Drop `--concurrency` or raise the server's limit. **Results vary run-to-run on the same server** Two runs with different `--random-seed` values will land on different trajectories and different `k_i`s, so some variation is expected. To reproduce exactly, capture the seed AIPerf logged at startup and pass it back via `--random-seed`. Server-side sampling stochasticity (generation temperature) also contributes; more profile runs (`--num-profile-runs`) give the percentiles more data to stabilize on. --- ## See Also - [Weka Agentic Coding Traces](weka-trace.md) — the underlying trace format and SPAWN/JOIN subagent mechanics. - [Metrics Reference](../metrics-reference.md) — definitions and formulas for TTFT, ITL, throughput, and every other reported metric. - [Effective vs Active Metrics](../reference/effective-vs-active-metrics.md) — the time-weighted EFFECTIVE and ACTIVE console tables printed at the end of a run. - [Timing Modes Reference](../benchmark-modes/timing-modes-reference.md) — the other AIPerf timing modes (request-rate, concurrency, fixed-schedule); it does not cover `agentic_replay`, which is described in this tutorial and in [Weka Agentic Coding Traces](weka-trace.md). - [Warmup Phase tutorial](warmup.md) — the generic AIPerf warmup mechanism (the agentic-replay warmup is a specialization of this). - [Input Sequence Length (ISL) Tokenization](../reference/isl-tokenization.md) — how `--isl`, `--apply-chat-template`, and the locked `--cache-bust first_turn_prefix` marker interact in the reported metric. - [ISL Budget Compensation Derivation](../reference/isl-budget-compensation.md) — the math behind chat-template + marker overhead compensation. - [CLI Options Reference](../cli-options.md) — the auto-generated reference for `--scenario`, `--unsafe-override`, `--system-idle-gap-cap-seconds`, and every other flag.