--- name: perf-regression-triage description: > Handle a Perfherder performance regression bug end to end: read the alert bug, confirm whether the regression is real, find the cause, and iterate to a fix. Use when the user has a Perfherder/browsertime regression bug (the "N% ... regression on " bugs filed in Testing::Performance), needs to establish whether an alert summary is a real regression, is interpreting a PerfCompare baseRev/newRev comparison, or asks what to do about a patch of theirs that regressed a benchmark. Not for simply running perf tests or pushing them to try — including running an alert summary's tests with `mach try perf --alert` (use perftest) — nor for choosing or writing perf tests (use perftest), analyzing a profile you already have in hand (use profiler-analysis), SpiderMonkey microbenchmarks (use js-perf-investigation), or non-performance regressions such as failing tests, crashes, or build bustage. allowed-tools: - Bash(./mach try perf --help:*) - Bash(./mach try perf --no-push:*) - Bash(treeherder-cli:*) - Bash(profiler-cli:*) - Bash(git log:*) - Bash(git show:*) - Bash(git status:*) - Bash(jj log:*) - Bash(hg log:*) - mcp__moz__get_bugzilla_bug - Read - Grep - Glob --- # Triaging a Perfherder regression bug ## Tooling Three tools cover the three kinds of evidence in a regression bug. Use them rather than fetching URLs by hand — `WebFetch` on a Treeherder or profiler URL returns a UI shell with no data in it. - **The regression bug** — `mcp__moz__get_bugzilla_bug`, or the `@moz:bugzilla://bug/{id}` resource. There is no Bugzilla CLI; this is the supported path. - **Push and job data** — `treeherder-cli `. Use `--perf` for performance and resource data, `--repo autoland` to inspect the culprit push, and `--watch --notify` to wait on a running try push instead of polling it. - **Before/After profiles** — `profiler-cli`, driven through the **profiler-analysis** skill. If it is not installed: `npm install -g @firefox-devtools/profiler-cli@latest`. `profiler-analysis` owns the profiler-cli protocol (run `profiler-cli guide` first, stop the daemon when done). Hand profiles to that skill instead of reimplementing it here. The regression policy gives the patch author **3 business days** to acknowledge and start investigating before the patch may be backed out. Establish early whether that clock is running, and tell the user if it is close to expiring. ## Step 1: read the bug Fetch it with `@moz:bugzilla://bug/{id}`. Every Perfherder alert bug carries the same machine-readable payload. Pull out all of it before doing anything else: - **Alert summary ID** — the `perfherder/alerts?id=NNNNN` link. Feeds `--alert` in step 2. - **Culprit push revision** — the `pushloghtml?changeset=...` link in comment 0. - **Base / new revisions** — `baseRev=` and `newRev=` in the PerfCompare link. - **Regressed tests** — first column of the alert table. - **Platform and options** — e.g. `linux2404-64-shippable`, `fission webrender`. - **Before/After profiles** — the profiler.firefox.com links in the last column. The culprit push usually contains several patches. Inspect it directly rather than reading the pushlog HTML: ``` treeherder-cli --repo autoland --perf ``` Identify which patch is the user's and which files it touched (`git show ` / the linked Phabricator revision). If the push has multiple candidate patches and it is not obvious which is responsible, say so — the confirmation push below tests the user's patch specifically, which is what settles it. Summarize for the user: how big the regression is, which tests, which platform, and what the patch changed. Then move to confirmation. ## Step 2: confirm it Do this before any investigation. A meaningful fraction of alerts do not reproduce. The whole confirmation is **one command**, not two manual pushes. `--alert` runs exactly the tests in the alert summary and compares your working revision against the base revision your patch sits on, pushing both sides for you: ``` ./mach try perf --alert --rebuild 10 ``` Read `references/confirm.md` before running it — it covers getting the local repo onto the right base revision, how many retriggers are actually needed, the pgo/shippable trap, and how to read the result. **Never push to try without explicit approval from the user.** Show the exact command, say roughly what it will cost in CI, and wait. Then branch on the outcome: - **Reproduces** — continue to step 3. - **Does not reproduce** — do not start optimizing. Go to `references/close-out.md`; this is likely noise or an unrelated patch in the same push. - **Ambiguous / overlapping distributions** — more retriggers, or narrow to the single most-regressed test. See `references/confirm.md`. ## Step 3: find the cause Start with the Before/After profiles already linked in the bug — they are free and specific to the regressed test. Hand them to the **profiler-analysis** skill rather than fetching them yourself; `WebFetch` on a profiler URL only retrieves the UI shell. Compare against what the patch actually changed. Most Perfherder regressions on speedometer-class benchmarks come from work added to a hot path, a lost fast path, extra allocation, or added main-thread sync work. ## Step 4: iterate to a fix Read `references/iterate.md`. The loop is change → narrow try push → check, and the entire point of that file is keeping each round cheap: narrow the test set, drop to one platform, and only re-run the full alert set for the final confirmation. ## Step 5: close out Read `references/close-out.md` for the possible resolutions, the bug fields to set, and who to talk to when the right answer is "this regression is acceptable." ## Cost discipline This skill exists partly to keep regression work cheap, in CI and in tokens. - One push per question. `--alert` gives base and new together; do not push twice. - Confirm broadly once, then iterate narrowly. Full alert-set reruns are for the final check only. - Do not read the reference files up front. Read the one for the step you are on. - Read profiles through profiler-analysis, not by downloading them into context.