name: Mutation testing # Repo-wide, sliced, incrementally cached mutation testing via Stryker. Every package is gated on its own merged score against the break threshold recorded in its stryker.config.ts (derived by the rule documented on `thresholds.break` in stryker.shared.ts), and the mutation-result job at the bottom is the one stable check context that reports the outcome. # # Three kinds of run share this file, and what each gates differs. A pull request run covers the packages the change can move a mutation result for (mutation-scope.ts decides, from the `_test:mutation` inputs in turbo.json, the packages' dependency graph and the lockfile) and, of those, only the ones that plan into a single slice; a change that cannot alter any result, such as documentation, starts no mutation job. A scheduled run covers every package, and is the run that gates the large ones and gives main a completed result. A dispatched run covers every package, or the ones named in its `packages` input, on whatever ref it is started against, which is how a large package is measured for a branch before it merges. Nothing runs on a push to main, so a threshold is checked before a merge only for the packages a pull request runs, and for every other package on the next scheduled run. A scheduled or dispatched run on main that finds a package failing opens, or updates, the one issue titled "Mutation testing is failing on main" with the packages currently failing (report-mutation-outcome.ts merges a run's own result into whatever the issue already listed, so a run scoped to only some packages neither drops nor hides a failure in one it did not cover), and closes the issue once none remain. # # Every run draws on one pool of runners that the whole account shares, the required checks included, so the parallelism of each kind of run is derived in plan-mutation-slices.ts from the account's documented concurrent-job limit rather than chosen here. on: pull_request: schedule: # Once a day, off the hour so it does not queue with the many other schedules that fire on it. - cron: "23 2 * * *" workflow_dispatch: inputs: packages: description: Space-separated workspace package names to mutate. Empty means every package. required: false default: "" tracking_issue_title: description: Only for exercising the tracking issue from a branch. A title to manage instead of the real one, which only a run on main manages. required: false default: "" # GitHub allows one running and one pending run per concurrency group, and a newer run replaces the pending one (docs.github.com, "Control the concurrency of workflows and jobs"). A single group for every ref therefore starves everything but the first two runs: each new push cancels the pending run before it starts a job. So a pull request gets a group of its own, and a newer push to that pull request cancels only that pull request's stale run. Every other run (scheduled, dispatched) shares one group per ref and is never cancelled, so a full run always finishes and at most one more waits behind it. concurrency: group: ${{ github.event_name == 'pull_request' && format('mutation-pr-{0}', github.event.pull_request.number) || format('mutation-{0}', github.ref) }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} permissions: contents: read env: # Mirrors ci.yml's own TURBO_FLAGS exactly: a pull request scopes to the packages it actually touches (and their dependents, via turbo's own dependency-aware --affected), a scheduled or dispatched run covers the whole workspace. TURBO_FLAGS: ${{ github.event_name == 'pull_request' && '--affected' || '' }} # Mirrors ci.yml's own TURBO_SCM_BASE exactly, and is required for --affected to mean anything on a pull_request: actions/checkout's detached-HEAD checkout has no local branch named `main` (only `origin/main`), so without an explicit base turbo cannot resolve the literal `main` ref it falls back to, warns "unable to detect git range, assuming all files have changed", and silently treats every package as affected. TURBO_SCM_BASE: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || '' }} jobs: plan: name: Plan mutation slices runs-on: ubuntu-latest timeout-minutes: 10 outputs: matrix: ${{ steps.plan.outputs.matrix }} has-packages: ${{ steps.plan.outputs.has-packages }} max-parallel: ${{ steps.plan.outputs.max-parallel }} steps: - uses: actions/checkout@v7 with: # --affected compares against the base branch; the plan script's own turbo dry-run needs that history present, same reason ci.yml's own affected-scoped jobs fetch it. fetch-depth: 0 # Not the setup-workspace composite action here: it declares no outputs of its own, so a step's $GITHUB_OUTPUT writes inside it never reach this job's `outputs:` block above. commitlint's own affected-range step has the identical shape (needs both install and a step output) and solves it the same way: plain checkout/setup steps, then the command inline. - uses: pnpm/action-setup@v6 - uses: actions/setup-node@v7 with: node-version-file: .tool-versions cache: pnpm - run: pnpm install --frozen-lockfile - name: Plan the mutation slices id: plan env: MUTATION_EVENT: ${{ github.event_name }} # The dispatch input reaches the script as data through the environment, never as text in the command line. MUTATION_PACKAGES: ${{ inputs.packages }} run: node --experimental-strip-types .github/scripts/plan-mutation-slices.ts $TURBO_FLAGS # One job per package slice: a package whose estimated cold cost fits the slice budget is one job, a larger one is split by file (plan-mutation-slices.ts holds the estimate and the derivation of both the slice size and each job's own timeout). A job's incremental cache is keyed by its package and slice, so it restores its own previous report rather than another package's. mutation-test: name: Mutation test (${{ matrix.package }} ${{ matrix.slice }}/${{ matrix.sliceCount }}) needs: plan if: needs.plan.outputs.has-packages == 'true' runs-on: ubuntu-latest timeout-minutes: ${{ matrix.timeoutMinutes }} strategy: fail-fast: false # A ceiling on how many of this run's jobs occupy the account's shared runner pool at once, computed by the plan job for the kind of run (see maxParallelFor in plan-mutation-slices.ts): the account's documented concurrent-job limit, a stated share of it for mutation as a whole, split between full runs and pull request runs. It bounds one run, not the repository, which is why the pull request share is divided by the number of pull request runs expected at once. max-parallel: ${{ fromJSON(needs.plan.outputs.max-parallel) }} matrix: ${{ fromJson(needs.plan.outputs.matrix) }} env: # The package and its file list reach the shell as environment variables, never interpolated into the script text: a slice's file list is built from file names in the checked-out change, which a pull request controls. PACKAGE: ${{ matrix.package }} MUTATE: ${{ matrix.mutate }} # Each job mutates only part of a package, so its own score is not the package's; the mutation-result job gates the merged score instead (see stryker.shared.ts). MUTATION_DEFER_BREAK: "true" steps: - uses: actions/checkout@v7 with: fetch-depth: 0 # Stryker's own incremental report: reuses a prior run's per-mutant kill or survive result for anything unchanged since it was written, instead of re-testing every mutant every run. Restored by the slice's own key first, then by any earlier slice layout of the same package, since a report for a file stays valid wherever the file was sliced, and for either fallback only within the same configHash (plan-mutation-slices.ts's strykerConfigHash), a fingerprint of the package's stryker.config.ts and the shared stryker.shared.ts it is built through. Stryker's incremental mode decides what to reuse purely from whether a mutated file's own content changed, with no idea the configuration governing what gets mutated changed too, so without configHash in the key a mutate-glob or shared-option change can restore a report computed under the old configuration and silently carry its mutants forward as though nothing changed. # # run_attempt in the save key because actions/cache silently drops a save whose key already exists, and a re-run of the same run reuses run_id. - uses: actions/cache/restore@v6 with: path: ${{ matrix.directory }}/reports/stryker-incremental.json key: mutation-incremental-${{ matrix.cacheKey }}-${{ matrix.configHash }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | mutation-incremental-${{ matrix.cacheKey }}-${{ matrix.configHash }}- mutation-incremental-${{ matrix.package }}-${{ matrix.configHash }}- - uses: ./.github/actions/setup-workspace with: # matrix.cacheKey in the turbo cache's own task name for the identical reason as the Stryker cache above: the composite action's key is task-scoped, so slices of different packages must not share one. task: mutation-${{ matrix.cacheKey }} # `--` hands the file list to Stryker's own --mutate, replacing the package's default globs for this slice only. An empty list means the package runs whole. command: | args=() if [ -n "$MUTATE" ]; then args=(-- --mutate "$MUTATE") fi pnpm exec turbo run _test:mutation --filter="$PACKAGE" "${args[@]}" # Saved only by scheduled and dispatched runs, never by a pull request: a pull request restores what the last full run on main saved, so an entry it wrote would only be a write-once one, since the pull request's next run restores main's newest anyway and nothing else can see a pull request ref's entry. The prune-caches job keeps the newest entry per slice on main. # Saved by a step that runs even when an earlier step failed or was cancelled: GitHub runs `always()` steps on cancellation, and a job that exceeds its timeout is cancelled the same way. What it can save is limited by Stryker, which writes its incremental report only when its run finishes, so a run killed by the timeout leaves nothing to save. That is why slice size and timeout are derived (plan-mutation-slices.ts) so a slice finishes well inside its limit, and why a slice that does not finish costs one slice's rerun rather than the whole package's. - uses: actions/cache/save@v6 if: always() && github.event_name != 'pull_request' with: path: ${{ matrix.directory }}/reports/stryker-incremental.json key: mutation-incremental-${{ matrix.cacheKey }}-${{ matrix.configHash }}-${{ github.run_id }}-${{ github.run_attempt }} # The json report is renamed after its slice before upload, so that every slice's report lands in one flat directory with a distinct name when the result job downloads them. The layout of a downloaded artifact otherwise depends on how many artifacts matched, which a gate reading fixed paths cannot tolerate. - name: Name the json report after its slice if: always() env: DIRECTORY: ${{ matrix.directory }} CACHE_KEY: ${{ matrix.cacheKey }} run: | if [ -f "$DIRECTORY/reports/mutation/mutation.json" ]; then cp "$DIRECTORY/reports/mutation/mutation.json" "$DIRECTORY/reports/mutation/$CACHE_KEY.json" fi - uses: actions/upload-artifact@v7 if: always() with: name: mutation-json-${{ matrix.cacheKey }} path: ${{ matrix.directory }}/reports/mutation/${{ matrix.cacheKey }}.json if-no-files-found: ignore - uses: actions/upload-artifact@v7 if: always() with: name: mutation-html-${{ matrix.cacheKey }} path: ${{ matrix.directory }}/reports/mutation/mutation.html if-no-files-found: ignore # The single check context that reports the outcome. The mutation-test job above is a matrix whose width the plan job computes per run from the affected package set, so no slice's own context could be a stable name to require. This job always runs, and succeeds when nothing needed mutating. It is not one of the required checks on main and this workflow does not make it one. mutation-result: name: Mutation testing result needs: [plan, mutation-test] if: always() runs-on: ubuntu-latest timeout-minutes: 15 permissions: contents: read # To open, update and close the one tracking issue below; nothing else in this workflow writes. issues: write steps: - uses: actions/checkout@v7 - uses: pnpm/action-setup@v6 - uses: actions/setup-node@v7 with: node-version-file: .tool-versions cache: pnpm - run: pnpm install --frozen-lockfile if: needs.plan.outputs.has-packages == 'true' - uses: actions/download-artifact@v8 if: needs.plan.outputs.has-packages == 'true' with: pattern: mutation-json-* path: mutation-reports # All slices' reports into the one directory, whatever their number: without this the layout differs between one matching artifact and several. merge-multiple: true - name: Gate each package's merged score if: needs.plan.outputs.has-packages == 'true' env: # The matrix crosses into the script as data through the environment, not as text in the command line. MATRIX: ${{ needs.plan.outputs.matrix }} run: node --experimental-strip-types .github/scripts/gate-mutation-scores.ts mutation-reports "$MATRIX" "$RUNNER_TEMP/mutation-gate-result.json" # A red scheduled or dispatched run on main must not go unnoticed, so one tracking issue, titled "Mutation testing is failing on main", lists the packages currently failing (report-mutation-outcome.ts): a package this run covers is updated to its own result, one it does not cover keeps whatever the issue already said, so a run scoped to only some packages can neither hide nor falsely clear a failure in one it left untouched. The issue is opened the first time the merged list is non-empty, updated while it stays non-empty, and closed once it empties and the run itself did not break outright. It is found by title rather than by number, since a new one is opened after one is closed. Only a run on main manages it: a run on any other ref manages nothing, or the test title it was given, and can never target the real one. - name: Keep the tracking issue in step with this run if: always() && (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') env: GH_TOKEN: ${{ github.token }} PLAN_RESULT: ${{ needs.plan.result }} SLICE_RESULT: ${{ needs.mutation-test.result }} MUTATION_ISSUE_TITLE: ${{ inputs.tracking_issue_title }} RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} run: node --experimental-strip-types .github/scripts/report-mutation-outcome.ts "$RUNNER_TEMP/mutation-gate-result.json" "$RUN_URL" - name: Fail when the plan failed # Last, so the tracking issue above still hears about a failed plan. A plan failure is a real failure (nothing can be said about the mutation state); a plan that found no affected packages is a pass (nothing to mutate). if: always() && needs.plan.result != 'success' run: | echo "::error::Mutation slice planning failed; see the Plan mutation slices job." exit 1 # Keeps the newest incremental entry of each slice on main and deletes the older ones, so the entries stay at one per slice however many full runs there are. The key ends in the run id and attempt, so entries are grouped by the key with that suffix removed. It is its own job so that actions: write is held by a job that runs no workspace code and only on main, the same shape as ci.yml's prune-caches. prune-caches: name: Prune mutation caches needs: mutation-test if: always() && github.event_name != 'pull_request' && github.ref == 'refs/heads/main' runs-on: ubuntu-latest timeout-minutes: 10 permissions: actions: write env: GH_TOKEN: ${{ github.token }} GH_REPO: ${{ github.repository }} steps: - name: Keep the newest incremental cache entry of each slice run: | gh cache list --ref refs/heads/main --key mutation-incremental- --limit 1000 --json id,key,createdAt \ --jq 'group_by(.key | sub("-[0-9]+-[0-9]+$"; "")) | .[] | sort_by(.createdAt) | .[:-1] | .[].id' \ | while read -r id; do gh cache delete "$id"; done