---
name: cms-stuck-pr-triage
description: Diagnose "stuck" or "repeatedly failing" runs of cms-publish-loop-host.yml, cms-publish-loop-prod.yml, canary-prod.yml, or any workflow that opens a `cms/<col>/<slug>` PR via Decap and waits for it to auto-merge. The workflow itself is rarely the bug — almost always the cause is a long-lived BLOCKED PR whose CI ran on a stale base, or a PR stuck BLOCKED with all checks green. Use when the user says a publish-loop is "stuck", cancels & restarts a run, or asks why a daily canary keeps failing.
---

# CMS publish-loop / canary stuck-PR triage

The publish-loop and canary workflows in this platform (`cms-publish-loop-host.yml`, `cms-publish-loop-prod.yml`, `canary-prod.yml`, plus the preview loops in `cms-preview-loops.yml`) all follow the same shape: open a `cms/<col>/<slug>` PR via Decap CMS → wait for the editorial-workflow auto-merge to fire → wait for `deploy-production.yml` to finish → assert the live URL.

When a run "gets stuck" it almost never means the workflow is misbehaving. The default failure mode is:

> A *prior* run opened a `cms/<col>/<slug>` PR, that PR's required checks fail or hang, auto-merge can't fire, the spec waits up to 13–40 minutes, and times out with `Timed out waiting for PR #N to merge`. The current run then opens a *new* PR (or pushes onto the same one) — same fate.

Most user-facing symptoms ("stuck for 30 min", "cancelled and restarted three times", "Docker config warning at top of log") are not the bug. The bug is in the open-PR queue.

## When to invoke

- The user says any of: "publish-loop is stuck / failing / not succeeding", "I cancelled a stuck run", "the canary keeps failing", "this workflow has been running for an hour".
- A `gh run view` log ends with `Error: Timed out waiting for PR #N to merge`.
- A workflow run was just cancelled and a new one was kicked off — before suggesting any workflow changes, finish this triage.
- After landing changes that affect e2e spec selection (lane filtering, the spec set itself), proactively audit open `cms/*` PRs whose CI ran against the pre-fix tree.

## A diagnostic script exists, but the procedure below is still manual

The platform *does* ship `scripts/diagnose-stuck-pr.js` — a read-only diagnostic that enumerates the suspect PRs, classifies each (`BLOCKED` by failing checks, `DIRTY` with a real conflict, `DIRTY` with a newline-only conflict the auto-resolver will fix, a queued `deploy-production` run holding the lane), and prints a Markdown report. It is wired up two ways: (1) `e2e/with-stuck-pr-diagnostic.js` is the in-spec wrapper (Layer 1) — when a wait helper (`waitForMerge`, `waitForCmsPullRequest`, etc.) is about to throw a `Timed out waiting for …` error, the wrapper spawns the script and appends its Markdown output to that error's message; (2) `cms-publish-loop-host.yml` references it in a comment for the same failure class. There is no separate standalone `Diagnose stuck PRs` workflow step, and no auto-generated PR *comment* from the script itself — the diagnostic output lands in the spec's own error message / test-failure output, which is what `post-failure-comment` then surfaces as a PR comment (markers `host-loop-failure-summary`, `prod-mutate-failure-summary`, `preview-loop-failure-summary`).

So when a publish-loop spec times out, that failure-summary comment often already contains the offending PR number, its merge state, and the diagnostic's classification. Read that comment first — but the script only diagnoses the one PR's context around a specific timeout; it doesn't run the fleet-wide enumeration below. The enumeration / classification / remediation procedure below is still manual.

## Procedure

### 1. List the open `cms/*` PRs and their merge state

```bash
gh pr list --state open --search "head:cms" --limit 1000 \
  --json number,title,mergeStateStatus,createdAt \
  --jq '.[] | [.createdAt, .number, .mergeStateStatus, .title] | @tsv'
```

A PR with `mergeStateStatus: BLOCKED` is the prime suspect, especially if it's been open for more than an hour. `UNKNOWN` is also worth checking — GitHub sometimes returns that when checks are pending or the cached status is stale.

### 1b. If the BLOCKED PR's only red check is ONE `e2e / project (…)` lane, suspect the install, not the tests

This is the most common cause of a loop timing out, and the loop's own error points
somewhere else entirely. The signature:

- The loop fails with `Timed out waiting for the URL to reflect the change … NO
  deploy-production run fired for your merge — the chain never fired`, with
  `in-flight: 0, queued: 0`.
- Its canary PR is `BLOCKED`, and the ONLY failures are
  `e2e / project (<one lane>)` plus the aggregating `e2e / e2e`.
- That lane's log ends in the install step, not in a test:

```
E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process NNNN (apt-get)
Error: Installation process exited with code: 100
::error::playwright install failed 3 time(s)
```

**What happened:** a GitHub-hosted runner runs its own apt
(`unattended-upgrades` / `apt-daily`) for the first minute after boot, and
`playwright install --with-deps` shells out to apt-get itself, so it races that
holder. It is almost always a **webkit** lane, because webkit's dep set is the
biggest (~2x chromium's apt time), giving it the widest window to overlap.

The blast radius is what makes this worth its own step: one lane's install failure
fails the aggregating gate, a red required check BLOCKS the canary PR, and the loop
then waits out its whole budget for a merge that can never happen. On 2026-08-07 it
took out the host loop AND prod-mutate in the same hour (adamdaniel.ai jobs
92892148211 and 92898220347).

**Fix:** re-run the failed jobs (`gh run rerun <run-id> --failed`) — the lock is
gone by then, so the canary PR unblocks and auto-merges on its own. Platform
v0.1.71+ prevents it: the install composite drops `DPkg::Lock::Timeout` into
`apt.conf.d` so apt WAITS for the lock instead of erroring, and retries back off
(15 s / 30 s) instead of firing back-to-back into the same window. If you see this
on v0.1.71 or later, the holder outlasted `apt-lock-timeout-seconds` (300 s) —
raise it rather than adding more retries.

### 1c. Read the loop's OWN verdict — since v0.1.76 it says whether the PR merged

The loop's timeout message now distinguishes "the PR has not merged" from "the
deploy chain missed", so it usually tells you which leg to triage before you
enumerate anything:

| Verdict in the message | What it means | Where to go |
|---|---|---|
| `the cms PR #N had NOT MERGED yet` (`pr-awaiting-required-check`) | a required check is still pending, or the merge mechanism has not fired. **An unmerged PR cannot have a deploy**, so the deploy chain is NOT the failing leg | §2/§3 below — the PR queue |
| `the cms PR #N could NOT MERGE` (`pr-required-check-red`) | a required check went **red**; the message names it | §1b (an install failure?) then §2 |
| `NO deploy-production run fired for your merge — the chain never fired` (`no-deploy-fired`) | the PR **IS** merged and no deploy ran — a genuine trigger problem | the deploy trigger / editorial-workflow, not the PR queue |
| `your deploy-production run … DID complete, but the URL never served the marker` (`deploy-completed-url-missing`) | S3 sync / CloudFront / cache in the serve layer | not a PR problem at all |

**In a log from BEFORE v0.1.76, `no-deploy-fired` is NOT evidence of a trigger
problem.** The extender asked only "is the deploy lane idle?", and before the
merge lands the lane is idle for an entirely innocent reason — nothing can
deploy yet — so a merely-SLOW auto-merge was reported as "the chain never
fired". Live instance: adamdaniel.ai run **31107474927** (2026-08-06 host loop)
killed `cms-tags-lifecycle` at exactly that message after **908 s**, `in-flight:
0, queued: 0`, while the auto-merge was simply still pending — well inside the
documented ~30-min latency, and the other two host-loop specs passed in the same
run. Same class in runs 30915982319 and 30822288078. So on an old log, re-read
the message as "the URL did not reflect", check whether the PR had merged at
that timestamp, and triage the queue from §2.

Two scoping notes worth knowing when a verdict looks absent or vague. **Only 6
of the 15 extender call sites are PR-anchored** (the forward/create legs,
including the one that failed live); the other 9 stay bare on purpose because no
PR for that leg is in scope — the delete legs discover their recovered PR inside
a poll loop without capturing it, and `cms-unpublish-republish` never opens a
tracked PR at all. A bare leg keeps the pre-#215 behaviour, so it can still emit
`no-deploy-fired` without having asked the merge question. And the verdict is
deliberately **list-free** — it never checks a `requiredContexts` list, because
there is no single source of one in this repo (the harness default is the bare
`["validate-content"]`, adamdaniel's nudge lists 6, jodidaniel's 1, and the
`examples/site` template still ships a stale 9-context list matching no real
check-run name). "Has the PR merged?" needs none of it, and check-run STATE
alone separates *awaiting* from *red*.

### 2. For each BLOCKED PR, find the failing checks and the base it ran against

```bash
gh pr view <N> --json mergeStateStatus,statusCheckRollup,baseRefOid,autoMergeRequest \
  --jq '.mergeStateStatus, .baseRefOid, .autoMergeRequest, [.statusCheckRollup[] | select(.conclusion=="FAILURE" or .status=="IN_PROGRESS") | {name, conclusion, status}]'
```

Two diagnostic questions the output answers:

- **Is the failure a current bug or a stale-base artefact?** Compare `baseRefOid` to current `origin/main` (`git log origin/main --oneline -1`). If the PR was opened against an older base than a recent fix that landed on main, its CI ran against the pre-fix tree. Re-running the same checks against current main would likely pass.
- **Is auto-merge enabled?** If `autoMergeRequest` is null, the spec failed to enable auto-merge in the first place — that's a different bug (look at the spec's `gh api PUT pull/N/merge` shim error). If it's set, the PR is waiting on its required checks to pass before GitHub fires the merge.

### 3. The "BLOCKED but every check is green" case

A distinct failure mode: `mergeStateStatus: BLOCKED`, `autoMergeRequest` already populated, yet **every** required check's latest run is SUCCESS / NEUTRAL / SKIPPED. This is a GitHub merge-state-evaluator caching bug — two auto-merge-enabling label events landed in the same second, GitHub cached the BLOCKED snapshot taken mid-mutation, and no later event re-triggers evaluation. The PR sits green-but-BLOCKED until the nightly sweep closes it.

```bash
# Confirm the pattern: BLOCKED + auto-merge on + no non-green required check
gh pr view <N> --json mergeStateStatus,autoMergeRequest,statusCheckRollup \
  --jq '{state: .mergeStateStatus, automerge: (.autoMergeRequest != null),
         non_green: [.statusCheckRollup[] | select(.conclusion=="FAILURE" or .conclusion=="CANCELLED" or .status=="IN_PROGRESS") | .name]}'
```

`cms-automerge-nudge.yml` handles this automatically. It declares a `*/5` cron, but GitHub throttles scheduled workflows under load — in practice it fires roughly every 45-90 min, not every 5, so worst-case time-to-merge drops from "until the sweep closes it" to that ~45-90 min window. Its recovery mechanism is an explicit synchronous `pulls.merge` (SQUASH) against any `automated-test`-labelled PR matching exactly this pattern — a real merge forces GitHub to re-evaluate mergeability fresh, dislodging the stale BLOCKED snapshot that a no-op `enablePullRequestAutoMerge` re-call was observed NOT to reliably dislodge. Re-enabling auto-merge is kept only as a **fallback**, tried when the explicit merge momentarily 405s. It only touches PRs that (1) carry `automated-test`, (2) already have auto-merge enabled OR have a non-`main` base (`basePreviewOnly` — since v0.1.52: a `cms/preview-only` PR can never get `autoMergeRequest` populated in the first place, since its base branch has no required-status-check protection for GitHub's auto-merge to key off, so the null-`autoMergeRequest` check alone would skip it forever), (3) are BLOCKED-or-unevaluated, and (4) have all required checks green — so it never re-enables auto-merge a human disabled on a main-based PR and never touches a real editor's draft.

As a manual fallback (or if you don't want to wait for the next cron pass), nudge the stuck PR by hand — re-enabling auto-merge re-evaluates the merge state:

```bash
gh pr merge <N> --auto --merge   # no-op re-enable; re-triggers GitHub's merge-state eval
```

### 4. Decide: rebase, close, nudge, or wait

- **Stale-base, fix is on main**: the cleanest move is to *close* the stale PR (Decap will open a fresh one on the next workflow run, on top of current main):
  ```bash
  gh pr close <N> --delete-branch --comment "closing stale CMS PR; CI ran on pre-fix tree, next workflow run opens a fresh one"
  ```
  Rebase + force-push also works but is more fragile — if the spec used a content-based slug, the next run rewrites the same branch and races with the rebase.

- **BLOCKED with all checks green**: nudge auto-merge (§3) rather than closing — the PR is mergeable, GitHub just didn't notice.

- **Real failure, fix not yet on main**: investigate the failing check. `e2e`/`parity` failures on a CMS PR are usually content-spec drift (a spec hardcodes a fixture that was deleted from main); see `e2e/content-fixtures.js`'s discovery helpers for the dynamic-discovery pattern.

- **Pending checks, recent PR**: if the PR was opened in the last few minutes and `statusCheckRollup` shows `IN_PROGRESS`, just wait — the publish-loop run that's currently watching it should succeed when the checks settle.

- **Stale Decap workflow state on a fixed-branch PR (`decap-cms/pending_publish` etc.)**: Decap reuses a fixed branch per entry — `cms/e2e/canary-post` for the e2e canary, `cms/posts/<slug>` for posts. When a prior run leaves the PR open in a non-Draft editorial-workflow state (e.g. `decap-cms/pending_publish` or `decap-cms/pending_review`), the next run's spec edits the entry, Decap pushes onto the SAME branch, the existing PR's labels stick around, and Decap's UI shows "Status: Ready" (or "In Review") rather than "Status: Draft". The publish-loop spec waits for Status: Draft and times out. Symptom in the test log:

  ```
  Error: locator.click: Test timeout of 1200000ms exceeded.
  Call log:
    - waiting for getByRole('button', { name: /^Status:\s*Draft$/i })
  ```

  Fix: close the stale PR with `gh pr close <N> --delete-branch`. Decap will create a fresh branch on the next Save, with `cms/draft` from `cms-editorial-workflow.yml`'s opened-event handler, and the spec sees Status: Draft. To check if a Decap PR is in this state:

  ```bash
  gh pr view <N> --json labels --jq '[.labels[].name]'
  # Look for any of: decap-cms/pending_publish, decap-cms/pending_review,
  # decap-cms/ready  →  next run's Status:Draft wait will block
  # cms/draft, decap-cms/draft  →  fine; spec will see Status:Draft
  ```

- **`dirty` (merge conflict)**: a `cms/*` PR can go `dirty` when its branch and main both touched the same trailing newline. `auto-resolve-newline-conflict.yml` re-resolves newline-only conflicts on the next run and lets the PR merge; if the conflict is more than whitespace, a manual rebase is needed. Check the diff before assuming it's auto-resolvable.

### 5. After cleaning the queue, re-trigger the workflow

```bash
gh workflow run cms-publish-loop-host.yml         # or cms-publish-loop-prod.yml / canary-prod.yml
gh run watch                                       # follow the new run live
```

The new run opens a fresh `cms/<col>/<slug>` PR on top of current main; with the queue clean it should auto-merge cleanly.

### 6. Delete-spec specific: `delete:` flag on the collection

`cms-delete-published-preview.yml`'s delete spec clicks the Decap UI's "Delete published entry" menuitem. Decap renders that menuitem ONLY when the entry's collection has `delete: true` in `admin/config.yml`. If the collection is `delete: false` the status menu opens but renders an empty list, the menuitem never appears, and `getByRole("menuitem", { name: /delete (published )?entry/i }).click()` times out at the action-timeout.

When triaging a stuck delete-spec, before chasing infrastructure, check the `delete:` flag for the relevant collection: `grep -A1 "name: e2e\|name: posts\|name: <collection>" admin/config.yml`. NOTE (v0.1.4+): the live `config.yml` is **rendered** by the theme gem at build time — there is no hand-authored source `admin/config.yml` in a consuming site. Grep the **rendered** output `_site/admin/config.yml` (or fetch the served `/admin/config.yml`), or the platform's template `theme/admin/config.base.yml`.

### 7. Empty status-menu pattern (Published button opens, nothing inside)

The artifact's `error-context.md` snapshot looks like:

```yaml
- button "Published" [expanded] [active] [ref=...]
  - menu:
    - list      # ← empty, no items
```

This is the same `delete: false` symptom from §6 — the menu rendered, but Decap had no items to put in it because the collection's capability flags forbade them. Could also indicate the spec's selector matched a non-clickable element (rare; check the UI's actual structure for the entry state).

## What ISN'T the bug (red herrings to ignore unless you've ruled out the above)

- **`WARNING: Error loading config file: open /root/.docker/config.json: permission denied`** at the top of a job log. A benign GHA quirk. Ignore.
- **Workflow run dispatched against `main` rather than a feature branch.** The cron trigger fires from `main` daily; `workflow_dispatch` from `main` is identical. The user's intuition that "running it from main caused this" is a mis-attribution; the cause is in the open-PR queue regardless of which ref dispatched the run.
- **Concurrency cancellation.** The publish-loop workflows use `cancel-in-progress: false`, so cron + dispatch + PR runs queue rather than killing each other. A run of "cancelled" runs in the run-list is almost always user-cancellations of stuck runs, not concurrency interference.

## Why this matters

The publish-loop is the only test that exercises the live Decap → editorial-workflow → auto-merge → deploy-production chain. When it's broken, the only pre-prod safety net for the actual publish flow is broken. Every minute spent restarting cancelled runs without diagnosing the open PR queue is a minute the canary is silently red. The first action when one of these workflows looks "stuck" is *always* `gh pr list --state open --search "head:cms"`.

## Cleanup vs triage

Don't manually close every orphan PR you find — `.github/workflows/sweep-stale-cms-prs.yml` runs nightly and handles the routine cases. Triage that finds the *root cause* of why the sweep alone isn't enough is the goal; if you're just closing leftovers, use the workflow's `workflow_dispatch` with `dry_run: false` instead.

The sweep has three tiers: branch-prefix safelist (closes + deletes), `automated-test` label (closes only — Decap reuses branches), and orphaned branches with no open PR (deletes). Per-PR opt-out via the `keep` label; per-branch opt-out via `[sweep-keep]` in the tip commit message. See `sweep-stale-cms-prs.yml`'s header for the full table.

**Pagination rule** (applies to any future addition): `gh pr list` defaults to `--limit 30` and silently truncates above that. `--paginate` is NOT a flag for `gh pr list` — that's gh-api-only. Always `--limit 1000` for top-level listings, `--limit 1` for existence checks. The sweep workflow's comments document this inline; mirror it in any new sweep tier or related cleanup script.

## Reference

- Triage workflows the platform ships: `.github/workflows/sweep-stale-cms-prs.yml` (nightly cleanup), `.github/workflows/auto-resolve-newline-conflict.yml` (re-resolves newline-only `cms/*` conflicts), `.github/workflows/cms-automerge-nudge.yml` (recovers green-but-BLOCKED CMS PRs via an explicit merge; its `*/5` cron is throttled to roughly 45-90 min in practice).
- Loop workflows: `.github/workflows/cms-publish-loop-host.yml`, `cms-publish-loop-prod.yml`, `canary-prod.yml`, `cms-preview-loops.yml`.
- Failure surfacing: `post-failure-comment` composite action (markers `host-loop-failure-summary`, `prod-mutate-failure-summary`, `preview-loop-failure-summary`); inline error augmentation by `e2e/with-stuck-pr-diagnostic.js`.
- Spec-helper that owns the timeout: `e2e/github-actions-poll.js` (the `Timed out waiting for PR #N to merge` error originates in its `waitForMerge`).
- Content-fixture discovery (the de-coupled pattern that prevents PR-CI failures when fixtures change): `e2e/content-fixtures.js`.