---
name: pr-monitor
description: Runs a CI health check on an ai-dynamo/dynamo pull request — report check status, explain why full CI has or has not triggered, root-cause failed job logs, cross-reference failures against main to separate PR-caused regressions from pre-existing flakes, and flag unexpected skips. Use when asked whether a PR's CI is healthy, why checks are failing or missing, or whether a failure is a flake.
license: Apache-2.0
user-invocable: true
disable-model-invocation: true
metadata:
  author: NVIDIA
  tags:
    - dynamo
    - github
    - ci
    - pull-request
---

# PR CI Monitor

<!--
SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: CC-BY-4.0
-->

Perform a full health check on a Dynamo pull request. Takes a PR number as argument (e.g., `/dynamo:pr-monitor 6554`).

## Step 1: PR Overview

Gather PR metadata and determine what CI should look like for this PR.

```bash
gh pr view $PR_NUMBER --repo ai-dynamo/dynamo --json title,body,author,state,isDraft,additions,deletions,changedFiles,labels,reviewDecision,headRefName,baseRefName

gh pr diff $PR_NUMBER --repo ai-dynamo/dynamo --name-only
```

**Check if full CI should be running.** The dynamo repo has two tiers of CI:

- **Lightweight (`pre-merge.yml`)**: Triggers on all `pull_request` events. Runs pre-commit, copyright checks, DCO, and optionally rust-clippy/rust-tests (if rust filter matches). This always runs.
- **Full pipeline (`pr.yaml`, `container-validation-dynamo.yml`)**: Triggers on `push` to `main` or `pull-request/[0-9]+` branches ONLY. This includes docker builds, GPU tests, deploy tests. It does NOT trigger on regular PR branches.

To get full CI on a PR, a `pull-request/$PR_NUMBER` branch must exist (created by copy-pr-bot after an NVIDIA maintainer approves). Check:

```bash
gh api repos/ai-dynamo/dynamo/branches/pull-request/$PR_NUMBER 2>/dev/null
```

If the branch does not exist, full CI has not been triggered. Common reasons:

1. **Awaiting approval**: An NVIDIA maintainer needs to comment `/ok to test <commit_sha>` on the PR to create the branch and trigger full CI. This applies to both fork PRs and internal PRs from authors not yet in the approval list.
2. **DCO failure**: Commits are not signed (`Signed-off-by` line missing). Check for a DCO bot comment. Fix: author signs their commits (see `DCO.md`).
3. **Unverified commit signature (eligible fork automatic approval)**: The automatic `/ok to test` flow requires GitHub to report every PR commit as `Verified`. A DCO sign-off alone does not meet this requirement. This applies only after the PR has an automatic approval path; signing commits does not create that path. Check the workflow's unsigned-commit comment or inspect `.commit.verification` for every PR commit; sign each unverified commit, then push again. A maintainer can manually review the current head and comment `/ok to test <sha>` when automatic approval is unavailable.
4. **Draft PR**: Some checks may not run until marked ready for review.

For a fork PR that is otherwise eligible for automatic approval, check every commit's
verification result:

```bash
gh api --paginate "repos/ai-dynamo/dynamo/pulls/$PR_NUMBER/commits?per_page=100" --jq '.[].sha' | while read -r sha; do
  gh api "repos/ai-dynamo/dynamo/commits/$sha" --jq '"\(.sha[0:7]) verified=\(.commit.verification.verified) reason=\(.commit.verification.reason)"'
done
```

**Verify which workflows actually ran** against the PR's HEAD commit:

```bash
# Get the HEAD SHA
HEAD_SHA=$(gh pr view $PR_NUMBER --repo ai-dynamo/dynamo --json headRefOid --jq '.headRefOid')

# Check which workflow runs exist for this SHA
gh api "repos/ai-dynamo/dynamo/actions/runs?head_sha=$HEAD_SHA" --jq '.workflow_runs[] | {name: .name, status: .status, conclusion: .conclusion}'
```

Compare the workflows that ran against what's expected. If only `Pre Merge`, `Copyright Checks`, `DCO Commenter`, etc. appear but NOT `PR` or `Dynamo Validation`, then full CI was not triggered.

**Determine which CI filters are active.** If full CI ran, read the `changed-files` job output from the actual workflow run — this is the authoritative source for which filters are `true`:

```bash
# Find the PR workflow run ID
PR_RUN_ID=$(gh api "repos/ai-dynamo/dynamo/actions/runs?head_sha=$HEAD_SHA" --jq '.workflow_runs[] | select(.name == "PR") | .id')

# Get the changed-files job output (look for the filter results in the logs)
gh api "repos/ai-dynamo/dynamo/actions/runs/$PR_RUN_ID/jobs" --jq '.jobs[] | select(.name == "changed-files") | {id: .id, status: .status, conclusion: .conclusion}'
```

If full CI hasn't run, fall back to fetching `filters.yaml` and matching manually:

```bash
gh api repos/ai-dynamo/dynamo/contents/.github/filters.yaml --jq '.content' | base64 -d
```

Key rules for manual matching:
- `core` being true triggers ALL framework pipelines (vllm, sglang, trtllm).
- Filters use YAML anchors (e.g., `*ci`) — resolve these when reading.
- Negation patterns like `!**/*.md` in a filter mean markdown-only changes do NOT trigger that filter.

**Jobs that always run in `pr.yaml` regardless of filters:**
- `changed-files`, `deploy-operator`, `backend-status-check`, `clean-k8s-builder`, `cleanup` — these run unconditionally or with `if: always()`.

## Step 2: CI Status Dashboard

Fetch all check runs. Note: `gh pr checks` returns non-zero exit codes when checks are pending or failed — this is normal, not an error.

```bash
gh pr checks $PR_NUMBER --repo ai-dynamo/dynamo
```

If no checks are returned at all, refer back to the Step 1 diagnosis (DCO, approval, draft status).

**Ignore external CI checks** (e.g., GitLab mirror `ci/gitlab/*`). These are NVIDIA-internal pipelines that cannot be inspected from GitHub. Only analyze GitHub Actions checks.

**Distinguish two situations:**
1. **Full CI triggered** (workflow runs include `PR`, `Dynamo Validation`): Analyze all jobs normally.
2. **Only lightweight CI ran** (`Pre Merge` and utility workflows only): Report this clearly. The filter predictions from Step 1 describe what *would* run once full CI triggers, but there's nothing to analyze yet beyond the lightweight checks.

If full CI ran, **identify the critical path** — which checks are most relevant to this PR's changes based on the filter mapping from Step 1.

Produce a concise dashboard grouped by status:
- **Failed** — needs immediate attention
- **Pending/In-progress** — still running (note which are critical path)
- **Passed** — healthy (just count, don't enumerate unless asked)
- **Skipped** — handled in Step 4

## Step 3: Failure Analysis

For each failed GitHub Actions job, drill into the logs to extract root cause.

First, identify the failed jobs within each run:

```bash
gh api repos/ai-dynamo/dynamo/actions/runs/$RUN_ID/jobs --jq '.jobs[] | select(.conclusion == "failure") | {name: .name, id: .id, html_url: .html_url}'
```

Then fetch logs for specific failed jobs. The check URL format from `gh pr checks` is `https://github.com/.../actions/runs/{RUN_ID}/job/{JOB_ID}`. Extract the `RUN_ID` (first number):

```bash
gh run view $RUN_ID --repo ai-dynamo/dynamo --log-failed 2>&1 | tail -200
```

Note: `--log-failed` concatenates all failed job logs, which can be noisy. For multi-failure runs, prefer fetching per-job to isolate root causes.

For each failure, report:
- **Job name** and which workflow it belongs to
- **Root cause** — the actual error (compilation error, test assertion, timeout, infra issue, DCO sign-off failure, etc.)
- **Relevant log excerpt** — the key lines (max 20 lines), not the full dump
- **Suggested fix** — concrete action the PR author can take
- If it looks like an infra flake, include: `gh run rerun $RUN_ID --repo ai-dynamo/dynamo --failed`

If there are no failures, say so and move on.

## Step 3b: Main Branch Cross-Reference

For each failed job identified in Step 3, check if the same job also fails on main. This distinguishes PR-caused regressions from pre-existing flakes.

**First, check how far behind the PR is from main:**

```bash
gh api "repos/ai-dynamo/dynamo/compare/main...$HEAD_SHA" --jq '{behind_by: .behind_by, ahead_by: .ahead_by, status: .status}'
```

If the PR is significantly behind main (>20 commits), note this in the report — some failures may already be fixed on main, and some apparent "regressions" may just be the PR missing recent fixes.

**Then, check recent main branch CI results:**

```bash
# Get last 3 completed PR workflow runs on main
MAIN_RUNS=$(gh api "repos/ai-dynamo/dynamo/actions/workflows/pr.yaml/runs?branch=main&per_page=3&status=completed" --jq '[.workflow_runs[].id] | join(" ")')

# For each failed job name from Step 3, check if it also failed on main
for RUN_ID in $MAIN_RUNS; do
  gh api "repos/ai-dynamo/dynamo/actions/runs/$RUN_ID/jobs?per_page=100&filter=latest" \
    --jq ".jobs[] | select(.name == \"$FAILED_JOB_NAME\") | {run: $RUN_ID, conclusion: .conclusion}"
done
```

**Classification:**
- **`[PR-CAUSED]`** — Fails on this PR but passes on all recent main runs → Likely a regression introduced by this PR. Needs attention.
- **`[PRE-EXISTING]`** — Also fails on recent main runs → Pre-existing flake, not caused by this PR. Suggest rerun.
- **`[UNCLEAR]`** — Mixed results on main (sometimes passes, sometimes fails) → Flaky test. Note flakiness and suggest rerun.
- **`[NEW JOB]`** — Job doesn't exist on main runs → New CI job added by this PR, can't compare.

**Important caveats to include in the report:**
- If the PR is behind main by many commits, a `[PR-CAUSED]` classification may be a false positive — the failure could already be fixed on main. Suggest: "Consider rebasing on main before investigating."
- If the PR is ahead of main (includes main's latest), the classification is reliable.
- Only compare job names, not test-level results. A job passing on main doesn't guarantee the same tests pass — just that the overall job succeeds.

## Step 4: Skip & Discrepancy Analysis

This step is **exception-based only**. Do NOT enumerate expected skips — that's noise.

**If full CI did not trigger**, skip this step entirely — there are no jobs to analyze. The absence of full CI was already explained in Steps 1-2.

**If full CI ran**, compare the filter results from Step 1 against actual CI results from Step 2. Only report **surprises**:

- A filter should be `true` (files matched its paths) but the corresponding job was skipped or missing
- A gate job (`backend-status-check`, `dynamo-status-check`) was skipped — these use `if: always()` and should always run
- All framework pipelines skipped when `core` files changed (core should trigger all frameworks)

**Things that are NOT surprises** (do not report):
- Jobs skipped because their filter is `false` (e.g., docs jobs skipped when no docs changed)
- Multi-GPU tests skipped on pre-merge (these are gated to post-merge/nightly)
- arm64 copy-to-acr jobs skipped (only triggered on merge to main)
- arm64 GPU tests skipped (GPU tests are amd64-only)
- Downstream jobs skipped because their upstream was legitimately skipped
- External CI (GitLab) status — ignored entirely
- `deploy-operator` running despite `operator=false` — this job always runs in `pr.yaml`

If everything matches expectations, say "No unexpected skips or discrepancies" and move on.

## Step 5: Actionable Summary

Synthesize into a concise report:

**PR Health: [PASSING | FAILING | PENDING | CI NOT TRIGGERED | PARTIAL — lightweight only]**

**If full CI not triggered:**
- "Full CI awaiting approval — an NVIDIA maintainer needs to comment `/ok to test <sha>` to create the `pull-request/$PR_NUMBER` branch."
- "DCO check failed — commits need to be signed. See DCO.md."
- "Eligible fork automatic approval is blocked — every PR commit needs a cryptographic signature that GitHub reports as `Verified`; DCO sign-off alone is insufficient."
- "Draft PR — some checks may not run until marked ready for review."
- Note which lightweight checks passed/failed.

**Blocking issues (PR-caused)** — failures that pass on main but fail here:
- One-line root cause + suggested fix for each
- Note if `backend-status-check` or `dynamo-status-check` gate is failing

**Pre-existing failures** — also failing on main, not caused by this PR:
- One-line description + rerun command for each
- If PR is significantly behind main: "Consider rebasing — this may already be fixed on main."

**Non-blocking issues** (if any):
- Flaky tests (`[UNCLEAR]` from Step 3b), infra timeouts, unexpected skips

**Critical path status** — the checks most relevant to this PR's changes:
- List them with current status (passed/failed/pending/not triggered)
- If pending, suggest re-checking in ~15 minutes

**Next steps** — concrete actions ordered by priority:
- "An NVIDIA maintainer should comment `/ok to test <sha>`" if full CI hasn't triggered
- "Sign your commits with `git commit --amend -s`" for DCO failures
- "Sign and rewrite every unverified fork-PR commit, then push" when eligible automatic approval is blocked by commit verification
- "Ask a maintainer to review the current head and comment `/ok to test <sha>`" when automatic approval is unavailable
- "Fix X in file Y" for code failures
- Re-run command for infra flakes: `gh run rerun $RUN_ID --repo ai-dynamo/dynamo --failed`
- "No action needed — CI is green" if everything passed

## Step 6: Monitor Pending Checks

If any checks are still pending or in-progress, offer to monitor them.

**List remaining checks:**

```bash
gh pr checks $PR_NUMBER --repo ai-dynamo/dynamo | grep -E 'pending|queued|in_progress'
```

Report:
- How many checks are still pending
- Which ones are on the critical path
- Estimated wait time based on similar completed jobs (e.g., if `vllm-cuda12.9-amd64 / Test` took 20m and `vllm-cuda13.0-amd64 / Test` is still running, estimate ~20m remaining)

If the user wants to wait, poll periodically:

```bash
# Re-check status
gh pr checks $PR_NUMBER --repo ai-dynamo/dynamo | grep -cE 'pass|fail|skipped'  # completed count
gh pr checks $PR_NUMBER --repo ai-dynamo/dynamo | grep -cE 'pending|queued|in_progress'  # remaining count
```

When all checks complete, re-run the summary from Step 5 with final results. Report any checks that changed from pending to failed since the last check.

## Behavior Notes

- **Concurrency cancellation**: If a PR has rapid pushes, earlier runs get cancelled. Note if you see cancelled runs and suggest checking the latest run instead.
- **Large log output**: Always truncate to relevant sections. Never dump more than 50 lines of raw log in the summary.
- **Rate limits**: If `gh` commands fail due to rate limiting, report what you could gather and suggest retrying later.
- **Multiple workflows**: A single push can trigger `pr.yaml`, `pre-merge.yml`, and `container-validation-dynamo.yml`. Check all of them.
- **`pull-request/[0-9]+` branches**: Created by copy-pr-bot after maintainer approval. Required for full CI — applies to both fork and internal PRs.
- **`external-contribution` label**: Fork PRs get this label automatically. Its presence confirms the PR is from an external contributor.
- **External CI (GitLab)**: Ignore `ci/gitlab/*` checks entirely. These are NVIDIA-internal and cannot be diagnosed from GitHub.