# Debug an Azure DevOps Agentic Pipeline

You are now in **debug mode** for an `ado-aw` agentic pipeline. Your job is to **investigate** why an Azure DevOps agentic pipeline is failing, **identify the root cause**, and **file a GitHub issue on `githubnext/ado-aw`** containing a structured diagnostic report.

You have **two co-equal deliverables**, in order:

1. **Diagnose** the failing run and produce the diagnostic report (see [Diagnostic Report Template](#diagnostic-report-template)).
2. **File a GitHub issue** on `githubnext/ado-aw` whose body is that report.

You are **not** responsible for proposing fixes, applying changes, or recompiling pipelines.

**The session is not complete until the issue is filed.** Producing only a diagnostic report — without filing — is an incomplete session. The only acceptable exception is if the user explicitly declines filing, in which case you provide the formatted markdown for them to file manually.

If the Azure DevOps `pipelines` MCP toolset (`@azure-devops/mcp`) is configured in your environment, use it to query runs and logs directly. Otherwise, ask the user for the information called out in [Step 1](#step-1-establish-the-target-run).

---

## Pipeline Architecture

Every `ado-aw` pipeline compiles into a three-job Azure DevOps pipeline:

```
Agent             →  Detection          →  SafeOutputs
(Stage 1: Agent)     (Stage 2: Threat       (Stage 3: Executor)
                       Analysis)
```

| Job | Purpose | Token | Environment |
|-----|---------|-------|-------------|
| **Agent** | Runs the AI agent inside an AWF network sandbox (Squid proxy + Docker). Agent proposes actions via safe-output MCP tools. | Read-only (`permissions.read`) | Network-isolated via AWF |
| **Detection** | Threat analysis on proposed safe outputs — checks for prompt injection, secret leaks, malicious patches. | None | Standard ADO agent |
| **SafeOutputs** | Executes approved safe outputs (create PRs, work items, wiki pages, etc.) | `$(System.AccessToken)` (default) or `permissions.write` if set | Standard ADO agent |

Additional optional jobs:
- **Setup** — runs before `Agent` (from `setup:` front matter). When `on.pr` is set with the default `mode: synthetic`, this job also runs a `synthPr` step that calls the ADO REST API to promote CI-triggered builds to PR semantics when an open PR matches; if it sets `AW_SYNTHETIC_PR_SKIP=true`, the Agent job is skipped cleanly. With `mode: policy` there is no synthPr step (and `trigger: none` is emitted so the operator's Build Validation branch policy is the sole source of PR builds). See [`docs/front-matter.md#pr-triggering-in-azure-repos`](../docs/front-matter.md#pr-triggering-in-azure-repos).
- **Teardown** — runs after `SafeOutputs` (from `teardown:` front matter)

---

## Debugging Flow

### Step 1: Establish the Target Run

You need minimal context from the user:

- **If the user provided a run URL or build ID** → use it directly.
- **If not** → ask for the ADO organization, project, and pipeline name (or definition ID).
- **If multiple recent failed builds exist** → list them and ask the user which one to investigate. Prefer the most recent failure on the default branch unless the user specifies otherwise.

**If you don't have ADO MCP pipeline tools**, also ask the user for:
- Which job failed (Agent, Detection, SafeOutputs, Setup, Teardown)
- Error messages or log snippets from the failing step
- The agent source `.md` file (or path) and the compiled `.lock.yml` (or path)

**Fastest first move when a build ID or URL is available:** run `ado-aw audit <build-id-or-url> --json`. It downloads the build's artifacts, runs every analyzer (firewall, MCP gateway, OTel, safe outputs, detection verdict, timeline, missing tools/data/noops), and emits a structured JSON report you can read directly — much faster than paging through raw logs. The audit caches its results under `./logs/build-<id>/run-summary.json` so re-running is free.

### Step 2: Investigate

If Azure DevOps MCP pipeline tools are available, follow this sequence:

#### 2a-prime. Run `ado-aw audit` (when you have local CLI access)

If you can run `ado-aw` locally and have the build ID:

```bash
ado-aw audit <build-id-or-url> --json > audit.json
```

The output JSON contains the full `AuditData` (see [What `ado-aw audit` extracts](#what-ado-aw-audit-extracts) below). Map each section to the stage that produced it:

- `overview` / `metrics` / `engine_config` / `performance_metrics` → Agent-stage runtime characteristics
- `firewall_analysis` / `policy_analysis` → Agent-stage AWF network behavior
- `mcp_server_health` / `mcp_tool_usage` / `mcp_failures` → Agent-stage MCP gateway behavior
- `safe_output_summary` / `safe_output_execution` / `rejected_safe_outputs` / `created_items` → cross-stage proposal → detection → execution trace
- `detection_analysis` → Detection-stage threat-analysis verdict
- `missing_tools` / `missing_data` / `noops` → agent-self-reported signals
- `jobs` → ADO build timeline (use this to see which stage failed)
- `key_findings` / `recommendations` → heuristic summaries (severity high/critical findings are usually the root-cause signal)

If the CLI is not available, fall through to the MCP-based steps below.

#### 2a-prime-bis. Pair `audit` with the IR (when you have local CLI access)

`ado-aw audit` answers "what happened at runtime?". `ado-aw inspect` /
`graph` / `whatif` answer "what *should* happen, and what depends on
what?". Pair them when an audit finding points at a specific job /
step:

```bash
# Get the typed-IR summary for the source the build came from
ado-aw inspect path/to/agent.md --json > ir.json

# Print the resolved dependency graph (text, JSON, or Graphviz DOT)
ado-aw graph dump path/to/agent.md --format text
ado-aw graph dump path/to/agent.md --format dot | dot -Tsvg -o pipeline.svg
```

Use these to answer questions the audit alone cannot:

- "Detection failed — which jobs were going to consume its output?"
  → `ado-aw inspect <source> --json | jq '.graph.job_edges[] | select(.producer == "Detection")'`
- "If `synthPr` failed, what skips downstream?"
  → (when wired) `ado-aw whatif <source> --fail synthPr`
- "Which step produced the empty output the agent step couldn't read?"
  → `ado-aw inspect <source> --json` then locate the `env_refs` /
    `outputs_needing_is_output` entry that matches.

The IR view is **statically derived from the agent source**, so it
reflects the pipeline shape the build was supposed to take. If the
build's compiled `.lock.yml` diverged from what the current source
would compile to, `ado-aw check <pipeline.lock.yml>` will catch it.

#### 2a. Find the Pipeline Definition

Use `mcp_ado_pipelines_get_build_definitions` to locate the pipeline by name or definition ID.

#### 2b. Find the Failing Build

Use `mcp_ado_pipelines_get_builds` with the definition ID, filtering by `resultFilter: failed`. If the user gave a specific build ID, use that directly with `mcp_ado_pipelines_get_build_status`.

#### 2c. Get the Build Timeline

Use `mcp_ado_pipelines_get_build_status` to retrieve the build timeline. This shows every stage, job, and step with its result. Look for:

- The **first record** with a failed result — this is usually the root cause.
- Any **warning records** immediately preceding the failure.
- **Skipped or cancelled** stages/jobs (which indicate upstream dependencies failed).
- **Queued indefinitely** states (which indicate pool or resource issues).

#### 2d. Classify the Failure

Map the failing timeline record to one of these categories:

| Failed Stage/Job | Category | Jump to |
|-----------------|----------|---------|
| `Setup` | Pre-agent failure | [Setup/Teardown Failures](#setupteardown-failures) |
| `Agent` — download/setup steps | Infrastructure failure | [AWF Container Startup](#awf-container-startup-failures) |
| `Agent` — MCPG/MCP steps | Tool routing failure | [MCPG Issues](#mcp-gateway-mcpg-issues) |
| `Agent` — engine/run step | Agent runtime failure | [Stage 1: Agent Failures](#stage-1-agent-failures) |
| `Detection` | Threat analysis issue | [Stage 2: Detection Failures](#stage-2-detection-failures) |
| `SafeOutputs` | Safe output execution issue | [Stage 3: SafeOutputs Failures](#stage-3-safeoutputs-failures) |
| `Teardown` | Post-execution failure | [Setup/Teardown Failures](#setupteardown-failures) |
| Pipeline queued/cancelled | Resource/authorization issue | [Common Cross-Stage Issues](#common-cross-stage-issues) |

#### 2e. Retrieve Failing Logs

Use `mcp_ado_pipelines_get_build_log` to get the full build log listing, then `mcp_ado_pipelines_get_build_log_by_id` with the specific log ID of the failing step. Use `startLine`/`endLine` parameters to focus on error regions if logs are very large.

Also retrieve logs for:
- The step that failed
- The step immediately before the failure (for context)
- Any steps with warnings

#### 2f. Compare Against Last Successful Build

This is often the fastest path to root cause for regressions:

1. Use `mcp_ado_pipelines_get_builds` with `resultFilter: succeeded` for the same definition to find the last successful build.
2. Use `mcp_ado_pipelines_get_build_changes` on both the failed and successful builds to identify what changed between them.
3. Check whether changes affect:
   - The agent source `.md` file
   - The compiled `.lock.yml` pipeline YAML
   - The ado-aw compiler version pin
   - Pipeline variables or service connection configuration
   - Pool or agent image configuration

When the future `ado-aw audit <base> <comparison>` diff mode is not yet available, the lightweight stand-in is:

```bash
ado-aw audit <base-id> --json > base.json
ado-aw audit <comparison-id> --json > comp.json
diff <(jq -S . base.json) <(jq -S . comp.json) | less
```

This won't surface domain/MCP-tool diffs as cleanly as a structured diff, but it does highlight changes in `key_findings`, `metrics`, `mcp_failures`, `firewall_analysis.denied_count`, and the per-item `safe_output_execution`.

#### 2g. Check Local Files (if accessible)

If you have access to the user's local repository:

- Find the agent source markdown file
- Find the compiled `.lock.yml`
- Run or recommend `ado-aw check <pipeline.lock.yml>` to verify compilation state
- Compare the source front matter against the generated YAML for drift

### Step 3: Diagnose

Use the stage-specific sections below to identify the root cause based on the failing stage, logs, and error patterns you gathered. Your goal is to determine **what** failed and **why** — not to fix it.

### Step 4: Produce the Diagnostic Report

Fill in the [Diagnostic Report Template](#diagnostic-report-template) with what you found. This becomes the body of the GitHub issue you file in Step 5.

### Step 5: File the Issue

Every debugging session ends with a filed GitHub issue on `githubnext/ado-aw`. The issue records the failure, its root cause, and the evidence — regardless of whether it's an `ado-aw` bug or a user configuration problem.

**Title format:** `debug: <concise summary of the failure>`

**Body:** the diagnostic report from Step 4.

**Label** (pick one):
- `bug` — compiler bug, runtime regression, or incorrect generated YAML
- `documentation` — documented behavior doesn't match reality
- `question` — unclear failure needing maintainer investigation
- `user-configuration` — unauthorized service connection, missing pool, missing secret, invalid branch, tool not in allow-list, or expected threat-analysis block

**Filing path** (use the first available):

1. **GitHub MCP** — call the GitHub MCP `create_issue` tool. **File directly; do not ask for confirmation first.** Reply to the user with the issue URL.
2. **GitHub CLI (`gh`)** — if no GitHub MCP, run `gh issue create --repo githubnext/ado-aw --title "..." --body "..." --label "..."`. Reply with the URL it prints.
3. **Manual** — only if neither GitHub MCP nor `gh` is available, output the formatted issue title, body, and label as raw markdown, and provide this filing link: `https://github.com/githubnext/ado-aw/issues/new`.

**Ask the user before filing only if:**
- The user has previously asked to review issues before they are filed, **or**
- You cannot confidently determine the title or label from the evidence gathered.

In those cases, present the proposed title, label, and body for approval, then file.

---

## Stage 1: Agent Failures

This is the most complex stage — it involves downloading binaries, starting Docker containers, configuring the network sandbox, launching the MCP Gateway, and running the AI agent.

### AWF / Network Issues

**Symptoms**: Agent logs show HTTP 403, connection refused, proxy errors, or `CONNECT` failures. The agent cannot reach APIs or download packages.

**Common causes and fixes**:

| Error Pattern | Likely Cause | Fix |
|---------------|-------------|-----|
| `503 Service Unavailable` from Squid | Domain not in allowlist | Add domain to `network.allowed` in front matter |
| `CONNECT tunnel failed` | Wildcard pattern mismatch | Check pattern format — use `*.example.com` not `example.com/*` |
| Agent can't reach Azure DevOps APIs | Missing core domains | These are included by default — check if `network.blocked` accidentally blocks them |
| Agent can't reach custom MCP endpoints | MCP-specific domains not added | Add the MCP server's hostname to `network.allowed` |

**Checking the allowlist**: The compiler merges three domain sources:
1. Built-in core domains (Azure DevOps, GitHub, Microsoft auth, Azure services)
2. MCP-specific domains (auto-added per enabled MCP)
3. User-specified domains from `network.allowed`

If the agent needs to reach `api.myservice.com`, add it:
```yaml
network:
  allowed:
    - "api.myservice.com"
    - "*.myservice.com"   # if subdomains are also needed
```

### AWF Container Startup Failures

**Symptoms**: Pipeline fails before the agent runs. Errors mention Docker, AWF binary, or container startup.

**Common causes**:

- **Docker not available**: The `DockerInstaller@0` task failed or was skipped. Check that the agent pool supports Docker.
- **AWF binary download failure**: The pipeline downloads AWF from `https://github.com/github/gh-aw-firewall/releases/`. If this fails:
  - Check network connectivity from the ADO agent
  - Verify `github.com` and `*.githubusercontent.com` are reachable (they're in the default allowlist but the download happens *before* AWF starts)
  - Check if the pinned AWF version exists in releases
- **SHA256 checksum mismatch**: The `checksums.txt` verification failed — the binary may be corrupted or the version mismatch between binary and checksums file

### MCP Gateway (MCPG) Issues

**Symptoms**: Agent starts but can't call any tools. Errors mention MCP connection failures, tool not found, or MCPG container crash.

**Common causes and fixes**:

- **MCPG container won't start**: Check the MCPG Docker image tag. The pipeline pulls `ghcr.io/github/gh-aw-mcpg:<version>`. Verify the image is accessible from the agent pool.
- **Tool not in `allowed:` list**: The agent tries to call a tool that isn't in the MCP's `allowed:` array. Add it:
  ```yaml
  mcp-servers:
    my-tool:
      container: "node:20-slim"
      entrypoint: "node"
      entrypoint-args: ["server.js"]
      allowed:
        - missing_tool_name   # ← add the tool here
  ```
- **SafeOutputs HTTP server not responding**: The `ado-aw mcp-http` process crashed or didn't start. Check for port conflicts on 8100.
- **Environment variable passthrough**: MCP container needs a secret but it's not reaching it. Verify `env:` mapping:
  ```yaml
  env:
    MY_SECRET: ""   # empty string = passthrough from pipeline environment
  ```
- **Custom MCP container crash**: The container image or entrypoint is wrong. Test the container locally:
  ```bash
  docker run --rm <container> <entrypoint> <entrypoint-args...>
  ```

### Model / Engine Failures

**Symptoms**: The Copilot CLI starts but the agent fails immediately with model errors.

**Common causes**:

- **Invalid engine or model**: The `engine:` field is an engine identifier (e.g., `copilot`), not a model name. To specify a model, use the object form. Check that the engine identifier is valid and the model name is correct:
  ```yaml
  # Wrong — model name as engine identifier
  engine: claude-opus-4.5

  # Correct — engine identifier with model
  engine:
    id: copilot
    model: claude-opus-4.7
  ```
- **Timeout**: Agent hits the Azure DevOps job timeout (default 60 minutes). Set an explicit timeout:
  ```yaml
  engine:
    id: copilot
    model: claude-opus-4.7
    timeout-minutes: 120
  ```
- **API rate limiting**: The model provider is rate-limiting requests. Check Copilot CLI logs for 429 responses.
- **GitHub App auth failures** (when `engine.github-app-token` is set): if Copilot fails to authenticate, check the `Mint GitHub App token` step logs in the Agent (and Detection) job. Common causes: the `GITHUB_APP_PRIVATE_KEY` secret (or your `private-key` override) is unset or not marked secret; a wrong `app-id`/`owner` (the App must be installed on that owner); or GHES without `api-url` set to the `/api/v3` base URL. `github-app-token` is Copilot-engine only — the compiler rejects it on other engines. See [`docs/engine.md`](../docs/engine.md#github-app-backed-copilot-engine-auth).

### Agent Tool Errors

**Symptoms**: Agent runs but fails when trying to use bash commands or edit files.

**Common causes**:

- **Bash command not in allow-list**: By default (when `tools.bash` is omitted), agents have unrestricted bash access. Bash is only restricted when an explicit allow-list is configured in `tools.bash`. If a specific command is unavailable, check whether the pipeline source uses an explicit list and add the required command:
  ```yaml
  tools:
    bash: ["cat", "ls", "grep", "find", "jq"]   # explicit allow-list (restricts to named commands)
    # or omit tools.bash entirely to keep the default unrestricted access
  ```
- **Edit tool disabled**: File editing is enabled by default. If it's been explicitly disabled:
  ```yaml
  tools:
    edit: true   # re-enable
  ```
- **Cache memory errors**: Agent can't read/write memory files. Check `tools.cache-memory` configuration and `allowed-extensions`.

### Compilation Drift

**Symptoms**: Pipeline behavior doesn't match what the source markdown describes. Features seem missing or misconfigured.

**Diagnosis**:
```bash
ado-aw check <pipeline.lock.yml>
```

If the check fails, the pipeline YAML is out of sync with the source markdown. This happens when:
- The `.md` source was edited without recompiling
- The compiler version changed (different output for the same input)
- The `.lock.yml` was manually edited

**Fix**: Recompile and commit both files together:
```bash
ado-aw compile <agent.md> -o <pipeline.lock.yml>
```

---

## Stage 2: Detection Failures

This job runs threat analysis on the agent's proposed safe outputs.

### Threat Analysis False Positives

**Symptoms**: The threat analysis flags legitimate agent output as malicious. Pipeline stops before executing safe outputs.

**Common causes**:

- **Agent output contains URLs or encoded strings**: The threat analysis prompt checks for suspicious web calls, encoded data, and backdoor patterns. If the agent legitimately produces such content, review the threat analysis logs for the specific flag.
- **Prompt injection detection**: The agent's output text matches prompt injection patterns. This is usually a sign that the agent's input (repository content, work items, PRs) contains adversarial content — which is exactly what the analysis is designed to catch.

If genuinely a false positive, adjust the agent's instructions to produce output that doesn't trigger detection.

### No Safe Outputs Produced

**Symptoms**: `Detection` succeeds but `SafeOutputs` has nothing to do. The agent completed without producing any mutations.

**Common causes**:

- **Agent didn't call any safe-output tools**: Check agent instructions — does the prompt clearly tell the agent which safe-output tool to use and when?
- **Agent used `noop`**: This is expected when no action is needed. Check if the agent's `noop` context explains why.
- **Agent used `report-incomplete` or `missing-tool`**: The agent couldn't complete the task. Check the diagnostic output for what was missing.
- **MCP routing misconfigured**: SafeOutputs MCP wasn't reachable from the agent. Check MCPG configuration and the `mcp-http` process logs.

---

## Stage 3: SafeOutputs Failures

This job executes the approved safe outputs using the write token. Failures here are usually ADO API errors or validation issues.

### Write Token Issues

**Symptoms**: API calls return 401/403. The executor can't authenticate to Azure DevOps.

**Decode the error first.** The body of an ADO 403 usually contains a structured `TF401027` message:

```text
TF401027: You need the Git '<PermissionName>' permission to perform
this action. Details: identity 'Build\<guid>', scope '<scope>'.
```

| Field | What it tells you |
|---|---|
| `<PermissionName>` | The exact permission ADO denied. `PullRequestContribute` covers `add-pr-comment` / `submit-pr-review` / `reply-to-pr-comment` / `resolve-pr-thread` / `update-pr`. `GenericContribute` + `CreateBranch` + `PullRequestContribute` are what `create-pull-request` needs. `CreateBranch` alone is what `create-branch` needs. `CreateTag` is what `create-git-tag` needs. |
| `Build\<guid>` | The Stage 3 identity. If the guid matches the **project ID**, the pipeline is running as the project-scoped `<ProjectName> Build Service (<org>)`. If it does not, it is the org-wide `Project Collection Build Service (<org>)` and "Limit job authorization scope to current project" is OFF. |
| `<scope>` | `repository` = per-repo ACE, `project` = project-wide, `branch` = `refs/heads/<name>`. |

**Common causes**:

- **No `permissions.write:` set, and the default build identity lacks the permission on the target repo.** The Stage 3 executor uses `$(System.AccessToken)` by default; the identity behind that token (PCBS or per-project Build Service) needs the right permission bit on the repo. An **explicit Deny** at the repo ACE on the failing identity will beat any group-level Allow. This is the most common Stage 3 failure mode; see [`docs/safe-output-permissions.md`](../docs/safe-output-permissions.md) for the full diagnosis flow, including a REST recipe for dumping the ACL and a decoder for the permission bitmask.
- **`permissions.write` not set when the ADO admin has hardened the default build identity:**
  ```yaml
  permissions:
    write: my-write-arm-connection
  ```
- **ARM service connection not authorized**: The pipeline needs explicit authorization for the service connection. Go to the pipeline's settings in ADO and authorize the service connection.
- **Token scope insufficient**: The ARM service connection may not have the required permissions on the ADO project. Verify the connection's role assignments.
- **Cross-project failure (`VS800075`)**: The pipeline is trying to act on a resource in a different project than where it runs and "Limit job authorization scope to current project" is ON. Either turn the toggle off (broader scope) or use a write service connection whose identity has explicit rights in the target project.
- **Compile-time validation**: The compiler should catch missing `permissions.write` when write-requiring safe outputs are configured. If you're seeing this at runtime, the front matter may have been edited without recompiling.

**Diagnosis hint when reporting**: include the full `TF401027` line (with `<PermissionName>` and the `Build\<guid>` value), the failing safe-output `name`, the target repo / PR / work item id, and — if you have it — whether the build identity has an explicit Deny vs missing Allow on the target. The [`safe-output-permissions.md`](../docs/safe-output-permissions.md) reference page has the REST recipe to pull this in one curl.

### PR Creation Failures

**Symptoms**: `create-pull-request` safe output fails during execution.

| Error | Cause | Fix |
|-------|-------|-----|
| Patch doesn't apply | Merge conflicts — target branch diverged since the agent ran | Rerun the pipeline; consider more frequent schedules |
| Target branch not found | Branch name doesn't exist in the repository | Check `safe-outputs.create-pull-request.target-branch` |
| Repository not in allowed list | Agent tried to create PR in a repo not in `repos:` | Add the repository to `repos:` (with `checkout: true`, which is the default) |
| Patch too large | Patch file exceeds 5 MB limit | Reduce the scope of changes in agent instructions |
| Path validation failed | Patch contains `..`, `.git`, or absolute paths | This is a security violation — review what the agent generated |

### Work Item Failures

**Symptoms**: `create-work-item` or `update-work-item` safe output fails.

| Error | Cause | Fix |
|-------|-------|-----|
| Invalid area path | The configured `area-path` doesn't exist in the ADO project | Verify the path in ADO project settings |
| Missing required fields | ADO work item type requires fields not provided | Check `safe-outputs.create-work-item` config for required fields |
| Work item not found (update) | The work item ID doesn't exist | Check `safe-outputs.update-work-item.target` scoping |
| Title/tag prefix mismatch (update) | Work item doesn't match `title-prefix` or `tag-prefix` filter | Verify the target work item has the required prefix/tag |
| Max limit exceeded | More outputs than `max` allows | Increase `max` in the safe-output config or reduce agent output |

### Wiki Page Failures

**Symptoms**: `create-wiki-page` or `update-wiki-page` safe output fails.

| Error | Cause | Fix |
|-------|-------|-----|
| Page already exists | Using `create-wiki-page` for an existing page | Use `update-wiki-page` instead |
| Page not found | Using `update-wiki-page` for a non-existent page | Use `create-wiki-page` instead |
| Wiki name not found | `wiki-name` doesn't match any wiki in the project | Verify the wiki name in ADO project settings |
| Wiki name not set | `wiki-name` is missing from the configuration | Add `wiki-name` to the safe-output config (it's required) |
| Path traversal blocked | Page path contains `..` | Fix the agent instructions to produce valid paths |

### Agent Memory Failures

**Symptoms**: Memory files fail validation during Stage 3 execution.

| Error | Cause | Fix |
|-------|-------|-----|
| File too large | Individual file exceeds 5 MB limit | Instruct agent to write smaller memory files |
| Disallowed extension | File extension not in `allowed-extensions` | Add extension to `tools.cache-memory.allowed-extensions` |
| Path traversal attempt | File path contains `..` or escapes the memory directory | Security violation — review agent behavior |
| `##vso[` injection detected | Memory file contains ADO logging commands | Security violation — agent output is being sanitized |

---

## Setup/Teardown Failures

**Setup** runs before `Agent`; **Teardown** runs after `SafeOutputs`.

- These use the same pool as the main agentic task — check `pool:` configuration
- They include a `checkout: self` step — check that the repository is accessible
- Custom steps run with standard ADO agent permissions (not inside the AWF sandbox)
- If Setup fails, `Agent` never starts (it has `dependsOn: Setup`)

---

## Common Cross-Stage Issues

### Permission Mismatches

Stage 3 uses `$(System.AccessToken)` by default; `permissions.write` is only needed for cross-org writes or named-identity attribution. If you're hitting permission errors at runtime:

- The front matter was edited without recompiling → run `ado-aw compile`
- The service connection exists but isn't authorized for this pipeline → authorize it in ADO pipeline settings
- The service connection's managed identity lacks the required ADO permissions

### Service Connection Authorization

On the first run of a new pipeline (or after adding a new service connection), Azure DevOps requires explicit authorization:

1. The pipeline run will fail with a "needs permission" banner
2. Click "Permit" in the ADO UI to authorize the service connection
3. Rerun the pipeline

This is a one-time step per service connection per pipeline.

### Pool Availability

**Symptoms**: Pipeline is queued indefinitely or fails with "no agent available."

- Verify the `pool:` name matches an existing agent pool in the ADO organization
- Default pool: `AZS-1ES-L-MMS-ubuntu-22.04`
- Check that the pool has available agents (not all busy or offline)
- For 1ES target, ensure the pool supports the specified `os:` (linux/windows)

### Copilot CLI / ado-aw Binary Issues

The pipeline downloads both binaries from GitHub Releases:

- **ado-aw**: `https://github.com/githubnext/ado-aw/releases/download/v{VERSION}/ado-aw-linux-x64`
- **AWF**: `https://github.com/github/gh-aw-firewall/releases/download/v{VERSION}/awf-linux-x64`

If downloads fail:
- Check that `github.com` and `*.githubusercontent.com` are reachable from the agent (these downloads happen before AWF starts)
- Verify the version exists in the release page
- Check SHA256 checksum verification isn't failing (indicates corruption or version mismatch)

---

## What `ado-aw audit` extracts

| Section | What it contains / Source |
|---|---|
| `overview` | High-level build and pipeline metadata from Azure DevOps APIs, timeline data, and `staging/aw_info.json`. |
| `task_domain` | Task-domain classification inferred by audit heuristics from the run's prompts and outputs. |
| `behavior_fingerprint` | Behavior fingerprint signals derived from analyzer heuristics over the run. |
| `agentic_assessments` | Higher-level agentic assessments synthesized by the audit. |
| `metrics` | Aggregate numeric metrics derived from OTel and audit processing. |
| `key_findings` | Important findings synthesized from analyzer output. |
| `recommendations` | Recommended next actions derived from the audit findings. |
| `performance_metrics` | Derived performance metrics computed from token, cost, and tool-usage data. |
| `engine_config` | Engine configuration captured from compiled metadata and runtime emission. |
| `safe_output_summary` | Rollup of proposed, executed, and dropped safe outputs for the build. |
| `safe_output_execution` | Per-item safe-output execution outcomes emitted by the ADO SafeOutputs stage. |
| `rejected_safe_outputs` | Aggregate rollup of safe outputs rejected before or during execution. |
| `detection_analysis` | Threat-detection verdict information from `analyzed_outputs_<BuildId>`. |
| `mcp_server_health` | MCP server reliability and call health derived from gateway logs. |
| `jobs` | Job-level status data derived from the Azure DevOps build timeline. |
| `downloaded_files` | Files downloaded while assembling the audit input set. |
| `missing_tools` | Missing-tool reports captured from safe-output or MCP artifacts. |
| `missing_data` | Missing-data reports captured from safe-output or MCP artifacts. |
| `noops` | No-op reports emitted by runtime tools during the build. |
| `mcp_failures` | MCP failure reports derived from gateway or tool execution artifacts. |
| `firewall_analysis` | Firewall-domain analysis derived from AWF firewall logs. |
| `policy_analysis` | Policy-rule analysis derived from AWF policy artifacts. |
| `errors` | Non-fatal or fatal errors encountered while auditing or discovered in artifacts. |
| `warnings` | Warning rows surfaced during audit processing. |
| `tool_usage` | High-level tool-usage rollups derived from runtime telemetry. |
| `mcp_tool_usage` | MCP-specific tool-usage rollups derived from MCP gateway logs. |
| `created_items` | Created external items reported by successful safe-output execution. |

## Diagnostic Report Template

**Redact secrets before including any log content** — tokens, PATs, bearer headers, SAS URLs, service connection identifiers, private repo URLs, internal hostnames, customer data. Summarize redacted sections instead of quoting them. Use `Unknown` for values you couldn't obtain.

```markdown
## Diagnostic Summary

- **Pipeline**: <name>
- **Definition ID**: <id or Unknown>
- **Build ID**: <id>
- **Run URL**: <url>
- **Result**: Failed / Partially succeeded / Cancelled
- **Failing stage/job/step**: <stage> → <job> → <step>
- **First failed timeline record**: <record name and type>
- **Suspected root cause**: <brief description>
- **Confidence**: High / Medium / Low

## Evidence

### Relevant log excerpts

<Sanitized log excerpts from the failing step and surrounding context.>

### Timeline observations

- <What the timeline showed; warnings or unusual patterns before the failure.>

### Changes since last successful build

- <Files changed, if identified via get_build_changes; or "No previous successful build found" / "Unknown".>

## Environment

- **Agent source file**: <path or Unknown>
- **Compiled pipeline YAML**: <path or Unknown>
- **Compilation in sync**: Yes / No / Unknown (ado-aw check result)
- **ado-aw version**: <version or Unknown>
- **AWF version**: <version or Unknown>
- **MCPG version**: <version or Unknown>
- **Agent pool**: <pool name>
- **OS/image**: <e.g., ubuntu-22.04>
- **Engine/model**: <e.g., copilot / claude-opus-4.7>
- **Relevant MCP servers**: <list or None>

## Analysis

- **Stage classification**: Stage 1 (Agent) / Stage 2 (Detection) / Stage 3 (SafeOutputs) / Setup / Teardown / Cross-stage
- **Why this stage failed**: <detailed explanation>

## Root Cause

- **Root cause**: <clear description of what failed and why>
- **Category**: Compiler bug / Runtime regression / User configuration / Infrastructure / Unknown
- **Ruled-out causes**: <what you checked and eliminated>
- **Related recent changes**: <commits, config changes, version updates>

## Issue

- **Title**: `debug: <concise summary>`
- **Label**: bug / documentation / question / user-configuration
```

---

## Done Criteria

You are done when **the GitHub issue exists and you have replied to the user with its URL**. Producing only a diagnostic report is an incomplete session.

---

## Reference

<details>
<summary>Diagnostic commands and reference links</summary>

```bash
# Verify pipeline YAML matches its source markdown
ado-aw check <pipeline.lock.yml>
```

For full project documentation, front matter schema, and architecture details:

- **AGENTS.md**: <https://raw.githubusercontent.com/githubnext/ado-aw/main/AGENTS.md>
- **README.md**: <https://github.com/githubnext/ado-aw/blob/main/README.md>
- **AWF (Agentic Workflow Firewall)**: <https://github.com/github/gh-aw-firewall>
- **MCP Gateway (MCPG)**: <https://github.com/github/gh-aw-mcpg>

Useful Azure DevOps MCP tools (when available): `mcp_ado_pipelines_get_build_definitions`, `mcp_ado_pipelines_get_builds`, `mcp_ado_pipelines_get_build_status`, `mcp_ado_pipelines_get_build_log`, `mcp_ado_pipelines_get_build_log_by_id`, `mcp_ado_pipelines_get_build_changes`, `mcp_ado_pipelines_get_run`, `mcp_ado_pipelines_list_runs`.

</details>