--- name: troubleshoot description: >- Diagnose and fix common ABCA issues: deployment failures, preflight errors, authentication problems, agent failures, and build issues. Use when the user says "troubleshoot", "debug", "not working", "error", "failed", "help me fix", "preflight_failed", "task failed", "deploy failed", "auth error", "401", "422", "503", or describes something not working as expected. --- # ABCA Troubleshooting You are diagnosing an issue with the ABCA platform. Follow a systematic approach: gather symptoms, check the most common causes, and apply targeted fixes. > **Running the CLI:** commands below call `node cli/lib/bin/bgagent.js …`. In a non-interactive or mise-managed shell `node` may not be on `PATH` — prefix with `mise exec --`. Ironically, `node: command not found` is itself a common symptom (the shell hasn't activated mise); that's a missing prefix, not a broken install. ## Step 1: Identify the Problem Category Determine which area the issue falls into: 1. **Build/Compilation** — TypeScript errors, test failures, lint issues 2. **Deployment** — CDK deploy/synth failures, CloudFormation errors 3. **Authentication** — Cognito errors, token issues, 401 responses 4. **Task Submission** — 422 errors, validation failures, guardrail blocks 5. **Task Execution** — Preflight failures, agent failures, timeouts 6. **Local Agent Testing** — Docker issues, run.sh problems ## Build/Compilation Issues ```bash export MISE_EXPERIMENTAL=1 mise //cdk:compile 2>&1 | tail -50 # TypeScript errors mise //cdk:test 2>&1 | tail -50 # Test failures ``` **Common causes:** - Missing `mise run install` after pulling changes - `yarn: command not found` — Run `corepack enable && corepack prepare yarn@1.22.22 --activate` - Type mismatches after editing `cdk/src/handlers/shared/types.ts` without updating `cli/src/types.ts` ## Deployment Issues ```bash # Check CloudFormation events for the failed stack aws cloudformation describe-stack-events --stack-name backgroundagent-dev \ --query 'StackEvents[?ResourceStatus==`CREATE_FAILED` || ResourceStatus==`UPDATE_FAILED`].[LogicalResourceId,ResourceStatusReason]' \ --output table ``` **Common causes:** - Docker not running — Required for CDK asset bundling - Missing CDK bootstrap — Run `mise //cdk:bootstrap` - IAM permission issues — Check `aws sts get-caller-identity` - Region mismatch — Ensure consistent region across all commands ## Authentication Issues ```bash # Verify credentials aws sts get-caller-identity # Check Cognito user exists aws cognito-idp admin-get-user \ --user-pool-id $USER_POOL_ID \ --username user@example.com ``` **Common causes:** - "App client does not exist" — Region mismatch between CLI config and stack deployment - Token expired — Re-authenticate with `bgagent login` - 401 on API calls — Token not included or malformed in Authorization header - User not created — Self-signup is disabled; admin must create users ## Task Submission Issues (422 / 400) **"Repository not onboarded" / `REPO_NOT_ONBOARDED` (422):** - The repo isn't registered. Fastest fix: `bgagent repo onboard ` (operator path — writes the RepoTable record at runtime, no redeploy). A CDK Blueprint is only needed for declarative config. Use the `onboard-repo` skill for details. - Also confirm the `owner/repo` matches **exactly** what you pass to `bgagent submit --repo`. **"GUARDRAIL_BLOCKED" (400):** - Task description triggered Bedrock Guardrails content screening - Review and rephrase the task description to remove potentially flagged content **Validation errors:** - Check required fields: `repo` is required, plus at least one of `issue_number`, `task_description`, `pr_number` - `max_turns` range: 1-500 - `max_budget_usd` range: $0.01-$100 ## Task Execution Issues ```bash # Check task events for details node cli/lib/bin/bgagent.js events --output json ``` **`preflight_failed`:** - GitHub PAT lacks permissions for the repo - Repository doesn't exist or is private without proper token scope - Check event `reason` and `detail` fields for specifics - Verify PAT: fine-grained token must include the target repository with Contents (read/write), Pull Requests (read/write), Issues (read) **`task_failed` / task completes with 0 tokens and no PR:** - Agent encountered an error during execution - Check CloudWatch logs for the session: ```bash aws logs filter-log-events \ --log-group-name "/aws/vendedlogs/bedrock-agentcore/runtime/APPLICATION_LOGS/jean_cloude" \ --filter-pattern "" \ --region us-west-2 --query 'events[*].message' --output text ``` - Common: repo build/test commands not documented in CLAUDE.md **403 "not authorized to perform bedrock:InvokeModelWithResponseStream":** - The repo's `model_id` is a model the runtime IAM role wasn't granted. The runtime only has `grantInvoke` for the models in the stack's configured set — read it from the `BedrockModelIds` stack output rather than a list here (Sonnet 4.6, Opus 4.8, Opus 5, Haiku 4.5 by default). - **Quick fix:** point the repo at an already-granted model — `bgagent repo onboard --model global.anthropic.claude-opus-5` (no redeploy). - **To add a new model to the runtime:** grant it in the stack and redeploy. The model set is the shared list in `cdk/src/constructs/bedrock-models.ts` — add the model via the `bedrockModels` CDK context (`cdk.json`) so both the AgentCore and ECS backends grant it (#433). Adding a model also requires **account-level Bedrock access** for it (separate from IAM — see the next row). **Model not enabled / "not available on your Bedrock deployment" (often immediate failure, few turns, zero or near-zero tokens):** - **IAM is necessary but not sufficient.** The AgentCore role may already have `bedrock:InvokeModel*`, but the **account** must also satisfy [Amazon Bedrock model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html): Marketplace subscription flow on first serverless use (with `aws-marketplace:Subscribe` / `ViewSubscriptions` where needed), Anthropic **first-time use** details (`PutUseCaseForModelAccess` or the console model catalog), and a valid payment method for Marketplace-backed models. - **Use an inference profile ID** in the Blueprint / DynamoDB `model_id` when Bedrock requires it for on-demand invocation (for example `global.anthropic.claude-opus-5` for global Opus 5). See [Use an inference profile in model invocation](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-use.html). Raw `anthropic.*` IDs often hit "on-demand not supported" or wrong routing — see the **400** row below. - **Cross-Region profiles** route across Regions in a geography; ensure IAM and any SCPs allow Bedrock in **all destination Regions** for that profile. See [Supported Regions and models for inference profiles](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-support.html). - **Task status:** When the Claude CLI reports a terminal error via `ResultMessage.is_error`, the agent marks the task **FAILED** (not COMPLETED) and persists `error_message` in DynamoDB. **400 "Invocation with on-demand throughput isn't supported":** - The Blueprint `modelId` uses a raw foundation model ID (e.g. `anthropic.claude-opus-4-8`) - Fix: change to the inference profile ID, prefixed with the geography the stack grants — its `BedrockGeoRegion` output (e.g. `global.anthropic.claude-opus-4-8`) — then update DynamoDB via redeploy. A prefix from a *different* geography raises `AccessDenied` rather than this 400, since the IAM grant is scoped per geography. **503 "Too many connections" / task completes with 0 tokens after long duration:** - Bedrock is throttling model invocations. The agent retries for minutes then gives up. - Symptoms: task runs for 10-15 minutes, may end with `COMPLETED` if the SDK does not flag `ResultMessage.is_error` (unlike hard Bedrock entitlement errors, which surface as **FAILED** once the CLI sets `is_error` on the result) - Diagnosis: 1. Check application logs for `"text": "API Error: 503 Too many connections"` 2. **Check what model_id is actually being passed** — the DynamoDB record may have a stale model override: ```bash aws dynamodb get-item \ --table-name \ --key '{"repo": {"S": "owner/repo"}}' \ --query 'Item.model_id' --output text ``` - Causes: - **Stale model_id in DynamoDB** (most common) — the Blueprint `onUpdate` only sets fields present in props; removing a `modelId` prop does NOT remove the field from DynamoDB. The task keeps using the old model. - Bedrock service-level throttling for the specific model (Opus-class models have tighter limits than Sonnet or Haiku) - Account quota limits reached - Fix: 1. **Check and fix the DynamoDB record first** — remove stale `model_id` if present: ```bash aws dynamodb update-item \ --table-name \ --key '{"repo": {"S": "owner/repo"}}' \ --update-expression "REMOVE model_id" ``` 2. If model_id is correct, wait and retry — throttling is often transient 3. Switch to a model with higher availability (Haiku 4.5 > Sonnet 4.6 > Opus) 4. Request a Bedrock quota increase for `InvokeModel` RPM on your model **`task_timed_out`:** - 9-hour maximum exceeded - Consider reducing scope or increasing `max_turns` for complex tasks - Check if the agent is stuck in a loop (review logs) **Concurrency limit:** - Default: 3 concurrent tasks per user - Wait for running tasks to complete or cancel them ## Local Agent Testing Issues ```bash # Verify Docker is running docker info # Test locally with dry run DRY_RUN=1 ./agent/run.sh "owner/repo" "Test task" ``` **Common causes:** - Missing environment variables: `GITHUB_TOKEN`, `AWS_REGION` - Docker not running or insufficient resources (needs 2 vCPU, 8 GB RAM) - Missing AWS credentials for Bedrock access ## Diagnostic Commands Quick Reference ```bash # Stack status aws cloudformation describe-stacks --stack-name backgroundagent-dev --query 'Stacks[0].StackStatus' # Stack outputs aws cloudformation describe-stacks --stack-name backgroundagent-dev --query 'Stacks[0].Outputs' --output table # Task status (use --verbose for HTTP-level debug output) node cli/lib/bin/bgagent.js --verbose status node cli/lib/bin/bgagent.js events --output json # Watch task progress in real time node cli/lib/bin/bgagent.js watch # Download full execution trace (task must have been submitted with --trace) node cli/lib/bin/bgagent.js trace download # List running tasks node cli/lib/bin/bgagent.js list --status RUNNING # Build health mise run build ``` **Tip:** Add `--verbose` to any `bgagent` command to see the full HTTP request/response cycle on stderr. This is the fastest way to diagnose auth, network, or API contract issues.