{ "skill_name": "foundry-agentic-loop", "evals": [ { "id": 1, "mode": "sdd", "capabilities": [ "architecture", "grounding", "placement", "contract", "sdk-versions" ], "targets": [ "references/specify.md", "references/foundry-models.md", "references/implementation-contract.md", "references/foundry-iq-grounding.md", "references/copilot-sdk-foundry.md" ], "risk": "medium", "prompt": "Post-process this Agentic Loop SDD specification into concise architecture and planning guidance: Build a Zava HR Concierge on Microsoft Foundry. Use a GitHub Copilot SDK hosted agent, enterprise HR documents in Foundry IQ, one toolbox MCP endpoint, a React frontend, a Python backend, ingestion, and azd/Bicep deployment. Multiple engineers will implement components in parallel. The target subscription is not known to have model, Search, hosted-agent, or preview capacity in East US 2. Do not add document-level or caller-aware authorization. Return a Markdown policy addendum only; do not create or deploy resources.", "expected_output": "A concise policy addendum that applies the one-toolbox/one-knowledge-base architecture, joint live placement preflight, frozen implementation contract, latest-stable SDK compatibility, citations, and deterministic no-source refusal without caller-aware document controls.", "files": [], "expectations": [ "The addendum defaults to one governed toolbox MCP endpoint and one Foundry IQ knowledge base.", "Before persisting azd environment values, placement jointly validates the exact model/version/SKU/capacity, Azure AI Search SKU, hosted-agent availability, and required preview features.", "The plan freezes a shared cross-component contract before parallel work and includes service names, environment variables, protocol/endpoint, MCP schemas, model configuration, response/citation/refusal shapes, and runtime/package versions.", "Supported claims require citations, and no cited evidence produces a deterministic refusal.", "The addendum does not require caller identity propagation, group/persona filters, entitlement knowledge bases, or document allow/deny evaluation.", "The SDK guidance resolves and verifies the latest stable github-copilot-sdk API before generating code." ] }, { "id": 2, "mode": "shared", "capabilities": [ "sdk-versions", "companion-skills" ], "targets": [ "references/copilot-sdk-foundry.md", "references/build-skills-catalog.md" ], "risk": "low", "prompt": "Review this generated Python reference against the latest stable github-copilot-sdk release and return a corrected code block plus a short compatibility note. The current code is: `client = CopilotClient(); await client.start(); session = await client.create_session(config); await session.send_and_wait({\"prompt\": prompt}); await client.stop()`. Preserve tool, model, provider, and permission options from `config`. Prefer safe lifecycle cleanup. Also state what to do if an installed companion skill's example disagrees with the package API.", "expected_output": "Lifecycle-safe code verified against the latest stable package API, plus accurate Python/session/response-event compatibility and upstream-defect guidance.", "files": [], "expectations": [ "The latest stable github-copilot-sdk release is resolved from the official package source before API guidance is produced.", "The corrected code follows the session creation, prompt sending, response/event, and lifecycle APIs exposed by that resolved release.", "The compatibility note states the resolved package version and supported Python version.", "Compatibility checks are assigned to the generated application's implementation or CI workflow, not to the agent skill itself.", "If companion guidance conflicts with the resolved package API, the package source is authoritative and the defect is handled through an upstream issue or PR rather than a local third-party skill patch." ] }, { "id": 3, "mode": "sdd", "capabilities": [ "placement", "companion-skills", "rbac" ], "targets": [ "references/specify.md", "references/foundry-models.md", "references/build-skills-catalog.md", "references/rbac-preflight.md" ], "risk": "medium", "prompt": "Prepare the Specify preflight for an Agentic Loop SDD run that needs gpt-5.4-mini at 100K TPM, Azure AI Search, Foundry hosted agents, Application Insights, and least-privilege RBAC. The project has copilot-sdk and microsoft-foundry installed from moving main branches, but their freshness is unknown. Explain how to validate placement and companion skills before persisting azd environment values. Return Markdown only and do not install or modify third-party skills.", "expected_output": "A live joint placement check, preferred/deployed location recording, dynamic skill existence and freshness checks, supported refresh workflow, and use of microsoft-foundry/rbac rather than standalone azure-rbac.", "files": [], "expectations": [ "Placement is one live subscription-specific decision covering model/version/SKU/100K capacity, Search SKU, hosted agents, and required preview features before azd environment persistence.", "The plan records preferred and deployed locations and immediately reconciles spec, plan, and deployment-plan artifacts when they differ.", "Installed skill source URL, version/tree SHA, pin state, and update status are inspected; installation alone is not treated as freshness.", "Catalog candidates are validated against the target repository before use, and stale approved unpinned skills are refreshed through gh skill update.", "The output does not modify third-party installed skills and directs defects to an upstream issue or PR.", "Foundry RBAC guidance uses the maintained microsoft-foundry/rbac sub-skill rather than standalone azure-rbac." ] }, { "id": 4, "mode": "rapid", "capabilities": [ "rbac", "contract", "observability", "deploy", "integration-checkpoint" ], "targets": [ "references/rapid.md", "references/rbac-preflight.md", "references/hosted-integration-checkpoint.md", "references/observability.md", "references/implementation-contract.md" ], "risk": "high", "prompt": "Use Agentic Loop rapid prototyping to plan a basic service-desk agent and React/FastAPI app in my existing Foundry project. I am subscription Owner but data-plane access is unknown, and the backend managed identity has not been created. Keep the Basic sample's model, do not use runtime skills or private document retrieval, and do not provision anything. Return only a concise architecture and acceptance checklist.", "expected_output": "A concise plan using MAF, the sample's model and source deployment, with consolidated deployer permission gaps, planned runtime assignments, a README implementation contract, a hosted checkpoint limited to the required boundaries, correlated privacy-preserving telemetry, and separate approval for cloud app deployment.", "files": [], "expectations": [ "Uses Microsoft Agent Framework for the basic agent, Python 3.13, Responses API and the Basic sample's model without substituting a hard-coded model or region.", "Checks management-plane and data-plane access separately; Owner is not accepted as proof of data-plane access.", "Checks ability to assign planned runtime roles without requiring a not-yet-created identity to already have effective permissions.", "Records shared service, environment, protocol/schema, authentication, streaming/error and dependency contracts under ## Implementation contract in the root README, and plans to record the hosted checkpoint verdict under ## Hosted checkpoint.", "Acceptance verifies a correlated backend/agent/tool trace in Application Insights, with sensitive content capture off unless explicitly approved and controlled.", "Preserves local E2E testing and a separate approval gate before frontend/backend deployment with `az containerapp up --name --source --resource-group --environment `.", "Does not require Search, Foundry IQ, the foundry-agentic-loop-workflow, a new Foundry project or an audit of every installed skill.", "The hosted integration checkpoint proves only the required boundaries: model inference always, toolbox skill loading only when runtime skills are required and an MCP call only when an MCP tool is required. It records each excluded boundary (here runtime skills) as N/A with a reason, and does not add an unused skill or switch frameworks to satisfy the checkpoint." ] }, { "id": 5, "mode": "shared", "capabilities": [ "runtime-skills", "grounding", "companion-skills", "placement" ], "targets": [ "references/run-skills-catalog.md", "references/foundry-iq-grounding.md", "references/copilot-sdk-foundry.md", "references/foundry-models.md", "references/build-skills-catalog.md" ], "risk": "medium", "prompt": "Plan a skill-driven incident assistant using the existing incident-postmortem runtime skill and private operational runbooks. The user experience is React with FastAPI. There is an existing Foundry project, a pinned Copilot SDK build skill and no approved Search resource yet. Do not install, update, rewrite skills or provision anything; return concise guidance covering reuse, grounding, compatibility, placement and acceptance.", "expected_output": "A Copilot SDK plan that reuses the named runtime skill, preserves its governed version, respects the pinned build skill, validates SDK APIs and joint placement conditionally, and includes cited and insufficient-evidence acceptance cases.", "files": [], "expectations": [ "Selects Copilot SDK for the skill-driven agent and distinguishes the build skill from the incident-postmortem runtime skill.", "Resolves incident-postmortem from the run catalog rather than generating a replacement and plans to publish/version it in the agent's toolbox.", "Respects the pinned companion skill and does not update it, patch third-party copies or audit unrelated installed skills.", "Checks installed SDK APIs and locks compatible dependency versions.", "Plans governed grounding with citations and an explicit insufficient-evidence response, testing both paths.", "Checks new resource availability, model/SKU/quota and required service placement jointly without changing the existing project, model or region silently.", "Does not provision an unapproved Search resource or send runtime test prompts.", "Plans the deployed identity, named skill load, required MCP call and model request checkpoint before full application implementation." ] }, { "id": 6, "mode": "shared", "capabilities": [ "lessons", "deploy-approval" ], "targets": [ "SKILL.md", "assets/loop-template.md", "references/rapid.md" ], "risk": "medium", "prompt": "Finish the local acceptance handoff for an Agentic Loop build. The UI-to-agent E2E checks and correlated telemetry verification have passed, and the verified local frontend URL has just been saved as LOCAL_FRONTEND in root .env. During this build, a backend/frontend API schema mismatch caused a failed E2E test; updating both sides to the contract in the root README.md fixed it and the test passed. Existing docs/LOOP.md contains a user-authored note and a proposed early contract check that we tried successfully. There are no measured time or token savings. Return the proposed docs/LOOP.md update and handoff only; do not run services, modify installed skills or deploy anything.", "expected_output": "An evidence-grounded docs/LOOP.md update after the .env write, preserving prior content, evaluating the earlier lesson, and proposing generalizable changes to the foundry-agentic-loop skill with regression checks, followed by the separate app deployment approval question.", "files": [], "expectations": [ "Creates or updates docs/LOOP.md after writing LOCAL_FRONTEND and before the frontend/backend deployment approval pause.", "Preserves the existing user-authored note and prior entries rather than replacing the feedback file.", "Records the observed schema mismatch, its correction and passing E2E result without inventing commands, file paths or measurements.", "Proposes a specific improvement to foundry-agentic-loop instructions with a target section, expected benefit and regression check, rather than only summarizing the application.", "Revisits the earlier early-contract-check proposal and records its supported outcome.", "Separates proposals from validated changes and does not claim measured time or token savings.", "Does not disclose secrets or raw .env contents, rewrite installed skills or treat docs/LOOP.md as authorization to relax approval gates.", "Retains the explicit confirmation gate before backend/frontend deployment." ] }, { "id": 7, "mode": "shared", "capabilities": [ "integration-checkpoint", "rbac", "observability", "contract" ], "targets": [ "references/hosted-integration-checkpoint.md", "references/copilot-sdk-foundry.md", "references/rbac-contract.md", "references/implementation-contract.md" ], "risk": "high", "prompt": "Use Agentic Loop to plan a skill-driven operations assistant with one existing toolbox skill, one read-only MCP tool, a React UI and FastAPI backend. We previously built the whole app before finding hosted identity failures. The agent principal does not exist yet. Return the ordered implementation and acceptance plan only; do not provision or invoke anything. Keep Python 3.13, the Basic sample's model and code deployment.", "expected_output": "An integration-first plan that blocks full app work until a minimal deployed skill/tool/model path passes, distinguishes planned from actual identity checks, and persists four separate endpoint contracts without reserved-variable collisions.", "files": [], "expectations": [ "Consults hosted-integration-checkpoint.md and copilot-sdk-foundry.md at the relevant steps rather than inventing platform mechanics.", "Orders minimal MCP/toolbox and hosted-agent deployment before full UI, backend or orchestration implementation.", "Plans to discover the actual agent principal after creation and verifies principal/resource/role combinations separately from the toolbox connection identity.", "Distinguishes the full project endpoint, model API base URL, hosted Responses endpoint and toolbox MCP endpoint; persists discovered values and uses a nonreserved project-endpoint variable.", "Requires actual deployed-identity skill loading, MCP execution and model-request evidence with terminal completion and schema-valid output before expansion.", "Configures hosting and Copilot runtime telemetry before deployment and requires correlated hosting, model and required tool spans at the checkpoint rather than deferring telemetry until full application work.", "Preserves Python 3.13, code deployment, the sample's model and the separate frontend/backend deployment approval gate.", "Does not execute deployment or runtime probes for this plan-only request." ] }, { "id": 8, "mode": "shared", "capabilities": [ "integration-checkpoint", "rbac", "diagnostics" ], "targets": [ "references/hosted-integration-checkpoint.md", "references/rbac-contract.md", "references/foundry-hosted-agent.md" ], "risk": "high", "prompt": "Our minimal hosted agent is active. The CLI exits zero and returns HTTP 200, but its SSE stream ends in response.failed after a model 403. The same prompt works with my az login credentials. An agent identity now exists, the toolbox connection has a different identity, and the application uses a full project URL stored in FOUNDRY_PROJECT_ENDPOINT. Can we proceed with the full app? Give a bounded diagnostic and acceptance plan only. Include tests for missing upstream fields, JSON versus SSE, concurrent response IDs and async tool-call limits.", "expected_output": "A blocked checkpoint with targeted identity/endpoint and direct-versus-SDK diagnosis, bounded propagation retries, safe correlation evidence and exact contract tests instead of repeated full deployments.", "files": [], "expectations": [ "Explicitly blocks full application expansion; active, CLI zero and HTTP 200 do not override response.failed.", "Requires response.completed for SSE or final completed status for JSON, schema-valid output and actual required skill/tool/model execution evidence.", "Uses the actual deployed principal and verifies project Foundry User, model-host account Cognitive Services OpenAI User and authenticated telemetry emitter Monitoring Metrics Publisher where applicable, without blanket grants.", "Keeps the agent-to-toolbox and toolbox-connection-to-downstream identities distinct and does not substitute developer credentials.", "Checks the reserved endpoint-variable collision and moves the application full project URL to a validated nonreserved variable without overwriting platform configuration.", "Compares minimal direct and SDK requests under the same hosted identity/context, uses bounded RBAC propagation retries and avoids unchanged redeployments.", "Requests response, hosted and SDK session, agent-version and service request IDs in safe diagnostics, without tokens, sensitive bodies or opaque caller-context values.", "Tests missing fields, JSON/SSE errors, interleaved response-ID matching and tool-budget reservation before asynchronous dispatch.", "Includes failure rendering, persistence, export, mobile and keyboard acceptance and cleanup restricted to task-created debug sessions." ] }, { "id": 9, "mode": "shared", "capabilities": [ "sdk", "auth", "mcp", "packaging" ], "targets": [ "references/copilot-sdk-foundry.md", "references/copilot-sdk-with-toolbox.py", "references/foundry-toolbox.md", "references/foundry-hosted-agent.md" ], "risk": "medium", "prompt": "Write an implementation recipe, not a deployment, for our Python 3.13 code-deployed Copilot SDK agent. Use the versioned Agentic Loop reference as a source-checked baseline and explain what to verify in the installed SDK/runtime. Model requests use a discovered /openai/v1/ URL; toolbox requests pass on the first turn but fail after token expiry. Concurrent hosted calls carry different x-agent-foundry-call-id and x-agent-user-id headers. Remote pip resolution also fails, although our locked Linux wheels are available locally. Preserve code deployment and our pinned dependencies.", "expected_output": "A version-aware OpenAI Responses provider recipe with refreshed workload tokens and request-isolated allowlisted context for model and toolbox, plus a conditional reproducible bundled-dependency fallback.", "files": [], "expectations": [ "Names the source-checked SDK reference version without claiming the whole recipe was deployment-tested; checks installed signatures and logs the actual running SDK/runtime versions before troubleshooting.", "Uses type openai, wire_api responses and the discovered /openai/v1/ base, with bearer_token_provider for refresh rather than a startup token.", "Forwards each request's x-agent-foundry-call-id unchanged to model and toolbox requests, never forwards x-agent-user-id outbound and never trusts arbitrary browser headers as platform identity.", "Explains that model bearer-token callbacks do not refresh static MCP headers and chooses a supported refreshing or lifecycle-managed request-scoped MCP transport.", "Isolates concurrent caller contexts and tests token renewal without modifying global mutable headers or weakening tool approvals.", "Uses bundling only after remote-resolution diagnosis; retains Python 3.13 and source-code deployment with pinned hash-verified Linux-compatible dependencies.", "Includes extracted packages and compatible runtime assets in the flat deployment archive while excluding them from Git and pytest discovery.", "Does not copy a macOS virtualenv, discard hashes, upgrade pinned dependencies silently or claim upload/active status proves runtime success.", "Requires the hosted checkpoint to pass after the changed bundled artifact is deployed, without performing that deployment in this recipe-only task." ] }, { "id": 10, "mode": "shared", "capabilities": [ "subagents", "integration-checkpoint", "contract" ], "targets": [ "SKILL.md", "references/implementation-contract.md", "references/hosted-integration-checkpoint.md" ], "risk": "medium", "prompt": "Use Agentic Loop to plan parallel implementation of a skill-driven operations assistant with React, FastAPI and a Foundry hosted agent/toolbox. Coding subagents are available, but the minimal hosted checkpoint has not passed yet. The UI and API share schemas; all components share .env and a lockfile. Explain what can run in parallel now and after the checkpoint, worker ownership and integration gates. Return a plan only; do not launch workers or provision resources.", "expected_output": "A dependency-aware subagent plan that preserves integration-first ordering, gives workers non-overlapping ownership and complete context, and reserves shared configuration, approvals and final acceptance for coordinated integration.", "files": [], "expectations": [ "Limits pre-checkpoint parallel work to independent research/preflight checks and blocks full application implementation until hosted proof passes.", "Plans concurrent substantial frontend, backend or agent/tool tasks only after freezing the shared contract and resolving dependencies.", "Gives each worker bounded goals, exclusive files, relevant skill references, verified context, acceptance checks and stop conditions.", "Assigns shared schemas/contracts, .env, lockfiles and deployment configuration to one owner rather than allowing conflicting edits.", "Keeps the worker set small, honors configured limits, and avoids recursive delegation or duplicating worker tasks.", "Serializes dependent work and shared Azure mutations without extending provisioning or deployment approval.", "Requires worker evidence and central cross-component verification; retains hosted proof, local E2E acceptance, docs/LOOP.md and separate app-deployment approval.", "Does not actually launch subagents or provision anything for this plan-only request." ] }, { "id": 11, "mode": "shared", "capabilities": [ "observability", "integration-checkpoint", "privacy" ], "targets": [ "references/observability.md", "references/copilot-sdk-foundry.md", "references/hosted-integration-checkpoint.md" ], "risk": "high", "prompt": "Our Python Copilot SDK hosted agent completes its minimal skill/tool/model probe. The Foundry project has Application Insights linked and APPLICATIONINSIGHTS_CONNECTION_STRING is set. We see hosting requests but no model or tool spans. CopilotClient currently has no telemetry options; two hosted requests can run concurrently. Can we pass the Agentic Loop checkpoint and build the UI? Give a bounded configuration and acceptance plan only. We have not approved a new collector or Azure resources; do not provision, deploy, invoke agents or change dependencies.", "expected_output": "A blocked telemetry checkpoint and version-aware two-producer configuration plan, with a verified OTLP export route, correct exporter identity scope, privacy-preserving trace correlation and deployed span evidence before expansion.", "files": [], "expectations": [ "Blocks full application expansion despite terminal probe completion, a linked project and hosting spans because required model/tool trace evidence is missing.", "Uses the telemetry section of copilot-sdk-foundry.md and checks actual hosting/Copilot SDK/runtime versions rather than claiming every installed version supports current source behavior.", "Configures the host's supported observability once and explicit Copilot runtime telemetry without duplicate providers or runtime clients.", "Uses a real reachable OTLP HTTP receiver with a supported onward export path, not a connection string, ordinary ingestion URL or gRPC-only endpoint as the runtime receiver.", "Records the destination, export routes and actual exporting identities; distinguishes direct Application Insights emitter scope from native OTLP collector DCR scope without unconditional grants.", "Reuses an approved route or reports the missing route as a blocker; does not silently provision a collector or change dependencies.", "Preserves W3C traceparent/tracestate and request-local parent relationships across concurrent runtime/tool callbacks, separately from Foundry authentication caller context.", "Keeps both content-capture settings off and avoids sensitive explicit logs; checks sampling, graceful flush and bounded ingestion delay.", "Requires deployed hosting, model and required tool spans in the destination's actual schema, with correct trace relationships and agent/version identification, not mocks or exporter startup messages.", "Keeps diagnosis bounded and does not invoke probes, deploy or repeatedly redeploy unchanged artifacts for this plan-only request." ] }, { "id": 12, "mode": "shared", "capabilities": [ "lessons" ], "targets": [ "SKILL.md", "assets/loop-template.md" ], "risk": "low", "prompt": "We are using Agentic Loop and have not built the frontend or written LOCAL_FRONTEND. A hosted model request failed because our app read an account-only project URL. We switched to the nonreserved full project endpoint variable and reran the same deployed probe: it now completes with schema-valid output. docs/LOOP.md already has a user note and a proposed endpoint-contract check. No time or token savings were measured, and no skill changes have been applied. Return the immediate next action and proposed docs/LOOP.md update only; do not deploy, edit installed skills or implement the UI.", "expected_output": "Immediate lesson capture before further implementation, updating the existing endpoint lesson with the verified fix and evidence while preserving the user note and proposal status.", "files": [], "expectations": [ "Calls for updating the workspace's docs/LOOP.md now, without waiting for local acceptance, LOCAL_FRONTEND or full application success.", "Records the failure, supported cause, nonreserved endpoint fix and supplied passing deployed-probe evidence without inventing identifiers, commands or measurements.", "Updates the matching endpoint lesson rather than duplicating it and preserves the user-authored note.", "Includes a proposed skill change/section, expected benefit, regression check and status, distinguishing the verified app fix from an unvalidated skill proposal.", "Requires read-back verification of the write and explicit reporting of persistence failures.", "Does not claim all other hosted checkpoint requirements passed or authorize implementation, deployment or installed-skill edits." ] }, { "id": 13, "mode": "shared", "capabilities": [ "lessons", "subagents" ], "targets": [ "SKILL.md", "assets/loop-template.md" ], "risk": "low", "prompt": "During an Agentic Loop build, a worker reports that a retry fix was committed and the agent deployed successfully, but the original failing probe still times out. Another worker fixed an independent schema mismatch and reran its failing contract test successfully. Both workers want to edit the shared docs/LOOP.md now, and local acceptance is tomorrow. Explain the immediate lesson-capture decisions only; do not execute anything.", "expected_output": "Keep the timeout unresolved, immediately record the verified schema fix through the coordinator, and preserve shared-file ownership and evidence boundaries.", "files": [], "expectations": [ "Does not treat a commit or successful deployment as resolution of the still-failing timeout.", "Requires successful rerun of the failing probe or equivalent boundary verification before marking that failure resolved.", "Captures the independently verified schema fix immediately rather than waiting for local acceptance or every failure to be fixed.", "Has workers report evidence promptly to the coordinator, who serializes shared docs/LOOP.md updates instead of allowing competing edits.", "Preserves existing lessons, excludes secrets and separates application-fix verification from skill-change validation." ] }, { "id": 14, "mode": "sdd", "capabilities": [ "mode-selection", "workflow-start", "preflight" ], "targets": [ "SKILL.md", "references/sdd.md" ], "risk": "high", "prompt": "Use Agentic Loop to build and deploy a claims triage assistant for Contoso Insurance with spec-driven development. Use subscription 00000000-0000-0000-0000-000000000000 and region eastus2. Adjusters upload a claim summary; the agent classifies severity, checks the policy rules skill and routes the claim. Start the delivery now.", "expected_output": "SDD mode: a read-only preflight, at most one clarification round, then run_dynamic_workflow for foundry-agentic-loop-workflow with a self-contained brief and the Azure target, followed by a short report of the run ID, canvas progress and checkpoints without polling.", "files": [], "expectations": [ "Chooses SDD mode, not rapid prototyping, because the user asks for spec-driven development and deployment.", "Runs read-only preflight checks (az account, azd auth status, azd Foundry extension, microsoft-foundry availability) and reports all gaps together without installing or upgrading tools unasked.", "Asks at most one round of clarifying questions, only about decisions that change the spec.", "Starts run_dynamic_workflow with name foundry-agentic-loop-workflow and args containing a self-contained brief, subscriptionId, location eastus2 and an environmentName of at most 20 characters.", "Does not create Foundry resources, scaffold code or run azd provision in Chat before the run; the workflow phases do that.", "Does not pass limits the user did not specify.", "Reports the run ID, that the Foundry Agentic Loop canvas shows progress and that the run pauses at approve-spec-v1 and approve-deploy-v1, and does not poll the run." ] }, { "id": 15, "mode": "sdd", "capabilities": [ "canvas-action", "workflow-gates", "lessons" ], "targets": [ "references/sdd.md", "references/verify.md" ], "risk": "medium", "fixture": "sdd-paused", "prompt": "[Agentic Loop canvas action] The foundry-agentic-loop-workflow run wr_42 is paused at blocked-verify-r2. Investigate the Verify gate issues, fix the root cause and record evidence. Do not resume the run.", "expected_output": "Reads the run's gate issues, the artifacts and docs/LOOP.md, fixes and re-verifies the root cause, records a lesson and leaves the run paused for the canvas to resume.", "files": [ "AGENTS.md", "docs/spec.md", "docs/verify.md", "docs/LOOP.md", ".agentic-loop/runs/wr_42.json", "src/backend/routing.py" ], "expectations": [ "Reads the run with dynamic_workflows_manage inspect-run and the gate issues before changing code.", "Reads docs/verify.md, the related artifacts and docs/LOOP.md for context.", "Fixes the root cause in the smallest scope and reruns the failing check, then the related suite, reporting real results.", "Upserts a docs/LOOP.md lesson only after the fix is verified.", "Does not call run_dynamic_workflow or resume the run, because the canvas prompt says not to.", "Does not weaken tests, thresholds or gates, skip the phase or report unverified checks as passed." ] }, { "id": 16, "mode": "sdd", "capabilities": [ "canvas-action", "spec", "placement" ], "targets": [ "references/specify.md", "assets/spec-template.md", "assets/deployment-plan-template.md", "references/foundry-models.md" ], "risk": "medium", "prompt": "[Agentic Loop canvas action] The foundry-agentic-loop-workflow run wr_7 is paused for spec approval. Apply this feedback to docs/spec.md, and to .azure/deployment-plan.md when affected: use swedencentral for EU data residency and add a requirement that adjusters can export the triage decision as PDF. Do not resume the run.", "expected_output": "Consistent edits to the spec and deployment plan, with a new EARS requirement, updated placement and verification plan, and no resume.", "files": [], "expectations": [ "Adds the export requirement in EARS form with a new stable ID, an acceptance criterion and a verification plan entry.", "Updates the region in both docs/spec.md and .azure/deployment-plan.md and reruns or flags the joint placement check for swedencentral before persisting AZURE_LOCATION.", "Keeps existing requirement IDs and the frozen implementation contract consistent; contract changes are explicit.", "Summarizes the changes, including any cost or quota impact.", "Does not resume the run and does not start implementation, provisioning or deployment." ] }, { "id": 17, "mode": "sdd", "capabilities": [ "workflow-gates", "deploy-approval" ], "targets": [ "references/sdd.md", "references/deploy.md" ], "risk": "high", "prompt": "My foundry-agentic-loop-workflow run wr_9 is paused at approve-deploy-v1. Approve it and deploy now.", "expected_output": "A concise pre-deployment summary followed by resuming the same run, without deploying directly from Chat.", "files": [], "expectations": [ "Summarizes the resources, cost drivers, security exposure and region from the deployment plan and docs/verify.md before resuming.", "Resumes with run_dynamic_workflow and resumeFromRunId wr_9 because the user explicitly asked.", "Does not run azd up, azd deploy or az containerapp commands directly in Chat.", "Does not start a new run or raise workflow limits without a workflow_limit_reached error and approval.", "Reports that the run continues in the canvas and notifies on completion, without polling." ] }, { "id": 18, "mode": "improve", "capabilities": [ "improve", "evidence", "approval", "eval" ], "targets": [ "references/improve.md" ], "risk": "medium", "fixture": "improve-history", "prompt": "Improve the Agentic Loop skill and workflow from our last runs in this repository.", "expected_output": "Read-only analysis of flight records, docs/LOOP.md, workflow runs and sessions, clustered by root cause, with minimal evidence-backed diffs and regression checks, applied only after approval.", "files": [ "docs/LOOP.md", ".agentic-loop/runs/wr_101.json", ".agentic-loop/runs/wr_117.json", ".agentic-loop/runs/wr_123.json" ], "expectations": [ "Builds candidate clusters from .agentic-loop/runs/*.json, docs/LOOP.md and dynamic_workflows_manage runs and inspect-run, then uses time-bounded session_store_sql queries with LIMIT only to confirm or quantify them, never reading raw events.jsonl transcripts end to end.", "Clusters findings by root cause and ranks them by frequency and impact, keeping only clusters with evidence from two runs or one verified severe failure.", "Proposes minimal diffs with target file and section, expected benefit and a regression eval, preferring tightening or deleting text.", "Never proposes weakening approvals, RBAC, safety or evidence gates, or adding secrets or tenant-specific values.", "Does not edit any file before approval; in a solution repository, applies changes as project overrides under .github/skills instead of editing installed plugin files.", "After applying, selects regression cases with `node /scripts/evals.mjs index --target ` and runs them with `run ... --base `; accepts only no regressions with every new or changed case passing, reports failures that also fail at the base as pre-existing, and stops with \"evals unavailable\" if the runner exits 2.", "Marks matching docs/LOOP.md lessons adopted only after a passing regression run, and validated only after a later real run confirms the benefit." ] }, { "id": 19, "mode": "rapid", "capabilities": [ "mode-selection", "rbac", "integration-checkpoint", "local-e2e" ], "targets": [ "references/rapid.md", "references/rbac-preflight.md", "references/hosted-integration-checkpoint.md" ], "risk": "medium", "prompt": "Quickly prototype a retail returns assistant in my Foundry project proj-retail-dev. I want to try it locally today; no pipelines.", "expected_output": "Rapid prototyping in the existing project, driven from Chat, with project validation, RBAC preflight, the hosted checkpoint and a locally tested app.", "files": [], "expectations": [ "Chooses rapid prototyping and does not start foundry-agentic-loop-workflow or create a new Foundry project.", "Validates the named project and records its ARM ID as FOUNDRY_PROJECT in the root .env only after validation.", "Runs the RBAC preflight with management-plane, data-plane and role-assignment checks reported together.", "Keeps the Basic sample's model deployment and passes the hosted integration checkpoint before building the full app.", "Tests the app locally end to end, upserts LOCAL_FRONTEND with the verified URL and treats cloud deployment as optional with separate approval." ] }, { "id": 20, "mode": "shared", "capabilities": [ "bootstrap", "extensions" ], "targets": [ "SKILL.md" ], "risk": "low", "prompt": "Start an Agentic Loop session. extensions_manage list shows foundry-agentic-loop-canvas loaded from the foundry-agentic-loop plugin, but no foundry-agentic-loop-workflow.", "expected_output": "Explains that the workflow is missing, asks before reinstalling the plugin, reloads once after approval and opens the canvas once.", "files": [], "expectations": [ "Does not reinstall or duplicate foundry-agentic-loop-canvas.", "Tells the user foundry-agentic-loop-workflow is missing and asks before running copilot plugin install foundry-agentic-loop@awesome-copilot; never calls install_extension or edits Copilot settings files, and calls extensions_reload once only after approval.", "If installation fails, inspects the extension and reports its log instead of retrying blindly.", "Opens the canvas once with canvasId foundry-agentic-loop-canvas and instanceId foundry-agentic-loop.", "Ensures microsoft-foundry is available, asking before installing it." ] }, { "id": 21, "mode": "sdd", "capabilities": [ "rbac", "mcp", "auth", "infra" ], "targets": [ "references/azd-layout.md", "references/rbac-contract.md", "references/hosted-integration-checkpoint.md" ], "risk": "high", "prompt": "Our Agentic Loop SDD claims solution provisioned, but the hosted integration checkpoint fails. The toolbox's `tools/list` against our keyless MCP server returns HTTP 403, and that container app's authentication shows `allowedApplications: []`. The first hosted invoke returns 500 while the agent downloads its skill. Our script that grants Azure roles to the agent blueprint principal fails with `PrincipalTypeNotSupported`. The Foundry account and project are in `rg-claims-dev-foundry`, the container apps are in `rg-claims-dev`, and the app layer's role assignments on the account failed. We will re-provision the app layer later in Verify. Give a bounded fix plan only; do not run anything.", "expected_output": "A fix plan that grants runtime roles to the agent instance identity only, adds project Foundry User for skill download, persists the MCP allow-list through an azd env var, Bicep and a live patch, scopes account grants to the Foundry resource group, keeps deployed images across re-provision and retries to a deadline instead of redeploying.", "files": [], "expectations": [ "Reads the `instance_identity` and `blueprint` principal and client IDs from `azd ai agent show --output json` and grants Azure roles to the runtime instance principal only, never the blueprint.", "Grants the runtime instance identity Foundry User on the project so the skill download succeeds, plus only the other RBAC contract rows the agent needs, without blanket or subscription-wide grants.", "Allows the instance and blueprint client IDs of every agent whose toolbox reaches the MCP app, stores the sorted list in an azd env var that Bicep maps to `allowedApplications`, and patches the live `authConfigs/current`; it explains the 403 by the empty list.", "Creates the account and project role assignments in a module scoped to the Foundry resource group (`resourceGroup(aiResourceGroupName)` from `AZURE_FOUNDRY_RESOURCE_GROUP`) or in the postdeploy hook, not in the app resource group.", "Sets each container app image from `${SERVICE__IMAGE_NAME=}` with a placeholder fallback, so the planned re-provision keeps the deployed images.", "Puts the post-deploy grants in an idempotent `postdeploy` hook that checks before creating, then retries the first hosted invoke with backoff until a fixed deadline instead of redeploying unchanged code.", "Records the actual resource groups and regions, including `AZURE_FOUNDRY_RESOURCE_GROUP`, in the deployment plan." ] }, { "id": 22, "mode": "sdd", "capabilities": [ "spec", "seed-data" ], "targets": [ "references/specify.md", "assets/spec-template.md" ], "risk": "medium", "prompt": "Review this Agentic Loop spec excerpt before the Specify gate and return only the corrected rules and seed rows. Today is 2026-10-08. (1) Seed S2: a prior claim dated 2021-05-14, 'outside 36 months; bureau count 1'; `get_bureau_reports` defaults to `months=36`. (2) Seed S7 has loss type `water_damage (flood)` and seed S8 has `water_damage`. (3) Fraud signal WEATHER fires on any 'weather-related' loss type. (4) Fraud signal OVER_LIMIT fires when the claimed amount exceeds `limit_usd` from `get_coverage`, but only the adjudicator agent has `get_coverage`. (5) Fraud signal RECENT_POLICY fires when the policy started within 30 days. (6) Routing queues claims by `max(decision.fraud_score, assessment.fraud_score)`, sorted by `routing.priority`; `decision` is null when the adjudicator fails. (7) The two-column RBAC table (Principal | Role) has the row `| backend MI | AcrPull | grant after provision |`. (8) The toolbox note says to call the MCP server with `Authorization: Bearer `.", "expected_output": "Executable seed and decision rules: frozen counts and anchored windows, closed trigger lists, inputs traced to the consuming agent's tools, null-safe routing, well-formed tables, credentials described in words and recomputed seed outcomes.", "files": [], "expectations": [ "S2 states bureau count 0 for the default 36-month window, because 2021-05-14 falls outside it, or explicitly freezes a different window.", "Loss types come from one closed enum and labels carry no decision facts: S7 and S8 either share one value with the flood detail in a separate frozen field, or flood becomes its own enum value, never a parenthetical such as `water_damage (flood)`.", "WEATHER fires on a closed list of loss types instead of an open 'weather-related' category.", "OVER_LIMIT reads only the fraud agent's own tools or frozen facts, for example by adding `get_coverage` to its tool list or freezing `limit_usd` per seed, not a tool that only the adjudicator has.", "RECENT_POLICY names its anchor date, such as the loss date, and each seed freezes the dates the rule depends on.", "Routing defines the result when `decision` is null or invalid, and its sort key covers every enum value plus null.", "Every RBAC row has as many cells as its header, whether the table keeps two columns or adopts the plan template's columns, and the post-provision instruction moves out of the row.", "The `Authorization` header is described in words, as the Bearer scheme with an access token, not written as a literal.", "Each seed's expected outcome is recomputed from the corrected rules: which signals fire and why, plus the score and routing wherever the excerpt supplies the weights they need; missing weights are flagged, not invented." ] }, { "id": 23, "mode": "shared", "capabilities": [ "dependencies", "sdk-versions" ], "targets": [ "SKILL.md", "references/implementation-contract.md", "references/implement.md" ], "risk": "medium", "prompt": "Implement is pinning Python dependencies for a hosted agent and a FastMCP server. The official index has github-copilot-sdk 1.0.17 and fastmcp 4.0.11 as the latest stable releases. The build uses an approved package proxy that serves only up to github-copilot-sdk 1.0.16 and fastmcp 4.0.10, plus a 1.0.17rc1 pre-release of the SDK. A `--generate-hashes` lock compiled through the proxy contains `--hash=md5:` entries, and the ACR remote build fails. The spec's Runtime versions row says `azure-identity` 'latest stable'. What exactly should we pin and record? Return the plan only.", "expected_output": "Pins the newest stable releases the build's index serves (never a pre-release), inspects that API, keeps only sha256/sha384/sha512 hashes, replaces 'latest' with exact versions and records the gap to the official latest release.", "files": [], "expectations": [ "Resolves the latest stable releases from the official package index and also checks what the build's proxy index serves before pinning.", "Pins github-copilot-sdk 1.0.16 and fastmcp 4.0.10, the newest stable releases the proxy serves, and never 1.0.17rc1 or another pre-release.", "Inspects the API of the pinned github-copilot-sdk release instead of assuming the 1.0.17 API.", "Records the gap between each official latest release and its pinned version in the contract or implementation notes.", "Regenerates or filters the hashed lock so it keeps only sha256, sha384 or sha512 hashes, matching pip's allowed hash algorithms.", "Replaces 'latest stable' with an exact version for every direct dependency and exact image tags, and confirms the build's index or registry serves each pin." ] }, { "id": 24, "mode": "sdd", "capabilities": [ "e2e", "concurrency" ], "targets": [ "references/verify.md", "references/deploy.md" ], "risk": "medium", "prompt": "Verify is about to run 10 multi-turn E2E claim scenarios against `gpt-5.4-mini` (GlobalStandard, capacity 100). One scenario took 3 minutes, and its Responses `usage` totaled 60K tokens. An earlier run at concurrency 10 hit 429s and produced AGENT_ERROR results. How should we run and record the E2E suite now, and what does Deploy reuse?", "expected_output": "Parallel E2E at a measured concurrency c = 4, computed from capacity, measured tokens and duration, recorded in docs/verify.md and reused by Deploy, with a 429 after retries counted as a failure.", "files": [], "expectations": [ "Runs the E2E cases in parallel at a measured concurrency, not sequentially by default or at a hard-coded concurrency of 1.", "Computes c = max(1, ⌊0.8 × TPM × minutes per case ÷ tokens per case⌋) with TPM = capacity × 1000, which gives c = 4 for these measurements.", "Takes tokens per case and duration from one measured case's Responses `usage` and timing, not from a guess.", "Records c and its inputs (TPM, tokens per case and minutes per case) in `docs/verify.md`.", "Treats a 429 that persists after retries as a test failure, not as a pass or a skip.", "Has Deploy rerun the E2E suite at the concurrency recorded in `docs/verify.md`." ] }, { "id": 25, "mode": "sdd", "capabilities": [ "scaffold", "azd" ], "targets": [ "references/azd-layout.md", "references/implement.md" ], "risk": "medium", "prompt": "Implement must scaffold the first hosted agent, `fraud`, from the Basic sample into our existing SDD workspace. It already has `docs/`, a root `azure.yaml` with the foundry layer and `.azure/deployment-plan.md`. Everything runs with `--no-prompt`, and `azd ai agent init .` failed with 'confirmation declined; app was not initialized'. A teammate suggests adding `--force` or running `azd init -t` with the sample template first. Give the exact steps only.", "expected_output": "Scaffolds in an empty `.scaffold` folder, moves the sample to `src/agents/fraud`, merges root files, copies only the azd environment, unstages and removes `.scaffold` and writes `.agentignore`, without `--force` or `azd init -t`.", "files": [], "expectations": [ "Runs `azd ai agent init .` with `--no-prompt`, `-t ` from the sample's `initCommand` (not the deprecated `-m `) and `--agent-name fraud` inside a new, empty `.scaffold` folder.", "Rejects both `--force` and `azd init -t