---
name: "testing-course-samples"
description: "Use when asked to validate, test, smoke-test, or run the course's notebook and code samples against a live Microsoft Foundry / Azure OpenAI configuration. Covers environment setup (.env, az login, packages), the scripts/validate-notebooks.ps1 runner, interpreting PASS/FAIL results, and which lessons need extra resources (Azure AI Search, GitHub MCP, Foundry Local, Playwright)."
---

# Testing the Course Samples

Validate that the lesson notebooks and code samples run against a live
Microsoft Foundry / Azure OpenAI setup. The repo ships a runner at
[`scripts/validate-notebooks.ps1`](../../../scripts/validate-notebooks.ps1) that
executes every Python notebook headlessly and prints a PASS/FAIL matrix.

## When to use
- "Validate all the notebooks / samples against my Azure subscription."
- "Smoke-test the course after upgrading packages or changing models."
- "Which lessons still pass / fail live?"

Do **not** use this for the AI Smoke Test GitHub Action (that validates *deployed*
hosted agents — see [`tests/README.md`](../../../tests/README.md)). This skill
runs the notebooks locally.

## Prerequisites (check first)
1. **Python 3.12+** with course deps: `python -m pip install -r requirements.txt`
   plus the executor: `python -m pip install nbconvert ipykernel`.
2. **`.env` at the repo root** (copy from [`.env.example`](../../../.env.example)) with at least:
   - `AZURE_AI_PROJECT_ENDPOINT` — Foundry project endpoint
     (`https://<account>.services.ai.azure.com/api/projects/<project>`)
   - `AZURE_AI_MODEL_DEPLOYMENT_NAME` — a non-deprecated deployment (e.g. `gpt-5-mini`)
   - `AZURE_OPENAI_ENDPOINT` (`https://<account>.openai.azure.com`) and `AZURE_OPENAI_DEPLOYMENT`
     for lessons that call Azure OpenAI directly (Lesson 06, 02-azure-openai, 14 handoff/human-loop).
3. **`az login`** completed — samples authenticate with `AzureCliCredential` (Entra ID, keyless).
4. Verify the model deployment exists:
   `az cognitiveservices account deployment list -g <rg> -n <account> -o table`.

## Running the validation
```powershell
# All Python notebooks (skips .NET, .venv, site-packages, translations, skill assets)
pwsh scripts/validate-notebooks.ps1

# A single lesson, with a longer per-cell timeout
pwsh scripts/validate-notebooks.ps1 -Filter '08-*' -Timeout 600

# Just list what would run (no execution)
pwsh scripts/validate-notebooks.ps1 -List

# Explicit interpreter (if `python` is not on PATH, e.g. Windows Store alias)
pwsh scripts/validate-notebooks.ps1 -Python "C:/path/to/python.exe"
```
The script writes executed copies, per-notebook logs, and `results.json` to
`$env:TEMP\aiab-nbval` and exits with the number of failures.

Transient failures (shared-subscription HTTP 429 rate limits, an occasional
`AzureCliCredential` token hiccup, or a timeout) are retried automatically
(`-Retries`, default 2, with `-RetryDelaySeconds` backoff, default 20). If a
model deployment is regularly 429-ing, check the subscription's GlobalStandard
TPM quota (`az cognitiveservices usage list -l <region>`) — raising a single
deployment's capacity does not help when the *subscription* quota is exhausted.

## Interpreting results
- `PASS` — the notebook ran end-to-end with no cell error.
- `FAIL` — the first `*Error` / `*Exception` line is shown; open the matching
  `log_*.txt` in the output dir for the full traceback.
- A single notebook's failure is bounded by `-Timeout` (per cell), so a hung
  human-in-the-loop cell surfaces as `StdinNotImplementedError` rather than hanging.

## Lessons that need extra resources (expected to fail without them)
| Lesson | Extra requirement |
|--------|-------------------|
| 05 Agentic RAG | Azure AI Search (`AZURE_SEARCH_SERVICE_ENDPOINT`, key) — has an in-memory fallback path |
| 11 MCP / GitHub | GitHub MCP server + PAT |
| 13 memory (cognee) | `cognee` configured with a model provider |
| 15 browser-use | Playwright browsers installed (`playwright install`) + `AZURE_OPENAI_CHAT_DEPLOYMENT_NAME` |
| 17 local agent | Foundry Local runtime + a downloaded Qwen model (on-device, no cloud) |
| `*-dotnet-*` notebooks | .NET Interactive kernel (excluded by default; use `-IncludeDotnet`) |

## Reporting back
Summarise as a PASS/FAIL table grouped by lesson. Separate genuine regressions
(code/config bugs to fix) from environment gaps (missing Search/Foundry Local/PAT),
and cite the failing `log_*.txt` for each real failure.