# Contributing to AIPerf For technical architecture, see [`docs/architecture.md`](docs/architecture.md). For AI assistant instructions, see `CLAUDE.md`. ## Development Setup ### Prerequisites - **Python 3.11+** - **uv**: Package manager (installed automatically by `make first-time-setup`) - **pre-commit**: For automated code quality checks - **Kubernetes tests only**: Docker, Kind, kubectl, and Helm ### Initial Setup ```bash # Clone the repository git clone cd aiperf # One-command setup: creates venv, installs project + mock server + pre-commit hooks make first-time-setup # Or manually: make setup-venv # Create virtual environment make install # Install project + mock server + fake amdsmi bindings pre-commit install # Install pre-commit hooks ``` ### Available Commands | Command | Description | |---------|-------------| | `make first-time-setup` | Full environment setup (venv + install + hooks) | | `make install` | Install project, mock server, and fake amdsmi bindings in editable mode | | `make install-app` | Install project only | | `make install-app-locked` | Install project only, at the exact versions in `uv.lock` (what CI's `make ci-install` uses) | | `make install-mock-server` | Install mock server only | | `make install-mock-amdsmi` | Install fake `amdsmi` bindings to exercise the AMD telemetry path on non-AMD hardware (see [Mocking a ROCm Environment](docs/reference/mock-amdsmi.md)) | | `make test` | Unit tests (parallel, excludes integration) | | `make test-verbose` | Unit tests with DEBUG logging | | `make test-all` | All tests (unit + component integration + integration) | | `make test-integration` | Integration tests with mock server | | `make test-integration-verbose` | Integration tests with real-time output | | `make test-component-integration` | Component integration tests | | `make test-kubernetes-ci` | Serial Kubernetes PR gate on a fresh isolated Kind cluster | | `make test-ci` | CI mode: unit + component integration with coverage | | `make test-imports` | Verify all modules can be imported | | `make test-stress` | Stress tests with mock server | | `make coverage` | Unit tests with HTML/XML coverage report | | `make lint` | Run ruff linter | | `make lint-fix` | Auto-fix linter errors | | `make fmt` | Format code with ruff | | `make check-fmt` | Check formatting without changes | | `make validate-plugin-schemas` | Validate plugins.yaml against schemas | | `make generate-all-plugin-files` | Regenerate plugin enums, overloads, schemas | | `make generate-all-docs` | Regenerate CLI + env var documentation | | `make generate-cli-docs` | Regenerate CLI documentation | | `make generate-env-vars-docs` | Regenerate environment variable documentation | | `make generate-crd` | Regenerate the Helm AIPerfJob and AIPerfSweep CRD templates | | `make check-crd` | Verify the generated CRD templates match the Python models | | `make crd-release` | Render standalone CRD manifests into `dist/` (override with `HELM_DIST_DIR`) | | `make helm-lint` | Lint the bundled AIPerf operator Helm chart | | `make helm-template` | Render the bundled Helm chart without cluster access | | `make helm-package` | Package the bundled Helm chart into `dist/` (override with `HELM_DIST_DIR`) | | `make docker` | Build Docker image | | `make docker-run` | Run Docker container | | `make clean` | Clean caches and build artifacts | | `make version` | Print project version | Direct pytest commands: ```bash uv run pytest tests/unit/ -n auto # Unit tests (parallel) uv run pytest -m integration -n auto # Integration tests (multiprocess) uv run pytest -m component_integration -n auto # Component integration tests make test-kubernetes-ci # Kubernetes acceptance tests (serial Kind) ``` The CI workflow builds the local runtime and mock-server images before invoking this target. The test run loads those images, creates a uniquely named Kind cluster with an isolated kubeconfig, and deletes that cluster afterward. It excludes the opt-in GPU, slow, audit, and chaos suites. Keep the gate serial (`-n 0`): its tests intentionally share one operator installation and exercise ordered cluster lifecycle behavior. ### Pre-Commit Hooks The repository uses pre-commit hooks defined in `.pre-commit-config.yaml`: **General hooks:** - `check-ast` - Verify Python AST validity - `debug-statements` - Detect leftover debug statements - `detect-private-key` - Prevent committing private keys - `check-added-large-files` - Fail if files > 5MB added - `check-case-conflict` - Detect case-insensitive filename conflicts - `check-merge-conflict` - Detect merge conflict markers - `check-json` - Validate JSON syntax - `check-toml` - Validate TOML syntax - `check-yaml` - Validate YAML syntax - `end-of-file-fixer` - Ensure files end with newline - `trailing-whitespace` - Remove trailing whitespace - `mixed-line-ending` - Enforce consistent line endings - `no-commit-to-branch` - Prevent direct commits to main **Code quality hooks:** - `codespell` - Spell checking - `ruff` - Lint with auto-fix - `ruff-format` - Format with ruff **Project-specific hooks:** - `add-license` - Add SPDX copyright headers - `generate-cli-docs` - Regenerate CLI documentation when Python files change - `generate-env-vars-docs` - Regenerate env var docs when environment.py changes - `generate-plugin-artifacts` - Regenerate plugin enums/overloads/schemas - `validate-plugin-schemas` - Validate plugin YAML against schemas - `test-imports` - Verify all modules can be imported Run pre-commit after every code change, even before creating commits. Do not wait until commit time to discover problems. ### Bundled Skills The repository ships agent skills under `.agents/skills/` (surfaced to Claude Code through the `.claude/skills` symlink). Running `/skills` inside the repository lists them: ``` Project skills (.claude/skills) aiperf-code-review review a branch against origin/main aiperf-llm-ergonomics-review review CLI/API surfaces for LLM ergonomics aiperf-kube-run run a benchmark on Kubernetes end to end aiperf-kube-setup prepare a cluster and install the operator aiperf-kube-triage diagnose a stuck or failed Kubernetes run aiperf-kube-sweep run parameter sweeps on Kubernetes bump-version version bump helper cherry-pick cherry-pick helper docs-to-fern docs site conversion helper linear-issue Linear issue helper ``` #### Code review When creating a PR, you can run the `aiperf-code-review` skill yourself within your branch (or inside of a worktree) once your pull request is created by prompting Claude similar to the example below: ``` ❯ Can you run a code review with the aiperf-code-review skill? ⏺ Skill(aiperf-code-review) ⎿  Successfully loaded skill ``` You are encouraged to use this to self-review as a first pass review before a maintainer reviews your PR. Please note, the skill does run `aiperf` and utilizes a mock server. If you are working on a laptop or personal work station, be aware that this may slow down your computer during review. #### Kubernetes The `aiperf-kube-*` skills cover the Kubernetes path and are grounded in [`docs/kubernetes/`](docs/kubernetes/). Start with `aiperf-kube-run`, which links out to the other three. They quote CLI flags, container defaults, and Helm values as literals, so they can go stale when those change; the pack ships a verifier that checks every one of them against the live CLI and the working tree: ```bash uv run python .agents/skills/aiperf-kube-run/verify_pack.py ``` Run it after changing an `aiperf kube` flag, a `AIPERF_K8S_*` default, or `deploy/helm/aiperf-operator/values.yaml`, and update the skill text when it fails. ### Package Management Always use `uv` (never pip): `uv add package`, `uv run pytest`. ## Contribution Guidelines Contributions that fix documentation errors or that make small changes to existing code can be contributed directly by following the rules below and submitting a PR. Contributions intended to add significant new functionality must follow a more collaborative path. Before submitting a large PR, submit a GitHub issue describing the proposed change so the AIPerf team can provide feedback: - A design for your change will be agreed upon to ensure consistency with AIPerf's architecture. - The Dynamo project is spread across multiple GitHub repositories. The AIPerf team will provide guidance about how and where your enhancement should be implemented. - Testing is critical. Plan on spending significant time on creating tests. The team will help design testing compatible with existing infrastructure. - User-visible features need documentation. ## Contribution Rules - Code style is enforced by `ruff` (formatting + linting). Follow existing conventions. - Avoid introducing unnecessary complexity. - Keep PRs concise and focused on a single concern. - Build log must be clean: no warnings or errors. - All tests must pass. - No license or patent conflicts. You must certify compliance with the [license terms](https://github.com/ai-dynamo/aiperf/blob/main/LICENSE) and sign off on the [Developer Certificate of Origin (DCO)](https://developercertificate.org). ## Git Workflow Feature branches use `/feature-name` format, forked from `main`. ## Required Checks A PR can merge into `main` only after the `pre-commit` and `Unit Tests Gate` checks pass. `Unit Tests Gate` is the last job in `.github/workflows/run-unit-tests.yml` and fails unless every unit test job passed. Unit tests run on NVIDIA runners, so copy-pr-bot copies each PR to a `pull-request/` branch to run them. A PR is copied automatically only when its author is trusted and every commit is cryptographically signed (GPG or SSH; this is separate from the DCO sign-off below). Fork PRs, PRs with unsigned commits, and bot PRs such as Dependabot's wait for a maintainer to review them and comment `/ok to test `. Until then the PR shows `Unit Tests Gate` as expected and cannot merge. Docs-only PRs are not exempt, because several unit tests check the docs. Repository admins can bypass the requirement when needed. A PR must also be up to date with `main` before it merges, and every branch update creates a new head commit that needs a full unit test run. For a PR that is not copied automatically, a maintainer must comment `/ok to test ` again after each update. If a job fails from a known flake, rerun the failed jobs with `gh run rerun --failed`, which also reruns `Unit Tests Gate`. ## Documenting a Runnable Command A tutorial's `aiperf profile` command is executed in CI against a real inference server only if the block carrying it is tagged, so an untagged guide rots silently -- a renamed flag leaves a copy-pasteable command that no longer works, and nothing fails until a user tries it. Tag at least one command in a new guide by wrapping its fenced block in a pair of HTML comments named `aiperf-run-{server}-endpoint-server`, the closing one prefixed with a slash. `docs/tutorial.md` shows the exact syntax, and [`tests/ci/test_docs_end_to_end/README.md`](tests/ci/test_docs_end_to_end/README.md) documents the available server groups, the `weight=` attribute, and how to run the suite locally. `tools/check_docs_e2e_tags.py` fails a PR that adds a doc with an untagged `aiperf profile` command. > Spelling a tag out in full here would be a mistake: the harness parses every > markdown file in the repo and does not skip code fences, so the example would > be collected and run as a real benchmark. ## Running GitHub Actions Locally You can use the `act` tool to run GitHub Actions locally. See [act usage](https://nektosact.com/introduction.html). ```bash act -j run-integration-tests ``` You can also use the Visual Studio Code extension [GitHub Local Actions](https://marketplace.visualstudio.com/items?itemName=SanjulaGanepola.github-local-actions). ## Nightly Slack Alerts The nightly workflow ends with a `notify-slack` job that calls the reusable `.github/workflows/notify-slack.yml`. It lists the run's jobs through the Actions API and, when any job failed, timed out, or was cancelled, posts one Slack alert per configured webhook. Advisory jobs listed in the caller's `ignored_jobs` input (currently Suggest Shard Weights) are not reported, and nothing is posted for a green run or when the run itself was cancelled. Configuration lives in repository settings. `SLACK_NOTIFY_NIGHTLY_WEBHOOK_URL` (secret) is the incoming webhook for the release automation channel and falls back to the legacy `NIGHTLY_SLACK_WEBHOOK` secret. `SLACK_NOTIFY_AIPERF_DEV_WEBHOOK_URL` (secret) is the webhook for the AIPerf dev channel. `NIGHTLY_SLACK_MENTION` (variable) holds optional Slack IDs to mention, `S...` for a team or `U...` for a user, comma separated. Every webhook is optional; with none configured the job logs a warning and stays green. ## Signing Off Your Work We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have the right to submit it under the project's [Apache-2.0 license](LICENSE). - Any contribution which contains commits that are not Signed-Off will not be accepted. - To sign off on a commit you simply use the `--signoff` (or `-s`) option when committing your changes: ```bash $ git commit -s -m "Add cool feature." ``` This will append the following to your commit message: ```text Signed-off-by: Your Name ``` - Full text of the DCO (https://developercertificate.org/): ```text Developer Certificate of Origin Version 1.1 Copyright (C) 2004, 2006 The Linux Foundation and its contributors. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Developer's Certificate of Origin 1.1 By making a contribution to this project, I certify that: (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. ``` If your pull request fails the DCO check, see the [DCO Troubleshooting Guide](DCO.md).