--- name: rez-test-ci description: "Declaring and running tests from a package definition — the `tests` attribute and its `command` / `requires` / `run_on` / `on_variants` fields, why a bare `rez-test` runs only tests tagged `default`, every `rez-test` flag, how the exit code is derived, and how to drive package tests from CI. Use when a package needs tests, when `rez-test` runs fewer tests than you declared, or when you are wiring rez package tests into a pipeline. Covers Rez 3.4.0." --- # Rez package tests and CI > **One-sentence summary**: tests live in the `tests` dict of `package.py` and run through > `rez-test`, which resolves a **separate environment per test** from that test's own `requires` — > so a test is a resolve, not just a command. Skill scope: **declaring, running and automating package tests**. For the rest of the build/release/test loop and the other commands' flags see `rez-cli`; for the `package.py` attributes in general and `@early`/`@late` on `tests` see `rez-package-definition`; for the mistakes that make a `package.py` fail to load see `rez-package-pitfalls`; for driving tests from Python see `rez-python-api`. ## Declaring tests `tests` is a dict mapping a test name to either a command, a list of commands, or a dict: ```python name = "toolcheck" version = "1.0.0" tools = ["toolcheck"] tests = { # shorthand: a command string "unit": "python -c \"import sys; sys.exit(0)\"", # or a list of commands "lint": ["python -m compileall -q python"], # full form "integration": { "command": "python -m pytest {root}/tests", "requires": ["pytest"], "run_on": ["pre_release"], "on_variants": { "type": "requires", "value": ["python-3"], }, }, } ``` | Field | Meaning | |---|---| | `command` | the command, or a list of them; run inside the test's own environment | | `requires` | extra request added to the test environment | | `run_on` | a tag or list of tags; a test with no tag is `default` | | `on_variants` | `True` to run on every variant, or `{"type": "requires", "value": [...]}` to run only on variants matching a request | `tests` may be an `@early()` or `@late()` function, which is how a package can declare different tests per variant. Command strings automatically expand references such as `{root}`, so a test can reach files inside its own package without hard-coding a path: ```python tests = { "unit": "python -m unittest -s {root}/tests", } ``` The command otherwise runs as an ordinary shell command in the test environment. Use `{root}` in preference to an absolute path — a test that hard-codes a repository location breaks the moment the package is released somewhere else. ## Running tests ```bash rez-test toolcheck # every test tagged 'default' rez-test toolcheck unit # one named test rez-test toolcheck -l # list every test, including non-default ones rez-test toolcheck --dry-run # show what would run, run nothing rez-test toolcheck -s # stop at the first failure ``` ### A bare `rez-test` runs only `default` tests This is the single most common surprise. Without a test name, `rez-test` selects tests whose `run_on` tags include `default` — and a test declares no `run_on` at all **is** `default`. Anything tagged otherwise, such as `run_on: ["pre_release"]`, is skipped. So given a package with `unit`, `failing` and a `pre_release`-tagged `integration`: ```text $ rez-test toolcheck -l failing integration unit $ rez-test toolcheck 1 succeeded, 1 failed, 0 skipped ``` `integration` is listed but never run. To run it, name it: ```bash rez-test toolcheck integration ``` ### Output and exit code Every run ends with a summary table: ```text Test results: -------------------------------------------------------------------------------- 1 succeeded, 1 failed, 0 skipped Test Status Variant Description ---- ------ ------- ----------- failing failed /packages/toolcheck/1.0.0 Test failed with exit code 3 unit success /packages/toolcheck/1.0.0 Test succeeded ``` The exit code is the exit code of the **first** test that failed, or `0` if none did. A test that could not run at all — its environment would not resolve — yields `-1`. That is all a CI job needs: a non-zero exit is a failure, and the exit code may be the test's own. `--dry-run` reports each test as `skipped` with the description `Dry run mode`, and exits `0`. Use it to check which tests *would* run before you trust a CI job's coverage. Extra arguments after a single named test are passed through to that test's command, which is how you forward flags to the tool a test wraps: ```bash rez-test toolcheck unit -- --verbose ``` The `--` is **required**. Without it the arguments are consumed and the test command receives nothing at all — silently, with the test still passing. Forwarding a flag to the tool a test wraps therefore always needs the separator, and the flags behind it must not be `rez-test` flags you also meant to set. You can only pass extra arguments to **one** named test; naming two is a parser error. ### Flags | Flag | Effect | |---|---| | `-l, --list` | list the package's tests and exit | | `--dry-run` | show what would run without running it | | `-s, --stop-on-fail` | stop on the first failure | | `--inplace` | run in the **current** environment; tests whose requirements are unmet are skipped | | `--extra-packages PKG ...` | add requests to every test environment | | `--paths PATHS` | override the package search path | | `--nl, --no-local` | ignore `local_packages_path` | | `-v, --verbose` | verbose, repeatable | `--inplace` is mutually exclusive with `--extra-packages`, `--paths` and `--no-local`: ```text rez test: error: Cannot use --inplace in combination with --extra-packages/--paths/--no-local ``` `--inplace` also needs an active rez context — outside one it fails on a missing current package, so it belongs inside a `rez-env`, not in a fresh CI shell. A package with no tests is not an error. `rez-test` prints `No tests found in ` to stderr and exits `0`. If your CI must fail when a package declares no tests, check with `-l` and assert on the output yourself. ## Wiring into CI The pattern that works on every CI provider: make the runner rez-aware once, then run `rez-test` as an ordinary command. Rez only needs to be on `PATH` and the package repository reachable. ```bash python -m pip install "rez==3.4.0" export REZ_PACKAGES_PATH=/path/to/packages rez-test toolcheck unit rez-test toolcheck integration ``` Name the tests you want rather than relying on the default tag — an explicit list is self-documenting and cannot silently lose coverage when someone adds a `run_on` tag later. A minimal GitHub Actions job: ```yaml jobs: rez-tests: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.11" - name: Install rez run: python -m pip install "rez==3.4.0" - name: Build and install the package run: rez-build --install - name: Run package tests run: | rez-test toolcheck unit rez-test toolcheck integration ``` Build before you test. `rez-test` reads an **installed** package from the package path, so a package that has only been edited and not built is invisible to it. Verify visibility first when a CI job cannot find a package: ```bash rez-search toolcheck ``` ### Making CI reflect what the package declares Because a bare `rez-test` skips non-default tests, a job that runs `rez-test ` alone will happily report success while running a subset. Prefer enumerating tests, and gate the set in one place so it cannot drift: ```bash rez-test toolcheck -l # what exists rez-test toolcheck --dry-run # what a bare run would do ``` Compare the two when a tag is added. The gap between them is exactly the coverage a bare run would silently drop. ## Troubleshooting | Symptom | Cause | |---|---| | Fewer tests ran than `-l` lists | a bare run only runs `default`-tagged tests; name them | | `No tests found in ` | the package declares none, **or** `rez-test` is reading a different installed copy than you edited | | Test cannot find the package's files | the command runs in a fresh environment; paths must resolve there | | `Cannot use --inplace in combination with ...` | `--inplace` excludes `--extra-packages` / `--paths` / `--no-local` | | `--inplace` fails outside a context | it needs an active rez context; wrap it in `rez-env` | | `You can only pass extra arguments to a single, specified test` | extra args are allowed for exactly one named test | | A test ignores the arguments I passed it | the `--` separator is missing, so they were swallowed and the test still passed | | Exit code is an odd number like `3` | that is the failing test's own exit code, passed through | | CI cannot see the package | it was not built and installed; check `rez-search` | | A test's dependency is missing | add it to the test's own `requires`, not the package's | ## Agent workflow 1. Confirm the package is installed and visible first: `rez-search `. 2. List what exists with `rez-test -l`, then check what a bare run would do with `--dry-run`. 3. Run named tests rather than relying on the default tag, so coverage cannot drift silently. 4. Read the summary table and the exit code — non-zero is the first failing test's own code. 5. Tag slow or release-only tests with `run_on` deliberately, and enumerate them in CI so the tag never removes coverage by accident. 6. Reach for `--inplace` only inside an active `rez-env`; in CI prefer a normal resolved environment.