# Visual Regression Test Runner This directory contains both the code for the Renderer's Visual Regression Test Runner and a directory of certified snapshot images (`visual-regression/certified-snapshots`) for each of the defined Visual Regression Test Case Snapshots (Snapshots). Within the `certified-snapshots` directory are subdirectories defining which browser and runtime environment (RUNTIME_ENV environment variable) was used to generate their contained snapshots in the format `${browser}-${env}` (WebGL backend) or `${browser}-${env}-canvas` (Canvas2D backend). The supported runtime environments are `ci` (generated from a Linux-based Docker container) and `local` (generated by the local machine environment). Both render backends are tested by default (`--renderMode all`). Use `--renderMode webgl` or `--renderMode canvas` to run a single backend. Tests can opt out of a backend by exporting `renderModes: ('webgl' | 'canvas')[]` from their module in `examples/tests/` (e.g. the `rtt-*` tests are WebGL-only because `renderToTexture` is a noop on the Canvas2D backend). `ci` snapshots are checked into the repo and are the basis of a GitHub Actions PR status check. `local` snapshots are never checked into the repo and are used strictly to speed up local development. See instructions below. NOTE: Currently these tests only run the Chromium web browser as a baseline. Other browser support may come in the future. ## How to run ``` Visual Regression Test Runner Options: --help Show help [boolean] --version Show version number [boolean] -c, --capture Capture new snapshots [boolean] [default: false] -o, --overwrite Overwrite existing snapshots (--capture must also be set) [boolean] [default: false] -v, --verbose Verbose output [boolean] [default: false] -s, --skipBuild Skip building renderer and examples[boolean] [default: false] -p, --port Port to serve examples on [number] [default: 50535] -i, --ci Run in docker container with `ci` runtime environment [boolean] [default: false] -f, --filter Tests to run ("*" wildcard pattern) [string] [default: "*"] ``` The test runner may be launched in local mode with: ``` pnpm test:visual ``` To run the tests in a Docker container with the `ci` runtime environment, use `--ci` mode: ``` pnpm test:visual --ci ``` NOTE: For this to work you need a container runtime **installed and running** (Docker Desktop, or [Colima](https://colima.dev/) via `colima start`, or Podman) and the Visual Regression Docker Image built. On a fresh shell or after a reboot the runtime is usually stopped — if `docker` can't reach the daemon, start it first. See [DOCKER.md](./DOCKER.md), including its [Troubleshooting](./DOCKER.md#troubleshooting) section. ### Comparison Mode (Default) By default, the runner will build the Renderer, then the Example Tests, and then serve/launch the tests in a headless browser. The actual screenshot output from the headless browser for each defined Visual Regression Snapshot will then be compared pixel-by-pixel to the certified expected snapshot images. If a difference is detected, the test will fail. To make it easy to see what went wrong in any failed Snapshot, three images will be saved to the `visual-regression/failed-results`: the certified expected snapshot image, the actual snapshot image, and a difference between the two of them. As a protection, a Git hook exists that will prevent a commit if this directory contains any failed result files. The failed results are cleared before every comparison run. If any test fails the exit code of the test runner will be `1` to indicate there was a failure. Otherwise it will be `0`. ### Capture Mode In order to capture new Snapshots (or overwrite existing ones), you need to run the Visual Regression Test Runner in Capture mode: ``` pnpm test:visual [--ci] --capture ``` This will do everything that the Comparison mode does, but skip the actual comparing and instead capture and save the Snapshot image data files to files in the `visual-regression/certified-snapshots` directory. As a safety feature, by default, Capture mode will not overwrite any existing certified snapshot files. So in order to capture new ones it is recommended to delete the specific certified snapshot files you'd like to update before running Capture mode. If you are sure you'd like to overwrite all of the certified snapshots, you can add the `--overwrite` CLI argument to the command. ### Local Snapshot Testing When developing locally, running the tests in Docker CI mode can be significantly slower than than running them on the local runtime environment. Local snapshots are never checked into the repository, however you can utilize local snapshots to speed up the time it takes to verify if your changes cause any regressions. 1. After creating a new branch for a new feature or bug fix, capture a new set of local snapshots by running: `pnpm test:visual --capture --overwrite` 2. Implement the new feature or bug fix. 3. Before pushing new commits to the remote PR branch, compare your changes to the snapshots you took prior to starting your PR: `pnpm test:visual` ## GitHub Actions Workflow The Visual Regression Tests run as a status check on every pull request update via the GitHub Actions workflow defined in `.github/workflows/tests.yml`. The tests that run are compared with the snapshots in the `certified-snapshots/chromium-ci` directory. When tests failed, the failure results that you normally find in `visual-regression/failed-results` are uploaded to the workflow run as a zip file artifact named **failed-results**. See [Where does the upload go?](https://github.com/actions/upload-artifact#where-does-the-upload-go) for more on how to find it. ## How to define Snapshots The Snapshots themselves are defined in the individual Example Tests located in the `examples/tests` directory. Note that not all Example Tests need to define Snapshots. For an Example Test to define Snapshots, it must export an `automation()` function. Here's an example of one from the `alpha` Example Test: ```ts export async function automation(settings: ExampleSettings) { // Launch the test await test(settings); // Take/define a snapshot await settings.snapshot(); } ``` This method is called only when the Visual Regression Tests are run. Here the Example Test only defines one Snapshot. It first runs the Example Test's renderer code by calling `test()` (defined later in the Example Test's code), and then takes a single snapshot by calling `settings.snapshot()`. When the Visual Regression Tests are run, this Snapshot will be given the name `alpha-1` since it defines the 1st (and only) Snapshot of the `alpha` Example Test. Addtional snapshots can be defined by calling `settings.snapshot()` additional times, while of course making changes to the Renderer state in between calls. A name may be optionally provided in the snapshot call: ```typescript settings.snapshot({ name: 'myname' }); ``` This name will be appended to the name of the Example Test. For example, if run in the `alpha` Example Test, the name of Snapshot will be `alpha_myname-1`. The same name may be used multiple times. Example Tests that utilize the `PageContainer` class to define separate pages of static content may use the `pageContainer.snapshotPages()` helper method to automatically take snapshots of each of the pages defined in the container. Here's an example from the `text-rotation` Example Test: ```ts export async function automation(settings: ExampleSettings) { // Snapshot all the pages (`await test()` resolves to a PageContainer instance) await (await test(settings)).snapshotPages(); } ``` ### Defining Smaller Snapshots By default, when calling `settings.snapshot()` a snapshot will be created of the entire app/testRoot area at 1920x1080 (downscaled to canvas/PNG at 1280x720). Large snapshots like the default require more processing to load and do a diff on. If your test only requires a small snapshot area you can communicate this to the Visual Regression Test Runner in two ways: **Method 1:** Set the dimensions of the testRoot. ```ts // Do this in the actual test itself (not the automation function) for the best // results. export default function test(settings: ExampleSettings) { const { testRoot } = settings; // Set a smaller snapshot area testRoot.width = 200; testRoot.height = 200; // Set a color on the test root so its more obvious when running the test in // the browser what the snapshot area is. testRoot.color = 0xffffffff; } ``` **Method 2:** Provide a `clip` rectangle prop to the `settings.snapshot()` method. ```ts export async function automation(settings: ExampleSettings) { // Launch the test await test(settings); // Take/define a snapshot await settings.snapshot({ clip: { x: 0, y: 0, width: 200, height: 200, }, }); } ``` Method 2 allows more flexibility in how snapshots are defined but should only be used if Method 1 does not satisfy the requirements for the snapshots to be taken. ### Math.random() In order to allow for consistent snapshots when random numbers are convenient in writing tests, Example Tests that are run by the Visual Regression Test Runner receive a consistent constant [seeded implementation](https://github.com/stdlib-js/random-base-mt19937) of `Math.random()`. This means that for each Example Test that is run, `Math.random()` will produce the same sequence of random numbers. `Math.random()` operates as it normally does when the Example Tests are run directly in the browser (without the VRT Runnner).