# Visual Regression Testing (Automated)

This document outlines the automated visual regression testing process implemented in the UQ Design System. It provides an overview for developers contributing components and detailed information for maintainers responsible for the testing infrastructure.

---

## 1. Purpose of Visual Regression Testing

Visual regression testing is crucial for the UQ Design System to ensure that visual changes to components are intentional and do not introduce unintended side effects across different parts of the system or break existing components. By comparing screenshots of components over time, we can quickly identify and address unwanted visual deviations, maintaining a consistent and high-quality user interface.

---

## 2. Overview of the Automated Process

Our visual regression testing is integrated into the GitHub Actions workflow. When changes are pushed to specific branches (e.g., `pull requests`, `master`, or `releases`), a dedicated GitHub Action is triggered. This action automates the following steps:

1.  **Builds Storybook instances:** It compiles both the HTML and React Storybooks into static bundles.
2.  **Generates Test Configuration:** A script automatically discovers all active stories from the built Storybooks and generates a `screenshot.config.json` file. This file includes the specific `localhost` URLs for each story (based on their respective Storybook ports) and defined viewport settings.
3.  **Runs Regression Tests:** `reg-suit` visual regression testing tool (driven by `screenshot.js`) consumes the generated `screenshot.config.json` to navigate to each Storybook story at specified viewports, capture screenshots, and compare them against a baseline.
4.  **Reports Results:** The results, including any detected visual differences, are reported back in the GitHub Action run and posted as a comment in the `pull request`.

This automation ensures that every new component or story is automatically included in the regression test suite, removing the need for manual configuration updates and guaranteeing comprehensive coverage.

---

## 3. Key Files and Their Roles

Understanding these files is essential for both using and maintaining the system:

- **`.github/workflows/reg.yml`**:
  - **Role:** The GitHub Actions workflow file that orchestrates the entire regression testing process. It defines the sequence of steps, including installing dependencies, building Storybooks, generating the test configuration, and running the screenshot tests.
  - **Location:** `.github/workflows/reg.yml`
- **`scripts/generate-screenshot-config.js`**:
  - **Role:** This Node.js script is the core of the automation. It reads the `index.json` manifest files from both the HTML and React Storybook static builds. It then extracts all individual Storybook stories, constructs their full `localhost` URLs (applying the correct port for HTML or React Storybook), and compiles them into the `screenshot.config.json` file, along with the predefined viewports.
  - **Location:** `scripts/generate-screenshot-config.js`
- **`scripts/screenshot.config.json`**:
  - **Role:** This is the _generated_ configuration file for the visual regression testing tool. It lists all the Storybook story URLs to be tested and defines the screen resolutions (viewports) at which screenshots should be taken. **This file should generally not be manually edited.**
  - **Location:** `scripts/screenshot.config.json`
  - **Example Structure:**
    ```json
    {
      "uris": {
        "introduction-overview--overview": "http://localhost:6006/iframe.html?id=introduction-overview--overview",
        "core-colour--brand": "http://localhost:6006/iframe.html?id=core-colour--brand",
        "components-button--primary": "http://localhost:6007/iframe.html?id=components-button--primary"
        // ... more URIs, correctly prefixed
      },
      "viewports": {
        "mobile": { "width": 375, "height": 932 },
        "desktop": { "width": 1280, "height": 900 }
      }
    }
    ```
- **`scripts/screenshot.js`**:
  - **Role:** This script acts as the runner for your chosen visual regression testing tool. It reads the `uris` and `viewports` from the `screenshot.config.json` and executes the screenshot comparison logic.
  - **Location:** `scripts/screenshot.js`
- **`package.json`**:
  - **Role:** Defines the `npm` scripts used in the GitHub Action, such as `npm run build-storybook`, and any other commands necessary for the setup (e.g., `http-server`).
  - **Location:** `package.json` (at the repository root)
- **`packages/storybook-html/storybook-static/index.json`**:
  - **Role:** This is a manifest file generated by the HTML Storybook build. It lists all the stories and their metadata for the HTML components.
  - **Location:** `packages/storybook-html/storybook-static/index.json`
- **`packages/storybook-react/storybook-static/index.json`**:
  - **Role:** Similar to the HTML version, this manifest file is generated by the React Storybook build, listing all React component stories.
  - **Location:** `packages/storybook-react/storybook-static/index.json`

---

## 4. How the Automation Works (Detailed Flow in `reg.yml`)

When the `reg.yml` workflow is triggered:

1.  **Checkout Code:** The repository code is checked out.
2.  **Node.js Setup:** Node.js is installed to run the scripts.
3.  **Install Dependencies:** `npm ci` installs all project dependencies.
4.  **Build Storybook (React):** The command `npm run build-storybook` is executed, compiling the React Storybook and generating its static assets, including `index.json` in `packages/storybook-react/storybook-static/`.
5.  **Build Storybook (HTML):** The command `npm run build-storybook` is executed, which compiles the HTML Storybook and generates its static assets, including `index.json` in `packages/storybook-html/storybook-static/`.
6.  **Generate Screenshot Config:** `node scripts/generate-screenshot-config.js` is run. This script:
    - Reads `packages/storybook-react/storybook-static/index.json`.
    - Reads `packages/storybook-html/storybook-static/index.json`.
    - Combines all `type: "story"` entries from both files.
    - For each story, it creates a full URL:
      - `http://localhost:6007/iframe.html?id=<storyId>` for React stories.
      - `http://localhost:6006/iframe.html?id=<storyId>` for HTML stories.
    - Writes the final `uris` (with correct base URLs) and `viewports` to `scripts/screenshot.config.json`.
7.  **Run Regression Tests:** `node scripts/screenshot.js` is executed. This script then uses the newly generated `screenshot.config.json` to instruct your visual regression tool to:
    - Spin up local web servers for `packages/storybook-html/storybook-static` on port `6006` and `packages/storybook-react/storybook-static` on port `6007` (or connect to pre-existing ones if your `screenshot.js` handles that).
    - Visit each URL listed in `uris`.
    - Capture screenshots at the specified `viewports`.
    - Compare them against baseline images.
    - Report differences.

---

## 5. How to Add a New Component/Story for Testing

One of the main benefits of this automation is simplicity for developers:

1.  **Create your new component/story** as you normally would within your Storybook setup (either HTML or React).
2.  **Ensure your story is correctly configured** in your Storybook's `main.js` and is discoverable by Storybook during its build process.
3.  **Commit and push your changes.**
4.  When the GitHub Action for regression testing runs, the `generate-screenshot-config.js` script will automatically:
    - Detect your new story from the updated `index.json` file.
    - Add its corresponding `localhost` URL to `screenshot.config.json` with the correct port.
    - Your new component/story will then be included in the subsequent visual regression test run.

**No manual updates to `screenshot.config.json` are required when adding new stories!**

---

## 6. Understanding Test Results

After a GitHub Action run completes, you can review the Visual Regression results:

### GitHub Action logs

The run summary will indicate whether the `Visual Regression Testing` job passed or failed.

- **Previous Snapshot** detects the previous visual regression.
  ```javascript
  [reg-suit] info Detected the previous snapshot key: '206e0b3a360c71f3524fb6d25ff38cf4f6275773'
  [reg-publish-s3-plugin] info Download 442 files from uq-ds-regsuit.
  ```
- **Comparison and current snapshot** Will indicate changed, new, deleted and passed items
  ```javascript
  [reg-suit] info    Changed items: 0
  [reg-suit] info    New items: 80
  [reg-suit] info    Deleted items: 22
  [reg-suit] info    Passed items: 420
  [reg-suit] info The current snapshot key: 'ecca89a09e154107b250f8668371c7134794170c'
  ```
- **Publish snapshot** indicates successful publish to S3 bucket and report URL.
  ```javascript
  [reg-publish-s3-plugin] info Upload 946 files to uq-ds-regsuit.
  [reg-suit] info Published snapshot 'ecca89a09e154107b250f8668371c7134794170c' successfully.
  [reg-suit] info Report URL: https://uq-ds-regsuit.s3.amazonaws.com/regression/ecca89a09e154107b250f8668371c7134794170c/index.html
  ```

### GitHub pull request's comment

`reg-notify-github-plugin` will send the comparison result as the GitHub pull request's comment.

- **Pull request comment** comment will typically indicate which tests were new, deleted, passed, or had differences.
  ![Regression summary](https://github.com/reg-viz/reg-suit/blob/master/packages/reg-notify-github-plugin/images/capt_pr_comment.png)

- **View report** If changes have been flagged the report will provide a visual comparison of artifacts, including difference images (highlighting visual deviations).
  ![Difference images](https://raw.githubusercontent.com/reg-viz/reg-suit-lp/master/src/images/chrome_mobile.png)

---

## 7. Troubleshooting Common Issues

- **`index.json` not found:**
  - **Cause:** The Storybook build step (`npm run build-storybook`) might have failed, or the `storybookStaticDir` paths in `generate-screenshot-config.js` are incorrect.
  - **Solution:**
    - Check the build steps in the GitHub Action logs for errors.
    - Verify that the `build-storybook` commands in `package.json` correctly deploys the output directory (e.g., `storybook-static`).
    - Double-check the paths in `scripts/generate-screenshot-config.js` are accurate relative to the `scripts` directory.
- **Stories not appearing in `screenshot.config.json`:**
  - **Cause:** The story might not be correctly configured in Storybook, or its `type` is not `story` (e.g., `docs`), or the Storybook build is not fresh.
  - **Solution:**
    - Verify the story appears correctly when you run Storybook locally.
    - Ensure the `type` property for your story in `index.json` is `"story"`.
    - Confirm the Storybook build step in `reg.yml` ran successfully.
- **Test Failures (Visual Differences):**
  - **Cause:** An actual visual change has occurred (intentional or unintentional), or there's an environmental difference between the CI and baseline environment.
  - **Solution:**
    - Inspect the difference images/reports (if available as artifacts) to understand the change.
    - If the change is intentional and desired, update your visual regression baseline images. (The method for updating baselines depends on your specific visual regression tool, e.g., running with an `--update` or `--diff` flag).
    - If unintentional, debug the component or style changes that caused the regression.
- **`localhost` server issues (relevant if `screenshot.js` directly serves):**
  - **Cause:** The Storybook static servers are not starting correctly, or are not accessible on the specified ports (`6006`, `6007`).
  - **Solution:** Check the logs for the `Serve Storybook HTMLfor VRT` and `Serve Storybook REACT for VRT` steps in `reg.yml`. Ensure `npx http-server` (or your chosen server) is running and accessible on the correct ports. Verify firewall settings or container network configurations if running in complex environments.

---

## 8. Maintenance and Future Considerations

- **Updating Storybook Versions:** When upgrading Storybook, always check for changes in the `index.json` format or build output locations. The `generate-screenshot-config.js` script might need minor adjustments.
- **Adding New Storybook Instances:** If you introduce another Storybook instance (e.g., for a different framework), you will need to:
  1.  Add its build step to `reg.yml`.
  2.  Add its path and `baseUrl` to the `storybookBuildConfigs` array in `scripts/generate-screenshot-config.js`.
  3.  Ensure its `index.json` is correctly generated.
  4.  Update `reg.yml` to start/manage its static server.
- **Changing Ports:** If the ports (`6006`, `6007`) for serving Storybook instances change, update them in `scripts/generate-screenshot-config.js` (within `storybookBuildConfigs`) and in your `reg.yml` workflow (`http-server` commands).