---
name: phoenix-playwright-tests
description: Write Playwright E2E tests for the Phoenix AI observability platform. Use when creating, updating, or debugging Playwright tests, or when the user asks about testing UI features, writing E2E tests, or automating browser interactions for Phoenix.
metadata:
  internal: true
---

# Phoenix Playwright Test Writing

Write end-to-end tests for Phoenix using Playwright. Tests live in `app/tests/` and follow established patterns.

## Timeout Policy

- Do not pass timeout args in test code under `app/tests`.
- Tune timing centrally in `app/playwright.config.ts` (global `timeout`, `expect.timeout`, `use.navigationTimeout`, and `webServer.timeout`).

## Quick Start

```typescript
import { expect, test } from "@playwright/test";
import { randomUUID } from "crypto";

test.describe("Feature Name", () => {
  test.beforeEach(async ({ page }) => {
    await page.goto(`/login`);
    await page.getByLabel("Email").fill("admin@localhost");
    await page.getByLabel("Password").fill("admin123");
    await page.getByRole("button", { name: "Log In", exact: true }).click();
    await page.waitForURL("**/projects");
  });

  test("can do something", async ({ page }) => {
    // Test implementation
  });
});
```

## Test Credentials

| User   | Email                | Password  | Role   |
| ------ | -------------------- | --------- | ------ |
| Admin  | admin@localhost      | admin123  | admin  |
| Member | member@localhost.com | member123 | member |
| Viewer | viewer@localhost.com | viewer123 | viewer |

## Selector Patterns (Priority Order)

1. **Role selectors** (most robust):

   ```typescript
   page.getByRole("button", { name: "Save" });
   page.getByRole("link", { name: "Datasets" });
   page.getByRole("tab", { name: /Evaluators/i });
   page.getByRole("menuitem", { name: "Edit" });
   page.getByRole("cell", { name: "my-item" });
   page.getByRole("heading", { name: "Title" });
   page.getByRole("dialog");
   page.getByRole("textbox", { name: "Name" });
   page.getByRole("combobox", { name: /mapping/i });
   ```

2. **Label selectors**:

   ```typescript
   page.getByLabel("Email");
   page.getByLabel("Dataset Name");
   page.getByLabel("Description");
   ```

3. **Text selectors**:

   ```typescript
   page.getByText("No evaluators added");
   page.getByPlaceholder("Search...");
   ```

4. **Test IDs** (when available):

   ```typescript
   page.getByTestId("modal");
   ```

5. **CSS locators** (last resort):
   ```typescript
   page.locator('button:has-text("Save")');
   ```

## Common UI Patterns

### Dropdown Menus

```typescript
// Click button to open dropdown
await page.getByRole("button", { name: "New Dataset" }).click();
// Select menu item
await page.getByRole("menuitem", { name: "New Dataset" }).click();
```

### Nested Menus (Submenus)

```typescript
// Open menu, hover over submenu trigger, click submenu item
await page.getByRole("button", { name: "Add evaluator" }).click();
await page
  .getByRole("menuitem", { name: "Use LLM evaluator template" })
  .hover();
await page.getByRole("menuitem", { name: /correctness/i }).click();

// IMPORTANT: Always use getByRole("menuitem") for submenu items, not getByText()
// Playwright's auto-waiting handles the submenu appearance timing
// ❌ BAD - flaky in CI:
// await page.getByText("ExactMatch").first().click();
// ✅ GOOD - reliable:
// await page.getByRole("menuitem", { name: /ExactMatch/i }).click();
```

### Dialogs/Modals

```typescript
// Wait for dialog
await expect(page.getByRole("dialog")).toBeVisible();
// Fill form in dialog
await page.getByLabel("Name").fill("test-name");
// Submit
await page.getByRole("button", { name: "Create" }).click();
// Wait for close
await expect(page.getByRole("dialog")).not.toBeVisible();
```

### Tables with Row Actions

```typescript
// Find row by cell content
const row = page.getByRole("row").filter({
  has: page.getByRole("cell", { name: "item-name" }),
});
// Click action button in row (usually last button)
await row.getByRole("button").last().click();
// Select action from menu
await page.getByRole("menuitem", { name: "Edit" }).click();
```

### Tabs

```typescript
await page.getByRole("tab", { name: /Evaluators/i }).click();
await page.waitForURL("**/evaluators");
await expect(page.getByRole("tab", { name: /Evaluators/i })).toHaveAttribute(
  "aria-selected",
  "true",
);
```

### Form Inputs in Sections

```typescript
// When multiple textboxes exist, scope to section
const systemSection = page.locator('button:has-text("System")');
const systemTextbox = systemSection
  .locator("..")
  .locator("..")
  .getByRole("textbox");
await systemTextbox.fill("content");
```

## Serial Tests (Shared State)

Use `test.describe.serial` when tests depend on each other:

```typescript
test.describe.serial("Workflow", () => {
  const itemName = `item-${randomUUID()}`;

  test("step 1: create item", async ({ page }) => {
    // Creates itemName
  });

  test("step 2: edit item", async ({ page }) => {
    // Uses itemName from previous test
  });

  test("step 3: verify edits", async ({ page }) => {
    // Verifies itemName was edited
  });
});
```

## Assertions

```typescript
// Visibility
await expect(element).toBeVisible();
await expect(element).not.toBeVisible();

// Text content
await expect(element).toHaveText("expected");
await expect(element).toContainText("partial");

// Attributes
await expect(element).toHaveAttribute("aria-selected", "true");

// Input values
await expect(input).toHaveValue("expected value");

// URL
await page.waitForURL("**/datasets/**/examples");
```

## Navigation Patterns

```typescript
// Direct navigation
await page.goto("/datasets");
await page.waitForURL("**/datasets");

// Click navigation
await page.getByRole("link", { name: "Datasets" }).click();
await page.waitForURL("**/datasets");

// Extract ID from URL
const url = page.url();
const match = url.match(/datasets\/([^/]+)/);
const datasetId = match ? match[1] : "";

// Navigate with query params
await page.goto(`/playground?datasetId=${datasetId}`);
```

## Running Tests

Before running Playwright tests, build the app so E2E runs against the latest frontend changes:

```bash
pnpm run build
```

```bash
# Run specific test file
pnpm exec playwright test tests/server-evaluators.spec.ts --project=chromium

# Run with UI mode
pnpm exec playwright test --ui

# Run specific test by name
pnpm exec playwright test -g "can create"

# Debug mode
pnpm exec playwright test --debug
```

### Avoiding Interactive Report Server

By default, Playwright serves an HTML report after tests finish and waits for Ctrl+C, which can cause command timeouts. Use these options to avoid this:

```bash
# Use list reporter (no interactive server)
pnpm exec playwright test tests/example.spec.ts --project=chromium --reporter=list

# Use dot reporter for minimal output
pnpm exec playwright test tests/example.spec.ts --project=chromium --reporter=dot

# Set CI mode to disable interactive features
CI=1 pnpm exec playwright test tests/example.spec.ts --project=chromium
```

**Recommended for automation**: Always use `--reporter=list` or `CI=1` when running tests programmatically to ensure the command exits cleanly after tests complete.

## Phoenix-Specific Pages

| Page                 | URL Pattern                  | Key Elements                                       |
| -------------------- | ---------------------------- | -------------------------------------------------- |
| Datasets             | `/datasets`                  | Table, "New Dataset" button                        |
| Dataset Detail       | `/datasets/{id}/examples`    | Tabs (Experiments, Examples, Evaluators, Versions) |
| Dataset Evaluators   | `/datasets/{id}/evaluators`  | "Add evaluator" button, evaluators table           |
| Playground           | `/playground`                | Prompts section, Experiment section                |
| Playground + Dataset | `/playground?datasetId={id}` | Dataset selector, Evaluators button                |
| Prompts              | `/prompts`                   | "New Prompt" button, prompts table                 |
| Settings             | `/settings/general`          | "Add User" button, users table                     |

## UI Exploration with agent-browser

When selectors are unclear, use agent-browser to explore the Phoenix UI. For detailed agent-browser usage, invoke the `/agent-browser` skill.

### Quick Reference for Phoenix

```bash
# Open Phoenix page (dev server runs on port 6006)
agent-browser open "http://localhost:6006/datasets"

# Get interactive snapshot with element refs
agent-browser snapshot -i

# Click using refs from snapshot
agent-browser click @e5

# Fill form fields
agent-browser fill @e2 "test value"

# Get element text
agent-browser get text @e1
```

### Discovering Selectors Workflow

1. Open the page: `agent-browser open "http://localhost:6006/datasets"`
2. Get snapshot: `agent-browser snapshot -i`
3. Find element refs in output (e.g., `@e1 [button] "New Dataset"`)
4. Interact: `agent-browser click @e1`
5. Re-snapshot after navigation/DOM changes: `agent-browser snapshot -i`

### Translating to Playwright

| agent-browser output       | Playwright selector                              |
| -------------------------- | ------------------------------------------------ |
| `@e1 [button] "Save"`      | `page.getByRole("button", { name: "Save" })`     |
| `@e2 [link] "Datasets"`    | `page.getByRole("link", { name: "Datasets" })`   |
| `@e3 [textbox] "Name"`     | `page.getByRole("textbox", { name: "Name" })`    |
| `@e4 [menuitem] "Edit"`    | `page.getByRole("menuitem", { name: "Edit" })`   |
| `@e5 [tab] "Evaluators 0"` | `page.getByRole("tab", { name: /Evaluators/i })` |

## File Naming

- Feature tests: `{feature-name}.spec.ts`
- Access control: `{role}-access.spec.ts`
- Rate limiting: `{feature}.rate-limit.spec.ts` (runs last)

## Common Gotchas

1. **Dialog not closing**: Wait for a deterministic post-action signal (e.g., dialog hidden + success row visible)
2. **Multiple elements**: Use `.first()`, `.last()`, or `.nth(n)`
3. **Dynamic content**: Use regex in name: `{ name: /pattern/i }`
4. **Flaky waits**: Prefer `waitForURL` over `waitForTimeout`
5. **Menu not appearing**: Wait for specific menu state/element visibility

## Debugging Flaky Tests

### Critical Lessons Learned

1. **Don't assume parallelism is the problem**
   - Phoenix tests run with 7 parallel workers without issues
   - The app handles concurrent logins, database operations, and session management properly
   - If tests fail with parallelism, it's usually a test timing issue, not infrastructure
   - Playwright's browser context isolation is robust - each worker gets isolated cookies/sessions

2. **waitForTimeout is almost always wrong**
   - `page.waitForTimeout()` is the #1 cause of flakiness in Phoenix tests
   - Arbitrary timeouts race against rendering and network speed
   - **Always replace with state-based waits:**
     ```typescript
     // ❌ BAD - flaky, races against rendering
     await page.waitForTimeout(500);
     await element.click();

     // ✅ GOOD - waits for actual state
     await element.waitFor({ state: "visible" });
     await element.click();
     ```

3. **Test the actual failure before fixing**
   - Run tests with parallelism enabled to see what actually fails
   - Check error messages - they often point to the real issue
   - Don't optimize prematurely (e.g., caching auth state) if it's not the problem

4. **Phoenix test infrastructure is solid**
   - In-memory SQLite works fine with parallel tests
   - No need for per-worker databases
   - No need for auth state caching
   - Tests use `randomUUID()` for data isolation - this works well

### Debugging Workflow

When tests are flaky:

1. **Run with parallelism multiple times** to catch intermittent failures:
   ```bash
   for i in 1 2 3 4 5; do
     pnpm exec playwright test --project=chromium --reporter=dot
   done
   ```

2. **Look for `waitForTimeout` usage** - replace with proper waits:
   ```bash
   grep -r "waitForTimeout" app/tests/
   ```

3. **Check for race conditions** in element interactions:
   - Wait for element visibility before interacting
   - Wait for network idle when needed: `page.waitForLoadState("networkidle")`
   - Use `waitForURL` after navigation actions

4. **Verify selectors are stable**:
   - Avoid CSS selectors that depend on DOM structure
   - Use role/label selectors that match ARIA attributes
   - Test selectors don't break when UI updates

5. **Run with trace on failure** to see what happened:
   ```bash
   pnpm exec playwright test --trace on-first-retry
   ```

### Common Flaky Patterns and Fixes

| Flaky Pattern | Root Cause | Fix |
|--------------|------------|-----|
| Submenu item not found | Using `getByText()` instead of `getByRole()` | Use `getByRole("menuitem", { name: /pattern/i })` for submenu items |
| Menu click fails | Menu not fully rendered | `await menu.waitFor({ state: "visible" })` before click |
| Dialog assertion fails | Dialog animation not complete | Assert specific completion signal (hidden dialog + next-state element) |
| Navigation timeout | Page still loading | Remove `waitForLoadState("networkidle")` - it's flaky in CI |
| Element not found | Dynamic content loading | Wait for element visibility, not arbitrary timeout |
| Stale element | Re-render between locate and click | Store locator, not element handle |

### Test Stability Best Practices

1. **Use proper waits**:
   ```typescript
   // Wait for element state
   await element.waitFor({ state: "visible" | "hidden" | "attached" })

   // Wait for network
   await page.waitForLoadState("networkidle" | "domcontentloaded" | "load")

   // Wait for URL change
   await page.waitForURL("**/expected-path")
   ```

2. **Use unique test data**:
   ```typescript
   const uniqueName = `test-${randomUUID()}`;
   ```

3. **Prefer role selectors** - they're less brittle:
   ```typescript
   page.getByRole("button", { name: "Save" }) // ✅ Good
   page.locator('button.save-btn') // ❌ Brittle
   ```

4. **Don't fight animations** - wait for them:
   ```typescript
   await expect(dialog).not.toBeVisible();
   ```

5. **Verify URL changes** after navigation:
   ```typescript
   await page.waitForURL("**/datasets");
   ```