# ai-wright

AI steps (`ai.act`, `ai.verify`) in your Playwright tests - open source, vision-enabled, with no vendor lock-in.

## Introduction

`ai-wright` lets you include AI-native actions, verifications, and data extraction in any Playwright test. It supports:
- BYOL (bring-your-own-license) to use your own [OpenAI](./src/llm-providers/openai.md), [Google Gemini](./src/llm-providers/gemini.md), or [Anthropic Claude](./src/llm-providers/claude.md) API keys.
- or use your [TestChimp](https://testchimp.io) license key to avoid paying separately for token usage. 

Unlike other solutions, `ai-wright` relies on *vision intelligence*: screenshots are annotated with Set-of-Marks (SoM) overlays and combined with DOM element maps for disambiguation, so the LLM can navigate complex UIs with far greater accuracy and resilience.

**Why teams adopt `ai-wright`:**
- **Vendor flexibility & BYOL** - use your own [OpenAI](./src/llm-providers/openai.md), [Gemini](./src/llm-providers/gemini.md), or [Claude](./src/llm-providers/claude.md) keys, or your [TestChimp](https://testchimp.io) license (to avoid separate token usage costs).
- **Vision-first semantics** - SoM overlays + DOM metadata give the model precise context.
- **Resilient prompting** - pre-action planning (eg: handling blockers like modals before addressing the actual requirement step), retry guidance, ability to handle coarse-grained steps with multi-step planning.
- **Open source** - complete transparency and community support.
- **Pluggable LLM providers** - extend to any LLM provider (eg: a local LLM) by implementing a provider (see [src/llm-providers/README.md](./src/llm-providers/README.md)).

## Usage Guide

### Installation

```bash
npm install ai-wright
# or
yarn add ai-wright
```

Then import the library inside your Playwright tests:

```ts
import { ai } from 'ai-wright';
```

### AI Commands

#### `ai.act(objective, {page,test})`
Executes one or more UI actions to satisfy the given objective. The library:
1. Waits for page stability.
2. Generates a SoM map + screenshot.
3. Queries the LLM for pre-actions necessary (e.g., close modals) and main commands.
4. Runs each command sequentially with detailed retries, to achieve the given objective.

```ts
await ai.act('Log in as alice@example.com with password TestPass123', {
  page,
  test,
});
```

#### `ai.verify(requirement, {page,test}, options?)`
Vision-driven assertion that works like `expect`. It fails the Playwright step if the LLM reports `verificationSuccess = false` or if the reported `confidence` falls below `options.confidence_threshold` (default 70%).

```ts
await ai.verify('The toast should say "Message sent"', {
  page,
  test,
}, {
  confidence_threshold: 85,
});
```

#### `ai.extract(requirement, context, options?)`
Pulls structured data from the page. Set `options.return_type` to shape the output (`'string' | 'string_array' | 'int' | 'int_array'`).

```ts
const orderIds = await ai.extract('List the order IDs from the table', {
  page,
  test,
}, {
  return_type: 'string_array',
});
```

### Authentication

`ai-wright` chooses credentials in priority order:

1. **OpenAI API key**
   - Set `OPENAI_API_KEY` env var. (and optionally `OPENAI_MODEL`, defaults to `gpt-5-mini`).
2. **TestChimp API keys**
   - Set `TESTCHIMP_API_KEY` (optional `TESTCHIMP_PROJECT_ID` for legacy clients), _or_ `TESTCHIMP_USER_AUTH_KEY` + `TESTCHIMP_USER_MAIL`.
   - Benefit: reuse your existing TestChimp account, letting their backend proxy the LLM and cover token costs.
3. **Google Gemini API key**
   - Set `GEMINI_API_KEY` (and optionally `GEMINI_MODEL`, defaults to `gemini-3.8-flash`).
4. **Anthropic Claude API key**
   - Set `CLAUDE_API_KEY` (and optionally `CLAUDE_MODEL`, defaults to `claude-sonnet-5-5`).

The selection order is configurable in [`src/llm-providers/config.ts`](./src/llm-providers/config.ts). See the [LLM provider guide](./src/llm-providers/README.md) for instructions on adding new providers.

### Example Playwright Test

```ts
import { test } from '@playwright/test';
import { ai } from 'ai-wright';

test('send message', async ({ page }) => {
  await page.goto('https://studio--cafetime-afg2v.us-central1.hosted.app/');

  await ai.act('Log in with alice@example.com / TestPass123', { page, test });

  await ai.act('Open the Messages tab and send "Hello"', { page, test });

  await ai.verify('The message input field should be empty afterwards', { page, test });
});
```

### Advanced Configuration

Environment variables:

| Variable | Description | Default |
| --- | --- | --- |
| `AI_PLAYWRIGHT_DEBUG` | Enable verbose logging (`1`, `true`, `on`, `yes`). | off |
| `AI_PLAYWRIGHT_TEST_TIMEOUT_MS` | Extend Playwright test timeouts automatically; `0` disables extension. | `120000` |
| `AI_PLAYWRIGHT_MAX_WAIT_RETRIES` | How many times the LLM may request additional waits. | `2` |
| `LLM_CALL_TIMEOUT` | Max duration (ms) for each LLM request. | `120000` |
| `COMMAND_EXEC_TIMEOUT` | Timeout (ms) for individual DOM actions. | `30000` |
| `NAVIGATION_COMMAND_TIMEOUT` | Timeout (ms) for navigation actions. | `30000` |
| `OPENAI_MODEL` | Override the OpenAI model (ignored when using TestChimp). | `gpt-5-mini` |
| `GEMINI_API_KEY` | Google Gemini API key used by the Gemini provider. | - |
| `GEMINI_MODEL` | Override the Gemini model. | `gemini-3.8-flash` |
| `CLAUDE_API_KEY` | Anthropic Claude API key used by the Claude provider. | - |
| `CLAUDE_MODEL` | Override the Claude model. | `claude-sonnet-5-5` |
| `CLAUDE_MAX_TOKENS` | Override the Claude response token limit. | `1024` |

Optional context/options:
- `context.logger`: `(message: string) => void` to receive internal log output.
- `options.confidence_threshold`: override verification threshold per call.
- `options.return_type`: control extraction result shape.

## Comparison with Other Solutions

### ZeroStep
- Requires a proprietary license key causing vendor-lock.
- Unmaintained and limited to GPT-3.5.
- Tied to CDP (Chrome DevTools Protocol), so works only with Chrome.
- Offers fewer resilience mechanisms than SoM plus multi-strategy retries.

### auto-playwright
- The agent relies on DOM context, significantly limiting its ability to navigate complex UIs.
- Reliance on DOM means the prompt sizes are unbounded.
- Verifications require you to parse the AI response manually and call `expect` yourself; `ai.verify` does this automatically.
- Project activity is minimal, raising maintenance concerns.

### Fully-Agentic Test Suites
- Vendor lock in. Fully agentic tests require proprietary runners and custom formats, which result in vendor lock-in.
- All steps are agentic, resulting in slow, costly, non-deterministic tests.
- `ai-wright` enables a hybrid approach: keep 90% of your test deterministic Playwright code, and inject AI only for the messy, nondeterministic UI flows / verifications.
- This balances speed and reliability while still unlocking AI flexibility where you _actually_ need it.

---
Start by installing the package, set either TestChimp or OpenAI credentials, and layer `ai.act`, `ai.verify`, or `ai.extract` onto the toughest parts of your Playwright suite. 

AI where it helps, plain Playwright everywhere else.

## How this helps in real testing scenarios

`ai-wright` shines where **selectors break** and **UI copy shifts** - chat widgets, canvas dashboards, CAPTCHA-gated flows, multi-step wizards, and AI-assisted surfaces. Use deterministic Playwright for stable paths; drop in `ai.act` / `ai.verify` only on the brittle steps. Pair with [@testchimp/playwright](https://www.npmjs.com/package/@testchimp/playwright) so AI steps report to TestChimp CI and count toward scenario traceability.

Start here: [Introduction to SmartTests](https://docs.testchimp.io/smart-tests/intro) · [TestChimp approach to test automation](https://docs.testchimp.io/qa-autopilot-claude/testchimps-approach-to-test-automation) · [Testing guides hub](https://docs.testchimp.io/guides/intro)

### AI & conversational UX (primary `ai-wright` use cases)

| Scenario | Testing guide |
|----------|----------------|
| Chatbots & multi-turn UI | [Conversational UI testing](https://docs.testchimp.io/guides/ai/testing-conversational-ui) |
| AI agent tool calling | [AI agent workflows](https://docs.testchimp.io/guides/ai/testing-ai-agent-workflows) |
| RAG / knowledge-base search | [RAG testing](https://docs.testchimp.io/guides/ai/testing-rag-search) |
| LLM output / JSON schema | [LLM output validation](https://docs.testchimp.io/guides/ai/testing-llm-output-validation) |
| Streaming token responses | [Streaming AI responses](https://docs.testchimp.io/guides/ai/testing-ai-streaming-responses) |
| When to use evals vs E2E | [AI testing guides hub](https://docs.testchimp.io/guides/ai/intro) · [Conversational UI](https://docs.testchimp.io/guides/ai/testing-conversational-ui) |
| AI-powered SaaS products | [AI web apps](https://docs.testchimp.io/guides/verticals/testing-ai-web-apps) |

### Visual & dynamic UI (vision-first `ai.verify` / `ai.act`)

| Scenario | Testing guide |
|----------|----------------|
| Canvas, charts, drag handles | [Canvas & visual interactions](https://docs.testchimp.io/guides/ai/testing-canvas-visual-interactions) |
| Google Maps & geo widgets | [Google Maps testing](https://docs.testchimp.io/guides/integrations/testing-google-maps) |
| Third-party iframes & embeds | [Third-party embeds](https://docs.testchimp.io/guides/integrations/testing-third-party-embeds) |
| Data grids & complex tables | [Data grids & tables](https://docs.testchimp.io/guides/patterns/testing-data-grids-tables) |
| Search with dynamic filters | [Search & filters](https://docs.testchimp.io/guides/patterns/testing-search-filters) |
| Form validation messages | [Form validation](https://docs.testchimp.io/guides/patterns/testing-form-validation) |

### Auth & flows where UI varies by provider

| Scenario | Testing guide |
|----------|----------------|
| CAPTCHA on signup/login | [CAPTCHA-enabled flows](https://docs.testchimp.io/guides/auth/testing-captcha-flows) |
| OAuth provider consent screens | [OAuth social login](https://docs.testchimp.io/guides/auth/testing-oauth-social-login) |
| MFA / TOTP flows | [MFA & 2FA](https://docs.testchimp.io/guides/auth/testing-mfa-2fa) |
| Firebase Auth edge cases | [Firebase authentication](https://docs.testchimp.io/guides/auth/testing-firebase-auth) |
| Magic links & passwordless | [Magic link testing](https://docs.testchimp.io/guides/auth/testing-magic-link-passwordless) |

### Complex business flows (hybrid Playwright + AI steps)

| Scenario | Testing guide |
|----------|----------------|
| HR / applicant pipelines | [HR applications](https://docs.testchimp.io/guides/verticals/testing-hr-applications) |
| Insurance quote wizards | [Insurance quotes](https://docs.testchimp.io/guides/verticals/testing-insurance-quotes) |
| Healthcare patient portals | [Healthcare portals](https://docs.testchimp.io/guides/verticals/testing-healthcare-portals) |
| SaaS onboarding & activation | [SaaS onboarding](https://docs.testchimp.io/guides/verticals/testing-saas-onboarding-flows) |
| E-commerce checkout | [Checkout flows](https://docs.testchimp.io/guides/verticals/testing-ecommerce-checkout-flows) |
| Legal e-signatures | [E-signature workflows](https://docs.testchimp.io/guides/verticals/testing-legal-esignatures) |
| Stripe payments | [Stripe in Playwright](https://docs.testchimp.io/guides/flows/testing-stripe-payments) |

### Close the loop with TestChimp

- Link hybrid tests to markdown plans - [Requirement traceability](https://docs.testchimp.io/test-planning/requirement-traceability)
- Run in CI with AI step reporting - [SmartTests in CI](https://docs.testchimp.io/smart-tests/run-in-ci-playwright) · [@testchimp/playwright](https://www.npmjs.com/package/@testchimp/playwright)
- Expand automation from TrueCoverage gaps - [/testchimp evolve](https://docs.testchimp.io/qa-autopilot-claude/evolve) · [TrueCoverage intro](https://docs.testchimp.io/truecoverage/intro)
- Capture manual paths first - [Chrome extension manual capture](https://docs.testchimp.io/chrome-extension/manual-test-capture)
- Hybrid vs record-replay - [Record-replay vs TestChimp](https://docs.testchimp.io/comparisons/record-replay-vs-testchimp)

More guides: [Integrations](https://docs.testchimp.io/guides/integrations/intro) · [UI patterns](https://docs.testchimp.io/guides/patterns/intro) · [Industry verticals](https://docs.testchimp.io/guides/verticals/intro) · [Fix flaky E2E](https://docs.testchimp.io/guides/verticals/testing-flaky-e2e-tests-startups)

## Author

`ai-wright` is an Open source project contributed by [TestChimp](https://testchimp.io) - an AI QA platform that learns your web app through explorations, to provide context-aware AI assistance to QA workflows.

## License

Distributed under the MIT License. See the [LICENSE](./LICENSE) file for full terms.