# five46

[![npm version](https://img.shields.io/npm/v/five46.svg)](https://www.npmjs.com/package/five46)
[![npm downloads](https://img.shields.io/npm/dm/five46.svg)](https://www.npmjs.com/package/five46)
[![CI](https://github.com/sekharsdet/five46/actions/workflows/ci.yml/badge.svg)](https://github.com/sekharsdet/five46/actions/workflows/ci.yml)
[![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)
[![node](https://img.shields.io/node/v/five46.svg)](https://www.npmjs.com/package/five46)

**Say what should work, in plain English. An AI agent checks it against your real app or API, using your own LLM key, and hands you a real Playwright test.**

![five46 driving a real browser through a login, add-to-cart, and checkout flow, then confirming the order was placed](./assets/demo.gif)

You just built something and want to know whether it actually works, before
writing a test. Give five46 a goal like *"log in and confirm the dashboard
loads"* or *"create a user via POST, then confirm it via GET"*. An LLM
(OpenAI, Anthropic, Gemini, Groq or AWS Bedrock, with your key) drives your
real app or API one action at a time. It tells you honestly whether the goal
was met, with a root-cause hypothesis when it wasn't. A successful run is
saved as a standalone Playwright (or `node:test`) spec, so today's check
becomes tomorrow's regression test, and running it needs no five46 or LLM.

## Quick start

```bash
npm install -g five46
npx playwright install chromium   # one-time browser download
five46 config                     # pick a provider and paste a key (Gemini's is free)
five46 doctor                     # confirms the key works
five46 test http://localhost:3000 --goal "log in as demo@example.com and confirm the dashboard says Welcome back"
```

You get a pass or fail (and a matching exit code), a report with
screenshots and, for failures, a root-cause hypothesis, plus a
`five46-agent-*.spec.ts` you can commit and run with `npx playwright test`.
Testing an API instead? Use `five46 api <base-url> --goal "..."`. It's
read-only unless you pass `--allow-writes`.

> **Status:** early and actively developed. Verified end-to-end against dozens of real-world sites and APIs.

## Check every pull request

The [five46 GitHub Action](https://github.com/sekharsdet/five46-action)
reads the **Acceptance criteria** section of a pull request's description
and checks each item against the PR's preview deployment. It keeps one PR
comment up to date with the verdict per criterion, why each failure failed,
and a ready-to-commit spec for every criterion that passed.

```yaml
on: deployment_status   # Vercel, Netlify and other preview deploys
jobs:
  five46:
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    permissions: { contents: read, pull-requests: write }
    steps:
      - uses: sekharsdet/five46-action@v1
        with:
          url: ${{ github.event.deployment_status.target_url }}
          api-key: ${{ secrets.FIVE46_API_KEY }}
```

## How it compares

| | five46 | Playwright Test Agents | `playwright codegen` | Cloud AI testing platforms |
|---|---|---|---|---|
| You provide | A goal or acceptance criteria in plain English | A seed test and a request, inside an AI coding tool | Your own clicks | Test descriptions in their app |
| Runs | CLI, CI, GitHub Action or MCP tool | Inside an AI coding tool, such as VS Code | Locally, while you click | Their cloud |
| Built to give a pass/fail verdict on the app | Yes, with a root-cause hypothesis | Not its focus: it generates and repairs tests | No | Yes |
| API testing | Yes (`node:test` output) | Browser-focused | No | Varies |
| Proves assertions catch failures | Yes (`--verify` mutation check) | No built-in check | No | Varies |
| LLM and cost | Your key, any of 5 providers | Your coding agent's model | None | Their subscription |
| Your app data goes to | Only your LLM provider | Your coding agent's provider | Nowhere | Their platform |
| Output | Plain Playwright or `node:test` files | Plain Playwright files | Plain Playwright files | Usually their runner |

Already working in an AI coding tool and growing a Playwright suite there?
Playwright's own agents are a great fit, especially the healer for keeping
existing tests green. five46 is for getting a verdict: is this feature
working right now, in CI or on a PR, for a browser flow or an API, with any
LLM you already pay for. More detail is in [Why five46](docs/why-five46.md).

## Highlights

- **Safe by default.** API runs are read-only unless you unlock writes;
  destructive-looking clicks ("Delete account") are blocked unless allowed.
- **Login credentials never reach the LLM.** It sees placeholders, and the
  real values are filled in locally. Log in once, then reuse the session.
- **Acceptance criteria in, verdicts out.** `--story` splits a user story
  or ticket into independent scenarios and reports pass/fail per criterion.
- **Specs you can trust.** `--verify-clean-session` re-runs the spec cold;
  `--verify` mutates its assertions and requires them to fail.
- **Works with any current key.** If your provider retires a model, five46
  finds a current one your key can use. You can also pin one with
  `FIVE46_LLM_MODEL`.
- **Fits your tools.** CI-friendly exit codes, `--report` JSON for
  dashboards, flaky-run detection (`--repeat`), video recording, and an
  MCP server for IDE assistants.

## Docs

- [Configuration](docs/configuration.md): providers, getting a key, `five46 doctor`, choosing models
- [CLI reference](docs/cli-reference.md): every command and flag, login sessions, API testing, story mode, verification, reports
- [MCP server](docs/mcp.md): use five46 from an IDE-embedded AI assistant
- [Security model](docs/security.md): what leaves your machine, what five46 guards, and what's your call
- [Why five46](docs/why-five46.md): design choices, the full feature list, comparisons
- [CHANGELOG](CHANGELOG.md)

## Known limitations

- **No *closed* shadow DOM traversal.** Elements inside a `mode: "closed"`
  shadow root aren't visible to the agent's page snapshot (a closed shadow
  root is deliberately inaccessible to external tooling in general, not
  just five46). *Open* shadow roots, the overwhelming majority of
  real-world web components, work fine, as do `<iframe>` and `<frame>`
  content.
- **Chromium only** for browser mode, with no Firefox or WebKit yet.
- **A run itself is not deterministic.** The same goal against the same page
  can take a different path next time. The generated spec is the frozen,
  repeatable artifact.

## Contributing

Issues and pull requests are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md).

```bash
git clone https://github.com/sekharsdet/five46.git
cd five46
npm install
npm run build
node dist/cli.js test <url> --goal "..."
npm test
```

If five46 is useful to you, a ⭐ on [GitHub](https://github.com/sekharsdet/five46) helps other people find it.

## License

[MIT](./LICENSE)