### The verification layer for the agentic coding era. AI ships code in minutes — verifying it hasn't. `testsprite` opens your live app, uses it like a real user, and shows your coding agent exactly what broke — so it fixes its own work before a bug ever reaches you. _{Proof, in public: verification beats model size. With this CLI in the loop, the cheapest model in the field shipped the most correct app on an open leaderboard — 89%, at half the cost of the priciest one. See the leaderboard →}

⭐ _Help us reach more developers and grow the TestSprite community. Star this repo!_

_{▶ Watch the launch video — the three hard limits every coding agent hits, and the loop that breaks them (4 min).}

--- ## What is it? [TestSprite](https://www.testsprite.com) is the AI testing platform 100,000+ teams use to test their software, frontend and backend — in the cloud, against the live product, not mocks. This repo is its official CLI. It puts that platform in your coding agent's hands: the agent verifies every behavior it ships, and what broke comes back as **one self-consistent bundle** it can act on — no dashboard scraping. Humans drive the same surface from a terminal or CI. ## ⭐️ Star the Repository testsprite-repo-star-fast

If you find `testsprite` useful, a GitHub Star ⭐️ would be greatly appreciated — it helps other builders (and their agents) find the project, and stars notify you about new releases. ## Quickstart Requires **Node.js 20.19+**, **22.13+**, or **24+**. (No global install? `npx @testsprite/testsprite-cli` works too.) ```bash npm install -g @testsprite/testsprite-cli testsprite setup ``` `testsprite setup` prompts for your [API key](https://www.testsprite.com/dashboard/settings/apikey), verifies it, and installs the verification-loop skill for the coding agents this project uses — one command, so your agent is wired to verify its own work. **Which agent it installs for.** Each of the eight supported targets — `claude`, `cursor`, `cline`, `windsurf`, `antigravity`, `kiro`, `copilot`, `codex` — lands its skill in a different place, so `setup` works out which you use: the agent calling it, if it identifies itself, together with every agent whose config is already in the project. It installs for all of them and prints what it found — in a terminal it shows you that list first, so you can accept it or narrow it before anything is written. If nothing identifies an agent, it installs for `claude` and says so — that is the one case where you want `--agent ` to name one yourself. Check the result any time with `testsprite agent status`. Non-interactive (CI / onboarding scripts): ```bash TESTSPRITE_API_KEY=sk-... testsprite setup --from-env --yes ``` > **Pointing a coding agent (Claude Code, Cursor, Codex, Cline, …) at TestSprite?** Have it run `testsprite setup` first — that installs the verification skill, so the agent knows how to create, run, and triage tests on its own (instead of guessing from this README). `setup` reports which agents it installed for; add `--agent ` if that list is not the one you wanted. New here? Start with the **[getting-started overview](https://docs.testsprite.com/cli/getting-started/overview)**. > **Privacy note:** interactive runs check the npm registry at most once per 24 h to offer a "new version available" notice — package name only, never your key or data; `TESTSPRITE_NO_UPDATE_NOTIFIER=1` disables it. The backend also advertises its minimum supported CLI version — a below-floor CLI prints a one-line upgrade advisory on stderr, and a too-old client may be rejected with exit 14 (`CLIENT_TOO_OLD`). Details in [DOCUMENTATION.md → Update notice](./DOCUMENTATION.md#update-notice). From there, the loop runs on its own — an example session, typed by the coding agent: ```bash # 1 — describe the behavior you want to guarantee, run it, wait testsprite test create --project proj_8f0f6 --type frontend \ --plan-from ./checkout-flow.plan.json --run --wait --output json # → exits 1: the run failed # 2 — pull ONE self-consistent failure bundle testsprite test failure get test_3a9f21c7 --out ./.testsprite/failure # 3 — the agent reads the bundle, fixes the code, then replays testsprite test rerun test_3a9f21c7 --wait --output json # → exits 0: passed. The test now lives in your durable suite. ``` `./checkout-flow.plan.json` (the `--plan-from` argument above) is a JSON file describing the test in plain language — no browser code required. This is the exact, byte-identical output of `testsprite test create --plan-template` (also embedded verbatim in `test create --help`): ```json { "$schema": "https://raw.githubusercontent.com/TestSprite/testsprite-cli/v0.4.0/schemas/plan.schema.json", "projectId": "prj_abc123", "type": "frontend", "name": "Login rejects an empty password", "planSteps": [ { "type": "action", "description": "Navigate to /login and submit the form with an empty password" }, { "type": "assertion", "description": "Verify an inline error says the password is required" } ] } ``` Get this exact skeleton without hand-copying it (and without the risk of it drifting from your installed version — see below): `testsprite test create --plan-template`. Full field reference (including the `{{...}}`-placeholder caveat, size caps, and the `$schema` hook for live editor validation): [Plan file format](./DOCUMENTATION.md#plan-file-format). > The `$schema` URL above is pinned to the CLI version that generated this doc (`v0.4.0`) — `--plan-template`'s live output always pins to **your installed version** instead, which is what actually resolves. If you're reading this on a later release, run the command yourself rather than trusting this snippet verbatim. **Don't want to write the first tests yourself?** For a deployed target, ask TestSprite to propose them. Local projects use the authored-plan flow below. `testsprite test plan generate --project ` explores your app (or reads the API spec you uploaded with `project docs upload`), drafts test cases, and stages them for review — nothing is written to your disk. You read the printed table and keep what you want: ```bash testsprite test plan generate --project proj_8f0f6 # stages proposals for review testsprite test plan accept --project proj_8f0f6 # → real tests testsprite test plan accept --project proj_8f0f6 --only prop_2 prop_5 # or just these ``` It only runs what the project is still missing, so a second call picks up where the first left off. Generation spends workspace credits per stage that actually runs (the result line reports the spend), and a PRD uploaded with `project docs upload --role prd` bills 0.5 credits for its embedding — an API spec is free. Full walkthrough, including what happens on Ctrl-C and every precondition error: [Generate commands](./DOCUMENTATION.md#generate-commands). Prefer to configure each step by hand (or learn the surface offline with `--dry-run` first)? See [Manual setup](./DOCUMENTATION.md#manual-setup) and [Install & verify](./DOCUMENTATION.md#install--verify). ## Testing an environment on your machine Frontend tests can reach your running app through a TestSprite tunnel, including on the Free plan (ordinary run credits apply). Use an API key with `run:tunnel`; the CLI names the missing scope if the key cannot open a tunnel. Existing keys receive this scope on backend v0.18.0 or later unless deliberately narrowed. ```bash testsprite project create --type frontend --name "My local app" --local 3000 testsprite test create --plan-from ./checkout.plan.json --project testsprite test run # the saved loopback URL opens a tunnel and waits testsprite test run --max-concurrency 5 testsprite test run --all --project # Or keep one tunnel open in terminal A (no port argument) testsprite tunnel start --ttl 3600 # Terminal B: borrow the clientId printed above; keep terminal A running testsprite test run --tunnel-client testsprite tunnel list testsprite tunnel stop # Stop every live tunnel on this account only when you intend to revoke them all testsprite tunnel stop --all --confirm ``` Run several local frontend tests in one invocation to share one tunnel binding. The batch runs up to 5 tests at once by default; `--max-concurrency` accepts 1–10. `--all --local` skips backend tests. An owned batch cancels unfinished runs on Ctrl-C or tunnel loss; JSON output lists every requested test in input order, including tests that did not start. The minted client id is printed before the tunnel connects. Hit the tunnel limit? Run `testsprite tunnel list`, then `testsprite tunnel stop ` for an unused binding, or `testsprite tunnel stop --all --confirm` to revoke them all. An older server without listing returns exit 4; a key without `run:tunnel` returns exit 3. A second `tunnel start` or process using the same credential takes over, and the first exits **10**. An environment is one origin (scheme, host and port) plus its sign-in configuration. Frontend runs automatically open a tunnel for a saved `localhost`, `127.0.0.1` or `[::1]` URL: `test run`, `test rerun`, `test create --run`, `test create-batch --run` and `testlist run` use this behavior. `--env ` selects the environment; omit it for the project default. `--no-wait` disables automatic tunnels; drop it to run through one. If environment reads are unavailable, the command keeps the previous server behavior. Several fresh-run IDs require a tunnel; public targets retain the exit-5 refusal. A named public environment used against a local origin prints a sign-in advisory on stderr. Reruns and test-list runs share one timeout including capacity waits and require the server to confirm tunnel support; an older server triggers a cancellation attempt and a clear error. `--local ` saves new project and environment URLs as `http://localhost:`. Use `--url http://127.0.0.1:` or `--url http://[::1]:` when the host matters (`--local-host` is deprecated). Dev servers can be slow through a tunnel, so for heavy apps test a production build (`npm run build` plus your framework's preview or start command). The tunnel uses verified TLS. On a network that re-signs TLS, set `NODE_EXTRA_CA_CERTS=/path/to/ca.pem` to your organisation's root CA. If the tunnel keeps failing to connect for 60 seconds, the command exits **10**, and an owned run is cancelled and refunded. On an owned tunnel, Ctrl-C or a timeout cancels unfinished runs by default, and a run cancelled before it finishes is refunded. A run on a borrowed `--tunnel-client` detaches on Ctrl-C instead. Local projects skip plan generation, so write plans with `test create --plan-from`. Portal runs stay blocked, at no charge, until the project has a public URL. See [the local-testing reference](./DOCUMENTATION.md#local-frontend-testing-and-tunnels) for the details. ## Commands | Group | Command | What it does | | -------------- | --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Setup** | `setup` | **Start here** — one command: configure your API key, verify it, and install the agent verification skill | | | `doctor` | Environment diagnostic — CLI/Node versions, profile, endpoint, credentials, connectivity, agent skill; exits non-zero on failure | | **Auth** | `auth status` | Resolve the active profile to its user, key, env, and scopes | | | `auth remove` | Remove the active profile from the credentials file | | | `usage` (alias `credits`) | Account pre-flight: identity, plus credit balance / plan info when the backend supplies them | | **Read** | `project list` / `project get` | List projects / fetch one by id, including the default environment and environment count when the server reports them | | | `test list` / `test get` | List tests under a project / fetch one by id; `test list --env ` selects frontend status and the ENV column names its source | | | `test code get` | Print (or write) the generated test source | | | `test steps` | List the latest run's steps with screenshot / DOM pointers | | | `test result` | Latest result; `--history` lists a test's prior runs (ENV column; `--env ` filters by environment) | | | `test failure get` | The agent entry point: one self-contained latest-failure bundle | | | `test failure summary` | One-screen triage card (no media download) | | | `test diff` | Compare two runs — verdict, failure kind, per-step status flips, code-version drift | | **Write** | `test scaffold` / `test lint` | Author plans locally: emit a schema-correct starter, validate plan files offline — no network, no credentials | | | `test create` / `test create-batch` | Create a test (or bulk-create from a plan file); `--step-timeout ` sets a per-test timeout; dependency flags wire BE metadata | | | `test update` / `test delete` / `test delete-batch` | Edit metadata, set/clear the step timeout (`--step-timeout` / `--clear-step-timeout`), or permanently delete (`--confirm` required) | | | `test code put` | Replace generated code (etag-guarded) | | | `test plan put` | Replace a frontend test's plan-steps | | | `project create` / `project update` / `project delete` | Manage projects; `--test-id-attributes ` registers the app's own test-hook attributes (priority order) for exported locators, `--clear-test-id-attributes` reverts; `delete` removes a project and everything under it (`--confirm` required, no restore window) | | | `project credential` / `project auto-auth` | Configure backend-test auth: a static injected credential, or auto-refresh login (Pro) | | | `project env list` / `get` / `create` / `update` / `delete` / `set-default` | Manage a project's named URL + login bundles. `get` shows safe login details and variables; `create` / `update` accept repeatable `--var KEY=VALUE` (plain text — not for secrets, use `--username`/`--password`), and `update --clear-credentials` disables account login. Use `--url http://localhost:` or `--local ` for an app on this machine; runs open its tunnel automatically. `delete` refuses the default environment. | | | `project sign-in get` / `set` | Inspect or change an environment's app sign-in mode; choose OTP with `project env create --sign-in otp`. SSO selection requires the per-account manual-login feature flag; capture the session in the Portal. | | **Generate** | `test plan generate` | Ask TestSprite to propose test cases for a project — proposals are staged server-side for review, never written to disk; only the stages the project is missing actually run | | | `test plan accept` | Turn staged proposals into real tests — all of them, or a subset with `--only` | | | `project docs upload` | Upload an API spec or PRD as a project source, so generation has something to read | | **Run** | `test run` | Trigger a fresh run; saved loopback environments open a tunnel and wait automatically; `--wait` blocks until terminal; `--all --project ` runs all tests in a project in wave order `--env ` runs against a named environment. | | | `test rerun` | Replay one/many tests (FE verbatim; BE with deps); billed as a run (0.5 credits FE / 0.2 BE); `--all --project ` reruns all tests `--env ` replays against a named environment; loopback URLs open a tunnel automatically when the server supports it. | | | `test flaky` | Replay a test several times (auto-heal off) and report a stability score | | | `test wait` | Block on one or more `runId`s until terminal | | | `test cancel` | Cancel one or more in-flight runs (Ctrl-C during `--wait` only detaches, `cancel` is the real stop — except a run with an owned tunnel, which Ctrl-C cancels by default) | | | `test artifact get` | Download the failure bundle for a specific `runId` | | **Test lists** | `testlist list` / `testlist get` | Inspect saved test lists — named, cross-project collections with per-project environment pins (V3-only) | | | `testlist create` / `update` / `add` / `remove` / `delete` | Manage a list and its cases; `--project-env :` pins environments; `delete` requires `--confirm` | | | `testlist run` | Run every case (or a `--case` subset) through its pinned environment; loopback environments open a tunnel automatically; `--wait` polls all runs, `--report junit` writes a CI sidecar | | **Schedule** | `schedule list` / `schedule get` | List schedules visible to the key / fetch one by id, including a project schedule's pinned or inherited environment (status is `ENABLED`, `PAUSED`, or `AUTO_PAUSED` after repeated failures) | | | `schedule create` | Run a project or test list on a 5-field cron; `--env ` pins a project schedule, while omission inherits the live default; states frequency and estimated cost | | | `schedule update` | Retime or rename a schedule, change recipients, pin a project schedule with `--env `, or `--pause` / `--resume` it | | | `schedule delete` | Remove a schedule and its run history, stopping future triggers (`--confirm` required, no restore window) | | | `schedule run list` | A schedule's past runs, with per-run pass / fail / blocked counts | | **CI** | `ci init github` | Write a GitHub Actions PR gate that runs `ci run` against each pull request's preview deployment — one placeholder step to fill in: your platform deploys every PR (`on: deployment_status`) or the workflow deploys | | | `ci run` | The CI gate in one command: every project test (or `--filter`) against `--target-url`, signed in as `--env`, in a temporary `github-pr-` environment, one execution; waits and exits with the verdict. Job summary = the execution report (the GitHub App comment); `--summary-file`, `--report junit`, step outputs | | **Agent** | `agent install` / `agent list` / `agent status` | Add, list, or health-check coding-agent skills (pure-local): `claude`, `codex`, `cursor`, `cline`, `antigravity`, `kiro`, `windsurf`, `copilot` | > The earlier command names — `init`, `auth configure`, `auth whoami`, `auth logout` — still work as hidden, deprecated aliases (each prints a one-line notice pointing at the new name), so existing scripts keep running. `auth configure` now runs the full `setup` (it also installs the skill). 📚 **Full reference — every command, flag, and example:** [DOCUMENTATION.md](./DOCUMENTATION.md), including [configuration & profiles](./DOCUMENTATION.md#configuration), [scripting](./DOCUMENTATION.md#output--scripting), and [exit codes](./DOCUMENTATION.md#exit-codes). ## Why a CLI for coding agents? - 🧪 **Tests like a real user.** Runs against a live browser or API in the cloud — real clicks, real navigation, real assertions. Not a mock. - 🤖 **Agent-shaped output.** `test failure get` returns **one bundle** — the failing step, its neighbors, screenshots, DOM snapshots, the test source, a root-cause hypothesis, and a recommended fix target — all sharing a single `snapshotId`. The CLI _refuses_ to stitch data from two different runs, so an agent never reasons over a frankenstein context. - ♻️ **A loop, not a one-shot.** `create → run → failure get → fix → rerun` — every pass is banked, not thrown away. - 📐 **Scriptable & deterministic.** Stable `--output json` contract, predictable [exit codes](./DOCUMENTATION.md#exit-codes), and a `--dry-run` that exercises the full code path offline with canned data. - 🚦 **CI-native.** `testsprite ci init github` writes a PR gate; `testsprite ci run --env --target-url "$PREVIEW_URL"` tests each pull request's preview deployment as one execution and exits with the verdict. The job summary is the execution report — the same content as the GitHub App's PR comment — plus `::error::` annotations on the checks tab, step outputs, `--summary-file` for a machine summary and `--report junit` for a JUnit sidecar. In any other CI, `testsprite ci run --project --env --target-url ` still runs the suite as one execution and makes the exit code the verdict — the pull-request context, draft skip and job summary are GitHub Actions only. [Details →](./DOCUMENTATION.md#continuous-integration) - 🔌 **One command to onboard your agent.** `testsprite setup` drops ready-made skill files into your repo — for the agents it detects you using — so your coding agent knows how to drive the loop on its own. ## How it works Every time your agent changes code, it asks one question: **is this behavior already covered by the suite?** - **Not yet covered** → `testsprite test create` — describe the new behavior, run it. - **Already covered** → `testsprite test rerun` — replay the existing tests, so nothing that used to work breaks silently. - **Something fails** → `testsprite test failure get` — one self-consistent bundle; the agent fixes the code and reruns. Every pass is banked into a durable suite, so coverage compounds as the project grows — a lasting record of every requirement it has ever gotten right, far bigger than any context window. ```mermaid flowchart TD A["🤖 Your coding agent
Claude Code · Codex · Antigravity · Kimi · Cursor · Trae …"] D{"behavior already
covered by the suite?"} B["testsprite test create
new behavior → new test"] R["testsprite test rerun
replay the existing tests"] C{{"☁️ TestSprite testing agent
runs the test like a real user against
real browsers & real APIs on Cloud"}} F["testsprite test failure get
ONE self-consistent bundle:
failing step · screenshots · DOM ·
root-cause · recommended fix"] S[("📚 Durable integration suite
grows with every pass")] A -->|"writes / changes code"| D D -->|"no — new behavior"| B D -->|"yes"| R B --> C R --> C C -->|"pass ✅"| S C -->|"fail ❌"| F F -->|"agent reads the bundle
& fixes the code"| A S -.->|"defines what's covered"| D ``` The cloud is a black box on purpose: your agent describes intent and reads results. It never has to know _how_ the test was driven — only _what_ a real user experienced. ## Proved in public On [**CoderCup**](https://codercup.ai) — an open leaderboard where frontier coding agents build the _same_ app under the _same_ rules, with TestSprite as the referee — the **cheapest** model in the field shipped the **most correct** app on the board: **89%**, at half the cost of the priciest one. That's the point of all of this: you no longer need the biggest, most expensive model to ship software you can trust — top-tier quality, without paying top-tier prices, within reach of every team. ## Getting help - 📚 **CLI reference** — [DOCUMENTATION.md](./DOCUMENTATION.md) - 🌐 **Platform docs** — [testsprite.com/docs](https://www.testsprite.com/docs) - 🐛 **Issues & feature requests** — [GitHub issues](https://github.com/TestSprite/testsprite-cli/issues) - 💬 **Quick questions** — [Discord](https://discord.gg/GXWFjCe4an), or `testsprite --help` / `testsprite test run --help` right in your terminal - 📝 **Changelog** — [CHANGELOG.md](./CHANGELOG.md) ## Contributing Contributions are welcome — the CLI is plain TypeScript/Node (20.19+, 22.13+, or 24+), tested with Vitest, built with `tsc`. Getting a dev loop running takes a minute: ```bash git clone https://github.com/TestSprite/testsprite-cli.git cd testsprite-cli && npm install npm run build # tsc → dist/ ; try it: node dist/index.js --help npm test # unit suite — no network or credentials needed npm run lint:fix # ESLint npm run typecheck # tsc --noEmit ``` Pull requests target the `main` branch. The full guide — build from source, test tiers, CI checks, branches, project layout — is in [CONTRIBUTING.md](./CONTRIBUTING.md). **Support** — if you need any help, we're responsive on [Discord](https://discord.gg/GXWFjCe4an), and feel free to email us at [contact@testsprite.com](mailto:contact@testsprite.com) too. ## License [Apache-2.0](./LICENSE) © TestSprite