# flaker New Project Onboarding Checklist [日本語版](new-project-checklist.ja.md) The checklist for introducing flaker to a new repository and getting value from it across the first day, first week, and first month. Assumes `0.7.0+`. Day 1 converges in five steps: `init -> doctor -> plan -> apply -> status`. If you follow it in order, initial setup takes about 30 minutes, your measurement baseline is in place within a week, and the repository is ready to promote CI gating in 2-4 weeks. --- ## Day 0: Preconditions (5 minutes) ```bash node --version # >= 24 pnpm --version # >= 10 git remote -v # origin should point at GitHub gh auth status # logged in (needed for flaker apply when pulling CI history) ``` Repositories with no GitHub Actions history are still fine. `flaker apply` detects an empty history and picks the cold-start path (self-seed from a local run). Real CI history accumulates naturally after Day 1. You do not need `moon` (MoonBit). flaker ships a bundled `dist/moonbit/flaker.js`, and falls back to TypeScript (`src/cli/core/loader.ts`) when needed. --- ## Day 1: Install through convergence (15 minutes) Day 1 is a single five-step flow. `flaker apply` handles collect / calibrate / quarantine sequencing internally, so you do not have to memorize the order. ### 1. Install ```bash pnpm add -D @mizchi/flaker ``` ### 2. Generate `flaker.toml` Choose the adapter and runner at init time: ```bash # vitest project pnpm flaker init --adapter vitest --runner vitest # playwright e2e pnpm flaker init --adapter playwright --runner playwright # jest pnpm flaker init --adapter jest --runner jest # actrun wrapping a GitHub Actions workflow for playwright pnpm flaker init --adapter playwright --runner actrun ``` `owner` and `name` are auto-detected from the git remote. Override with `--owner` / `--name` if needed. `flaker init` also writes default `[gate.release]` / `[gate.merge]` / `[gate.iteration]` blocks so that gates resolve immediately. ### 3. Check the environment with doctor ```bash pnpm flaker doctor ``` Expected output looks like: ```text OK config flaker.toml is readable OK config ranges all values within expected ranges OK duckdb DuckDB initialized successfully OK moonbit MoonBit JS build detected (or fallback) Doctor checks passed. ``` If DuckDB fails to initialize, the most likely cause is `node --version < 24`. ### 4. Configure the affected resolver To make `flaker run --gate iteration` and the `hybrid` strategy useful, you need an affected resolver. Edit `[affected]` in `flaker.toml` based on the repository shape: ```toml # pnpm workspaces / npm workspaces monorepo [affected] resolver = "workspace" config = "" # glob rules with a separate flaker.affected.toml [affected] resolver = "glob" config = "flaker.affected.toml" # bitflow repository [affected] resolver = "bitflow" config = "" ``` If you do not configure a resolver, `hybrid` still works through `weighted` fallback, but you lose the best change-aware behavior. Start with `workspace` if possible. ### 5. Preview with `flaker plan` ```bash export GITHUB_TOKEN=$(gh auth token) pnpm flaker plan ``` `flaker plan` reads `flaker.toml` as desired state and reports what the DB is currently missing. An empty-history repository usually plans `collect_ci` + `cold_start_run`; if `[quarantine].auto = true`, a `quarantine_apply` action shows up too. ### 6. Converge with `flaker apply` ```bash pnpm flaker apply ``` `flaker apply` is idempotent and runs `collect` / `calibrate` / `quarantine apply` on demand based on current state. It is safe to re-run from cron or nightly — repeated executions do not break anything. ### 7. Inspect with `flaker status` ```bash pnpm flaker status ``` A one-screen summary dashboard. On Day 1 you usually see `data confidence: insufficient`, which is expected. After a week of `flaker apply`, it lifts to `moderate` naturally. --- ## Day 2-3: Keep applying Because `flaker apply` is idempotent, "run it once a day" is the whole Day 2 story. You do not have to call `collect` or `calibrate` manually. ```bash export GITHUB_TOKEN=$(gh auth token) pnpm flaker apply pnpm flaker status # daily dashboard pnpm flaker status --detail # KPI view (formerly analyze kpi) ```
Drilling down into the individual steps `flaker apply` internally invokes the commands below on demand. They remain callable directly if you want to inspect one stage in isolation, but `flaker apply` is canonical. **Collect CI history:** ```bash export GITHUB_TOKEN=$(gh auth token) pnpm flaker import --ci --days 30 ``` **Calibrate:** ```bash pnpm flaker calibrate ``` This writes the recommended strategy and sampling percentage into `[sampling]` in `flaker.toml`. Add `--dry-run` if you want to preview without writing. If data is still thin (`commits < 20`), you may get `confidence: insufficient` or `low`. That is acceptable at this stage; `flaker apply` will recalibrate itself as history grows over the next week.
--- ## Day 3: Add package.json scripts Apply-first script layout for 0.7.0+: ```jsonc { "scripts": { "flaker": "flaker", "flaker:plan": "flaker plan", "flaker:apply": "flaker apply", "flaker:status": "flaker status", "flaker:run:iteration": "flaker run --gate iteration", "flaker:run:release": "flaker run --gate release", "flaker:eval": "flaker status --markdown", "flaker:doctor": "flaker doctor" } } ``` `pnpm flaker:run:iteration` works well from a pre-push hook via lefthook or husky. `pnpm flaker:apply` is a good target for a daily cron / launchd job. --- ## Day 5: Integrate with GitHub Actions (advisory mode) ### 1. PR advisory job Add this to `.github/workflows/ci.yml`: ```yaml - name: Setup Node uses: actions/setup-node@v4 with: node-version: 24 - name: Setup pnpm uses: pnpm/action-setup@v4 - name: Install run: pnpm install --frozen-lockfile - name: Run tests via flaker (advisory) run: pnpm flaker run --gate merge continue-on-error: true env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - name: Post status as PR comment if: github.event_name == 'pull_request' run: | pnpm flaker status --markdown > .artifacts/status.md pnpm flaker report report.json --summary --adapter vitest --pr-comment \ | gh pr comment ${{ github.event.pull_request.number }} --body-file - env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} ``` The key point is `continue-on-error: true`. For the first 2-4 weeks, do not make this a required check. ### 2. Nightly history job Create `.github/workflows/nightly-flaker.yml`: ```yaml name: nightly flaker on: schedule: [{ cron: "0 18 * * *" }] workflow_dispatch: jobs: apply: runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 - uses: pnpm/action-setup@v4 - uses: actions/setup-node@v4 with: { node-version: 24 } - run: pnpm install --frozen-lockfile - run: pnpm flaker apply env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: pnpm flaker status --markdown > .artifacts/flaker-status.md - uses: actions/upload-artifact@v6 with: name: flaker-nightly path: .artifacts/ ``` Every night, `flaker apply` converges state and `flaker status --markdown` emits a weekly review artifact. --- ## Week 1: Observe and tune Spend five minutes each morning on: ```bash pnpm flaker status # one-screen summary pnpm flaker status --list flaky # top flaky tests pnpm flaker explain insights # AI commentary on CI vs local drift ``` When something looks suspicious: ```bash # classify a single test as broken / flaky / transient pnpm flaker debug confirm "tests/api.test.ts:handles timeout" --runner local --repeat 10 # retry a failed CI run locally pnpm flaker debug retry --run # find the commit range where a test became flaky pnpm flaker debug bisect --test "tests/api.test.ts:handles timeout" ``` --- ## Week 2-4: When to promote to required Switch the merge gate from advisory to required when `pnpm flaker status` shows `ready` in its drift section. If you want the detailed actual values, use `pnpm flaker status --gate merge --detail`. The rough targets: | Metric | Target | |---|---| | Matched commits | >= 20 | | Recall (CI failures caught) | >= 90% | | False negative rate | <= 5% | | Pass correlation | >= 95% | | Holdout FNR (if enabled) | <= 10% | | Co-failure data | `ready` | | Data confidence | `moderate` or `high` | At that point, remove `continue-on-error: true` from the CI job. ### Re-run calibration as data grows As long as you run `flaker apply` regularly, calibration re-runs itself when the data warrants it. If you want to confirm explicitly: ```bash pnpm flaker apply git diff flaker.toml ``` --- ## Troubleshooting | Symptom | Cause and fix | |---|---| | `flaker.toml uses deprecated keys` | Config from 0.1.x or earlier. Use the rename table in `docs/how-to-use.md#config-migration`. | | `Config file not found` | You are not at the project root. `cd` there and start with `pnpm flaker init`. | | `actrun runner requires [runner.actrun] workflow` | Add `[runner.actrun]` to `flaker.toml`. | | `hybrid` selects 0 tests | Resolver not configured. Fill in `[affected].resolver`. | | `flaker apply` returns 0 runs | Missing or under-scoped `GITHUB_TOKEN`, often without `actions:read`. | | `flaker status` shows `data confidence: insufficient` | Fewer than 5 commits of history. Keep running `flaker apply`; usually resolves within a week. | | Parallel tests time out | DuckDB is single-writer. Serialize processes using the same `.flaker/data.duckdb`. | | `dist/moonbit/flaker.js` is missing | It should already be bundled by the npm package. If not, inspect the package build. | --- ## The ideal shape after one month - `flaker run --gate merge` is a required PR check - nightly `flaker apply` keeps history fresh every day - weekly reports (`flaker status --markdown`) are posted to Slack or issues - developers use `pnpm flaker:run:iteration` locally - flaky tests are auto-quarantined via `[quarantine].auto = true` + `flaker apply` At that point, many repositories can cut CI time by 30-70% while keeping missed failures under 5%. --- ## References - [README.md](../README.md) — project overview - [usage-guide.md](usage-guide.md) — user-facing entrypoint - [operations-guide.md](operations-guide.md) — operator-facing entrypoint - [how-to-use.md](how-to-use.md) — detailed commands and configuration - [migration-0.6-to-0.7.md](migration-0.6-to-0.7.md) — upgrading from 0.6.x - [migration-0.12-to-0.13.md](migration-0.12-to-0.13.md) — upgrading from 0.12.x - [contributing.md](contributing.md) — development and dogfood workflow - [CHANGELOG.md](../CHANGELOG.md) — release history and breaking changes