# Development Workflows

## Setup Commands

```bash
# Install dependencies and setup hooks
yarn setup

# Start full development stack (auto-assigns unique ports per worktree)
yarn dev        # or equivalently: make dev
```

## Key Development Scripts

- `yarn dev` / `make dev`: Start full dev stack with worktree-isolated ports. A
  dev portal at http://localhost:9900 auto-starts showing all running stacks.
- `yarn dev:down` / `make dev-down`: Stop the dev stack for the current worktree
- `make dev-portal`: Start the dev portal manually (auto-started by `yarn dev`)
- `yarn lint`: Run linting across all packages
- `yarn dev:unit`: Run unit tests in watch mode (per package)
- `yarn dev:mutation --mutate <file>`: Run mutation tests (common-utils only). A
  surviving mutant means the tests execute that code but don't assert on it.
  Always scope it with `--mutate`; a whole-package run takes tens of minutes.
  See [CONTRIBUTING.md](../CONTRIBUTING.md#mutation-tests)
- `yarn seed:alerts --count N` (in `packages/api`): Fill this worktree's
  dev-slot Mongo with N alerts for testing the alerts page at scale: half on
  saved searches, half on dashboard tiles. Seeded dashboards and saved searches
  are tagged `seeded`; `yarn seed:alerts --purge` removes them.

## Environment Configuration

- `.env.development`: Development environment variables
- Docker Compose manages ClickHouse, MongoDB, OTel Collector
- Hot reload enabled for all services in development

## Worktree Isolation (Multi-Agent / Multi-Developer)

When multiple git worktrees need to run the dev stack simultaneously (e.g.
multiple agents or developers working in parallel), use `make dev` instead of
`yarn dev`. This automatically assigns unique ports per worktree.

### How It Works

1. `scripts/slots.sh` computes a deterministic slot (0-99) from the worktree
   directory name (via `cksum`), or from its full path with `HDX_SLOT_FROM=path`
   (see below). It is the only place slots and ports are computed: `yarn dev`,
   `make dev-int`, `make dev-e2e`, and tools all use it.
2. Each service gets a unique port: `base + slot` (see table below)
3. Docker Compose runs with a unique project name (`hdx-dev-<slot>`)
4. Volume paths include the slot to prevent data corruption between worktrees

### Dev Port Mapping (base + slot)

Ports are allocated in the 30100-31199 range to avoid conflicts with CI
integration tests (14320-40098) and E2E tests (20320-21399).

| Service           | Base Port | Range         | Env Variable                  |
| ----------------- | --------- | ------------- | ----------------------------- |
| API server        | 30100     | 30100 - 30199 | `HYPERDX_API_PORT`            |
| App (Next.js)     | 30200     | 30200 - 30299 | `HYPERDX_APP_PORT`            |
| OpAMP             | 30300     | 30300 - 30399 | `HYPERDX_OPAMP_PORT`          |
| MongoDB           | 30400     | 30400 - 30499 | `HDX_DEV_MONGO_PORT`          |
| ClickHouse HTTP   | 30500     | 30500 - 30599 | `HDX_DEV_CH_HTTP_PORT`        |
| ClickHouse Native | 30600     | 30600 - 30699 | `HDX_DEV_CH_NATIVE_PORT`      |
| OTel health       | 30700     | 30700 - 30799 | `HDX_DEV_OTEL_HEALTH_PORT`    |
| OTel gRPC         | 30800     | 30800 - 30899 | `HDX_DEV_OTEL_GRPC_PORT`      |
| OTel HTTP         | 30900     | 30900 - 30999 | `HDX_DEV_OTEL_HTTP_PORT`      |
| OTel metrics      | 31000     | 31000 - 31099 | `HDX_DEV_OTEL_METRICS_PORT`   |
| OTel JSON HTTP    | 31100     | 31100 - 31199 | `HDX_DEV_OTEL_JSON_HTTP_PORT` |

### Dev Portal

The dev portal is a centralized web dashboard that discovers all running
worktree stacks by inspecting Docker container labels and slot files.

```bash
# Start the portal (runs on fixed port 9900)
make dev-portal

# Open in browser
open http://localhost:9900
```

The portal auto-refreshes every 3 seconds and shows each worktree's:

- Branch name and slot number
- All services with status (running/stopped) and clickable port links
- Separate cards for each active worktree

### Overriding the Slot

```bash
# Use a specific slot instead of the auto-computed one
HDX_DEV_SLOT=5 make dev
```

### Worktrees With the Same Folder Name

If your worktrees share a folder name (`a/hyperdx`, `b/hyperdx`), they get the
same slot and clash. Set `HDX_SLOT_FROM=path` in your shell profile to hash each
worktree's full path instead. It applies to the dev stack, integration tests,
E2E tests, and the dev portal.

```bash
export HDX_SLOT_FROM=path
```

Your existing stacks move to new slots once, so their dev data (in
`.volumes/*_<slot>`) starts empty. The default, `HDX_SLOT_FROM=name`, keeps
today's slots.

### Tools That Talk to the Dev Stack

Run them through `scripts/with-slots.sh`. It sets this worktree's dev ports
(`HDX_DEV_CH_HTTP_PORT`, `HYPERDX_API_PORT`, ...) and nothing else. Don't source
`scripts/dev-env.sh` for this: it also clears `packages/app/.next` and registers
the stack with the dev portal, which disrupts a running `yarn dev`.

```bash
# From a package script, e.g. packages/api's "seed:alerts"
../../scripts/with-slots.sh ts-node scripts/seed-alerts.ts

# Against another slot
HDX_DEV_SLOT=5 scripts/with-slots.sh <command>
```

## Testing Strategy

### Testing Tools

- **Unit Tests**: Jest with TypeScript support
- **Integration Tests**: Jest with database fixtures
- **Frontend Testing**: React Testing Library + Jest
- **E2E Testing**: Playwright (frontend) and Custom smoke tests with BATS
  (ingestion)

### Testing Patterns

- **TDD Approach**: Write tests before implementation for new features
- **Test organization**: Tests co-located with source files in `__tests__/`
  directories
- **Mocking**: MSW for API mocking in frontend tests
- **Database testing**: Isolated test databases with fixtures

### CI / Integration Testing

For integration testing:

```bash
# Build dependencies (run once before first test run)
make dev-int-build

# Run API integration tests (spins up Docker services, runs tests, tears down)
make dev-int FILE=<TEST_FILE_NAME>

# Run common-utils integration tests
make dev-int-common-utils FILE=<TEST_FILE_NAME>
```

**Multi-agent / worktree support:**

The `make dev-int` command automatically assigns unique Docker ports per
worktree directory, so multiple agents can run integration tests in parallel
without port conflicts.

- A deterministic slot (0-99) is computed from the worktree directory name
- Each slot gets its own Docker Compose project name and port range
- Override the slot manually: `make dev-int HDX_CI_SLOT=5 FILE=alerts`
- The slot and assigned ports are printed when `dev-int` starts

Port mapping (base + slot):

| Service         | Default port (slot 0) | Variable          |
| --------------- | --------------------- | ----------------- |
| ClickHouse HTTP | 18123                 | HDX_CI_CH_PORT    |
| MongoDB         | 39999                 | HDX_CI_MONGO_PORT |
| API test server | 19000                 | HDX_CI_API_PORT   |
| OpAMP           | 14320                 | HDX_CI_OPAMP_PORT |

**CI Testing Notes:**

- Uses separate Docker Compose configuration (`docker-compose.ci.yml`)
- Isolated test environment with unique `-p int-<slot>` project name
- Includes all necessary services (ClickHouse, MongoDB, OTel Collector)
- Tests run against real database instances for accurate integration testing

### E2E Testing

E2E tests use the same slot-based isolation pattern as integration tests, with
their own dedicated port range (20320-21399) so they can run simultaneously with
both the dev stack and CI integration tests.

E2E port mapping (base + slot):

| Service           | Base Port | Range         | Env Variable             |
| ----------------- | --------- | ------------- | ------------------------ |
| OpAMP             | 20320     | 20320 - 20419 | `HDX_E2E_OPAMP_PORT`     |
| ClickHouse HTTP   | 20500     | 20500 - 20599 | `HDX_E2E_CH_PORT`        |
| ClickHouse Native | 20600     | 20600 - 20699 | `HDX_E2E_CH_NATIVE_PORT` |
| API server        | 21000     | 21000 - 21099 | `HDX_E2E_API_PORT`       |
| MongoDB           | 21100     | 21100 - 21199 | `HDX_E2E_MONGO_PORT`     |
| App (local)       | 21200     | 21200 - 21299 | `HDX_E2E_APP_LOCAL_PORT` |
| App (fullstack)   | 21300     | 21300 - 21399 | `HDX_E2E_APP_PORT`       |

```bash
# Run all E2E tests
make e2e

# Run a specific test file (dev mode: hot reload, containers kept running)
make dev-e2e FILE=navigation

# Run a specific test by grep pattern
make dev-e2e FILE=navigation GREP="help menu"

# Grep across all files
make dev-e2e GREP="should navigate"

# Open HTML report after tests finish (screenshots, traces, step-by-step)
make dev-e2e FILE=navigation REPORT=1

# Or call the script directly for more control
./scripts/test-e2e.sh --ui --last-failed

# Override the slot manually
HDX_E2E_SLOT=5 ./scripts/test-e2e.sh
```

- A deterministic slot (0-99) is computed from the worktree directory name
- Each slot gets its own Docker Compose project name (`e2e-<slot>`) and port
  range
- The slot and assigned ports are printed when E2E tests start

**Port range safety:** E2E has its own dedicated port range (20320-21399) that
does not overlap with CI integration tests (14320-40098) or the dev stack
(30100-31199), so all three can run simultaneously from the same worktree.

## Common Development Tasks

### Adding New Features

1. **API First**: Define API endpoints and data models
2. **Database Models**: Create/update Mongoose schemas and ClickHouse queries
3. **Frontend Integration**: Build UI components and integrate with API
4. **Testing**: Add unit and integration tests
5. **Documentation**: Update relevant docs

### Debugging

- Check browser and server console output for errors, warnings, or relevant logs
- Add targeted logging to trace execution and variable states
- For persistent issues, check `fixes/` directory for documented solutions
- Document complex fixes in `fixes/` directory with descriptive filenames

## Code Quality

### Pre-commit Hooks

The project uses Husky + lint-staged to automatically run:

- Prettier for formatting
- ESLint for linting
- API doc generation (for external API changes)

These run automatically on `git commit` for staged files.

### Manual Linting (if needed)

If you need to manually lint:

```bash
# Per-package linting with auto-fix
cd packages/app && yarn run lint:fix
cd packages/api && yarn run lint:fix
cd packages/common-utils && yarn lint:fix

# Check all packages
yarn run lint
```

## File Locations Quick Reference

- **Config**: `packages/api/src/config.ts`, `packages/app/next.config.mjs`,
  `docker-compose.dev.yml`
- **Models**: `packages/api/src/models/`
- **API Routes**: `packages/api/src/routers/`
- **Controllers**: `packages/api/src/controllers/`
- **Pages**: `packages/app/pages/`
- **Components**: `packages/app/src/`
- **Shared Utils**: `packages/common-utils/src/` — no root barrel, so imports
  are deep `@hyperdx/common-utils/dist/...` paths. See
  [`code_style.md`](code_style.md#where-shared-code-already-lives) for the
  module map and the grep-first rule before you add a helper.

## Vercel Preview Deployments (Inline API)

The `hyperdx-private` Vercel project deploys the `@hyperdx/app` Next.js app for
preview environments. Normally `/api/*` requests proxy to a separately deployed
API service (`SERVER_URL`). For PR previews we instead **inline the entire
Express API** into the Next.js serverless function so a single Vercel deployment
serves both the app and the API.

### How it works

1. `packages/api/src/serverless.ts` exposes the Express app from `api-app.ts` as
   a stateless `(req, res) => Promise<void>` handler. It lazily connects to
   MongoDB on the first invocation and caches the connection across warm
   invocations.
2. `packages/app/pages/api/[...all].ts` branches on the `HDX_PREVIEW_INLINE_API`
   env var. When `true`, it `require()`s the compiled serverless handler and
   dispatches directly. Otherwise it falls back to the existing
   http-proxy-middleware path used by Docker/standalone builds.
3. `packages/app/next.config.mjs` adds a webpack `externals` rule that marks
   `@hyperdx/api` as a CommonJS external **unless** `HDX_PREVIEW_INLINE_API` is
   `true`. This keeps production app builds (Docker fullstack image, standalone
   Next output) byte-for-byte equivalent to before — they never bundle
   passport-saml, mongoose, AWS SDK, etc.
4. `vercel.json` declares the repo-root build commands so Vercel builds
   `common-utils` → `api` → `app` in order.

### Required Vercel project env vars (Preview scope only)

| Key                                          | Notes                                                             |
| -------------------------------------------- | ----------------------------------------------------------------- |
| `HDX_PREVIEW_INLINE_API`                     | `true` — turns on the inline path                                 |
| `MONGO_URI`                                  | Preview MongoDB connection string                                 |
| `EXPRESS_SESSION_SECRET`                     | Random 32+ char string shared by all instances; unset generates one per process, invalidating sessions on restart and between instances |
| `DISABLED_AUTH_METHODS`                      | `google,saml` (avoids per-preview OAuth callback URL config)      |
| `AI_PROVIDER`, `AI_API_KEY`, `AI_MODEL_NAME` | Optional, only if AI features should work                         |
| `FRONTEND_URL`                               | Optional. Leave unset to use host-only cookies on the preview URL |

Production env vars on the same project must leave `HDX_PREVIEW_INLINE_API`
unset (or `false`) so production deployments keep using the proxy path.

### Limitations

- **No background tasks.** Scheduled jobs (`packages/api/src/tasks/`, including
  `CheckAlertTask`) are not run in preview deployments.
- **No alerts.** Alert evaluation depends on the cron task runner.
- **No OPAMP server.** The OPAMP control-plane server (`APP_TYPE=opamp`) is not
  started in preview. The serverless entrypoint asserts `APP_TYPE=api` and
  refuses to boot otherwise.
- **Function size.** The bundled function should stay under Vercel's 50 MB
  unzipped limit. If it grows, mark heavy deps (`@aws-sdk/*`,
  `@node-saml/passport-saml`, AI SDKs) as `serverExternalPackages` in
  `next.config.mjs`.
- **Cold-start latency.** First request after idle pays ~500–1500 ms for the
  initial Mongo connection.
- **Shared preview MongoDB.** Unless you partition by database name, all preview
  deployments share the same Mongo state.