# AGENTS.md - Comprehensive Guide for AI Coding Agents

This file provides guidance for AI coding agents (Claude Code, Cursor, Codex, Amp, etc.) when working with code in this repository.

## Essential Commands - Quick Reference

### Build & Development
- **Build**: `pnpm build` (root) or `turbo build`; in `agents-private` (this file at `public/agents/`), `pnpm build:agents` from that repo's root
- **Dev**: `pnpm dev` (root) or navigate to package and run `pnpm dev`
- **Setup (core)**: `pnpm setup-dev` — core DBs (Doltgres, Postgres, SpiceDB), env config, migrations, admin user
- **Setup (isolated)**: `pnpm setup-dev --isolated <name>` — same as above but in a parallel environment (see [Isolated Environments](#isolated-parallel-environments))
- **Setup (optional services)**: `pnpm setup-dev:optional` — Nango + SigNoz + OTEL + Jaeger (run `setup-dev` first)
- **Optional services lifecycle**: `pnpm optional:stop` | `pnpm optional:status` | `pnpm optional:reset`

### Verification

**Iteration:** Fresh regression: `cd <pkg> && pnpm test --run <file>`; then touched lint/typecheck/guards, widened for inputs/consumers.

**Pre-push:** `pnpm format`. Optional: `pnpm check` (in `agents-private`, `pnpm check:agents` from its root, since `public/agents`' `precheck`/`prebuild` hooks and a real `install` there move links into a second store) runs lint, typecheck, tests, format:check, knip and the repo's guards; `agents-private` CI runs its parts.

### Database Operations (run from monorepo root)
- **Generate migrations**: `pnpm db:generate` - Generate Drizzle migrations from schema changes
- **Apply migrations**: `pnpm db:migrate` - Apply generated migrations to database
- **Drop migrations**: `pnpm db:drop` - Drop migration files (use this to remove migrations, don't manually delete)
- **Database studio**: `pnpm db:studio` - Open Drizzle Studio for database inspection
- **Check schema**: `pnpm db:check`
- **Initialize auth**: `pnpm db:auth:init` - Create default organization and admin user for local development

### Creating Changelog Entries (Changesets)

Create a changeset for any user-facing change to a published package:

```bash
pnpm bump <patch|minor|major> --pkg <package> "<message>"
```

**Examples:**
```bash
# Single package
pnpm bump patch --pkg agents-core "Fix race condition in agent message queue"

# Multiple packages (for tightly coupled changes)
pnpm bump minor --pkg agents-sdk --pkg agents-core "Add streaming response support"
```

**Valid package names:** `agents-cli`, `agents-core`, `agents-api`, `agents-manage-ui`, `agents-work-apps`, `agents-sdk`, `create-agents`, `ai-sdk-provider`

**Semver guidance:**
- **Major**: Reserved - do not use without explicit approval
- **Minor**: Schema changes requiring migration, significant behavior changes
- **Patch**: Bug fixes, additive features, non-breaking changes

#### Writing Good Changelog Messages

**Target audience:** Developers consuming these packages

**Style requirements:**
- Sentence case: "Add new feature" not "add new feature"
- Start with action verb: Add, Fix, Update, Remove, Improve, Deprecate
- Be specific about what changed and why it matters to consumers
- Keep to 1-2 sentences

**Good examples:**
- "Fix race condition when multiple agents connect simultaneously"
- "Add `timeout` option to `createAgent()` for custom connection timeouts"
- "Remove deprecated `legacyMode` option (use `mode: 'standard'` instead)"
- "Improve error messages when agent registration fails"

**Bad examples:**
- "fix bug" (too vague - which bug? what was the impact?)
- "update dependencies" (not user-facing, doesn't need changeset)
- "Refactored the agent connection handler to use async/await" (implementation detail, not user impact)
- "changes" (meaningless)

**When to create a changeset (MANDATORY):**
- Any bug fix, feature, or behavior change to a published package — even if the package is "internal-facing" (e.g., `agents-work-apps`, `agents-api`). If the code ships to users or affects runtime behavior, it needs a changeset.
- This includes work-app integrations (Slack, GitHub), API route changes, SDK changes, CLI changes, and core library changes.

**When NOT to create a changeset:**
- Documentation-only changes
- Test-only changes
- Internal tooling/scripts changes
- Changes to ignored packages (agents-ui, agents-docs, cookbook-templates, test-agents)

**Multiple changes in one PR:**
If a PR affects multiple packages independently, create separate changesets for each with specific messages. If changes are tightly coupled (e.g., updating types in core that SDK depends on), use a single changeset listing both packages.

### Running Examples / Reference Implementations
- Use `agents-cookbook/` for reference implementations and patterns.
- There is no `examples/` directory; prefer cookbook recipes or package-specific README files.

### Documentation Development
```bash
# From agents-docs directory
pnpm dev              # Start documentation site (port 3000)
pnpm build           # Build documentation for production
```

## Code Style (Biome enforced)
- **Imports**: Use type imports (`import type { Foo } from './bar'`), organize imports enabled, barrel exports (`export * from './module'`)
- **Formatting**: Single quotes, semicolons required, 100 char line width, 2 space indent, ES5 trailing commas
- **Types**: Explicit types preferred, avoid `any` where possible (warning), use Zod for validation
- **Naming**: camelCase for variables/functions, PascalCase for types/components, kebab-case for files
- **Error Handling**: Use try-catch, validate with Zod schemas, handle errors explicitly
- **React Compiler**: React Compiler is enabled for this repo. Do not add `memo`, `useMemo`, or `useCallback`; rely on the compiler unless a maintainer explicitly requests an exception
- **Function Arguments**: When a function has more than 3 parameters, prefer a single object argument so that parameters are well-labeled, ordering doesn't matter, and callers can benefit from spread operators
- **No Comments**: Do not add comments unless explicitly requested

## TypeScript Configuration

The root `typescript` devDependency in `public/agents/package.json` is a peer anchor, not a stray pin: it keeps `tsdown`, `rolldown-plugin-dts` and `knip` resolved against TypeScript 6 while another importer in the shared root lockfile declares `typescript@^7`. Do not remove it, and do not bump it past 6.x, without re-running `pnpm build` for every `public/agents` package first.

All TypeScript packages under `public/agents/packages/` and `public/agents/agents-*/` should extend the shared strict baseline at `public/agents/tsconfig.base.json`. This file defines the monorepo's strictness contract: `strict`, `noUncheckedIndexedAccess`, `exactOptionalPropertyTypes`, `noImplicitOverride`, `noFallthroughCasesInSwitch`, `forceConsistentCasingInFileNames`, `skipLibCheck`, `esModuleInterop`, `resolveJsonModule`, `isolatedModules`, `moduleDetection: force`, and `verbatimModuleSyntax: false`.

### New packages

When creating a new workspace package under `public/agents/packages/<name>/`, the `tsconfig.json` should extend the base:

```json
{
  "extends": "../../tsconfig.base.json",
  "compilerOptions": {
    "target": "ES2022",
    "module": "ESNext",
    "moduleResolution": "bundler",
    "outDir": "./dist",
    "rootDir": "./src",
    "declaration": true,
    "sourceMap": true
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules", "dist"]
}
```

Only override target/module/output settings in the per-package config. Strictness flags live in the base; do not duplicate them.

### Why this matters

- Enforces a consistent strictness floor across all TS packages (prevents drift over time)
- Strictness flags catch real runtime bugs at compile time (e.g., optional-prop-spread issues caught 3 real bugs in `agents-email` during the pilot)
- Bumping strictness later = edit one file instead of every package's tsconfig
- New packages inherit the baseline automatically

### Opt-out

A package that legitimately cannot extend the base (e.g., needs older target, needs non-strict mode for generated code) should:

1. Omit the `extends` line
2. Add a top-level comment explaining why: `// Opt-out rationale: <reason>`
3. Self-contain the compiler options it actually needs

Unexplained opt-outs drift the monorepo's strictness contract and should be caught in review.

### Path notes

The base lives at `public/agents/tsconfig.base.json`. After Copybara mirrors the subtree to `inkeep/agents`, the base lives at `/tsconfig.base.json` on the public repo (prefix stripped). The relative `extends` path from `packages/<name>/tsconfig.json` is `../../tsconfig.base.json` in both repos, portable across the mirror boundary.

## Testing (Vitest)
- Place tests in `__tests__/` directories adjacent to code
- Name: `*.test.ts` or `*.spec.ts`
- Pattern: `import { describe, it, expect, beforeEach, vi } from 'vitest'`
- Run with `--run` flag to avoid watch mode
- 60-second timeouts for A2A interactions
- Each test worker uses an embedded Postgres (pglite) database with manage/run Drizzle migrations applied in setup

### Visual Regression Tests
Browser-based visual tests (screenshot comparisons) run Chromium inside a Docker container so screenshots are identical across macOS and Linux CI.

Deliberately **not** chained into the default `test` script (that made `pnpm test` fail with an opaque `ws://127.0.0.1:3100 code=1006` on any machine without the server). CI runs them as an explicit step in `Agents Extended Validation`.

```bash
# Start the Playwright Docker server (one-time, stays running)
docker compose -f docker-compose.visual.yml up -d

# Run visual tests
pnpm --filter @inkeep/agents-manage-ui test:visual

# Update baselines after intentional UI changes
pnpm --filter @inkeep/agents-manage-ui test:visual:update

# Stop the server when done
docker compose -f docker-compose.visual.yml down
```

- Visual test files: `*.browser.test.tsx`
- Baselines stored in `src/__screenshots__/`
- Docker is **required** — without it, tests fall back to local Chromium and screenshots won't match CI

## Package Manager
- Always use `pnpm` (not npm, yarn, or bun)

### pnpm-lock.yaml Resolution Strategy

⚠️ **NEVER delete `pnpm-lock.yaml` and regenerate it from scratch.** Deleting the lockfile and running `pnpm install` allows the resolver to pick different (often lower) versions of transitive dependencies, causing silent downgrades that break tests or change runtime behavior.

**Correct approach when the lockfile has merge conflicts or needs updating:**

1. **Start from the base branch's lockfile** — check out the `pnpm-lock.yaml` from the target base (usually `main`):
   ```bash
   git checkout main -- pnpm-lock.yaml
   ```
2. **Re-install to layer your branch's dependency changes on top** (in `agents-private`, `pnpm install:all` from its root, after step 1 for each conflicted lockfile):
   ```bash
   pnpm install
   ```
   This preserves all existing resolutions from `main` and only adds/updates what your branch's `package.json` changes require.

3. **Commit the updated lockfile** with your other changes.

**Why this matters:** The lockfile pins exact transitive dependency versions. Regenerating from scratch lets the resolver freely re-resolve the entire tree, which can silently pick different versions even when `package.json` ranges haven't changed. Starting from the base lockfile ensures only your intentional changes affect resolution.

**When changing dependencies**: regenerate and commit every affected lockfile. In `agents-private`, run `pnpm install:all` from its root, where `LOCKFILE_ROOTS` in `scripts/check-monorepo-traps.mjs` defines the enforced install boundaries.

## Architecture Overview

This is the **Inkeep Agent Framework** - a multi-agent AI system with A2A (Agent-to-Agent) communication capabilities. The system provides OpenAI Chat Completions compatible API while supporting sophisticated agent orchestration.

### Core Components

#### Unified API (`agents-api`)
The `agents-api` package contains all API domains under a single service:
- **`/domains/manage/`** - Agent configuration, projects, tools, and administrative operations
- **`/domains/run/`** - Agent execution, conversations, A2A communication, and runtime operations
- **`/domains/evals/`** - Evaluation workflows, dataset management, and evaluation triggers

#### Multi-Agent Framework

#### Database Schema (PostgreSQL + Drizzle ORM)
- [manage-schema.ts](./packages/agents-core/src/db/manage/manage-schema.ts) - Config tables (Doltgres)
- [runtime-schema.ts](./packages/agents-core/src/db/runtime/runtime-schema.ts) - Runtime tables (Postgres)

## Key Implementation Details

### CRUD HTTP Method Conventions (RFC 9110, RFC 5789)

| Operation | Method | Path Pattern | Example |
|-----------|--------|-------------|---------|
| Create | POST | `/resources` | `POST /agents` |
| Read (list) | GET | `/resources` | `GET /agents` |
| Read (single) | GET | `/resources/{id}` | `GET /agents/{id}` |
| Update (partial) | PATCH | `/resources/{id}` | `PATCH /agents/{id}` |
| Delete | DELETE | `/resources/{id}` | `DELETE /agents/{id}` |

- **PATCH** for partial updates (RFC 5789) — canonical method for standard CRUD update operations
- **PUT** for full-resource replacement, upsert, and set/replace operations (RFC 9110 §9.3.4)
- **POST** for creates and non-idempotent actions (sync, reconnect, cancel, etc.)
- **GET** for reads — never mutates state
- **DELETE** for resource removal

#### Exceptions: PUT-canonical routes

The following routes use **PUT as the canonical method** because they perform full-resource replacement or upsert semantics, not partial updates:

| Route | Reason | OperationId |
|-------|--------|-------------|
| `PUT /project-full/{projectId}` | Upsert — creates or fully replaces a project | `update-full-project` |
| `PUT /agents-full/{agentId}` | Upsert — creates or fully replaces an agent | `update-full-agent` |
| `PUT /tools/{toolId}/github-access` | Set/replace — replaces entire GitHub access config | `set-mcp-tool-github-access` |
| `PUT /tools/{toolId}/slack-access` | Set/replace — replaces entire Slack access config | `set-mcp-tool-slack-access` |
| `PUT /projects/{projectId}/github-access` | Set/replace — replaces entire project GitHub access | `set-project-github-access` |

These routes still register a PATCH method for backward compatibility (with `x-speakeasy-ignore: true`).

#### Dual-registration helper

All update routes register both PUT and PATCH methods using `openapiRegisterPutPatchRoutesForLegacy()` from `agents-api/src/utils/openapiDualRoute.ts`. The legacy method gets `x-speakeasy-ignore: true` and a suffixed operationId (e.g., `update-agent-put`). To find all legacy routes, search for usages of this helper. When legacy methods are no longer needed, remove the helper calls and replace with single `app.openapi()` registrations.

#### Backward compatibility

All existing PUT update routes remain functional — they share the same handler as PATCH. New update routes must use PATCH as canonical (or PUT for upsert/set-replace semantics). Do not remove existing PUT routes without a deprecation period.

### Database Migration Workflow

#### Standard Workflow
1. Edit `packages/agents-core/src/db/manage/manage-schema.ts` or `packages/agents-core/src/db/runtime/runtime-schema.ts`
2. Run `pnpm db:generate` to create migration files in `drizzle/`
3. (Optional) Make minor edits to the newly generated SQL file if needed due to drizzle-kit limitations
4. Run `pnpm db:migrate` to apply the migration to the database

#### Important Rules
- ⚠️ **NEVER manually edit files in `drizzle/meta/`** - these are managed by drizzle-kit
- ⚠️ **NEVER edit existing migration SQL files after they've been applied** - create new migrations instead
- ⚠️ **To remove migrations, use `pnpm db:drop`** - don't manually delete migration files
- ✅ **Only edit newly generated migrations** before first application (if drizzle-kit has limitations)

### Testing Patterns (MANDATORY for all new features)
- **Vitest**: Test framework with 60-second timeouts for A2A interactions
- **Isolated Databases**: Each test worker gets in-memory SQLite database
- **Integration Tests**: End-to-end A2A communication testing
- **Test Structure**: Tests must be in `__tests__` directories, named `*.test.ts`
- **Coverage Requirements**: All new code paths must have test coverage

#### Shared Test Mocks (`@inkeep/agents-core/test-utils`)

Use the shared mock factories instead of defining inline mocks. This prevents drift when the real API changes.

```typescript
import { createMockLoggerModule } from '@inkeep/agents-core/test-utils';

// Simple — just suppress logger noise (most common):
vi.mock('../../logger', () => createMockLoggerModule().module);

// When asserting on logger calls:
const { mockLogger, module: loggerModule, clearAll } = createMockLoggerModule();
vi.mock('../../logger', () => loggerModule);
// In beforeEach: clearAll() — vi.clearAllMocks() does not reach nested mock fns

// Edge case: static imports after vi.mock (TDZ issue) — use vi.hoisted container:
const refs = vi.hoisted(() => ({ mockLogger: null as any }));
vi.mock('../../logger', async () => {
  const { createMockLoggerModule } = await import('@inkeep/agents-core/test-utils');
  const result = createMockLoggerModule();
  refs.mockLogger = result.mockLogger;
  return result.module;
});
```

**Do NOT** define inline logger mocks (`{ info: vi.fn(), warn: vi.fn(), ... }`) — use the factory.

#### Example Test Structure
```typescript
// src/builder/__tests__/myFeature.test.ts
import { describe, it, expect, beforeEach, vi } from 'vitest';

describe('MyFeature', () => {
  beforeEach(() => {
    vi.clearAllMocks();
  });

  it('should handle success case', async () => {
    // Test implementation
  });

  it('should handle error case', async () => {
    // Test error handling
  });
});
```

### Environment Configuration
Required environment variables in `.env` files:
```dotenv
ENVIRONMENT=development|production|test
INKEEP_AGENTS_MANAGE_DATABASE_URL=postgresql://appuser:password@localhost:5432/inkeep_agents
INKEEP_AGENTS_RUN_DATABASE_URL=postgresql://appuser:password@localhost:5433/inkeep_agents
PORT=3002
ANTHROPIC_API_KEY=required
OPENAI_API_KEY=optional
LOG_LEVEL=debug|info|warn|error
```

### Isolated Parallel Environments

Run multiple full dev stacks simultaneously with zero port conflicts. Each isolated environment gets its own Docker containers, volumes, and network with dynamically assigned ports.

**When to use:** Running multiple features in parallel, AI coding agents executing concurrently, or testing against an isolated database without affecting the default environment.

#### Quick Start

```bash
# Create an isolated environment (Docker + migrations + auth)
pnpm setup-dev --isolated my-feature

# Point your shell at it
source <(./scripts/isolated-env.sh env my-feature)

# Run the app (uses isolated databases + dynamic app ports)
pnpm dev

# Tear down when done
./scripts/isolated-env.sh down my-feature
```

#### Available Commands

| Command | Description |
|---------|-------------|
| `./scripts/isolated-env.sh setup <name>` | Full setup: Docker + health checks + migrations + auth init |
| `./scripts/isolated-env.sh up <name>` | Start containers only (no migrations) |
| `./scripts/isolated-env.sh down <name>` | Stop and remove containers + volumes |
| `./scripts/isolated-env.sh status` | List all running isolated environments with ports |
| `./scripts/isolated-env.sh env <name>` | Print source-able env var exports |

#### How It Works

- Uses `docker-compose.isolated.yml` with `COMPOSE_PROJECT_NAME` for namespace isolation
- Docker assigns random available host ports (no hardcoded bindings)
- Ports are discovered post-startup via `docker compose port` and saved to `.isolated-envs/<name>.json`
- The `env` command outputs `export` statements that override database URLs (`INKEEP_AGENTS_MANAGE_DATABASE_URL`, `INKEEP_AGENTS_RUN_DATABASE_URL`, `SPICEDB_ENDPOINT`) and app ports (`AGENTS_API_PORT`, `MANAGE_UI_PORT`, `INKEEP_AGENTS_API_URL`)
- Default environment (`docker-compose.dbs.yml` on fixed ports) continues to work unchanged

#### Key Files

| File | Purpose |
|------|---------|
| `docker-compose.isolated.yml` | Compose file with dynamic port allocation |
| `scripts/isolated-env.sh` | CLI for managing isolated environments |
| `.isolated-envs/<name>.json` | Port state files (gitignored) |

## High Product-level Thinking

This repo is a product with multiple user-facing surfaces and shared contracts. A “small” change in one place can have real side effects elsewhere.

**Rule:** Do NOT implement additional work beyond what the user requested. If you identify likely cross-surface impacts, **flag them** and **ask** the user to get full clarity and ensure high degree of product-level thinking. You can flag prior to starting new work/tasks, but also if you identify ambiguities or additional things to consider mid-execution. 

**If the request is NOT backed by a PRD, a well-specced artifact, or clearly scoped small task**, pause and ask targeted, highly relevant questions when the blast radius is unclear. When applicable, suggest user make a PRD; offer to help with the `spec` skill if it is installed.

Your goal is to **AVOID**:
- locking in unintended behavior
- making changes without thinking through all potential interaction points for how a user may consume or interact with a product change
- breaking changes or breaking data contracts without clear acknowledgement and plan
- missing critical dimensions of feature development, like authorization or security
- identify any side effects of the work that is being asked
- implementations that contradict or duplicate existing patterns/abstractions

Your responsibility is to think through the work that is being done from all dimensions and considerations.

### What to clarify (high-signal triggers)
- **Definition shapes / shared types / validation rules** (agent/project/tool/credential/etc.)
- **Runtime behavior or streaming formats** (responses, tool calls, artifacts/components)
- **Tracing / telemetry** (span names, attribute keys, correlation IDs, exporter config)
- **Resources, endpoints, or actions** (create/update/delete, new capabilities)
- **Auth / permissions / tenancy** (view/use/edit boundaries, RBAC, fine-grained authz, multi-tenant scoping)

### Surface area analysis (load before planning or implementing)

This product has **50+ customer-facing** and **100+ internal tooling/devops** surfaces with complex dependency chains. When planning or implementing any feature or change, load the relevant surface area skill to map the blast radius and plan "to-dos" of all areas that need to be addressed before writing a line of code:

| Skill | Scope | Load when |
|---|---|---|
| `audience-impact` | Who is affected and how fast does it reach them? Maps roles (Contributor, Builder, Platform User) × deployment modes (Cloud, Self-hosted) to impact propagation. | Start here — identifies which audiences a change affects and what deliverables are needed |
| `product-surface-areas` | APIs, SDKs, CLI, UIs, Widgets, Event Streams, docs, protocols, templates, etc. | Change affects anything a customer (developer or no-code admin) uses or depends on |
| `internal-surface-areas` | Build, CI/CD, DB, auth, runtime engine, test infra, internal AI tooling, etc. | Change affects infrastructure, tooling, or shared internals |

**Tip**: Start with `audience-impact` to quickly identify who cares and how changes propagate, then load the catalog skills for detailed surface tracing. Both catalog skills include dependency graphs, breaking change impact matrices, and transitive chain tracing.

## Development Guidelines

### ⚠️ MANDATORY: Required for All New Features

**ALL new work MUST include these three components - NO EXCEPTIONS:**

✅ **1. Unit Tests**
- Write comprehensive unit tests using Vitest
- Place tests in `__tests__` directories adjacent to the code
- Follow naming convention: `*.test.ts`
- Minimum coverage for new code paths
- Test both success and error cases

✅ **2. Agent Builder UI Components** 
- Add corresponding UI components in `/agents-manage-ui/src/components/`
- Include form validation schemas
- Update relevant pages in `/agents-manage-ui/src/app/`
- Follow existing Next.js and React patterns
- Use the existing UI component library

✅ **3. Documentation**
- Create or update documentation in `/agents-docs/content/docs/` (public-facing docs)
- Documentation should be in MDX format (`.mdx` files)
- Follow the **write-docs** skill whenever creating or modifying documentation

Fix a red in a test you added or edited, or naming a file you changed, at its cause; note any other in one line without chasing it, and make no green claim. Before marking any feature complete, verify:
- [ ] Your change's tests, lint and typecheck pass, bar noted reds
- [ ] **Changeset created** via `pnpm bump` for every published package with runtime behavior changes (see [Creating Changelog Entries](#creating-changelog-entries-changesets))
- [ ] UI components implemented in agents-manage-ui
- [ ] Documentation added to `/agents-docs/`
- [ ] Surface area and breaking changes have been addressed as agreed with the user (see “High Product-level Thinking”).

### 📋 Standard Development Workflow

1. Create a branch: `git checkout -b feat/your-feature-name` (in `agents-private`, create it in a new worktree, as below)
2. Create a changeset: `pnpm bump <patch|minor> --pkg <package> "<message>"` (required for any runtime behavior change to a published package — see [Creating Changelog Entries](#creating-changelog-entries-changesets))
3. Run [Verification](#verification) before pushing
4. Commit, then `gh pr create`

The user may override this workflow (e.g., work directly on main).

### 📁 Git Worktrees for Parallel Feature Development

Git worktrees allow you to work on multiple features simultaneously without switching branches in your main working directory. This is especially useful when you need to quickly switch context between different Linear tickets or have multiple features in progress.

#### Creating a Worktree

Cut each worktree from a fresh base (in `agents-private`, run every worktree command below from its root):

```bash
git fetch origin main && git worktree add ../pull-instrument -b feat/pull-instrument origin/main
```

**Important Conventions:**
- The directory name and branch name should match (e.g., `pull-instrument` matches `feat/pull-instrument`)
- Branch names should reference a Linear ticket when applicable (e.g., `feat/ENG-123-feature-name`)
- Worktree directories are temporary and should be removed after the work is complete

#### Working with Worktrees

```bash
# Create a new worktree for a feature
git fetch origin main && git worktree add ../my-feature -b feat/ENG-123-my-feature origin/main

# Enter it
cd ../my-feature

# Work normally
# ... make changes, commit, push, create PR ...

# List all worktrees
git worktree list

# Remove a worktree after PR is merged (run from main repo)
git worktree remove ../my-feature

# Remove the remote branch after cleanup
git branch -d feat/ENG-123-my-feature
git push origin --delete feat/ENG-123-my-feature

# Prune stale worktree references
git worktree prune
```

#### When to Use Worktrees

✅ **Use worktrees when:**
- Working on multiple features simultaneously
- Need to quickly test/review another branch without stashing current work
- Running long-running processes (tests, builds) while working on something else
- Comparing implementations across different branches side-by-side

**Reference**: [git-worktree documentation](https://git-scm.com/docs/git-worktree)

### When Working with Agents
1. **Always call `graph.init()`** after creating agent relationships to persist to database
2. **Use builder patterns** (`agent()`, `subAgent()`, `tool()`) instead of direct database manipulation
3. **Preserve contextId** when implementing transfer/delegation logic - extract from task IDs if needed
4. **Validate tool results** with proper type guards instead of unsafe casting
5. **Test A2A communication end-to-end** when adding new agent relationships
6. **Follow the mandatory requirements above** for all new features

### Performance Considerations
- **Parallelize database operations** using `Promise.all()` instead of sequential `await` calls
- **Optimize array processing** with `flatMap()` and `filter()` instead of nested loops
- **Implement cleanup mechanisms** for debug files and logs to prevent memory leaks

### Internal Self-Calls: `getInProcessFetch()` vs `fetch`
Any code in `agents-api` or `agents-work-apps` that makes **internal A2A calls or self-referencing API calls** (i.e. calling another route on the same service) **MUST** use `getInProcessFetch()` from `@inkeep/agents-core` instead of the global `fetch`.

- `getInProcessFetch()` routes the request through the Hono app's middleware stack **in-process**, guaranteeing it stays on the same instance.
- Global `fetch` sends the request over the network, where a load balancer may route it to a **different** instance — breaking features that depend on process-local state (e.g. the stream helper registry for SSE streaming).
- This bug only manifests under load in multi-instance deployments and is extremely difficult to diagnose.

**When to use:**
| Scenario | Use |
|---|---|
| Internal A2A delegation/transfer (same service) | `getInProcessFetch()` |
| Eval service calling the chat API on itself | `getInProcessFetch()` |
| Forwarding requests to internal workflow routes | `getInProcessFetch()` |
| Slack/work-app calls to `/run/api/chat` or `/manage/` routes | `getInProcessFetch()` |
| Calling an **external** service or third-party API | Global `fetch` |
| Test environments (falls back automatically) | Either (auto-fallback) |

### Route Authorization Pattern (`createProtectedRoute`)
All API routes in `agents-api` **must** use `createProtectedRoute()` from `@inkeep/agents-core/middleware` instead of the plain `createRoute()` from `@hono/zod-openapi`. This is enforced by Biome lint rules and ensures every route has explicit authorization metadata (`x-authz`) in the OpenAPI spec.

```typescript
import { createProtectedRoute, noAuth, inheritedAuth } from '@inkeep/agents-core/middleware';
import { requireProjectPermission } from '../../middleware/projectAccess';

// Standard protected route — pass the permission middleware directly
app.openapi(
  createProtectedRoute({
    method: 'get',
    path: '/',
    permission: requireProjectPermission('view'),
    // ... rest of route config
  }),
  handler,
);

// Public route (no auth) — use noAuth()
app.openapi(
  createProtectedRoute({
    method: 'get',
    path: '/callback',
    permission: noAuth(),
    security: [],
    // ... rest of route config
  }),
  handler,
);

// Route where auth is enforced by parent middleware — use inheritedAuth()
app.openapi(
  createProtectedRoute({
    method: 'get',
    path: '/',
    permission: inheritedAuth({
      resource: 'organization',
      permission: 'member',
      description: 'Auth enforced by parent middleware in createApp.ts',
    }),
    // ... rest of route config
  }),
  handler,
);
```

**Key helpers:**
| Helper | When to use |
|---|---|
| `requireProjectPermission('view' \| 'edit')` | Routes scoped to a project |
| `requirePermission({ project: 'create' })` | Org-level permission checks (admin) |
| `noAuth()` | Truly public endpoints (webhooks, OAuth callbacks) |
| `inheritedAuth(meta)` | Auth enforced by parent/global middleware |
| `inheritedRunApiKeyAuth()` | Run-domain routes behind API key middleware |
| `inheritedManageTenantAuth()` | Manage-domain routes behind session/API key middleware |
| `inheritedWorkAppsAuth()` | Work-apps routes behind OIDC/Slack middleware |

### Common Gotchas
- **Empty Task Messages**: Ensure task messages contain actual text content
- **Context Extraction**: For delegation scenarios, extract contextId from task ID patterns like `task_math-demo-123456-chatcmpl-789`
- **Tool Health**: MCP tools require health checks before use
- **Agent Discovery**: Agents register capabilities via `/.well-known/{subAgentId}/agent.json` endpoints

### File Locations
- **API Domains**: `agents-api/src/domains/` (unified API with manage/, run/, evals/ domains)
- **Core Agents**: `agents-api/src/domains/run/agents/Agent.ts`, `agents-api/src/domains/run/agents/generateTaskHandler.ts`
- **A2A Communication**: `agents-api/src/domains/run/a2a/`, `agents-api/src/domains/run/handlers/executionHandler.ts`
- **Evaluations**: `agents-api/src/domains/evals/` (evaluation workflows, dataset runs, triggers)
- **Management Routes**: `agents-api/src/domains/manage/routes/` (agent config, projects, tools, credentials)
- **Database Layer**: `packages/agents-core/src/data-access/` (agents, tasks, conversations, tools)
- **Builder Patterns**: `packages/agents-sdk/src/` (agent.ts, subAgent.ts, tool.ts, project.ts)
- **Schemas**: `packages/agents-core/src/db/manage/manage-schema.ts`, `packages/agents-core/src/db/runtime/runtime-schema.ts` (Drizzle), `packages/agents-core/src/validation/` (Zod validation)
- **Tests**: `agents-api/src/__tests__/` (unit and integration tests)
- **UI Components**: `agents-manage-ui/src/components/` (React components)
- **UI Pages**: `agents-manage-ui/src/app/` (Next.js pages and routing)
- **Documentation**: `agents-docs/` (Next.js/Fumadocs public documentation site)
- **Legacy Documentation**: `docs-legacy/` (internal/development notes)
- **Examples**: `agents-cookbook/` for reference implementations

## Feature Development Examples

### Example: Adding a New Feature

#### 1. Unit Test Example
```typescript
// agents-api/src/__tests__/manage/newFeature.test.ts
import { describe, it, expect, beforeEach } from 'vitest';
import { NewFeature } from '../newFeature';

describe('NewFeature', () => {
  let feature: NewFeature;
  
  beforeEach(() => {
    feature = new NewFeature({ tenantId: 'test-tenant' });
  });

  it('should initialize correctly', () => {
    expect(feature).toBeDefined();
    expect(feature.tenantId).toBe('test-tenant');
  });

  it('should handle errors gracefully', async () => {
    await expect(feature.process(null)).rejects.toThrow('Invalid input');
  });
});
```

#### 2. Agent Builder UI Component Example
```tsx
// agents-manage-ui/src/components/new-feature/new-feature-form.tsx
import { useState } from 'react';
import { z } from 'zod';
import { Button } from '../ui/button';
import { Input } from '../ui/input';
import { Form } from '../ui/form';

const newFeatureSchema = z.object({
  name: z.string().min(1, 'Name is required'),
  config: z.object({
    enabled: z.boolean(),
    value: z.string().optional()
  })
});

export function NewFeatureForm() {
  const [formData, setFormData] = useState({});
  
  return (
    <Form schema={newFeatureSchema} onSubmit={handleSubmit}>
      {/* Form fields */}
    </Form>
  );
}
```

#### 3. Documentation Example
````mdx
// agents-docs/content/docs/features/new-feature.mdx
---
title: New Feature
description: Brief description of what the feature does
---

## Overview
Brief description of what the feature does and why it's useful.

## Usage
```typescript
const feature = new NewFeature({
  id: 'feature-1',
  config: { enabled: true }
});

await feature.execute();
```

## API Reference
- `NewFeature(config)` - Creates a new feature instance
- `execute()` - Executes the feature
- `validate()` - Validates configuration

## Examples
[Include practical examples here]
````

// Also update agents-docs/source.config.ts to include the new page in navigation

## Debugging Commands

### Jaeger / OTLP Tracing Debugging
Replace `service` with the current service name (e.g., `inkeep-agents-api` in prod, `inkeep-agents-api-test` in tests). If using SigNoz/OTLP, point to that host/port instead of localhost.

```bash
# Get all services
curl "http://localhost:16686/api/services"

# Get operations for a service
curl "http://localhost:16686/api/operations?service=inkeep-agents-api"

# Search traces for recent activity (last hour)
curl "http://localhost:16686/api/traces?service=inkeep-agents-api&limit=20&lookback=1h"

# Search traces by operation name
curl "http://localhost:16686/api/traces?service=inkeep-agents-api&operation=agent.generate&limit=10"

# Search traces by tags (useful for finding specific agent/conversation)
curl "http://localhost:16686/api/traces?service=inkeep-agents-api&tags=%7B%22agent.id%22:%22qa-agent%22%7D&limit=10"

# Search traces by tags for conversation ID
curl "http://localhost:16686/api/traces?service=inkeep-agents-api&tags=%7B%22conversation.id%22:%22conv-123%22%7D"

# Get specific trace by ID
curl "http://localhost:16686/api/traces/{trace-id}"

# Search for traces with errors
curl "http://localhost:16686/api/traces?service=inkeep-agents-api&tags=%7B%22error%22:%22true%22%7D&limit=10"

# Search for tool call traces
curl "http://localhost:16686/api/traces?service=inkeep-agents-api&operation=tool.call&limit=10"

# Search traces within time range (Unix timestamps)
curl "http://localhost:16686/api/traces?service=inkeep-agents-api&start=1640995200000000&end=1641081600000000"
```

### Common Debugging Workflows

**Debugging Agent Transfers:**
1. View traces: `curl "http://localhost:16686/api/traces?service=inkeep-agents-api&tags=%7B%22conversation.id%22:%22conv-123%22%7D"`

**Debugging Tool Calls:**
1. Find tool call traces: `curl "http://localhost:16686/api/traces?service=inkeep-agents-api&operation=tool.call&limit=10"`

**Debugging Task Delegation:**
1. Trace execution flow: `curl "http://localhost:16686/api/traces?service=inkeep-agents-api&tags=%7B%22task.id%22:%22task-id%22%7D"`

**Debugging Performance Issues:**
1. Find slow operations: `curl "http://localhost:16686/api/traces?service=inkeep-agents-api&minDuration=5s"`
2. View error traces: `curl "http://localhost:16686/api/traces?service=inkeep-agents-api&tags=%7B%22error%22:%22true%22%7D"`

## AI Coding Assistant Guidance

`CLAUDE.md` is a symlink to `AGENTS.md`: edit `AGENTS.md`. AGENTS.md may load in part or not at all.

## Cursor Cloud specific instructions

Non-obvious caveats for Cursor Cloud agents bringing this subtree up in an ephemeral Linux VM (standard commands are documented above):

- Node: `nvm use default` (22.22.2) before pnpm; the VM base `/exec-daemon/node` (22.14.0) fails `engine-strict` (`agents-core` needs `>=22.18.0`). Login shells select it via `~/.bashrc`.
- Docker (Doltgres `:5432`, Postgres `:5433`, SpiceDB `:50051`, Mailpit) is preinstalled but not auto-started: `sudo dockerd &` then `sudo chmod 666 /var/run/docker.sock`. `/etc/docker/daemon.json` is preset for `fuse-overlayfs` (containerd snapshotter disabled, required on this kernel).
- Build `@inkeep/agents-core` before `pnpm setup-dev` (setup imports its `dist/setup`). Then `pnpm dev` serves API `:3002` (`/health` 204) and Manage UI `:3000` (local-dev auto-login `admin@example.com`).
- `.env` needs `ANTHROPIC_API_KEY`; a placeholder unblocks DB/API/UI + project CRUD, real agent chat needs a valid key. `setup-dev`'s sample-project push may fail (`inkeep` bin not resolvable) but is non-fatal.