--- name: databuddy-internal description: Work inside the Databuddy monorepo for internal implementation, debugging, review, and refactoring. Use only for repository code changes across dashboard, api, basket, links, docs, uptime, SDK, tracker, auth, RPC, database schema, ClickHouse, or shared packages. Do not use for external SDK, API, CDN, feature flag, or LLM observability integration guidance; use databuddy instead. --- # Databuddy Internal Databuddy is a Bun + Turborepo TypeScript monorepo. Start by locating the user request in one product surface, then trace its shared dependencies before editing. For **external** integrations (SDK, CDN, public APIs), use the **`databuddy`** skill; this skill is for **this repository**. ## Skill maintenance (required) When a mistake could have been avoided with better repo context (wrong app, package, port, or pattern), or when the user corrects you or asks you to fix something you got wrong, **update this skill** (`SKILL.md` or `references/codebase-map.md`) in the same turn when practical. Keep additions **minimal**: one bullet, a new `rg` hint, or a routing note—enough that the next session does not repeat it. If the lesson is for SDK/API customers, add it under `.agents/skills/databuddy/` instead. - Repo-local `.agents/skills` is versioned project guidance. Put new personal or experimental agent skills under `/Users/iza/.agents/skills` unless the user explicitly wants the skill committed with Databuddy. ## Quick Map - Work directly in the local `staging` checkout by default. Task branches, worktrees, and PRs require an explicit user request; preserve existing local edits. - Self-host changes must preserve hosted behavior with `SELFHOST` unset or `false`, including auth cookies/email, generated snippets, CSP, and image publication. Compare to the pre-change path; import guard-only helpers from `@databuddy/env/boolean` so checking the mode does not initialize unrelated URL config. - Before any PR merge, follow the AGENTS.md review-feedback gate: wait for configured reviewers on the final head, read all comment/review/thread pages, address each finding with evidence, and re-fetch to verify no unresolved feedback. Review bots can finish several minutes after a draft becomes ready; green CI does not establish completed review. - Prod infrastructure repo is local at `/Users/iza/Documents/GitHub/databuddy-infra` (`databuddy-analytics/infra`); ClickHouse cluster inventory is `clickhouse/ansible/inventory.yml`, not `/Users/iza/Dev/Databuddy/infra` or `DatabuddyOPS`. - Never use production/customer data as tests, fixtures, snapshots, examples, or copied output. Tests must use placeholders/mocks only (example.com, example IDs). If production ClickHouse is queried for investigation, summarize anonymized aggregates and do not paste customer domains, client IDs, emails, or other identifiers into code or responses. - `@databuddy/test/env` targets local `databuddy_test` unless `CI=true`, so a normal `db:push` may update a different database; sync that test database explicitly before debugging removed-column failures. - `apps/dashboard`: Next.js app on port `3000` (per-website **agent** chat: `@ai-sdk/react` `useChat` via `contexts/chat-context.tsx` — not the separate `chat-sdk` package; overlapping sends while streaming are queued client-side to mirror a “queue latest” strategy.) - Dashboard Playwright webServer commands run under CI PATH from setup-bun; avoid `bash -lc` because login shells can drop Bun from PATH. Build dist-only workspace packages such as `@databuddy/sdk` and `@databuddy/devtools` before starting the API/dashboard. Client `NEXT_PUBLIC_*` flags must use direct env access so Next can inline them. `readBooleanEnv` only treats the literal string `"true"` as enabled, so CI E2E booleans must use `"true"`/`"false"`, not `"1"`/`"0"`. - Dashboard pages that call `useSearchParams()` must keep that call inside a Suspense-rendered child; run the production dashboard build after adding URL-driven onboarding state. - Business context loads through `OrganizationProvider` before its page query and generation-access check; keep all three states on the same page/card geometry and test their transitions with delayed requests on desktop and mobile. - Business-context generation runs in the API; verify `AI_GATEWAY_API_KEY`, `CONTEXT_DEV_API_KEY`, and hosted `AUTUMN_SECRET_KEY` on the serving API deployment. Worker credentials and a green `/health` do not establish feature readiness. - Failed business-context generations survive refresh; dismiss them through the existing generation-ID-scoped cancel mutation while preserving unsaved editor drafts. Hiding the message in component state is not durable. - Playwright launched through Bun still uses the Node on PATH. CI installs Node 24; local Node 20 lacks `Promise.withResolvers`, so keep deferred E2E requests on portable Promise constructors. - The local `.env` sets `NODE_ENV=development`; override it with `NODE_ENV=production` when verifying `next build`, or root-error prerender behavior can produce a misleading local failure. - Local dashboard E2E tests that need `/api/test/e2e/*` should start the API/dashboard directly (or through Playwright's webServer command), not via `bun run dev:dashboard`; Turbo runs in strict env mode and drops `DATABUDDY_E2E_MODE`/`DATABUDDY_E2E_TEST_KEY` unless they are added to `turbo.json` `globalEnv`. - Dashboard Playwright public/demo analytics specs call API `/v1/query` anonymously from the browser; keep `DATABUDDY_E2E_MODE` query behavior isolated from production rate limits so CI retries do not exhaust `anon:unknown`. - `apps/api`: Elysia API on port `3001` - API tests use Vitest through `bun run test` inside `apps/api`; use Vitest test imports rather than `bun:test` in that package. - Public REST docs live in `apps/api/src/rpc/openapi.ts`: `/spec.json` is the generated spec, `/` is the reference UI, and hiding a router there also makes its top-level REST paths return 404 because `/*` uses the same filtered docs router. - `apps/slack`: Slack agent adapter; Slack installs resolve through org-scoped DB integration records, not a single env bot token/default website. Agent calls use the org-scoped internal principal synthesized from the active integration in `slack/installations.ts`, never a global internal secret. - Slack OAuth lives in `apps/api`, but slash commands/events require `apps/slack` to be running too; local `bun run dev:dashboard` runs dashboard + API only, so use `bun run dev:slack` when working on Slack. The Slack package scripts read the root `.env`. - Run Slack tests through `bun run test` inside `apps/slack`; the package script supplies only the inert Redis URL needed by eager shared imports. - Inspect root `package.json` overrides before interpreting Slack dependency manifest versions or apparent lockfile drift; the effective `@slack/web-api` version may be pinned there. - Slack routing is organization-scoped: OAuth binds a Slack workspace to a Databuddy organization, app mentions from the installed workspace auto-bind channels including Slack Connect, and `/bind` is now a manual fallback for unknown/unapproved channels. DMs/assistant threads work after workspace install. Analytics questions should go through app mentions/DMs using MCP-style website discovery inside the installed organization, never by fanning out across the message sender's user memberships. Slack emits evlog events under `apps/slack/.evlog/logs` in development/`SLACK_EVLOG_FS=1`; Axiom uses `AXIOM_TOKEN` and the `slack` dataset; reactions need the `reactions:write` bot scope. - Model evaluations should use the user’s agreed quality, latency, and cost tradeoff; an explicitly accepted accuracy regression is not an adoption veto. Keep authorization and execution validation deterministic, and report fresh held-out results with failures and model overhead included. - Slack scope changes require reinstalling/reauthorizing the workspace; updating the local/remote manifest alone does not grant newly-added bot scopes to an existing installation. - Slack agent billing flows through an org-scoped automation API key; existing keys may have `userId: null`, so the agent billing resolver must fall back to the organization owner when an API key has `organizationId`. - Slack memory is separate from billing/auth: pass a Slack-scoped `memoryUserId` such as `slack-{team}-{user}` plus current-speaker context so one Slack user's saved name/preferences do not bleed into another user's replies. - Slack agent tools use the scopes on the internal principal in `slack/installations.ts`; changing those Databuddy scopes applies without Slack reauthorization. Only changes to Slack OAuth scopes require reconnecting the workspace. - Shared agent integrations should call `@databuddy/ai/agent` (`askDatabuddyAgent` / `streamDatabuddyAgent`) instead of importing internal MCP run/history helpers directly. - Keep the Jev evaluation request in its owning classifier; do not add a provider-wrapper file for a single call. Slack message types already live in `ai/mcp/slack-context.ts`. - The shared conversational MCP runner (including Slack) uses native goal/link tools through `createMcpAgentTools`; the external MCP server has separate wrappers in `mcp/workspace-tools.ts`. - First-party ads attribution work should start by preserving UTMs into registration and signup events only; do not add RPC plumbing, conversion destinations, env hooks, tables, workers, or UI until explicitly needed. - Organization business-context drafting is an active streaming request owned by `apps/api/src/ai/organization-business-context.ts`, injected into oRPC by the API host. Its request abort cancels only the matching generation; consumed usage settlement stays independent. Investigation generation stays in `apps/insights`. - Insights generation logic belongs in `apps/insights` and should reuse `@databuddy/ai`; `apps/api` should only read insight data or queue runs, not own prompts, model calls, tool loops, validation, or persistence orchestration. - `SPEC.md` is the intelligence product contract. `insight_observations` is the readable Insights history; `analytics_insights` is the durable investigation projection. The agent outcome owns brief publication and `act`/`ask` promotion; do not replace either with frontend heuristics or collapse the feed into cases. Do not add a parallel agent, evidence API, fixed query choreography, or action-specific lifecycle. - Insights quality reviews must compare fresh baseline/candidate outputs and lead with the product verdict and concrete examples. Score usefulness, noise, reading effort, and retained useful findings separately from code tests and contract passes; preserve interrupted attempts instead of reporting retries as an uninterrupted pass rate. - Insights RPC helpers that take `{ context, ...input }` must strip `context` before parsing a `.strict()` Zod input schema (same pattern as `appendInvestigationReply` / `applyInsightGoalAction`); otherwise CI fails with `Unrecognized key: "context"`. - `insights.history` / MCP `list_investigations` hide cases while analysis or verification is queued/running; included clarifications use saved evidence and must not hide or mutate the case. - When reporting what an organization can see in Insights, follow the `insights.brief`/`history` visibility rules instead of counting `analytics_insights`; the projection can contain legacy rows without a readable or published `insight_observations` turn. - Production insight shadows must freeze `--reference-time`, retain a tool-name trace, and pass available GitHub context before supporting quality claims. Postgres and ClickHouse are read-only, but connector token refreshes or cache writes can still occur; never describe the whole run as zero-write. - Automatic investigations have one organization-wide schedule (`off`, `daily`, or `weekly`) and one organization-wide delivery set; website selection is only for manual runs. Do not reintroduce per-website overrides, hourly/custom cadence, or cron input. - A manual insight run is a deliberate recheck: it bypasses automatic cooldown only for currently detected signals, while retaining detector thresholds and normal signal ranking. Otherwise “Run now” can complete without producing an evaluable result. - Insight run items are execution metadata, not rendered insight content; previews should use run status/counts or query real insights, never infer titles or bodies from run items. - Insight Slack delivery must resolve each channel binding to its active same-organization integration; never choose an arbitrary organization bot token. - Replies beneath delivered Slack investigations must resolve the delivery and enter the existing durable reply/resume path; never route them through generic Slack chat or relevance scoring. - One-off insight previews must preserve the real signal entity and use customer-facing product output. Never hand-write Slack copy from eval metadata or expose evaluation and suppression mechanics. - Agent ClickHouse SQL must use the canonical analytics.events schema: `client_id`, `time`, `path`, `event_name`, and pageviews as `event_name = 'screen_view'`; never `website_id`, `created_at`, `page_path`, `event_type`, or `pageview`. - Agent `get_data` filters select rows; do not expose SQL CTE `target` or `having` as generic event/error scopes. Query discovery supplies accepted selectors, and result row counts describe the query output rather than a complete population. - Slack agent expected stops such as exhausted Databunny credits should throw `DatabuddyAgentUserError` from `@databuddy/ai/agent/errors`; Slack surfaces those messages directly and reserves the generic reconnect copy for real infrastructure failures. - Slack Docker builds use `bun build --compile --bytecode`; keep `apps/slack/src/index.ts` bootstrapping inside an async `main()` instead of top-level `await`, which can fail during compile even when typecheck passes. - Insights Docker builds also use `bun build --compile --bytecode`; keep `apps/insights/src/index.ts` startup work inside async functions instead of top-level `await`. - After Slack Docker changes, verify the full pruned image with `docker build --progress=plain -f slack.Dockerfile -t databuddy-slack:test .`; the inner Bun compile is not enough because prune can miss dependency build outputs and package exports. - Slack-reachable shared packages (`@databuddy/ai`, `@databuddy/rpc`) must not import `evlog/elysia`; use host-injected request logger providers from the API and plain evlog fallbacks elsewhere. - AI link tools must assign link folders by existing folder `id` or `slug` only; folder names are display text and must not be used for routing or dedupe. - `apps/basket`: ingest and LLM tracking service, Elysia app on port `4000` - Basket tests use Vitest; run them through `bun run test` inside `apps/basket`, not `bun test` directly. - `apps/docs`: Next.js + Fumadocs docs app on port `3005` - When a user drops a prototype, remove only prototype-specific wiring and preserve the existing product surfaces it temporarily reused. - `apps/links`: redirect/link service - `apps/uptime`: uptime monitoring service - `apps/uptime` BullMQ worker concurrency defaults high for Bun async I/O; do not lower it just because `10_000` looks large. Verify downstream saturation or lock/timeout evidence first. - Public status pages render from `apps/status`; `apps/dashboard` owns status-page management/config UI only. When cleaning public status UX, update shared `@databuddy/ui/uptime` pieces or `apps/status` wrappers instead of redesigning dashboard-only route remnants. - `packages/db`: Drizzle Postgres schema, client, and ClickHouse helpers - Keep Bun-only DQL provisioning code off the `@databuddy/db/clickhouse` barrel; dashboard Next routes run under Node and import the shared ClickHouse surface. - `packages/rpc`: shared oRPC router, procedures, auth-aware server context - `packages/rpc` must declare `drizzle-orm: "catalog:"` before importing `drizzle-orm/*` helpers such as `drizzle-orm/zod`; otherwise TypeScript can resolve a different Drizzle instance than `@databuddy/db` and reject table-derived schemas. - `packages/auth`: Better Auth setup, permissions, organization access - `packages/env`: shared URL, public, and boolean environment helpers - `packages/shared`: shared types, flags, analytics schemas, utilities - Analytics query builders live in `packages/ai/src/query`; there is no standalone `packages/query` directory. Tests are excluded from root Biome checks, so format changed test blocks explicitly. - `apps/insights` uses the ES2022 TypeScript library; Bun supporting `findLast` or newer Set methods does not make those APIs typecheck here. Check the package TS library before choosing newer built-ins. - `packages/sdk`: published analytics SDK for React, Vue, and Node - `packages/tracker`: internal tracker script build and release package - `packages/encryption`, `packages/notifications`, `packages/cache`, `packages/redis`, `packages/services`, `packages/validation`, `packages/api-keys`: shared infra and domain packages - Knip is configured in root `knip.json` (run `bun run knip`); per-workspace test globs are required because the root `test:watch` (`bun test --watch ./apps`) script shadows the Bun plugin's per-workspace script parsing; `apps/cron` is ignored (standalone scripts, no package.json) Read [codebase-map.md](./references/codebase-map.md) when you need deeper routing guidance. ## Workflow 1. Identify the runtime surface first: dashboard UI, API, ingest pipeline, docs site, tracker, or shared package. 2. Read the owning package's `package.json`, entrypoint, and direct dependencies before changing code. 3. If the change crosses app boundaries, trace the contract: `dashboard -> apps/dashboard/lib/orpc.ts -> packages/rpc -> apps/api` 4. If the change touches analytics ingestion or LLM observability, trace: `packages/sdk` or `packages/tracker` -> `apps/basket` -> `packages/db` / ClickHouse 5. If the change touches auth, org permissions, or session-aware server behavior, inspect `packages/auth` and `packages/rpc` together. 6. Validate with the smallest relevant command instead of running the whole monorepo by default. ## Repo Conventions - Package manager: `bun` - For ad-hoc read-only data scripts, run from an owning package and import the client through `@databuddy/db`; a root-level Bun eval cannot resolve workspace or transitive dependencies directly. - When running `bun install --lockfile-only`, preserve lockfile sync for pre-existing `package.json` changes instead of reverting them as unrelated. - Task runner: `turbo` - Run filtered Turbo commands such as `bun run check-types --filter=…` from the workspace root; inside a package, its local script invokes `tsc` directly and treats those flags as TypeScript options. - Formatting/linting: `bun run format`, `bun run lint` - Use neutral branch names, commit messages, and PR copy; do not include tool-attribution prefixes or generated-by language. - When the user names an integration branch such as `staging`, verify `HEAD`, the intended commit set, and upstream divergence on that branch before declaring the work complete. - Lefthook's `no-secrets` guard intentionally ignores the exact `.env.example` template; real `.env`, `.env.*`, key, and credential files should still be blocked. - Root dev orchestration: `bun run dev` - Dashboard + API together: `bun run dev:dashboard` - Tests at root currently target `./apps`: `bun run test` - Package test mocks can leak across files in one Bun process; when `packages/ai` broadly mocks `@databuddy/redis`, keep the mock export surface in sync with runtime imports from shared RPC/tool code. - Preserve tool input types in test helpers with `Tool`; do not cast mixed tool inputs to `never` to suppress union errors. AI package tests are excluded from its normal type check, so check changed test types explicitly. - Import new shared tool helpers directly from their defining modules; the repository prohibits adding barrel exports even when an existing barrel is nearby. - Database scripts are routed from root into `packages/db` - Runtime environment reads stay in the owning service; shared URL/public helpers live in `packages/env` - BullMQ queues use `BULLMQ_REDIS_URL`; generic Redis cache/pubsub code uses `REDIS_URL`. ## Code Standards - Keep one source of truth. If output is AI-generated copy, semantic labels, summaries, or recommendations, fix the upstream prompt/schema/validation contract; do not patch it later with frontend regex/string heuristics. - When an undescribed goal or funnel combines unrelated workflows, explain its broad scope and end a published insight with one concrete evidence-backed edit, rename, create, use, or cleanup recommendation even when resolving it. Recommend deletion only when evidence shows duplication or no distinct use. An explicit purpose/config mismatch is material decision harm and becomes an action; ambiguity alone does not. - Start small backend slices with one implementation file per owning package; split only when a concrete independent consumer or lifecycle appears. - Use deterministic transforms only for deterministic data: stable enums, IDs, namespaces, routes, schema fields, and typed status values. Do not guess meaning from free-form model/user text with regexes. - Prefer structured contracts over text parsing. If the UI needs a label, action, link, severity, or metric category, add it to the schema/tool output and validate it at the boundary. - Keep domain concerns at the owning seam. Routers/UI should call domain/service helpers, not know cache keys, raw Redis patterns, billing internals, or provider-specific lifecycle details. - Prefer direct, boring code. Use typed registries and small local helpers when they delete duplication; avoid generic job/facade abstractions, labeled pipelines, or framework-y wrappers unless they clearly reduce code and concepts. - Test invariants and contracts, not implementation trivia. Add guard tests for architectural rules only when they prevent repeat classes of bugs. - During cleanup, do not add per-consumer tests or test-only modules for schemas already covered at their shared boundary; keep one real end-to-end boundary regression when wiring itself changed. ## Change Routing ### Dashboard work - Start in `apps/dashboard` - Organization settings should match `organizations/components/general-settings.tsx`: the same `max-w-2xl` column, shared `Card` headers/content, compact fields, and `TopBar.Actions` for Save. Compare actual neighboring pages visually before claiming design consistency; shared inputs alone are not enough. - The Feature Flags list is a dense data table; keep flag identity metadata clean and place activity telemetry in a dedicated labeled column with its accuracy caveat in the Activity header tooltip. - Keep large page-owned settings sheets and dialogs in adjacent feature files; route components should own page data and layout rather than embedding unrelated form lifecycles. - For dashboard navigation audits, check all route surfaces: `components/layout/navigation/navigation-config.tsx`, `components/ui/command-search.tsx`, and local `PageNavigation` layouts under `app/**/layout.tsx` before calling a page orphaned. - When fixing broken dashboard links to moved sections, update the real docs/search/navigation links and section anchors directly; do not add compatibility redirect pages unless explicitly requested. - Custom events UI is shared in `apps/dashboard/components/events/custom-events`; keep many-series legends outside the Recharts plot, use compact controls for property-summary event selection, and avoid separate event-count chip/list sections. - Goals and Funnels are sibling conversion surfaces; keep Goals list-first and visually aligned with `app/(main)/websites/[id]/funnels` instead of adding separate summary-card chrome. - Funnel rows keep the action menu outside the main toggle button; put row padding on the sibling `Button`, not only on `List.Row`, so the visible row surface is clickable without nesting buttons. - Demo website navigation must be public-safe and route-backed; hide sensitive, configuration-heavy, or unavailable website features such as Agent, Feature Flags, Revenue, Users, Realtime, and website Settings instead of inheriting the full website nav. Goals and Funnels may be public demo surfaces, but keep them read-only. - Dashboard definitions for feature flags and target groups are admin surfaces; do not expose even sanitized rows to demo-tier/public website access. - Billing settings must render details only while their switch is on; hidden grid rows can retain blank height and focusable controls. Autumn `overageLimit` is additional feature units, not USD. Use the native `databunny_chat` flag to hide legacy credit purchases and guards for included-chat accounts; an investigation balance alone does not establish chat terms. - Insights history is grouped by its backend-owned subject key in the RPC layer so every client sees one current row per investigation; reads must not invoke AI generation. - An executable Insight action must apply its stored mutation and enqueue verification transactionally; never ask a teammate to mark it done manually. Show a verified result only after measured evidence supports it. - Insights case pages should not explain their own structure (for example, "context, investigation, and verification in one thread"). Keep only evidence, current state, and actionable controls visible. - Insight visual cleanup must preserve scan hierarchy: retain the compact visual/status anchors and clearly separated proof on dense feed rows; remove redundant explanatory copy before flattening the information structure. - Theme: `apps/dashboard/app/globals.css`. **`--border` is intentionally subtle**; do not crank it darker for “contrast” unless **iza** asks—prefer text tokens or layout for readability. - Website analytics filters are two-way synced between Jotai and the `filters` URL param in `app/(main)/websites/[id]/layout.tsx`; guard URL-driven atom writes from echoing stale atom state back into `nuqs`, or adding a filter can lock the page during form submit. - Do not centralize, relocate, or otherwise refactor dashboard E2E API route access gates during cleanup; keep test-only access checks local to each route unless iza explicitly asks for that change. - Integration catalog logos: use filled Simple Icons SVG path data (or equivalent filled brand SVG), store the path on each item as `iconPath`, render it through a shared logo tile with `bg-secondary/60`, `border-border/70`, `text-foreground`, and `fill="currentColor"`, then use brand color only as a small accent bar (`accent` or `accentClassName: "bg-foreground/70"` for black/near-black brands). Avoid raw brand-black icons or mixed line/filled icon sets that disappear in dark mode. - Organization integrations settings should stay list-first and operational: coming-soon integrations are static rows, Slack is the only expandable row for now, and connected integrations need obvious lifecycle controls such as uninstall/disconnect in the row details. - MCP setup UI should mirror the governed write metadata in `packages/ai/src/ai/mcp/tools.ts`: default to `read:data`, then expose explicit action bundles for workspace actions, feature flags, and short links with their required scopes and confirmation behavior. - Dashboard primitives now live in `packages/ui/src/components` and are consumed through `@databuddy/ui` (`@databuddy/ui/client` for client-only components); the old `apps/dashboard/components/ds/README.md` path no longer exists. Read the corresponding shared component implementation before extending UI. Dashboard UI must use these shared primitives exactly; feature code must not use raw form/control elements (`button`, `input`, `select`, `textarea`, native dialogs), Base UI/Radix primitives, or ad hoc styled controls directly. If a variant is missing, add or extend the DS component first. For menu-style folder/status/filter/sort/action pickers, use `components/ds/dropdown-menu.tsx`; use `Select` only when the established pattern is explicitly a select/combobox. Inspect the shared component APIs in `packages/ui/src/components` before creating new dashboard UI. - `DropdownMenu.GroupLabel` must be rendered inside `DropdownMenu.Group`; Base UI throws `MenuGroupRootContext is missing` when labels are placed directly under `DropdownMenu.Content`. - Traffic Trends chart annotations should use a chart-adjacent annotation rail for dense data; avoid in-plot labels, tall lines, or floating dots that compete with the chart tooltip/data layer. - Flags list rows (`app/(main)/websites/[id]/flags/_components/flags-list.tsx`) are clickable containers with nested controls; mark nested controls with `data-row-interactive="true"` and have the row ignore those targets instead of relying on broad cell-level `stopPropagation`. - Never put interactive controls inside another `