# AGENTS.md ## Project Overview LiveKit Agents for Node.js — a TypeScript framework for building realtime, multimodal, and voice AI agents that run on servers. This is the Node.js distribution of the [LiveKit Agents framework](https://github.com/livekit/agents) (originally Python). ## Monorepo Structure - **`agents/`** — Core framework (`@livekit/agents`). Agent orchestration, LLM/STT/TTS abstractions, voice pipeline, metrics, telemetry, IPC/process pooling, and the CLI. - **`plugins/`** — Provider plugins (`@livekit/agents-plugin-*`). Each implements one or more of: LLM, STT, TTS, VAD, EOU (end-of-utterance), Realtime, or Avatar. - **`examples/`** — Example agents (private, not published). Run with `pnpm build && node ./examples/src/.ts dev`. - **`tests/e2e/`** — End-to-end tests via Docker (separate from unit tests co-located in each package). **Tooling:** pnpm 11 workspaces, Turborepo for builds, tsup for bundling (CJS + ESM), TypeScript 5.4+, Vitest for tests, Changesets for versioning. ## Common Commands ```bash pnpm build # Build all packages (turbo) pnpm build:agents # Build only @livekit/agents and its deps pnpm build:plugins # Build only plugins and their deps pnpm clean:build # Clean dist/ dirs then rebuild pnpm test # Run all tests (vitest) pnpm test agents/src/llm # Run tests by path pnpm test -t "chat context" # Run tests by name pnpm test:watch # Watch mode pnpm lint # ESLint all packages pnpm lint:fix # ESLint with auto-fix pnpm format:check # Prettier check pnpm format:write # Prettier format pnpm api:check # API Extractor validation pnpm api:update # Update API declarations ``` ### Running an example agent ```bash pnpm build && node ./examples/src/basic_agent.ts dev --log-level=debug ``` Required env vars: `LIVEKIT_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET`, plus provider keys (e.g. `OPENAI_API_KEY`). > **Rule (must follow before handing off to the user to run any example/agent file):** > Always run `pnpm build` first and iterate on TypeScript / dependency errors until it exits 0. Never tell the user to `node ./examples/src/.ts` (or any `.ts` entrypoint) until the build succeeds — this includes after editing example files, after modifying framework code that breaks `tsc`, and after copy-pasting reproductions from external sources. If the build fails, fix the errors yourself (add missing deps, fix outdated API usage, update types) before asking the user to run anything. ### Pre-downloading plugin assets (for Docker builds) `@livekit/agents` ships a `livekit-agents` bin with a `download-files` subcommand that discovers installed `@livekit/agents-plugin-*` packages from `node_modules` and runs each plugin's `downloadFiles()` (e.g. HuggingFace models for the turn detector) — without loading the user's agent code. Use it in Dockerfiles to separate dependency installation from app code so the download layer can be cached: ```dockerfile COPY package.json pnpm-lock.yaml ./ RUN pnpm install --frozen-lockfile RUN npx livekit-agents download-files COPY src/ ./src RUN pnpm build ``` The `download-files` subcommand on `cli.runApp()` is deprecated for this purpose since it requires loading the agent file first. ### Debugging individual plugins Create a test file prefixed with `test_` in `examples/src/`. No `defineAgent` wrapper needed — just import the plugin directly and run: ```bash pnpm build && node ./examples/src/test_my_plugin.ts ``` ## Architecture ### Voice Pipeline (`agents/src/voice/`) The core of the framework. Pipeline flow: **Audio In → VAD → STT → LLM → TTS → Audio Out**. Key classes and their roles: - **`Agent`** — Base class holding instructions, tools, and model config. Subclass to override pipeline hooks (`ttsNode`, `realtimeAudioOutputNode`, `sttNode`, `llmNode`). Use `voice.Agent.default.(this, ...)` to call the base implementation from overrides. - **`AgentTask`** — Extends Agent for composable task workflows with isolated chat context and tool scoping. - **`AgentSession`** — Orchestrates the full session lifecycle: connects to LiveKit room, manages turn detection, handles interruptions, collects metrics. Entry point: `session.start({ agent, room })`. - **`AgentActivity`** (`agent_activity.ts`, ~100KB) — Complex state machine managing individual turns: VAD trigger → STT → endpointing → LLM inference → TTS generation → playout. Supports preemptive generation (starts LLM while user still speaking). - **`SpeechHandle`** — Represents a unit of agent speech with lifecycle tracking and priority levels (`LOW=0`, `NORMAL=5`, `HIGH=10`). Sources: `'say'`, `'generate_reply'`, `'tool_response'`. - **`AvatarSession`** — Base class for avatar plugin sessions. Registers `aclose()` as a job shutdown callback and warns when started after `AgentSession.start()` (which would replace the existing audio output). Plugin implementations extend this class and call `super.start(agentSession, room)` first. Turn detection modes: `"stt"` | `"vad"` | `"realtime_llm"` | `"manual"` (configured in `turn_config/`). Interruption detection: `"adaptive"` (ML-based via `inference/interruption/`) or `"vad"` mode. Subdirectories: `room_io/` (LiveKit Room I/O), `transcription/` (word-level sync), `avatar/` (avatar integration), `recorder_io/` (recording), `testing/` (test utilities), `turn_config/` (turn/interruption/endpointing config). ### LLM (`agents/src/llm/`) - **`ChatContext`** — Chronologically ordered conversation state. Supports action-aware history summarization via `_summarize(llm, { keepLastTurns })` to compress old messages while preserving tool execution results. - **Tool system** — `llm.tool()` with Zod schemas for parameters. `handoff()` for multi-agent transfers. - **`FallbackAdapter`** — Agent-side multi-provider LLM failover with availability tracking. - **Provider format adapters** — `provider_format/openai.ts`, `provider_format/google.ts`, and `provider_format/mistralai.ts` for model-agnostic code. ### Other Core Modules - **STT** (`stt/`): `SpeechStream` with automatic retry. `FallbackAdapter` for agent-side multi-provider failover. `StreamAdapter` converts non-streaming STT + VAD to streaming. - **TTS** (`tts/`): `SynthesizeStream`, `ChunkedStream`. `FallbackAdapter` for agent-side multi-provider failover. `StreamAdapter` for non-streaming providers. - **VAD** (`vad.ts`): Voice Activity Detection interface. Silero plugin is the primary implementation. - **Inference** (`inference/`): LiveKit Inference Gateway clients (LLM, STT, TTS, realtime) with server-side fallback support. Always use full `provider/model` format (e.g., `'openai/gpt-4o-mini'`). Also includes `interruption/` for adaptive interruption detection via ML models. - **Stream** (`stream/`): Composable Web Streams API primitives (`StreamChannel`, `DeferredStream`, `MultiInputStream`). - **IPC** (`ipc/`): Process pool for running agents in child processes. Two-way IPC: child sends inference requests back to parent. - **Worker** (`worker.ts`): Main process connecting to LiveKit server, receives job assignments, spawns agent processes. - **Telemetry** (`telemetry/`): OpenTelemetry tracing with custom span attributes (TTFT, TTFB, interruption probability, speech IDs, provider request IDs). Pino transport for structured logging. - **Metrics** (`metrics/`): `AgentMetrics` union type covering LLM, STT, TTS, VAD, EOU, Realtime, and Interruption metrics. `ModelUsageCollector` for aggregating per-provider usage. - **Beta** (`beta/`): `TaskGroup` for multi-task orchestration with optional chat context summarization between tasks. ### Remote Sessions (`voice/remote_session.ts`) Wire protocol for distributed agents. `SessionTransport` abstraction with `RoomSessionTransport` (LiveKit room channels) and `TcpSessionTransport` (raw TCP socket) implementations. ### Plugins (`plugins/`) Each extends `Plugin` base class, auto-registers on import via `Plugin.registerPlugin()`. Pattern: `@livekit/agents-plugin-`. Plugin capabilities by type: - **LLM**: openai, google, baseten, mistralai - **STT**: deepgram (v1+v2), openai, baseten, sarvam (v1/v2/v3), mistralai, inworld, cartesia - **TTS**: cartesia, elevenlabs, deepgram, openai, neuphonic, resemble, rime, inworld, baseten, sarvam (v1/v2/v3), mistralai, fishaudio, hume, speechify - **VAD**: silero (ONNX-based, local) - **EOU/Turn Detection**: livekit (HuggingFace + ONNX) - **Realtime**: openai (+ responses/, ws/ modules), google, xai, phonic - **Avatar**: trugen, lemonslice, bey, anam, liveavatar, did, protoface - **Test mocks**: test (private, for unit tests) ### AsyncLocalStorage Patterns The framework uses Node.js `AsyncLocalStorage` for implicit context passing: - `agentActivityStorage` — Access current `AgentActivity` in callbacks - `functionCallStorage` — Access current `FunctionCall` in tool handlers - `speechHandleStorage` — Access current `SpeechHandle` ## Code Conventions - **Lifecycle changes**: Follow the [lifecycle review checklist](CONTRIBUTING.md#review-lifecycle-changes) for session helpers, shutdown, and cleanup. Apply it to Python ports too. - **License header** required on every new file: ``` // SPDX-FileCopyrightText: 2026 LiveKit, Inc. // // SPDX-License-Identifier: Apache-2.0 ``` - **Prettier**: single quotes, trailing commas, 100 char width, sorted imports (`@trivago/prettier-plugin-sort-imports`). - **ESLint**: `@typescript-eslint` with strict rules. Prefix unused vars with `_`. Use `type` imports (`consistent-type-imports`). - **TypeScript**: strict mode, `noUncheckedIndexedAccess`, `verbatimModuleSyntax`, target ES2022, module node16. - **Time units**: Use milliseconds for all time-based values by default. Only use seconds when the name explicitly ends with `InS`. - **Changesets**: All packages in `agents/` and `plugins/` release together (fixed versioning). Run `pnpm changeset` to add a changeset before PRing. The examples package is ignored. - **API Extractor**: Public API surface is tracked. Run `pnpm api:check` after changing exports and `pnpm api:update` to update declarations. ## Testing - **Framework**: Vitest with 5s default timeout. - **Pattern**: `*.test.ts` files co-located with source. - **Snapshots**: Used in LLM chat/tool context tests (`agents/src/llm/__snapshots__/`). - **Inference LLM tests**: Always use full model names from `agents/src/inference/llm.ts` (e.g. `'openai/gpt-4o-mini'`, not `'gpt-4o-mini'`). Initialize logger first: `initializeLogger({ pretty: true })`. - **Test plugin**: `@livekit/agents-plugins-test` provides mock LLM, STT, TTS for unit tests without external APIs. - **STT testing utilities**: `stt.testing.FakeSTT` (from `@livekit/agents`) provides a configurable test harness for unit testing STT infrastructure (e.g. `FallbackAdapter`) with scripted transcripts, exceptions, timeouts, and observability channels. - **PR validation for major changes**: Verify `restaurant_agent.ts` and `realtime_agent.ts` work properly in [Agent Playground](https://agents-playground.livekit.io). ## Porting from Python (`livekit-agents`) When porting features or fixes from the Python `livekit-agents` repo to this JS/TS repo, follow these rules: ### Time unit unification Python uses **seconds** (`float`) for all time values. JS/TS uses **milliseconds** (`number`) by default. When porting a Python time parameter: - Multiply the Python default by `1000` for the JS default (e.g. `3.0 s` → `3000 ms`) - Use `setTimeout` / `clearTimeout` directly with the ms value — do **not** multiply by `1000` at call sites - Name the field in plain form (e.g. `aecWarmupDuration`, `userAwayTimeout`) — the ms convention is implied - Only use seconds as the unit if the variable name explicitly ends with `InS` (e.g. `delayInS`) Example mapping: | Python | JS/TS | | ------------------------------------------------- | ------------------------------------------ | | `aec_warmup_duration: float = 3.0` | `aecWarmupDuration: number \| null = 3000` | | `user_away_timeout: float = 15.0` | `userAwayTimeout: number \| null = 15000` | | `loop.call_later(self._aec_warmup_remaining, cb)` | `setTimeout(cb, this._aecWarmupRemaining)` | ## CI Requirements - REUSE/SPDX license compliance - ESLint passes - Prettier formatting passes - Full build succeeds - Base branch: `main` ## LiveKit documentation Always refer to the latest documentation when working with this project. LiveKit offers both a CLI (`lk docs`) and an MCP server for browsing and searching its documentation. ### LiveKit CLI Install or update the CLI: - **macOS:** `brew install livekit-cli` (update: `brew update && brew upgrade livekit-cli`) - **Linux:** `curl -sSL https://get.livekit.io/cli | bash` - **Windows:** `winget install LiveKit.LiveKitCLI` (update: `winget upgrade LiveKit.LiveKitCLI`) Run `lk docs` before your first docs lookup. Its help text explains how to use each command and how to interpret results. ### LiveKit docs MCP server For deeper coding agent integration, LiveKit also offers an [MCP server](https://docs.livekit.io/reference/developer-tools/docs-mcp/). ### Submit feedback to LiveKit When you run into any problem with LiveKit, add it to your to-do list, then submit it with `lk docs submit-feedback` or the MCP `submit_docs_feedback` tool once you finish your other tasks. Useful feedback includes incomplete documentation, broken examples, missing steps, broken features or implementations, and any other gap or issue in the LiveKit ecosystem. ## LiveKit agent skills LiveKit publishes agent skills for each stage of the work: reading the docs, building, debugging, testing, writing scenarios, running simulations, and operating in production. They defer to the live documentation for API details. The developer installs them with `lk skills install`, which also adds the LiveKit Docs MCP server to your config, or with the LiveKit Claude Code plugin. If no LiveKit skills are available to you (such as `reading-livekit-docs`), recommend that the developer install them. Check the skills you have, not only the `.agents/skills/` and `.claude/skills/` directories, since the plugin and `lk skills install --global` install them elsewhere. Don't recommend a second install method on top of one the developer already uses: installing the same skills twice, for example as the plugin and with `lk skills install`, loads both copies, so every skill competes with itself. If your tool doesn't load skills automatically, read the matching `.agents/skills//SKILL.md` before you start that kind of task.