# assistant > **⚠️ Experimental Software** > > This is personal software shared with the community for educational purposes. > Qustions and contributions are welcome, but don't expect stability or support. A personal AI assistant with a panel-based plugin system, multi-agent CLI integrations, and text/voice UI. Plugins define panels and operations so agents can collaborate on lists, notes, and other workflows, or extend the app with custom user-provided plugins. In-process chat uses the Earendil Pi SDK for model access, tool execution, and streaming.
Assistant Screenshot Assistant Mobile
## Table of Contents - [Features](#features) - [Quick Start](#quick-start) - [Repository Layout](#repository-layout) - [Documentation](#documentation) - [External Integrations](#external-integrations) - [Configuration](#configuration) - [Development](#development) - [Architecture](#architecture) - [License](#license) ## Features - **Text chat** with streaming responses - **Voice input** using browser-based speech recognition (Web Speech API) - **Voice output** via OpenAI TTS or ElevenLabs streaming TTS (optional) - **Android Thread voice** via [openai-speech-server](https://github.com/kcosr/openai-speech-server) for authenticated local Parakeet/Kokoro speech, with native token storage and model selection ([setup](packages/mobile-web/README.md#native-voice-runtime)) - **CLI agent integrations** (Claude, Codex, Pi) alongside built-in providers - **Scheduled sessions** for cron-driven CLI runs - **Panel plugins** for lists, notes, and custom workflows - **Tool integration** via MCP (Model Context Protocol) over stdio - **Built-in session tools** – agent can list, search, create, switch, rename, and pin sessions - **Persistent sessions** with JSONL event logs and optional naming/pinning - **Multi-client support** – multiple browser windows can share a session - **Theme + font preferences** (auto/light/dark + presets, including Herdr). Dark uses nearly black backgrounds, charcoal controls, and subtle borders; Auto uses the same palette in dark mode. ## Quick Start Requires Node.js **22.19.0 or newer**. The in-process Pi provider uses Pi SDK 1.5.2; CLI providers require their own separately installed executables. ```bash # Clone the repository git clone https://github.com/kcosr/assistant cd assistant # Install dependencies npm install # Build all packages npm run build # Build all packages in parallel npm run build:parallel # Build all packages in parallel on the remote builder npm run build:remote # Start the server (uses built-in config at packages/agent-server/data/config.json) npm run start -w @assistant/agent-server # Open http://localhost:3000 in your browser ``` In-process Pi agents reference named `chatProfiles` containing ordered `provider/model` choices, and per-model thinking levels. Shared request overrides supply optional sampling settings. Pi’s registry owns provider connections, authentication, capabilities, and limits, including custom models in `~/.pi/agent/models.json` on the service host. See [Pi configuration](docs/CONFIG.md#pi-provider). The built-in config includes Claude Code, Codex, and Pi CLI agents. For a full configuration example with all options, see `packages/agent-server/data/config.example.json`. ### Configuration Overrides Override the default paths using environment variables: ```bash # Use a custom data directory (sessions, events, config) export DATA_DIR=/path/to/data # Use a specific config file export APP_CONFIG_PATH=/path/to/config.json npm run start -w @assistant/agent-server ``` The server looks for config at `${DATA_DIR}/config.json` by default, or at `APP_CONFIG_PATH` if set. ### Building Skills for CLI Agents Plugins can be exported as skills for use with CLI agents that support skills (e.g., Claude Code, Codex, Pi): ```bash # Build all plugin skills to a directory npm run build:plugins -- --skills-dir /tmp/skills # Copy specific skills to your agent's skills directory # Claude: ~/.claude/skills/ | Pi: ~/.pi/skills/ | Codex: ~/.codex/skills/ cp -r /tmp/skills/{notes,lists} ~/.pi/skills/ # Or copy all skills (ensure no name conflicts with existing skills) cp -r /tmp/skills/* ~/.pi/skills/ # Build only specific skills directly to the skills directory npm run build:plugins -- --skills notes,lists --skills-dir ~/.pi/skills/ ``` Some skills include a CLI binary. Add the skill directories to your PATH, or reference CLIs by full path in your AGENTS.md: ```bash # Option 1: Add to PATH (in ~/.bashrc or ~/.zshrc) export PATH="$PATH:$HOME/.pi/skills/notes:$HOME/.pi/skills/lists" # Option 2: Reference full paths in AGENTS.md / SKILL.md documentation ``` ### Keyboard Shortcuts | Shortcut | Action | |----------|--------| | `Ctrl/Cmd + K` | Open command palette | | `Ctrl/Cmd + ]` / `Ctrl/Cmd + [` | Cycle focus between panels | | `Ctrl + P` | Toggle layout navigation mode | | `Ctrl + H` | Toggle header panel navigation mode | | `Ctrl + Shift + S` | Split active panel (placement mode) | | `Ctrl + I` | Toggle text input focus | | `Right Option` (macOS, opt-in) | Focus the selected chat input without consuming the key event | | `Ctrl + C/L/N` | Focus last-used chat/lists/notes panel (opens modal if none) | | `Ctrl + R` | Toggle speech recording (if available) | | `Cmd + Shift + S` (macOS) | Toggle sessions sidebar | | `Cmd/Ctrl + Shift + C` | Toggle chat panel | | `Cmd/Ctrl + Shift + ↑/↓` | Jump to top/bottom of chat | | `Ctrl + Shift + Cmd + W` (macOS) / `Ctrl + Shift + Alt + W` (others) | Close panel (replace with placeholder) | | `Ctrl + Shift + Cmd + X` (macOS) / `Ctrl + Shift + Alt + X` (others) | Remove panel from layout | | `Enter` | Replace active empty panel | | `Shift + Enter` | Submit modal form (e.g., from tags input in notes/lists) | | `Cmd + Click` (macOS) / `Ctrl + Click` (others) | Focus a non-chat panel (chat panels focus on regular click) | | `Tab` / `Shift + Tab` | Cycle focus between sidebar and input (when input is empty) | | `Esc` | Cancel active operations (on mobile, closes sidebar if open; in pinned or modal chat panels, closes the panel instead) | In layout navigation mode: arrows move between panels, `Enter` selects, `Esc` exits, `Tab` cycles tabs, `m` toggles split view, `1-9` select visible panels, `0` cycles pages. In header navigation mode: `←/→` (or `A/D`) cycle header panels, `Enter`/`↓` activates, `Esc` exits, `1-9` select visible panels, `0` cycles pages. In split placement mode: arrows or WASD choose region, `Enter` confirms, `Esc` cancels. **Sessions sidebar (when focused):** - `Arrow Up/Down` – Move between sessions (selection follows focus) - `Enter` – Focus input - `t` – Toggle pin - `d` / `Delete` / `Backspace` – Delete session (with confirmation) - `c` – Clear history (with confirmation) **Lists panel item selection:** - `Click` – Select item (when enabled in Settings menu) - `Shift + Click` – Range select from last selected item to clicked item - `Cmd + Click` (macOS) / `Ctrl + Click` (others) – Toggle individual item selection - `Double-click` – Edit item - `Arrow Up/Down` – Move selection - `Shift + Arrow Up/Down` – Extend selection - `Enter` – Edit focused item - `Space` – Toggle completed for selection - `n` – Add item - `a` – Toggle AQL mode - `w` / `s` – Move focused item up/down - `t` / `b` – Move focused item to top/bottom - `p` – Toggle pin - `Cmd/Ctrl + C` / `X` / `V` – Copy / cut / paste list items (cross-list) - `d` – Delete selection (with confirmation) - `Esc` – Clear selection **Lists/Notes browser (collection view):** - `Arrow keys` – Move focus - `Enter` – Open focused item - `p` – Toggle pin **Command palette (when open):** - `Arrow Up/Down` – Move focus - `Enter` – Launch selection - `Shift + Enter` – Replace active panel (when available) - `→` – Open action menu - `Esc` – Close palette or menus Palette commands: - `/pinned` – Show pinned lists and notes - `/favorites` – Show favorite lists and notes **Notes panel:** Drag to select text, then release with `Shift` held to add selection to agent context. ### ElevenLabs Streaming TTS (Optional) To enable ElevenLabs streaming text-to-speech (note: this can be costly): ```bash export TTS_BACKEND=elevenlabs export ELEVENLABS_API_KEY= export ELEVENLABS_TTS_VOICE_ID=VUGQSU6BSEjkbudnJbOj export ELEVENLABS_TTS_MODEL=eleven_multilingual_v2 ``` ### Git Versioning (Auto-Snapshots) Git versioning is **enabled by default** for lists, notes, and time-tracker plugins. This automatically commits changes to the plugin's data directory at regular intervals, providing version history. To disable git versioning for a plugin, remove or set `enabled: false` in the `gitVersioning` key: ```json { "plugins": { "lists": { "enabled": true, "gitVersioning": { "enabled": false } } } } ``` The default interval is 1 minute. You can adjust it with `intervalMinutes`. ### Recommended AGENTS.md Addition Add this to your project or user-level `AGENTS.md` to help AI agents understand and use the assistant skills: ```markdown ## Assistant Workspace The user runs a personal assistant workspace with panels for notes, lists, time tracking, and more. **Only use these skills when the user specifically mentions "assistant" or refers to the assistant workspace.** Otherwise, use standard file and shell operations. ### Skills Skills are packages of instructions and optional CLI tools that integrate with the assistant workspace. Each skill has a `SKILL.md` with full documentation. #### notes Markdown notes with tags and search. Use for: - Storing research, meeting notes, documentation - Capturing information the user wants to reference later - Organizing content with tags Commands: `list`, `read`, `write`, `search`, `rename`, `delete`, `tags-add`, `tags-remove` #### lists Structured lists with items, tags, and custom fields. Use for: - Task lists and to-dos - Reading lists, watch lists, bookmarks - Any ordered collection of items with metadata Commands: `list`, `create`, `item-add`, `item-update`, `item-remove`, `items-search` #### time-tracker Track time against tasks with timers and manual entries. Use for: - Starting/stopping work timers - Logging time spent on tasks - Querying time entries for reports Commands: `task_create`, `task_list`, `timer_start`, `timer_stop`, `entry_create`, `entry_list` #### panels Panel inventory and layout management. Use for: - Listing open panels - Opening/closing/moving panels - Sending events to panel instances Commands: `list`, `selected`, `open`, `close`, `move`, `tree` #### url-fetch Fetch and extract content from URLs. Use for: - Fetching web page content for analysis - Extracting metadata (title, description) from URLs - Getting readable text from articles Commands: `fetch` (modes: `extracted`, `raw`, `metadata`) ### Usage Run skill CLIs directly: notes-cli list lists-cli list time-tracker-cli timer_status All commands support `--json` for structured output. ``` ## Repository Layout Monorepo managed with npm workspaces: | Package | Description | | -------------------------- | ---------------------------------------------------------------------- | | `packages/agent-server/` | Node.js backend – WebSocket server, OpenAI integration, TTS, MCP tools | | `packages/web-client/` | Browser client – chat UI, speech input, audio playback | | `packages/assistant-cli/` | Internal runtime used to generate plugin CLIs | | `packages/shared/` | Shared types, protocol definitions, audio frame helpers | | `packages/plugins/` | Plugin packages (core, official) | | `packages/desktop/` | Canonical Electron desktop app wrapper | | `packages/desktop-tauri/` | Transitional Tauri desktop app wrapper | | `packages/mobile-web/` | Capacitor mobile app wrapper | | `packages/notify-proxy/` | Push notification proxy server | | `packages/push-cli/` | CLI for sending push notifications | | `docs/` | Design documents and specifications | ## Documentation For the full documentation map, see [docs/index.md](docs/index.md). ### Specifications | Document | Description | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [UI Specification](docs/UI_SPEC.md) | Common UI behavior and layout requirements for all client implementations (web, Android, iOS). Covers panel layouts, responsive behavior, toolbar controls, and interaction patterns. | | [Plugin SDK](docs/PLUGIN_SDK.md) | Canonical guide for plugin structure, operations, bundles, and CLI skill generation. | | [Config Reference](docs/CONFIG.md) | Comprehensive `config.json` reference, including agents, plugins, MCP servers, and CLI wrapper options. | ### Design Documents | Document | Description | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Agents](docs/design/agents.md) | Agent architecture and configuration. Covers agent definitions, sessions, tool allowlists, and the relationship between agents and conversations. | | [Preferences](docs/design/preferences.md) | Preferences storage architecture. Details server-side vs client-side storage, the `/preferences` API, per-list settings, and the optimistic update pattern. | | [External Agents](docs/design/external-agents.md) | Design for async external agent integration. Describes how to connect out-of-process agent systems to the assistant UI via HTTP callbacks. | | [Content Blocks](docs/design/content-blocks.md) | Output format documentation for Claude CLI and Codex CLI. Reference for implementing structured content block rendering. | ### Package Documentation | Document | Description | | ------------------------------------------------------ | ------------------------------------------------------------ | | [Agent Server](packages/agent-server/README.md) | Server configuration, plugin system, and MCP tool hosting. | | [Web Client](packages/web-client/README.md) | Browser client architecture and development. | | [Plugin CLI Runtime](packages/assistant-cli/README.md) | Internal CLI runtime used by plugin build output. | | [Shared](packages/shared/README.md) | Shared types, protocol definitions, and audio frame helpers. | | [Desktop](packages/desktop/README.md) | Canonical Electron desktop app wrapper and build steps. | | [Desktop Tauri](packages/desktop-tauri/README.md) | Transitional Tauri desktop app wrapper and build steps. | | [Mobile Web](packages/mobile-web/README.md) | Capacitor mobile app wrapper for Android/iOS. | | [Notify Proxy](packages/notify-proxy/README.md) | Push notification proxy server. | | [Push CLI](packages/push-cli/README.md) | CLI for sending push notifications. | ## External Integrations | Service | API | Purpose | | -------------- | ---------------------------- | ----------------------------- | | **OpenAI** | Chat Completions (streaming) | Text generation | | **OpenAI** | Audio Speech | TTS output (default backend) | | **ElevenLabs** | WebSocket Streaming TTS | TTS output (optional backend) | | [**openai-speech-server**](https://github.com/kcosr/openai-speech-server) | Authenticated HTTP + WebSocket audio API | Android Thread recognition and speech | | **MCP Server** | JSON-RPC 2.0 over stdio | Tool hosting | ## Configuration See `docs/CONFIG.md` for environment variables and the `config.json` schema (including CLI agent providers). Full examples live in `data/config.example.json`. ## Development ```bash # Run tests npm test # Lint npm run lint # Format check npm run format # Build all packages npm run build ``` ## Architecture ``` ┌─────────────────────────────────────────┐ ┌─────────────────┐ WebSocket + HTTP │ Agent Server │ │ Web Client │◄──────────────────►│ (Node.js) │ │ (Browser) │ │ │ └─────────────────┘ │ • WebSocket: chat, audio streaming │ │ • HTTP: static files, REST API │ ┌─────────────────┐ HTTP │ │ │ Plugin CLIs │◄──────────────────►│ │ │ (External Agent)│ REST API └──────────────────┬──────────────────────┘ └─────────────────┘ │ │ ┌─────────────────┬───────────────┬───────────────┬───────────────┬───────────────┬─────────────────┐ │ │ │ │ │ │ │ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Pi SDK Chat │ │ Claude CLI │ │ Codex CLI │ │ Pi CLI │ │ TTS Backend │ │ MCP Tools │ │ Plugins │ │ (API) │ │ (subprocess)│ │ (subprocess)│ │ (subprocess)│ │ (API) │ │ (subprocess) │ │ (files) │ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ ``` ### Client-Server Protocol The client and server communicate over WebSocket using: - **JSON messages** for control, text, and metadata - **Binary frames** for audio data (TTS output) See `packages/shared/src/protocol.ts` for message type definitions. ### Session Persistence - **Sessions index**: `${DATA_DIR}/sessions.jsonl` - **Session event logs**: `${DATA_DIR}/sessions//events.jsonl` Event logs are append-only ChatEvent records containing user messages, assistant output, tool calls/results, and agent callbacks. ## License MIT