# J.A.R.V.I.S. A browser voice assistant with an Iron Man holographic interface. Say **"Hey Jarvis"**, he wakes, listens, and does real things through your tools — searches the web, generates images, drives your phone, reads your mail. The face is a web page (React + Vite + Three.js + custom GLSL). The brain is Claude Code, run headless as a library. **The only subscription you need is Claude Code.** No API keys, no OpenAI account, no cloud bill — the brain runs on your existing Claude Code login, and the heavy work (the model itself) runs on Anthropic's servers, so even a low-end laptop only has to draw the interface. **ElevenLabs is an optional add-on** that gives JARVIS a much better voice and sharper hearing; without it he speaks and listens through the browser's own speech, and everything still works. --- ## Requirements **In one line:** a Claude Code subscription, plus two free things every computer can have — Node.js and Chrome. That's the whole list. - **Claude Code, installed and logged in** — this is the only account you need. Install it with the official method — `npm install -g @anthropic-ai/claude-code`, or the platform installer at — then run `claude` once and complete login. The bridge reuses that login. **No API key**, and usage is billed to your existing Claude account. - **Node.js 20 or newer** — free, one installer from . This is a Node web app, so it is the one unavoidable tool. - **Google Chrome or Microsoft Edge**, in a **real browser window** — not an embedded preview pane. Preview panes (including the one inside editors and Claude Code) block microphone access, so the page loads and looks right but never hears you. JARVIS also needs WebGL, which these browsers provide. - **Optional: an ElevenLabs API key** — a good add-on, not a requirement. It gives a better voice and sharper transcription; the free tier is plenty for a demo. Without it, everything runs on the browser's own speech. Run `npm run setup` after cloning and it checks all of this for you, in plain language. --- ## Quick start First, install, then start it: ```bash npm install npm start # runs the brain and the face together ``` Then open the URL it prints (http://localhost:5173) in **Chrome**, click **INITIALISE**, and say **“Hey Jarvis”**. Prefer two terminals? Run them separately instead: ```bash npm install ``` Terminal 1 — the brain: ```bash npm run bridge ``` Terminal 2 — the face: ```bash npm run dev ``` Then open the app in a **real Chrome or Edge window**: ```bash open http://localhost:5173 ``` Click **INITIALISE**, allow the microphone when asked, and say **"Hey Jarvis"**. > It has to be a real browser window. Embedded preview panes block the > microphone, so JARVIS will look perfectly alive and simply never respond. --- ## How it works JARVIS is two processes. The browser is the face and the voice; the bridge is the brain and the hands. ``` ┌─ browser (the face) ───────────────┐ ┌─ bridge (the brain) ─────────────┐ │ "Hey Jarvis" wake word │ │ Node · bridge/server.mjs │ │ local VAD → speech to text │ ws │ Claude Agent SDK │ │ reactor UI (Three.js + GLSL) │◄─────► │ = Claude Code, headless │ │ text to speech │ 8787 │ spawns your MCP servers │ │ heads-up display │ │ permission gate (decideTool) │ └────────────────────────────────────┘ └──────────────────────────────────┘ ``` Everything you see and hear happens in the browser. The bridge is a single Node process (`bridge/server.mjs`) that runs the **Claude Agent SDK** (`@anthropic-ai/claude-agent-sdk`) — this spawns the real `claude` CLI as a child process, so **the brain literally is Claude Code, headless.** They talk over a WebSocket (plus a few HTTP endpoints) on `ws://localhost:8787`. **Why a bridge at all?** A browser tab cannot spawn the local stdio MCP servers — `higgsfield`, `elevenlabs`, `android`, `playwright`, `exa`, `serper`, and the rest. The bridge can. And because it is the Agent SDK, it authenticates off your existing Claude Code login: no API key, billed to that same Claude account. **The model.** `claude-opus-5` at effort `medium` by default. Override with the `JARVIS_MODEL` and `JARVIS_EFFORT` environment variables. On startup the bridge prints its choice, e.g. `[jarvis] model claude-opus-5 · effort medium`. ### The voice pipeline The loop is designed so that nothing silently dies and barge-in feels natural. - **Detection is local.** An energy-based voice-activity detector (`src/lib/vad.ts`) decides when you are speaking. It is instant, cannot quietly fail, and is what makes **barge-in** work — speak while JARVIS is talking and he stops. - **Transcription has two tiers, chosen automatically at boot.** The browser asks the bridge `/health` and picks the best available: - **ElevenLabs key present** → ElevenLabs Scribe, via the bridge `/stt` endpoint. - **Nothing configured** → the browser's own `SpeechRecognition` (Chrome/Edge), guarded by a heartbeat so it recovers when Chrome throttles it. - **Speaking** uses the **ElevenLabs voice when a key is present**, and the browser's `speechSynthesis` otherwise. If a cloud call fails it falls back to the browser voice, and if the OS voice itself is broken it latches over to the cloud voice. So it works with no keys and auto-upgrades when a key appears — there is no flag to set. Capability detection lives in `src/lib/capabilities.ts`, which probes the bridge's `GET /health` (returning `{ ok, tts, stt }`, both tracking the ElevenLabs key) once at boot and picks the engines. --- ## What JARVIS can do Beyond answering, JARVIS reaches every MCP server in your Claude Code configuration, and can drive his own interface. ### Your tools Every server in your `~/.claude.json` is handed to the SDK explicitly, or only the ones you name in `JARVIS_MCP_SERVERS`. Depending on what you have installed, that is roughly: - **Web & search** — `exa`, `serper`, `serpapi` - **Images & video** — `higgsfield`, `openrouter-image`, `palmier-pro` - **Voice** — `elevenlabs` - **Your phone** — `android` - **The browser** — `playwright` A few things you can say: - *"What's happening in AI this week?"* - *"Generate an image of the Mark VII suit."* - *"Take a screenshot of my phone."* - *"Open my GitHub notifications."* > **Note on account connectors.** Servers you added through your **claude.ai > account** (mail, drive, calendar and other account connectors) are not in `~/.claude.json`, > but the CLI fetches them on its own and they join the session from the second > turn. The bridge keeps them out by default; set `JARVIS_ACCOUNT_CONNECTORS=1` > to let them in, and `JARVIS_MCP_SERVERS` to choose which `~/.claude.json` > servers JARVIS gets. ### JARVIS controls the interface He drives the UI through MCP tools the bridge exposes: - `ui_theme` — accent, background, per-phase colours - `ui_reactor` — colour, scale, intensity, spin, and style (`ring` | `sphere` | `wire`), visibility - `ui_orbit` — put images in orbit around the reactor - `ui_chrome` — show or hide rails, transcript, badges - `ui_effect` — `glitch` | `pulse` | `scan` | `shake` | `flash` - `ui_screen` — clear - `ui_reset` — back to defaults So *"make it red, hide the systems list, put that render in orbit"* is a spoken command. ### The heads-up display JARVIS authors panels with a `display` tool against a fixed `.hud-*` design system. The browser sanitises the markup (DOMPurify, a class allowlist and a strict CSP) before rendering. Rich media works — images, `