chimera-tts

# chimera-tts A text-to-speech voice for AI agents, made with the [Chimera Vocal Synth](https://github.com/jamesondh/chimera-vocal-synth) by Jameson Hodge: a physically modeled vocal tract, not a neural voice. It sounds like a friendly machine learning to talk. [Hear a sample](https://lowmagic.github.io/chimera-tts/sample.wav), or [talk to it in your browser](https://lowmagic.github.io/chimera-tts/). There are two ways to use it: - **A command that turns text into a WAV file**, for [Hermes Agent](https://hermes-agent.nousresearch.com) or anything else that can run a command for its speech. It needs only Node: no GPU, no network, no API key. Every client that plays Hermes's speech gets the voice (its own apps, and others such as Conduit). - **A browser demo and voice chat page**: hear the voice and its Markdown normalizer in one click with no API key and no network calls, or switch to live voice chat against any OpenAI-compatible endpoint, Hermes included. On top of Chimera this adds pitch and rhythm learned from recordings of people talking, clearer consonants, pronunciations for names Chimera's dictionary lacks, and Markdown made sayable. Chimera itself is used as it is, pinned as a submodule and not patched. Be warned that it is a characterful voice, not a clear one. A speech recognizer makes out about half of its words ([training/README.md](training/README.md), "Which words come out unclear"), and people do better with the sentence in front of them, which is why the chat page has captions. ## Text to speech as a command Branch `dist` of this repository holds everything ready-built, so there is nothing to compile. You need a recent Node (CI runs it on Node 26; 22 works too). ```sh git clone --depth 1 --branch dist https://github.com/lowmagic/chimera-tts /srv/chimera echo "Hello! This is how I sound." > reply.txt node /srv/chimera/speak/speak.mjs reply.txt reply.wav ``` `git -C /srv/chimera pull` updates it. The command is `speak/speak.mjs`, with everything it loads beside it: the engine, the pronunciation dictionary and the trained pitch model. It writes nothing but its output. (Built from source, the same files are in `dist/speak/`.) It reads the text (UTF-8) and writes one WAV of all of it (16-bit mono, 48 kHz), said as the chat page says a reply with its default settings: the same sentences, mood and emphasis marks, trained pitch and voice (adult male). Markdown is made sayable first (`src/reply.ts`): a link is its text, a bare URL is "link", a code block is "code block", a table row is its cells, emoji are dropped. If it can't, it prints one line to stderr and exits 1. It runs on one core at a little over a quarter of the audio's length: a 600-character reply (34 s of speech) takes 9 s on an Apple M4. ### In Hermes Agent Hermes can then say its replies in this voice wherever it speaks: its apps' voice mode and Read Aloud, voice messages, and other clients that play its speech. It runs the command as a TTS provider, in `config.yaml`: ```yaml tts: provider: chimera providers: chimera: type: command command: node /srv/chimera/speak/speak.mjs {text_path} {output_path} format: wav timeout: 600 # seconds without output; a long reply on a slow machine takes a while ``` Hermes cleans the text before it hands it over (Markdown, emoji and emphasis asterisks go; a mood tag stays) and converts the WAV as it needs. ### Changing the voice The chat page's voice sliders are flags on the command, anywhere on the line, and what you leave out stays at its default: | Flag | Default | The page's setting | | --- | --- | --- | | `--rate` | 0.95 | Speed | | `--pitch` | 1 | Pitch | | `--body` | male | Voice (`male`, `female`, `child`) | | `--plosives` | 1.5 | Harder p, t, k, b, d, g (1 to 2) | | `--voicing` | 1.5 | More voice in b, d, g, v, z (1 to 2) | | `--word-ends` | 1 | longer last syllable of each word (no slider on the page) | | `--timing` | on | Trained timing (`on` or `off`) | ```yaml command: node /srv/chimera/speak/speak.mjs --rate 0.8 --pitch 1.1 {text_path} {output_path} ``` Hermes reads the command for each reply, so a change takes effect on the next one. To find values you like, move the sliders in the chat page's Settings and copy them. ### Kept running A speech server that says many replies keeps one running instead, so the engine, dictionary and pitch model load once: ```sh node /srv/chimera/speak/speak.mjs --serve ``` reads one JSON request per line on stdin, `{"id": "1", "text": "Hello there."}`, and answers each, in order, with one line on stdout: `{"id": "1", "rate": 48000, "pcm": ""}`, the whole text as 16-bit little-endian mono PCM said as above, or `{"id": "1", "error": ""}` (a line that isn't a request is answered with an empty id). Nothing else goes to stdout; diagnostics go to stderr. It exits 0 when stdin closes. A request may carry `"options": {"rate": 0.9, "plosives": 1.5, "voicing": 1.5, "wordEnds": 1.3, "timing": false}` for itself; the flags above, before `--serve`, set what a request gets when it doesn't say. Loading takes about 0.16 s on an Apple M4, so a 10-word sentence takes 0.84 s served against 1.0 s as a command; the rest is the engine saying it. ## The browser demo and voice chat page The page opens in **Local Speak** mode: type or paste text or Markdown (or pick a preset: plain prose, Markdown with code, lists and URLs, a canned streamed reply, or a rotating funny line) and tap **Speak**. Everything runs locally in WebAssembly with no API key and no network calls, while Chimera's wireframe head mouths the words and a caption lights up each word as it is spoken. Switch to **Live LLM Chat** to talk (or type) to a model: Chimera says the reply sentence by sentence while the model is still writing it. ### How it works ``` mic → browser speech recognition → chat model (streamed) → sentences → Chimera → speakers └→ "is the user talking over it?" → stop ├→ caption, word by word └→ the 3D head ``` - **Voice**: Chimera is a git submodule in `vendor/chimera`, pinned to a commit. Its Rust engine is built to wasm and runs in an AudioWorklet; its own text front end (CMUdict, letter-to-sound rules) turns words into gestures. Nothing in Chimera is patched. - **Model**: any OpenAI-compatible streaming endpoint. The default is [OpenRouter](https://openrouter.ai), so the model is one setting. The API key is typed into Settings and stays in your browser's localStorage; the page sends it only to the endpoint. On OpenRouter the request asks for the fastest provider and no reasoning pass, and with "Zero data retention" on (the default) only for providers that keep nothing. OpenRouter's self-serve accounts still allow "safety retention" providers under that flag: to rule those out too, choose "ZDR only" in your OpenRouter privacy settings. If a model has no such provider the request fails and says so. The model is told which model and endpoint it is running on, so you can ask it, and the log notes it whenever that changes. - **Ears**: the browser's built-in speech recognition (Chrome, Edge, Safari). No key needed. Firefox has none, so there you type. - **Pitch**: with "Natural pitch" on, Chimera's own pitch gestures (the same bump on every stressed vowel, a full fall at every comma) are replaced with a contour shaped like spoken English: accents on content words only, a small rise before a comma, a climb at a yes/no question, a fall at a full stop. The model is asked to start a reply with a mood (`[excited]`) and to put asterisks round a word it would stress; both shape the pitch and neither is read aloud. The contour and the length of each syllable come from a small model trained on recordings of people talking (`public/models/`), with switches in Settings; without it, rules do the pitch (`src/prosody.ts`) and the timing is Chimera's. [training/README.md](training/README.md) has the steps to train your own (an NVIDIA GPU and under an hour). - **Consonants**: two sliders in Settings, from 1.00× (Chimera's own) to 2.00×, at 1.50× to start. "Harder p, t, k, b, d, g" pushes more breath behind each stop and widens its puff; "More voice in b, d, g, v, z" keeps the voice going under them. They adjust Chimera's finished speech plan (`src/articulation.ts`); `bun tools/consonants.ts --plosives 2 --voicing 2 --wav out.wav` measures and renders a setting. - **Names and initials**: words the dictionary lacks would be sounded out letter by letter, badly ("ChatGPT" as "chat-ch-p-t"). `src/lexicon.ts` holds pronunciations for such names: add a line whenever the voice mangles one. Initials it doesn't know are said as letters ("GPT"), and joined names are split ("DailyTalk"). - **Head**: Chimera's own 3D renderer (three.js), driven by the engine's live state. Drag to turn it. It can be turned off in Settings on a slow phone. - No server. The built site is static files, installable to a phone's home screen. Every push to `main` is also published ready-built on branch `dist` (below). ### Using it - **Talk**: tap, speak, pause. Tap again to send early. - While it is thinking or speaking the button is **Stop**: tapping it cuts the voice off and listens to you. - With "Keep listening after each reply" on, a spoken turn reopens the mic when the reply ends. - With "Interrupt it by talking" on, just start talking over it and it stops and listens. This watches the microphone for speech-like loudness while the voice speaks, relying on the browser's echo cancellation to keep the voice's own sound out. It works best with headphones; if it interrupts itself on speakers, turn it off. The first word or two of an interruption is lost while the recognizer starts. - On a phone, use the browser's "Add to Home Screen" to install it. The screen stays on during a conversation. - Settings has the voice (adult male, adult female, child), speed, pitch and the personality prompt. ## The chat page with Hermes Agent [Hermes Agent](https://hermes-agent.nousresearch.com/docs/user-guide/features/api-server) has an OpenAI-compatible API server, so the chat page can be its voice and face while Hermes keeps the memory, the tools and the conversations. In `~/.hermes/.env`: ```sh API_SERVER_ENABLED=true API_SERVER_KEY= ``` and run `hermes gateway`. Then either serve this app from Hermes's own address, or use it from another site. ### Served from Hermes's address This is how it runs on a home server, for a phone anywhere on the tailnet. Branch `dist` holds the built site: CI commits the build of every push to `main` to it, one commit each, whose message names the source commit's full SHA, with `index.html` at the root. The server pins a commit of `dist` and serves its files as they are (nothing is built there) from the Tailscale node that also passes `/v1/` and `/api/` to Hermes's API server (`http://127.0.0.1:8642/v1` and `/api`): the app and Hermes on one address, `https://..ts.net/`. Next to `index.html` it puts a `deploy.json`: ```json {"backend": "hermes", "baseUrl": "/v1"} ``` At startup the app reads it, and until you save other settings it talks to Hermes at that address's `/v1` (a relative `baseUrl` is resolved against the page). The API key still comes from you: paste the `API_SERVER_KEY` in Settings, once on each device. The page and Hermes share one origin, so Hermes needs no `API_SERVER_CORS_ORIGINS` and nothing else needs `tailscale serve` by hand. The public site has no deploy.json, so nothing changes there. ### From another site Add `API_SERVER_CORS_ORIGINS=` to Hermes's `.env`. In this app's Settings choose **Chat with: Hermes Agent** and paste the key. The choice fills in the endpoint (`http://127.0.0.1:8642/v1`) and model (`hermes-agent`), shows the exact origin to put in `API_SERVER_CORS_ORIGINS`, and keeps Hermes's key apart from OpenRouter's, so switching between them is one tap. - **Same computer, Chrome, Edge or Firefox**: `http://127.0.0.1:8642/v1` works from the HTTPS site. Chrome may ask once for permission to reach devices on your local network. - **Phone, another computer, or Safari**: browsers block an HTTPS page from calling plain HTTP on another machine, so Hermes needs an HTTPS address. On a Tailscale network, `tailscale serve --bg 8642` on the Hermes machine gives one (`https://..ts.net`); use that plus `/v1` as the Endpoint. Hermes can stay bound to 127.0.0.1. ### With Hermes chosen - Conversations live in Hermes, one Hermes session each. **Chats** lists them (Hermes's title, or the first words you said, and when), opens one with its earlier turns, starts a new one, renames and deletes them. A new conversation becomes a session with its first turn (`POST /api/sessions`). Each turn names it in `X-Hermes-Session-Id` and carries only the system prompt and the new turn, since Hermes takes the earlier ones from the session. Any device with the key can open a conversation and carry on with its context, and each device reopens the one it had open last. - Only conversations started through Hermes's API server are listed: Hermes won't let this API carry on a CLI, desktop or messaging-app session. - While the agent runs a tool the status line says what it is doing ("🔍 web_search: weather in Paris…") until the tool has finished. Tool output is never spoken. - The app still sends its personality prompt and the mood and emphasis instructions, which Hermes layers on top of its own prompt. - If your Hermes refuses the session header (an older one), the app notices and carries on without it, sending the whole conversation each turn. That key gives full access to the agent, terminal included. Never put the server on the open internet. The automated tests run against a stand-in written from Hermes v0.23.1's source and documentation. ## Build it yourself Needs Rust with the `wasm32-unknown-unknown` target, and [Bun](https://bun.sh). ```sh git clone --recurse-submodules https://github.com/lowmagic/chimera-tts cd chimera-tts bun install bun run dev # builds the engine, then serves http://127.0.0.1:5327 ``` Tap **Speak** (or any preset) to hear it right away, or switch to **Live LLM Chat**, open Settings, paste an API key, pick a model, and tap **Talk**. The microphone and the audio engine need a secure context: `localhost` or HTTPS. ```sh bun run test # unit tests (text, Markdown replies, sentences, pitch contours, the talk-over detector, Hermes's API) bun run build # typecheck and build to dist/ (with the speak command in dist/speak/) bun run e2e # after a build: the speak command, then headless Chromium, mocked model and recognizer make check # what CI runs: the tests, the build, the speak command, the training tools' tests, shellcheck ``` `make check` also needs the training tools' Python packages ([training/README.md](training/README.md)) and [uv](https://docs.astral.sh/uv/), which runs shellcheck. [AGENTS.md](AGENTS.md) says what to read before changing anything. `bun run e2e` needs Playwright with Chromium installed; it is not a dependency of this repo. To compare the two pitch modes by ear without the page, render a reply both ways to WAV files: ```sh bun tools/render.ts out female "[happy] Hello! This is *really* how I sound. Do you like it?" ``` ## Updating Chimera ```sh git -C vendor/chimera fetch && git -C vendor/chimera checkout bun run build && bun run e2e git add vendor/chimera && git commit ``` ## Not yet Conversation memory between visits without Hermes, a choice of speech recognizers, and dressing up the head. ## License [MIT](LICENSE). Chimera is MIT, by Jameson Hodge. The CMU Pronouncing Dictionary is copied from Chimera at build time under its own license (`public/data/CMUDICT-LICENSE.txt`). The trained pitch model in `public/models/` (`prosody.json`, `prosody.bin`) was trained on [DailyTalk](https://github.com/keonlee9420/DailyTalk) (Lee, Park and Kim, "DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech", ICASSP 2023), which is licensed [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/). Those two files are shared under the same license. They hold no audio or text from the dataset beyond a list of its commonest words.