**Open-source digital humans, ready to deploy.**
Realtime voice conversation on a published channel: ask, it answers with lip-synced video; interrupt it any time.
FlashHead with the built-in Luna half-body preset: one portrait, the whole head is rendered by the engine and pasted back onto a head-to-waist canvas; idle breathing and gesture clips (shy, laugh, shake, wink) are baked from the same canvas, and the instinct layer picks a gesture while the brain is still composing. Ships as a built-in preset; publish it as a channel from the console.
A self-hosted digital human you can actually hand to a customer: realtime voice
conversation, admin console, knowledge base, channel publishing — all open source.
One GPU, or none.
Stream a talking avatar over WebRTC — voice in, lip-synced video out — with
swappable avatar / TTS / ASR / LLM engines, avatar training from a video,
voice cloning, and a one-command setup wizard.
[](LICENSE)
[-green)](LICENSING.md)
· [Quickstart](#quickstart) · [Architecture](#architecture) · [Integration API](#integration--api) · [Licensing](LICENSING.md) · [Commercial support](#commercial-support)
> **Hardware, up front:** the default FlashHead avatar engine needs an
> **NVIDIA GPU (4090-class)**; the lighter wav2lipLS profile runs on more
> modest GPUs. **One avatar worker serves one live session at a time** — see
> [Capacity & scaling](#capacity--scaling) before planning a deployment.
> No GPU at all? Every component can be a remote endpoint.
---
## What it is
Mynah is a Go realtime core (**`cored`**) that streams synchronized
video + audio from a Python avatar worker to a browser over **WebRTC**. It
receives a WebRTC offer, opens a bidirectional gRPC session to the avatar
engine, feeds it **TTS audio** (from text, or from an **LLM** chat turn), and
paces the encoded VP8/H.264 + Opus frames back to the browser. Optional **voice
input** (ASR worker) and **semantic turn-taking** (a turn-detector worker that
decides when the user has actually finished speaking) make it a full
voice-to-voice loop. An optional control plane (`--db`) adds an admin API,
config hot-apply, a knowledge base (RAG), and publishable visitor "channels".
It is engine-agnostic by design: avatar, TTS, ASR, LLM and turn-detection are
**pluggable** behind small contracts (a gRPC streaming contract for workers, an
OpenAI-compatible HTTP surface for TTS/LLM). The default engines are chosen to
be **commercially usable** out of the box.
## Highlights
- **Realtime, not rendered video** — the cloud brain answers within ~0.2 s of
the user's last word and lip-synced video starts under a second; barge-in
stops it mid-sentence. A bundled audio end-of-turn model (Smart Turn v3.2,
BSD-2) decides when the user has actually finished speaking. Numbers in
[Performance](#performance).
- **One portrait, a half-body presenter** — FlashHead renders the whole head
and pastes it onto a 720×1280 head-to-waist canvas; idle breathing and
gesture clips (shy, laugh, shake, wink) are baked from the same canvas.
**Luna** and **Nova** ship as built-in presets; your own portrait takes one
prepare step.
- **Instinct, not just answers** — an instinct layer picks an expression or
gesture while the brain is still composing and plays it the moment the reply
ends. Backed by the Jev cognition model, with a zero-dependency rules
provider as the offline fallback. See `docs/guide/instinct.md`.
- **A console, not a script** — avatars, voices, knowledge base (pgvector RAG
with optional reranking), channel publishing with access control and
concurrency quotas, an engine pool with a VRAM budget, one-click avatar swap
per channel, session monitor.
- **Avatar training from a video** — upload a talking video, the pipeline
picks the best speaking-base frame (MediaPipe, no InsightFace), and the new
likeness becomes selectable with a baked living-standby idle loop.
Consent-gated (see `docs/compliance.md`).
- **Plugs into your stack** — brain options: local LLM, Qwen Realtime
speech-to-speech, or any **Pipecat** pipeline; avatar / TTS / ASR behind gRPC
and OpenAI-compatible contracts; Python SDK (`mynah`), `pipecat-ai-mynah`,
and a DeepSeek Harness plugin.
- **Commercially-clean defaults** — FlashHead (Apache-2.0) and MuseTalk 1.5
(MIT) avatar engines, a self-trained wav2lipLS lightweight profile. See
[LICENSING.md](LICENSING.md).
## Quickstart
**What you need.** One NVIDIA GPU for the avatar engine (6 GB for the
wav2lipLS profile, 12 GB for MuseTalk or FlashHead, 24 GB to run both side by
side), an LLM endpoint (local ollama/vLLM or any OpenAI-compatible API), and a
TTS. The stack starts with **EdgeTTS** (free Edge voices, CPU only, needs
internet) so the first conversation does not wait on a TTS model; switch to
Qwen3-TTS (voice cloning) or a cloud voice from the console without a restart.
**No GPU on this machine?** Every component is a network endpoint: run the
avatar worker on a GPU box and cored on anything. See the docs:
https://honwee.github.io/mynah/ (中文: /zh/)
**One-click cloud image (single-node trial).** On AutoDL, start a 24 GB
instance from the [Mynah community image](https://www.codewithgpu.com/i/honwee/mynah/Mynah), run
`~/mynah/mynahctl start`, open an SSH tunnel — see
[Cloud images](https://honwee.github.io/mynah/guide/cloud-images).
### Quickstart (original)
The fastest path is the **setup wizard** — it provisions LLM / TTS / ASR / avatar
for you (download + start, mirror-aware), then launches the stack:
```bash
python3 deploy/setup/helper.py # → http://127.0.0.1:9500
```
Open it in a browser, click **一键安装 / Install** on each component (or point it
at an existing endpoint), then **启动 / Launch**.
Prefer Docker Compose by hand? See **[deploy/compose/README.md](deploy/compose/README.md)**:
```bash
cd deploy/compose
cp .env.example .env && $EDITOR .env # set GPU index, ports, voice
bash setup-flashhead.sh # fetch FlashHead engine + weights
docker compose up -d --build # start cored + flashhead
# open https://
Layers with one contract between each pair (full write-up: [docs → Architecture](https://honwee.github.io/mynah/guide/architecture)):
```
Browser (WebRTC: video+audio, DataChannel)
│ /offer /human /interrupt_talk
▼
┌─────────────────────────────┐ ┌──────────────────────────────┐
│ cored (Go realtime core) │ gRPC │ Python workers │
│ WebRTC ⇄ session ⇄ pacing │◀──────▶│ • avatar (FlashHead/…) │
│ TTS · LLM · RAG · turn FSM │ stream │ • asr (SenseVoice) │
└──────────────┬──────────────┘ │ • turn (Smart Turn, audio)│
│ same process (--db) └──────────────────────────────┘
▼
┌─────────────────────────────┐ host-native (OpenAI-compatible):
│ control plane (admin API) │ TTS · ollama LLM · Postgres
│ config · KB/RAG · channels │
└─────────────────────────────┘
```
| Area | Package(s) | Role |
|---|---|---|
| Realtime core | `internal/session`, `internal/rtc`, `internal/mediaenc` | WebRTC peers, gRPC stream pacing, VP8/H.264/Opus encode, turn-taking FSM |
| Data plane | `internal/signaling` | Visitor HTTP: `/offer`, `/human`, `/interrupt_talk`, channels |
| Control plane | `internal/control`, `internal/store`, `internal/channel` | Admin API, config hot-apply, KB/RAG, publishable channels (needs `--db`) |
| Brains | `internal/llm` + `internal/tts` (local), `internal/qwenrt` (Qwen realtime), `internal/pipecatrt` (Pipecat bot) | Three interchangeable conversation brains behind one event surface |
| Instinct | `internal/instinct` | System One layer: rules or Jev picks a gesture clip while the brain talks; gated, paced, falls back |
| Providers | `internal/rag`, `internal/asrclient`, `internal/turnclient`, `internal/engineclient` | Pluggable RAG/ASR/turn/avatar clients |
| Feature extensions | `internal/{training,voiceclone,motion,avatarbake,rerank}` | Avatar training, voice clone, motion skeletons, idle + gesture bakes, RAG rerank |
| Avatars | `internal/avatarcatalog`, `workers/avatar/halfbody.py` | Built-in presets (incl. half-body FlashHead paste-back), per-avatar idle/gesture material |
| Assembly seam | `core/` | Interfaces (`AuthProvider`, `Reranker`, `AdminExtension`, …) for composing builds |
| Assembly / entry | `app/`, `cmd/cored` | Flag parsing, wiring, `app.Main(app.Options{...})` |
| Workers | `workers/{avatar,asr,turn,avatartrain,rerank}` | Python inference engines behind the gRPC/HTTP contracts |
| Contracts | `proto/{avatarengine,turn,asr}/v1` | gRPC service definitions (generated into `gen/`) |
## Performance
Measured on the public demo host (one RTX 4090, Qwen realtime brain, two engine workers online), 2026-10-11. Idle costs no GPU: silence is a baked loop replayed by cored.
| Metric | MuseTalk 1.5 (512) | FlashHead half-body (720×1280) |
|---|---|---|
| Turn end → first reply audio (cloud brain) | 1–170 ms | 1–170 ms |
| Turn end → first lip-synced audio from the engine | 0.40–0.72 s | 0.86–1.15 s |
| Engine: audio arrival → first frame (compute) | 0.32–0.55 s (0.24–0.48 s) | 0.79–0.90 s (0.62–0.76 s) |
| Local brain (LLM + EdgeTTS): request → first audio / first frame | 0.37–0.52 s / 0.73–1.07 s | — |
| Instinct decision (Jev via proxy / rules) | 0.4–0.85 s / 0 ms | same |
| Packet loss recovery (viewer PLI → keyframe) | ≤ 0.5 s | ≤ 0.5 s |
| VRAM, worker resident | 7.6 GB | 6.4 GB |
## Integration / API
cored exposes a small visitor-facing HTTP surface (default `:8020`, or `:8455`
TLS). Responses are wrapped `{code:0, data:…}` on success.
| Route | Method | Body | Purpose |
|---|---|---|---|
| `/offer` | POST | `{sdp, type}` | WebRTC offer → `{sdp, type:"answer", sessionid}`; starts a session |
| `/human` | POST | `{sessionid, text, type:"chat"\|"echo", interrupt}` | Make the avatar speak — `echo` = verbatim TTS, `chat` = through the LLM |
| `/interrupt_talk` | POST | `{sessionid}` | Barge-in: cancel in-flight speech |
| `/is_speaking` | POST | `{sessionid}` | Poll speaking state |
| `/channel/{slug}` · `/channel/{slug}/config` · `/channel/offer` | GET/POST | — | Published visitor channels (token / domain / CIDR access control) |
The admin/control plane (`/api/v1/...`, requires `--db` + bearer auth) adds
config, knowledge-base/RAG, channel CRUD, live-session introspection and a
playground. See `internal/control` for the full route set.
> **On "embeddable" today:** integration is via this **REST + WebRTC** surface
> (and the gRPC worker contracts), not a Go library import — `module mynah`
> is app-shaped, not yet published for `go get`. A documented HTTP API + a minimal
> web widget example are on the near-term roadmap.
### Pluggable engines (gRPC worker contracts)
Each worker speaks one streaming contract, so you can swap implementations:
- **`AvatarEngine`** (`proto/avatarengine/v1`) — `Session(stream ClientFrame) → stream ServerFrame)`: audio chunks in, encoded video+audio frames out, plus `Interrupt` / `Ready` / `Stats`. FlashHead and wav2lipLS both implement it.
- **`AsrEngine`** (`proto/asr/v1`) — audio packets in, `VadEvent` + `Transcript` (+ the utterance's `UtteranceAudio` for the turn detector) out.
- **`TurnDetector`** (`proto/turn/v1`) — `Predict(audio_pcm) → {eot_prob, end_of_turn}`. The reference worker is **Smart Turn v3.2** (Pipecat, BSD-2), bundled in-repo.
TTS and LLM are **OpenAI-compatible HTTP** (`/v1/audio/speech`, `/v1/chat/completions`),
so any compatible server or cloud endpoint drops in via config / `.env`.
## Commercial use & licensing
Mynah's own code is **Apache-2.0** ([LICENSE](LICENSE)). **The bundled
default avatar engines are usable commercially** — FlashHead is Apache-2.0,
wav2lipLS runs self-trained weights, and the only non-commercial engine (classic
Rudrabha Wav2Lip) has been removed. Full per-component provenance, caveats, and
the two items to verify for your deployment are in **[LICENSING.md](LICENSING.md)**.
If you clone real people's likenesses or voices, read
**[docs/compliance.md](docs/compliance.md)** — the training/cloning endpoints
are consent-gated by design, and deep-synthesis deployments are regulated in
most jurisdictions.
## Everything included, still composable
The full feature set — avatar training, idle baking, voice cloning, motion
skeletons, RAG reranking — ships in this one repository under Apache-2.0.
There is no paid edition and no feature wall. The assembly seam (`core/`
interfaces + `app.Main(app.Options{})`) remains public API: a distribution can
still compose a narrower build (drop an extension from `cmd/cored`, ship a
tighter `FeatureGate`) or inject richer implementations (SSO auth, custom
rerankers) without patching sources.
## Commercial support
Mynah is free and Apache-2.0, and will stay that way. If your company
wants help going to production, we offer paid engagements:
- **Deployment & tuning** — capacity planning, multi-GPU topologies, latency
tuning for your hardware.
- **Custom avatars & voices** — production-quality likeness pipelines beyond
the built-in trainer.
- **Integration & feature work** — bespoke engines, platform integrations,
priority fixes.
- **Hosted / managed** — we run it, you embed it.
Write to **mynah@au56.com**, or open an issue with the `commercial` label.
## Repository layout
```
app/ assembly + flags (app.Main seam)
cmd/cored/ binary entry point (registers the full feature set)
core/ assembly interfaces (compose narrower builds here)
internal/ realtime core, signaling, control plane, providers, features
proto/ gen/ gRPC contracts + generated stubs
web/ visitor widget (web/user) + admin console (web/admin)
workers/ Python engines (avatar / asr / turn / avatartrain / rerank)
deploy/compose/ docker-compose dev/prod stack (+ README)
deploy/setup/ first-run setup wizard (helper.py + scripts)
tools/probes/ headless acceptance probes (not shipped)
docs/ API contracts, compliance guide
```
## License
Apache-2.0 — see [LICENSE](LICENSE) and [LICENSING.md](LICENSING.md).
## Acknowledgments
Thanks to the [LINUX DO](https://linux.do) community for the early feedback and support.