# AssemblyAI Self-Hosted Services

Docker Compose stacks for running AssemblyAI transcription services on your own
infrastructure. The stacks are organized by **service** (the API you talk to)
and, within each service, by **model**.

## Choosing a service and model

| Service | Directory | API | Models | GPU requirement |
|---------|-----------|-----|--------|-----------------|
| **Streaming** | [`streaming/`](streaming/) | WebSocket, real-time | Universal English + Multilingual | NVIDIA T4+ per ASR container |
| **Streaming** | [`streaming/`](streaming/) | WebSocket, real-time | Universal-3.6 Pro | NVIDIA L40S, RTX PRO 4500, or RTX PRO 6000 (preferred) |
| **Sync** | [`sync/`](sync/) | Synchronous HTTP, full-file | Universal-3.6 Pro | NVIDIA L40S, RTX PRO 4500, or RTX PRO 6000 (preferred) |
| **Diarization** | [`diarization/`](diarization/) | Synchronous HTTP, full-file | Speaker diarization | NVIDIA T4 or newer, 16 GB+ VRAM |

- **Streaming** transcribes a live audio stream over a WebSocket connection. One
  stack serves multiple models; the client selects the model per session. See
  [`streaming/docker/README.md`](streaming/docker/README.md).
- **Sync** transcribes a complete file in a single HTTP request/response (audio
  ≤ 20 min by default). It is self-contained — a single GPU container plus the
  license-and-usage-proxy, no load balancer. See [`sync/docker/README.md`](sync/docker/README.md).
- **Diarization** adds speaker labels to a transcript (e.g. the sync response).
  See [`diarization/docker/README.md`](diarization/docker/README.md).

[`examples/`](examples/) combines services: [`examples/sync-diarization/`](examples/sync-diarization/)
runs sync and diarization with one license-and-usage-proxy.

Each stack is self-contained under `<service>/docker/`: its compose file(s),
`.env.example`, example client, and `README.md` live together. Run compose
commands from inside that directory.

Each stack can also run on serverless GPUs instead of hardware you manage. Every
stack is a self-contained Modal App deployed with a single `modal deploy` — see
[`sync/modal/`](sync/modal/) and
[`streaming/modal/`](streaming/modal/).

## Repository layout

```
.
├── diarization/
│   └── docker/  # compose stack (speaker diarization)
├── examples/
│   └── sync-diarization/  # sync + diarization with one proxy
├── streaming/
│   ├── docker/  # compose stacks (Universal English/Multilingual, Universal-3.6 Pro)
│   └── modal/   # serverless-GPU packages
└── sync/
    ├── docker/  # compose stack (Universal-3.6 Pro)
    └── modal/   # serverless-GPU package
```

## Prerequisites (all services)

1. **AssemblyAI license** valid for the self-hosted product (the `license.jwt` file).
2. **Docker & Docker Compose**.
3. **GPU support**: NVIDIA Container Toolkit for GPU-enabled containers.
4. **AWS access**: credentials that can pull images from AssemblyAI's ECR.

### GPU runtime setup

All GPU images ship CUDA 13 runtime libraries, so the host needs an NVIDIA
driver that supports CUDA 13 (R580 series or newer). Verify the driver:
```bash
nvidia-smi
```

Install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html),
then verify the Docker runtime has GPU access:
```bash
docker run --rm --gpus all nvidia/cuda:13.0.1-base-ubuntu24.04 nvidia-smi
```

### ECR authentication

```bash
aws ecr get-login-password --region us-west-2 \
  | docker login --username AWS --password-stdin 344839248844.dkr.ecr.us-west-2.amazonaws.com
```

### License file

Place your AssemblyAI `license.jwt` in the directory of the stack you are
running (`streaming/docker/`, `sync/docker/` or `diarization/docker/`), or point the `LICENSE_FILE_PATH`
environment variable in that stack's compose file at your license file's location.

If your license uses usage-based billing, also set `USAGE_TRACKING_API_KEY` to any API key from your
AssemblyAI dashboard: in `.env` for Docker, or in the `aai-license` secret for Modal. Without it, the
license-and-usage-proxy shuts down at startup (see [Usage reporting](#usage-reporting)).

## Shared component: license-and-usage-proxy

Every service runs the **license-and-usage-proxy** alongside it. It validates
your license and, for usage-based licenses, reports usage to AssemblyAI. Its
configuration and operational behavior are the same regardless of service.

### Usage reporting

The proxy supports two billing modes based on your license:

- **Flat billing** — usage tracking is disabled; no extra configuration needed.
- **Usage-based billing** — the proxy reports usage to AssemblyAI's usage
  tracker. Set `USAGE_TRACKING_API_KEY` (any key from the AssemblyAI dashboard)
  for the `license-and-usage-proxy` service.

**Behavior with usage-based billing:**
- At startup the proxy validates connectivity by registering with
  https://usage-tracker.assemblyai.com. If validation fails, the proxy shuts down.
- Usage is batched and reported every few seconds, with automatic retries.
- If https://usage-tracker.assemblyai.com stays unreachable and all retries fail
  (after 5–60 minutes), the proxy terminates itself as a fail-safe to protect
  usage-data integrity. Your orchestrator should replace the container.
- If the in-memory usage queue exceeds 1000 items, the proxy logs a warning
  suggesting you upscale.

### License status endpoint

`GET /v1/status` reports the live license-validation state:

```json
{
  "state": "Ready | Connected | TrustBased | Failed",
  "last_successful_checkin": "2025-01-01T12:00:00.000000Z",
  "trust_expiration": "2025-01-05T12:00:00.000000Z"
}
```

- `Ready` — initial state before any validation has occurred.
- `Connected` — last validation check succeeded.
- `TrustBased` — last validation failed but is within the trust-window grace
  period; services keep serving.
- `Failed` — last validation failed and the trust window expired; serving
  containers shut down.

`last_successful_checkin` and `trust_expiration` are ISO 8601 timestamps (null
until the first successful validation).

### Production recommendations (license-and-usage-proxy)

- **Deployment**: gradual rollouts; alert on service restarts.
- **Resources**: 1 CPU + ≥ 2 GB RAM per container; prefer more small containers
  over fewer large ones.
- **Monitoring**: alert on `/v1/status` transitions to `TrustBased` (warning)
  and `Failed` (critical). For usage-based billing, also monitor usage-reporting
  warnings and restarts.
- **Dependencies**: requires a valid license mounted on the container
  filesystem (set `LICENSE_FILE_PATH`). For usage-based billing, also requires
  connectivity to https://usage-tracker.assemblyai.com at startup.
- **Availability**: run a few containers behind a load balancer.

## Changelog

### v1.2.0

#### Diarization v0.2.1 (2026-11-02)

- Bump `self-hosted-diarization` to `release-v0.2.1`.

#### Diarization — New Self-Hosted Stack (added 2026-10-14)

- **New stack** — [`diarization/docker/`](diarization/docker/) adds speaker labels
  to a transcript (`self-hosted-diarization:release-v0.2.1`). Use it with
  `self-hosted-streaming-license-and-usage-proxy:release-v1.3.0`.
- **Sync + diarization example** — [`examples/sync-diarization/`](examples/sync-diarization/)
  runs sync and diarization with one license-and-usage-proxy and one license
  file (added 2026-10-21).
- **Sync project name** — `sync/docker/docker-compose.universal-3-6-pro.yml`
  now sets the Compose project name `sync` (it was `docker`), so the sync and
  diarization stacks do not share a proxy. Stop a running sync stack before you
  pull this change, or stop it after with
  `docker compose -p docker -f docker-compose.universal-3-6-pro.yml down`.

#### Streaming — Universal-3.5 Pro stack removed (2026-10-14)

The Universal-3.5 Pro streaming stack is superseded by the Universal-3.6 Pro
stack added on 2026-10-06 (below). If you are still on it:

- **Compose file removed** — `streaming/docker/docker-compose.universal-3-5-pro.yml`
  is gone; run `streaming/docker/docker-compose.universal-3-6-pro.yml` instead.
  Bring the old stack `down` before bringing the new one `up`.
- **Image env var** — `STREAMING_ASR_UNIVERSAL_3_5_PRO_IMAGE` is no longer read;
  set `STREAMING_ASR_UNIVERSAL_3_6_PRO_IMAGE` to
  `344839248844.dkr.ecr.us-west-2.amazonaws.com/self-hosted-streaming-asr-universal-3-6-pro:release-v1.2.0`
  (see `streaming/docker/.env.example`).
- **`speech_model` value** — clients must connect with
  `speech_model=universal-3-6-pro`. `nginx_streaming_asr.conf` no longer routes
  `universal-3-5-pro`, so sessions requesting it fail.
- **Modal** — `streaming/modal/modal_app_universal_3_5_pro.py` is removed; deploy
  `modal_app_universal_3_6_pro.py` (app `aai-streaming-u36pro`) and stop the old
  `aai-streaming-u3pro` app.
- Connection parameters, messages, mode names, and GPU requirements are
  unchanged. The Universal-3.5 Pro streaming features listed in this release
  carry over to Universal-3.6 Pro.

#### Sync — Universal-3.5 Pro replaced by Universal-3.6 Pro (added 2026-10-14)

The sync stack now serves **Universal-3.6 Pro** from
`self-hosted-sync-asr-universal-3-6-pro:release-v1.2.0`. If you are upgrading
from the Universal-3.5 Pro sync stack:

- **Compose file renamed** — `sync/docker/docker-compose.universal-3-5-pro.yml`
  is now `sync/docker/docker-compose.universal-3-6-pro.yml`. Bring the old
  stack `down` before bringing the new one `up`.
- **Image changed** — point `SYNC_API_IMAGE` at
  `self-hosted-sync-asr-universal-3-6-pro:release-v1.2.0` (the variable name is
  unchanged). The `release-v1.2.0` `self-hosted-streaming-license-and-usage-proxy`
  image is unchanged.
- **Drop-in for clients** — Same routes, request and response formats, and
  config options; no client change is needed. Usage reports are stamped
  `universal-3-6-pro`.
- **20-minute audio by default** — The stack now accepts audio up to 20
  minutes and request bodies up to 256 MB by default
  (`MAX_AUDIO_DURATION_MS=1200000`, `MAX_REQUEST_BYTES=268435456`, up from
  120 s / 40 MB). Both remain configurable in `.env`; set the old values to
  keep the previous limits.
- **Modal** — `sync/modal/modal_app.py` deploys Universal-3.6 Pro under the
  same app name (`aai-sync-u3pro`), so `modal deploy` upgrades an existing
  deployment in place. The image repository changed, so that deploy pulls and
  converts a new image; refresh the `aai-ecr-credentials` secret first if its
  session credentials have expired.

#### Universal-3.6 Pro — New Streaming Stack (added 2026-10-06)

- **New stack** — `streaming/docker/docker-compose.universal-3-6-pro.yml` serves
  Universal-3.6 Pro from `self-hosted-streaming-asr-universal-3-6-pro:release-v1.2.0`
  (`STREAMING_ASR_UNIVERSAL_3_6_PRO_IMAGE`). Clients connect with
  `speech_model=universal-3-6-pro`; `nginx_streaming_asr.conf` routes it to the
  new `streaming-asr-universal-3-6-pro` backend. A Modal package,
  `streaming/modal/modal_app_universal_3_6_pro.py`, is included.
- **Drop-in for Universal-3.5 Pro** — Same connection parameters, messages,
  modes, and GPU requirements. It runs with the existing `release-v1.2.0`
  `self-hosted-streaming-api` and `self-hosted-streaming-license-and-usage-proxy`
  images; nothing else needs upgrading. To switch, bring the Universal-3.5 Pro
  stack `down`, bring the Universal-3.6 Pro stack `up`, and change the clients'
  `speech_model`.
- **`max_accuracy` retuned for the new model** — Compared with Universal-3.5
  Pro, a turn that ends mid-entity is held for half as long, and an ordinary
  sentence waits about 0.1 s more silence before it ends. `min_latency` and
  `balanced` are unchanged.
- **No stray non-CJK output on streams pinned to CJK languages** — When
  `language_codes` lists only `zh`, `yue`, `ja`, or `ko`, a decoded segment with
  letters but no CJK characters is dropped. Segments that contain any CJK
  character are kept whole, so Latin words inside CJK speech survive.

#### Streaming API — New Features

- **`acknowledge_silence`** — Opt in to receive a `Silence` message about once
  per second while no speech is being transcribed, so clients can tell a quiet
  room apart from a stalled connection (Universal-3.5 Pro).
- **Heartbeat on Universal English and Multilingual** — The `session_heartbeat`
  opt-in now works on all three models.

#### Streaming API — Fixes and Improvements

- **Webhook hardening** — `webhook_url` targets on link-local or cloud-metadata
  addresses are rejected, and delivery no longer follows redirects.

#### Universal-3.5 Pro (streaming)

- **Entity-aware endpointing in all modes** — A turn that ends mid-number,
  mid-spelling or mid-address is held until it completes, while ordinary
  sentences still end at the mode's minimum silence.
- **Retuned mode thresholds** — The three modes now differ mainly in how long
  they wait for an entity to complete; `max_accuracy` ends ordinary sentences
  sooner than before.
- **Repetition-loop guardrails** — Runaway decodes that repeat a phrase dozens
  of times are collapsed and capped.
- **Speaker tags stripped** — Bracketed speaker tags no longer leak into
  transcript words.
- **13 more languages for `language_codes`** — Korean, Russian, Catalan,
  Galician, Romanian, Estonian, Persian, Cantonese, Afrikaans, Marathi, Zulu,
  Xhosa and Norwegian Nynorsk.
- **No CJK characters on streams pinned to other languages** — Stray Chinese,
  Japanese or Korean characters are dropped when `language_codes` names only
  non-CJK languages.
- **Multilingual sentence-final punctuation** — Turns in Japanese, Chinese,
  Arabic, Urdu and Hindi end on their native punctuation instead of waiting for
  the silence timeout.
- **More robust under load** — Timed-out inference requests are aborted, so
  overload degrades gracefully instead of failing sessions.
- **Working container health check** — The compose file's gRPC health probe now
  reports the container healthy.

#### Universal English and Multilingual ASR

No model change.

- **CUDA 13 runtime** — The host NVIDIA driver must now support CUDA 13 (R580
  or newer); T4 and newer GPUs remain supported.
- **Heartbeat support** — See Streaming API above.
- **Clean exit on fatal worker errors** — The container exits instead of
  hanging, so the restart policy takes over.

#### Sync — New Features

Shipped in `self-hosted-sync-asr-u3-pro:release-v1.1.0` (2026-09-11) and
carried forward.

- **Compressed audio** — `/transcribe` accepts AAC, MP3, M4A, OGG, WebM and
  FLAC in addition to WAV and raw PCM.
- **Streamed uploads** — `POST /v1/transcribe/live` transcribes while the audio
  is still uploading (WAV/PCM only).
- **Word timestamps** — Set `"timestamps": true` for per-word `start`/`end`;
  requires `SYNC_BFA_ALIGNMENT_ENABLED=true` on the container.
- **13 more languages** — The same list as streaming above.
- **Higher config limits** — Longer `prompt`, `keyterms_prompt` and
  `conversation_context` are accepted.
- **`GET /warm`** — Pre-warms a connection ahead of a latency-sensitive
  request.
- **`/v1` route prefix** — Routes are served at `/v1/...`; unprefixed paths
  continue to work.
- **`keyterms_prompt`** — Preferred name for the keyterms list; `keyterms` and
  `word_boost` remain accepted as aliases.

#### Sync — Fixes and Improvements

- Self-hosted usage reports are now accepted end to end; on v1.0.0 they were
  rejected, which on usage-based licenses also crash-looped the proxy.
- Guardrails against runaway decodes and hallucinated segments.
- Bracketed speaker tags no longer leak into transcripts.
- Faster large uploads and faster prompt trimming.

#### Sync — Behavior changes

- Words carry `start`/`end` only when `"timestamps": true` is requested.
- A caller-supplied `prompt` is added to the language directive instead of
  replacing it, and an explicit `language_code: "en"` now steers the model;
  output can differ for these requests.
- More than 100 keyterms are rejected with `400`.

#### License-and-usage-proxy

- No functional changes; rebuilt so every image shares one tag.

#### Also new since v1.0.0

Shipped in the `release-v1.0.1` images but not previously listed:

- **`session_heartbeat`** — Opt in to a periodic `Heartbeat` message
  (Universal-3.5 Pro).
- **`encoding=aac`** — AAC audio is accepted alongside PCM and Opus.
- **End-of-turn details on `Turn`** — Final turns carry `silence_before_eot_ms`
  and `turn_end_reason` (Universal-3.5 Pro).
- **Language detection covers every supported language code.**
- **Faster finals** — Speculative decoding roughly halves time-to-final after a
  `ForceEndpoint` with no change in accuracy (Universal-3.5 Pro).
- **Partials every second by default** — Send `continuous_partials=false` for
  the slower cadence (Universal-3.5 Pro).
- **English-first start for unsteered streams** — Prevents a first short turn
  from committing in the wrong language (Universal-3.5 Pro).
- **`VLLM_ATTENTION_BACKEND` override** — For GPUs the bundled kernels do not
  cover, such as NVIDIA B200 (Universal-3.5 Pro, Sync).
- **Bounded session lifetime** — A session whose client stopped sending closes
  within 60 s, and ASR streams that end without a completion signal are
  retried.

### v1.0.0

#### Repository restructure

The repository is now organized by **service**: [`streaming/`](streaming/)
(WebSocket, real-time) and [`sync/`](sync/) (synchronous HTTP, full-file). Each
service directory is self-contained — compose files, `.env.example`, example
client, and README live together. Compose files moved accordingly (e.g.
`docker-compose.yml` → `streaming/docker-compose.english-multilang.yml`).

#### Sync — New Self-Hosted Service (NEW)

This release introduces the **Sync self-hosted service**
(`sync/docker-compose.universal-3-5-pro.yml`), serving the Universal-3.5 Pro
model. It transcribes a complete audio file (≤ 120 s) in a single
`POST /transcribe` request/response — a single GPU container plus the
license-and-usage-proxy, no load balancer. It exposes `GET /readyz` (200 once
the model is warm) for readiness probes. See [`sync/docker/README.md`](sync/docker/README.md).

#### Streaming — U3 Pro replaced by Universal-3.5 Pro (BREAKING)

The U3 Pro stack introduced in v0.6.0 is replaced by the **Universal-3.5 Pro
stack** (`streaming/docker-compose.universal-3-5-pro.yml`). If you are
upgrading from the v0.6.0 U3 Pro stack:

- **`speech_model` value changed** — clients must connect with
  `speech_model=universal-3-5-pro`. The previous value `u3-rt-pro` is no longer
  routed by the load balancer and sessions requesting it will fail.
- **Image env var renamed** — `STREAMING_ASR_U3PRO_IMAGE` is now
  `STREAMING_ASR_UNIVERSAL_3_5_PRO_IMAGE`, pointing at the
  `self-hosted-streaming-asr-universal-3-5-pro` image.
- **Compose file renamed** — `docker-compose.u3pro.yml` is now
  `streaming/docker-compose.universal-3-5-pro.yml`.
- **Hardware recommendations updated** — tested on NVIDIA L40S (48 GB) and
  RTX PRO 6000 Blackwell (96 GB); the model weights use ~11 GB of VRAM, the
  rest becomes KV cache and sets max concurrency.

#### Images

`release-v1.0.1` is published for `self-hosted-streaming-api` (adds the
peer-aborted-handshake logging fix) and `self-hosted-streaming-asr-universal-3-5-pro`;
`release-v1.0.0` for `self-hosted-streaming-license-and-usage-proxy` and
`self-hosted-sync-asr-u3-pro`.
The English and Multilingual ASR images are unchanged since v0.6.0 — keep
`STREAMING_ASR_ENGLISH_IMAGE` and `STREAMING_ASR_MULTILANG_IMAGE` at
`release-v0.6.0` (see `streaming/.env.example`).

### v0.6.0

#### U3 Pro — New Self-Hosted Stack (NEW)

This release introduces the **U3 Pro self-hosted stack** (`docker-compose.u3pro.yml`), which serves the U3 Pro streaming model. U3 Pro delivers significant improvements over the universal English model on complex entities, short utterances, and end-of-turn (EOT) latency, and is targeted at voice agent scenarios.

Hardware: NVIDIA L4 / A10 / A100 / L40S / H100 (24 GB+ VRAM).

Highlights of U3 Pro behavior delivered with this release:

- **New transcription prompt** ("Transcribe verbatim with standard punctuation. Include filler words and incomplete utterances.") — 22% reduction in voice-agent hallucinations, 10% WER and 29% short-utterance error-rate reduction on voice-agent traffic, 5% improvement on medical, and improved EP F1.
- **Continuous partials during long turns** — partials are emitted incrementally instead of being delayed; turns now stitch up to 60s instead of hard-cutting at 16s/32s.
- **Early partial at 750ms** of detected speech for faster UI feedback.

#### Streaming API — New Features

- **`continuous_partials` query parameter** — clients can opt into continuous partials during long turns.
- **Structured logging** — both the U3 Pro ASR server and the universal ASR server now honor `USE_STRUCTURED_LOGGING`, matching the streaming-api behavior.

#### Other Improvements

- Various logging and metrics improvements across the streaming-api and ASR services.
- Bug fixes and stability improvements.

### v0.5.0

#### English ASR Model

A new English model is released, which produces already-formatted outputs directly and delivers large quality gains on digits, telephony, medical, and CI segments:

- **34% improvement on digit sequence error rate (DSER)**
- **17% improvement on telephony WER**
- **12% average improvement on medical WER**
- **10% average improvement on CI segments WER**
- **~2.4% absolute F1 score improvement on keyterms prompting**
- **Significantly improved timestamp accuracy** — resolves overlapping and zero-duration word issues

#### Multilingual ASR Model

- **~70% absolute improvement in timestamp accuracy** — fixes overlapping words and zero-duration word bugs

#### Streaming API — New Features

- **Error and Warning WebSocket message types** — Dedicated message types that let clients distinguish actionable errors from non-fatal warnings without relying on close codes.
- **Configuration echoed in SessionBegins** — The `SessionBegins` message now includes the resolved session configuration so clients can verify applied settings.
- **Explicit speech-model selection** — Clients explicitly select the speech model at session start.

#### Streaming API — Fixes and Improvements

- **More specific WebSocket close codes** for session termination scenarios, making client-side error handling more precise.
- **Improved `word_finalized` events** — All word finalizations are emitted (not only the last word of a turn).

#### Other Improvements

- Various logging, metrics, and observability improvements across the streaming-api and ASR services.
- Bug fixes and stability improvements.

### v0.4.0

#### English ASR Model

Major improvements to short utterance handling and hallucination reduction:

- **100% reduction in hallucinations**
- **12.8% improvement on short utterances** - Better performance for voice agent use cases
- **7.39% improvement on digit sequence error rate**
- **1.75% improvement on proper nouns**
- **0.46% improvement on CI segments**
- **0.39% improvement on accented speech**

#### Multilingual ASR Model

- **Context biasing support** - Customers can now use context biasing (model-based biasing) with the multilingual model

#### Other Improvements

- Increased concurrent session handling per container, leading to reduced deployment costs
- Improved observability for the license-and-usage-proxy service
- Various bug fixes and stability improvements