---
title: Realtime API
description: Low-latency, bidirectional voice conversations via WebSockets.
icon: Zap
---

import { Tab, Tabs } from 'fumadocs-ui/components/tabs';
import { TypeTable } from 'fumadocs-ui/components/type-table';
import { Callout } from 'fumadocs-ui/components/callout';
import { RealtimeVoiceDemo } from '@/components/realtime-voice-demo';
import { Mic, Activity, BookOpen, Tractor, Phone, Car } from 'lucide-react';

The **Realtime API** enables fluid, natural voice conversations. It uses **WebSockets** to stream audio bi-directionally, achieving response times under **300ms**.

It is powered by **`አሌፍ-1.2-realtime-audio`**, allowing for natural interruptions and back-channeling.

<Callout type="info" title="Stream speech with your chosen Voice 2 voice">
For text-to-speech over HTTP or a persistent WebSocket, see [Streaming text-to-speech](/docs/capabilities/text-to-speech#streaming). That guide covers voice selection, session tickets, saved clips, and minute billing. The page below covers microphone conversations.
</Callout>

## Endpoint

<div className="addis-panel flex items-center gap-2 p-3 font-mono text-sm my-6">
  <span className="bg-fd-primary text-fd-primary-foreground px-2 py-0.5 rounded text-xs font-bold">WSS</span>
  <span className="text-fd-foreground">wss://relay.addisassistant.com/ws</span>
</div>

## Authentication

For browser and mobile clients, pass your API key as a query parameter:

```text
wss://relay.addisassistant.com/ws?apiKey=YOUR_API_KEY
```

---

## Industry Use Cases

Our Realtime model is already deployed across key African sectors. You can test these specific **Agents** in our Playground.

<div className="addis-offset-shell addis-grid grid-cols-1 md:grid-cols-3 mt-8">
  <div className="p-4 border border-fd-border rounded-lg bg-fd-card">
    <div className="flex items-center gap-2 mb-2 font-bold text-fd-foreground">
      <Phone className="w-4 h-4 text-fd-muted-foreground" /> Telecom Assistant
    </div>
    <div className="text-xs text-fd-muted-foreground">
      Automated customer support for balance checks, package purchasing, and troubleshooting in Amharic.
    </div>
  </div>
  
  <div className="p-4 border border-fd-border rounded-lg bg-fd-card">
    <div className="flex items-center gap-2 mb-2 font-bold text-fd-foreground">
      <Car className="w-4 h-4 text-fd-muted-foreground" /> Ride Hailing
    </div>
    <div className="text-xs text-fd-muted-foreground">
      Voice-first driver negotiation and location finding. "Where are you?" "I'm at Bole near the bank."
    </div>
  </div>

  <div className="p-4 border border-fd-border rounded-lg bg-fd-card">
    <div className="flex items-center gap-2 mb-2 font-bold text-fd-foreground">
      <Tractor className="w-4 h-4 text-fd-muted-foreground" /> Farmer's Assistant
    </div>
    <div className="text-xs text-fd-muted-foreground">
      Agricultural advisory for weather updates, crop disease diagnosis, and market prices in local dialects.
    </div>
  </div>
</div>

---

## Audio Format Requirements

<Callout type="error" title="Critical: Audio Encoding">
  The API expects **16-bit Signed Integer PCM (PCM16)** audio.
  
  **Developer Warning:** The standard browser `AudioContext` returns **32-bit Float** data (-1.0 to 1.0) by default. If you send this directly, it will sound like static noise. You **must** convert Float32 to Int16 before sending it over the WebSocket.
</Callout>

| Property | Value |
| :--- | :--- |
| **Encoding** | PCM 16-bit (Little Endian) |
| **Client Upload** | `audio/pcm;rate=16000` (Mono) |
| **Server Audio Output** | PCM16 Mono, typically 24,000Hz |

---

## Integration Guide

### 1. Browser Implementation (JavaScript)
This example establishes a connection, captures microphone audio, **converts it from Float32 to PCM16**, and handles the audio stream.

<Tabs items={['JavaScript (Browser)', 'Python']}>
  <Tab value="JavaScript (Browser)">
    ```javascript
    const API_KEY = "sk_YOUR_KEY";
    const WS_URL = `wss://relay.addisassistant.com/ws?apiKey=${encodeURIComponent(API_KEY)}`;
    const INPUT_RATE = 16000;
    const OUTPUT_RATE = 24000;

    let socket;
    let inputContext;
    let outputContext;
    let stream;
    let source;
    let processor;
    let canStreamAudio = false;
    let nextStartTime = 0;

    async function startRealtime() {
      // Output context for AI playback
      outputContext = new (window.AudioContext || window.webkitAudioContext)({
        sampleRate: OUTPUT_RATE,
      });
      await outputContext.resume();
      nextStartTime = outputContext.currentTime;

      // Input context for mic capture
      inputContext = new (window.AudioContext || window.webkitAudioContext)({
        sampleRate: INPUT_RATE,
      });
      await inputContext.resume();

      stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: false });
      source = inputContext.createMediaStreamSource(stream);
      processor = inputContext.createScriptProcessor(2048, 1, 1);

      // Connect processor through a muted gain so callback keeps firing without mic echo
      const mute = inputContext.createGain();
      mute.gain.value = 0;
      source.connect(processor);
      processor.connect(mute);
      mute.connect(inputContext.destination);

      processor.onaudioprocess = (event) => {
        if (!canStreamAudio || socket?.readyState !== WebSocket.OPEN) return;

        // Browser gives Float32 (-1..1), convert to PCM16 and send as base64 JSON envelope
        const float32 = event.inputBuffer.getChannelData(0);
        const int16 = floatTo16BitPCM(float32);

        socket.send(
          JSON.stringify({
            data: arrayBufferToBase64(int16.buffer),
            mimeType: "audio/pcm;rate=16000",
          }),
        );
      };

      socket = new WebSocket(WS_URL);

      socket.onopen = () => {
        console.log("Connected. Waiting for setupComplete...");
      };

      socket.onmessage = async (event) => {
        const message = JSON.parse(event.data);
        console.log("Event:", message);

        if (message.setupComplete || (message.type === "status" && /ready/i.test(message.message || ""))) {
          canStreamAudio = true;
          return;
        }

        const b64 = message?.serverContent?.modelTurn?.parts?.[0]?.inlineData?.data;
        if (typeof b64 === "string" && b64.length) {
          await playPcm16Base64(b64, OUTPUT_RATE);
        }

        if (message?.error) {
          console.error("Realtime error:", message.error);
        }
      };

      socket.onerror = (event) => {
        console.error("WebSocket error:", event);
      };

      socket.onclose = (event) => {
        console.log(`WebSocket closed code=${event.code} reason=${event.reason || "n/a"}`);
      };
    }

    function stopRealtime() {
      canStreamAudio = false;

      if (socket && (socket.readyState === WebSocket.OPEN || socket.readyState === WebSocket.CONNECTING)) {
        socket.close(1000, "client-stop");
      }

      if (processor) processor.disconnect();
      if (source) source.disconnect();
      if (stream) stream.getTracks().forEach((t) => t.stop());

      if (inputContext) inputContext.close();
      if (outputContext) outputContext.close();
    }

    function floatTo16BitPCM(float32Array) {
      const int16Array = new Int16Array(float32Array.length);
      for (let i = 0; i < float32Array.length; i++) {
        const s = Math.max(-1, Math.min(1, float32Array[i]));
        int16Array[i] = s < 0 ? s * 0x8000 : s * 0x7fff;
      }
      return int16Array;
    }

    function arrayBufferToBase64(arrayBuffer) {
      const bytes = new Uint8Array(arrayBuffer);
      let binary = "";
      for (let i = 0; i < bytes.length; i++) {
        binary += String.fromCharCode(bytes[i]);
      }
      return btoa(binary);
    }

    function base64ToUint8Array(base64) {
      const binary = atob(base64);
      const bytes = new Uint8Array(binary.length);
      for (let i = 0; i < binary.length; i++) {
        bytes[i] = binary.charCodeAt(i);
      }
      return bytes;
    }

    async function playPcm16Base64(base64Audio, sampleRate = 24000) {
      const bytes = base64ToUint8Array(base64Audio);
      const pcm16 = new Int16Array(bytes.buffer);
      const float32 = new Float32Array(pcm16.length);

      for (let i = 0; i < pcm16.length; i++) {
        float32[i] = pcm16[i] / 32768;
      }

      const buffer = outputContext.createBuffer(1, float32.length, sampleRate);
      buffer.copyToChannel(float32, 0);

      const src = outputContext.createBufferSource();
      src.buffer = buffer;
      src.connect(outputContext.destination);

      const startAt = Math.max(outputContext.currentTime, nextStartTime);
      src.start(startAt);
      nextStartTime = startAt + buffer.duration;
    }
    ```
  </Tab>
  <Tab value="Python">
    ```python
    import asyncio
    import base64
    import json
    import pyaudio
    import websockets

    # Audio Config
    FORMAT = pyaudio.paInt16
    CHANNELS = 1
    RATE = 16000
    CHUNK = 1024  # ~64ms

    async def realtime_chat():
        api_key = "sk_YOUR_KEY"
        uri = f"wss://relay.addisassistant.com/ws?apiKey={api_key}"
        ready_event = asyncio.Event()
        
        async with websockets.connect(uri) as websocket:
            # 1. Setup Microphone
            p = pyaudio.PyAudio()
            stream = p.open(
                format=FORMAT,
                channels=CHANNELS,
                rate=RATE,
                input=True,
                frames_per_buffer=CHUNK,
            )

            async def receive_messages():
                async for raw in websocket:
                    message = json.loads(raw)
                    print("<", message)

                    if message.get("setupComplete"):
                        ready_event.set()
                        continue

                    if (
                        message.get("type") == "status"
                        and "ready" in message.get("message", "").lower()
                    ):
                        ready_event.set()
                        continue

            print("Listening... (Press Ctrl+C to stop)")

            try:
                receive_task = asyncio.create_task(receive_messages())
                await ready_event.wait()

                while True:
                    # Read raw PCM16 bytes from mic and send as base64 JSON message
                    data = stream.read(CHUNK)
                    await websocket.send(
                        json.dumps(
                            {
                                "data": base64.b64encode(data).decode("utf-8"),
                                "mimeType": "audio/pcm;rate=16000",
                            }
                        )
                    )
                    
            except KeyboardInterrupt:
                pass
            finally:
                receive_task.cancel()
                stream.stop_stream()
                stream.close()
                p.terminate()

    asyncio.run(realtime_chat())
    ```
  </Tab>
</Tabs>

---

## Protocol & Events

The WebSocket handles bidirectional traffic.

### 1. Connect + Setup Handshake
After connecting, wait for a readiness event before streaming microphone frames.

```json
{
  "setupComplete": true
}
```

Some integrations may also emit:

```json
{
  "type": "status",
  "message": "Ready to start conversation"
}
```

### 2. Client to Server Audio
Send JSON envelopes with base64 PCM16 chunks.

```json
{
  "data": "BASE64_ENCODED_PCM16_CHUNK",
  "mimeType": "audio/pcm;rate=16000"
}
```

### 3. Server to Client
Server events are JSON messages.

#### AI Audio Response
The model audio is returned as base64 PCM16 under `inlineData.data`.

```json
{
  "serverContent": {
    "modelTurn": {
      "parts": [
        {
          "inlineData": {
            "data": "BASE64_ENCODED_PCM16_AUDIO"
          }
        }
      ]
    }
  }
}
```

#### Turn Complete & Usage
When the AI finishes speaking, the server sends a completion event with billing details.

```json
{
  "serverContent": {
    "turnComplete": true
  },
  "usageMetadata": {
    "totalBilledAudioDurationSeconds": 5.2
  }
}
```
#### Warning Messages
Sent if there are non-critical issues, such as billing thresholds.

```json
{
  "type": "warning",
  "message": "Your wallet balance is low. Please top up to avoid service interruption."
}
```

#### Error Messages
Sent if a critical failure occurs.

```json
{
  "error": {
    "message": "AI service error",
    "status": 500,
    "timestamp": "2025-07-15T10:00:00.000Z"
  }
}
```

### Connection Troubleshooting
- If the socket closes with `1006`, verify:
  - You used the correct endpoint.
  - The `apiKey` query value is valid.
  - Client audio is sent as JSON (`data` + `mimeType`), not raw binary.

---

## Live Interactive Demo

Use the embedded tool below to test the Realtime API directly from this page.

<RealtimeVoiceDemo />

### Standalone HTML (Local Testing)

Use this standalone file if you want to run the same flow outside the docs site.

```html
<!doctype html>
<html>
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>Realtime Voice Test</title>
  </head>
  <body style="font-family: Arial, sans-serif; margin: 24px;">
    <h2>Realtime Voice Test</h2>
    <input id="apiKey" type="password" placeholder="sk_..." style="width: 340px; padding: 8px;" />
    <button id="startBtn">Start</button>
    <button id="stopBtn" disabled>Stop</button>
    <pre id="log" style="margin-top: 16px; border: 1px solid #ddd; padding: 12px; min-height: 140px;"></pre>

    <script>
      const WS_URL = "wss://relay.addisassistant.com/ws";
      const INPUT_RATE = 16000;
      const OUTPUT_RATE = 24000;

      let ws;
      let inputCtx;
      let outputCtx;
      let stream;
      let sourceNode;
      let processorNode;
      let muteNode;
      let canStream = false;
      let nextStartTime = 0;

      const logEl = document.getElementById("log");
      const startBtn = document.getElementById("startBtn");
      const stopBtn = document.getElementById("stopBtn");
      const apiKeyEl = document.getElementById("apiKey");

      function log(message) {
        const t = new Date().toLocaleTimeString();
        logEl.textContent += `[${t}] ${message}\n`;
        logEl.scrollTop = logEl.scrollHeight;
      }

      function f32ToPcm16(float32) {
        const out = new Int16Array(float32.length);
        for (let i = 0; i < float32.length; i++) {
          const s = Math.max(-1, Math.min(1, float32[i]));
          out[i] = s < 0 ? s * 0x8000 : s * 0x7fff;
        }
        return out;
      }

      function abToBase64(arrayBuffer) {
        const bytes = new Uint8Array(arrayBuffer);
        let binary = "";
        for (let i = 0; i < bytes.length; i++) binary += String.fromCharCode(bytes[i]);
        return btoa(binary);
      }

      function base64ToAb(base64) {
        const bin = atob(base64);
        const bytes = new Uint8Array(bin.length);
        for (let i = 0; i < bin.length; i++) bytes[i] = bin.charCodeAt(i);
        return bytes.buffer;
      }

      async function playPcm16(arrayBuffer) {
        if (!outputCtx) return;
        const pcm16 = new Int16Array(arrayBuffer);
        const f32 = new Float32Array(pcm16.length);
        for (let i = 0; i < pcm16.length; i++) f32[i] = pcm16[i] / 32768;

        const buffer = outputCtx.createBuffer(1, f32.length, OUTPUT_RATE);
        buffer.copyToChannel(f32, 0);
        const src = outputCtx.createBufferSource();
        src.buffer = buffer;
        src.connect(outputCtx.destination);
        const startAt = Math.max(outputCtx.currentTime, nextStartTime);
        src.start(startAt);
        nextStartTime = startAt + buffer.duration;
      }

      async function stop() {
        canStream = false;
        if (ws && (ws.readyState === 0 || ws.readyState === 1)) ws.close(1000, "client-stop");
        ws = null;

        try { if (processorNode) processorNode.disconnect(); } catch {}
        try { if (sourceNode) sourceNode.disconnect(); } catch {}
        try { if (muteNode) muteNode.disconnect(); } catch {}
        if (stream) stream.getTracks().forEach((t) => t.stop());

        if (inputCtx) await inputCtx.close().catch(() => {});
        if (outputCtx) await outputCtx.close().catch(() => {});
        inputCtx = outputCtx = null;
        stream = sourceNode = processorNode = muteNode = null;
        nextStartTime = 0;
        startBtn.disabled = false;
        stopBtn.disabled = true;
        log("Stopped.");
      }

      async function start() {
        const apiKey = apiKeyEl.value.trim();
        if (!apiKey) return log("Missing API key.");

        startBtn.disabled = true;
        stopBtn.disabled = false;

        outputCtx = new AudioContext({ sampleRate: OUTPUT_RATE });
        await outputCtx.resume();
        nextStartTime = outputCtx.currentTime;

        inputCtx = new AudioContext({ sampleRate: INPUT_RATE });
        await inputCtx.resume();
        stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: false });
        sourceNode = inputCtx.createMediaStreamSource(stream);
        processorNode = inputCtx.createScriptProcessor(2048, 1, 1);
        muteNode = inputCtx.createGain();
        muteNode.gain.value = 0;

        processorNode.onaudioprocess = (event) => {
          if (!canStream || !ws || ws.readyState !== 1) return;
          const f32 = event.inputBuffer.getChannelData(0);
          const pcm16 = f32ToPcm16(f32);
          ws.send(JSON.stringify({ data: abToBase64(pcm16.buffer), mimeType: "audio/pcm;rate=16000" }));
        };

        sourceNode.connect(processorNode);
        processorNode.connect(muteNode);
        muteNode.connect(inputCtx.destination);

        const url = new URL(WS_URL);
        url.searchParams.set("apiKey", apiKey);
        ws = new WebSocket(url.toString());

        ws.onopen = () => log("Connected. Waiting for setupComplete...");

        ws.onmessage = async (event) => {
          const msg = JSON.parse(event.data);
          if (msg.setupComplete || (msg.type === "status" && /ready/i.test(msg.message || ""))) {
            canStream = true;
            return log("Setup complete. Streaming live.");
          }
          const b64 = msg?.serverContent?.modelTurn?.parts?.[0]?.inlineData?.data;
          if (typeof b64 === "string" && b64.length) await playPcm16(base64ToAb(b64));
          if (msg?.error) log(`Server error: ${JSON.stringify(msg.error)}`);
        };

        ws.onclose = (e) => log(`Closed code=${e.code} reason=${e.reason || "n/a"}`);
        ws.onerror = () => log("WebSocket error");
      }

      startBtn.onclick = () => void start();
      stopBtn.onclick = () => void stop();
      window.addEventListener("beforeunload", () => void stop());
    </script>
  </body>
</html>
```

---
## Capabilities Roadmap
We are rapidly expanding the Realtime engine.
<div className="addis-offset-shell addis-grid grid-cols-1 md:grid-cols-2 mt-8">
<div className="p-4 border border-fd-border rounded-lg bg-fd-card">
<h3 className="font-bold flex items-center gap-2 mb-2 text-fd-foreground">
<Activity className="w-4 h-4 text-fd-muted-foreground" />
VAD (Voice Activity Detection)
</h3>
<div className="text-sm text-fd-muted-foreground">
The model currently uses server-side VAD to determine when the user has stopped speaking.
</div>
</div>
<div className="p-4 border border-fd-border rounded-lg bg-fd-card opacity-80">
<div className="flex items-center justify-between mb-2">
<h3 className="font-bold flex items-center gap-2 text-fd-foreground">
<BookOpen className="w-4 h-4 text-fd-muted-foreground" />
Knowledge Base
</h3>
<span className="text-xs text-fd-muted-foreground px-2 py-0.5 rounded-full border border-fd-border font-medium">Coming Soon</span>
</div>
<div className="text-sm text-fd-muted-foreground">
Soon you will be able to attach PDFs or Text documents to the Realtime session. This will allow the voice assistant to answer questions specifically based on your uploaded data (RAG).
</div>
</div>
</div>