{
"cells": [
{
"cell_type": "markdown",
"id": "3fc2fd8d",
"metadata": {},
"source": [
"# Low Latency Voice Assistant with ElevenLabs and Claude\n",
"\n",
"This notebook demonstrates how to build a low-latency voice assistant using ElevenLabs for speech-to-text and text-to-speech, combined with Claude for intelligent responses. We'll measure the performance gains from streaming responses to minimize latency.\n",
"\n",
"In this notebook, we will demonstrate how to:\n",
"\n",
"1. Convert text to speech using ElevenLabs TTS\n",
"2. Transcribe audio using ElevenLabs speech-to-text\n",
"3. Generate responses with Claude\n",
"4. Optimize latency using Claude's streaming API\n",
"\n",
"---\n",
"\n",
"## Installation\n",
"\n",
"First, install the required dependencies:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bf6d988c",
"metadata": {},
"outputs": [],
"source": [
"%pip install --upgrade pip"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bf25eeed",
"metadata": {},
"outputs": [],
"source": [
"%pip install -r requirements.txt"
]
},
{
"cell_type": "markdown",
"id": "9d383a4d",
"metadata": {},
"source": [
"## Imports\n",
"\n",
"Import the necessary libraries for ElevenLabs integration, Claude API access, and audio playback:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "2f00884d",
"metadata": {},
"outputs": [],
"source": [
"import io\n",
"import os\n",
"import time\n",
"\n",
"import anthropic\n",
"import elevenlabs\n",
"from dotenv import load_dotenv\n",
"from IPython.display import Audio"
]
},
{
"cell_type": "markdown",
"id": "86b29f9f",
"metadata": {},
"source": [
"## API Keys\n",
"\n",
"Set up your API keys for both ElevenLabs and Anthropic.\n",
"\n",
"**Setup Instructions:**\n",
"\n",
"1. Copy `.env.example` to `.env` in this directory\n",
"2. Edit `.env` and add your actual API keys:\n",
" - Get your ElevenLabs API key: https://elevenlabs.io/app/developers/api-keys\n",
" - Get your Anthropic API key: https://console.anthropic.com/settings/keys\n",
"\n",
"The keys will be automatically loaded from the `.env` file."
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "e16ac6e5",
"metadata": {},
"outputs": [],
"source": [
"# Load environment variables from .env file\n",
"\n",
"load_dotenv()\n",
"\n",
"ELEVENLABS_API_KEY = os.getenv(\"ELEVENLABS_API_KEY\")\n",
"ANTHROPIC_API_KEY = os.getenv(\"ANTHROPIC_API_KEY\")"
]
},
{
"cell_type": "markdown",
"id": "fe6020be",
"metadata": {},
"source": [
"## Initialize Clients\n",
"\n",
"Create client instances for both ElevenLabs and Anthropic services:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "1a68bea8",
"metadata": {},
"outputs": [],
"source": [
"assert ELEVENLABS_API_KEY is not None, (\n",
" \"ERROR: ELEVENLABS_API_KEY not found. Please copy .env.example to .env and add your API keys.\"\n",
")\n",
"assert ANTHROPIC_API_KEY is not None, (\n",
" \"ERROR: ANTHROPIC_API_KEY not found. Please copy .env.example to .env and add your API keys.\"\n",
")\n",
"\n",
"elevenlabs_client = elevenlabs.ElevenLabs(\n",
" api_key=ELEVENLABS_API_KEY, base_url=\"https://api.elevenlabs.io\"\n",
")\n",
"\n",
"anthropic_client = anthropic.Anthropic(api_key=ANTHROPIC_API_KEY)"
]
},
{
"cell_type": "markdown",
"id": "e6e89d06",
"metadata": {},
"source": [
"## List Available Models and Voices\n",
"\n",
"Explore the available ElevenLabs models and voices. We'll automatically select the first available voice for the assistant's responses:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "1daed00d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Available Models and Voices:\n",
"\n",
"Eleven v3 (alpha): eleven_v3\n",
"Eleven Multilingual v2: eleven_multilingual_v2\n",
"Eleven Flash v2.5: eleven_flash_v2_5\n",
"Eleven Turbo v2.5: eleven_turbo_v2_5\n",
"Eleven Turbo v2: eleven_turbo_v2\n",
"Eleven Flash v2: eleven_flash_v2\n",
"Eleven Multilingual v1: eleven_multilingual_v1\n",
"Eleven English v1: eleven_monolingual_v1\n",
"Eleven English v2: eleven_english_sts_v2\n",
"Eleven Multilingual v2: eleven_multilingual_sts_v2\n",
"\n",
"Rachel: 21m00Tcm4TlvDq8ikWAM\n",
"Drew: 29vD33N1CtxCmqQRPOHJ\n",
"Clyde: 2EiwWnXFnvU5JabPnv8n\n",
"Paul: 5Q0t7uMcjvnagumLfvZi\n",
"Aria: 9BWtsMINqrJLrRacOk9x\n",
"Domi: AZnzlk1XvdvUeBnXmlld\n",
"Dave: CYw3kZ02Hs0563khs1Fj\n",
"Roger: CwhRBWXzGAHq8TQ4Fs17\n",
"Fin: D38z5RcWu1voky8WS1ja\n",
"Sarah: EXAVITQu4vr4xnSDxMaL\n",
"\n",
"Selected voice: Rachel with ID: 21m00Tcm4TlvDq8ikWAM\n"
]
}
],
"source": [
"print(\"Available Models and Voices:\\n\")\n",
"for model in elevenlabs_client.models.list():\n",
" print(f\"{model.name}: {model.model_id}\")\n",
"\n",
"print()\n",
"\n",
"voices = elevenlabs_client.voices.search().voices\n",
"for voice in voices:\n",
" print(f\"{voice.name}: {voice.voice_id}\")\n",
"\n",
"# Select the first voice for assistant responses\n",
"selected_voice = voices[0]\n",
"VOICE_ID = selected_voice.voice_id\n",
"\n",
"print(f\"\\nSelected voice: {selected_voice.name} with ID: {VOICE_ID}\")"
]
},
{
"cell_type": "markdown",
"id": "d82d505c",
"metadata": {},
"source": [
"## Generate Input Audio\n",
"\n",
"Create a sample audio file using ElevenLabs text-to-speech. This will simulate user input for our voice assistant:"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "dada5148",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
" "
],
"text/plain": [
""
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"audio = elevenlabs_client.text_to_speech.convert(\n",
" voice_id=VOICE_ID, # Use the dynamically selected voice\n",
" output_format=\"mp3_44100_128\",\n",
" model_id=\"eleven_v3\",\n",
" text=\"Hello, Claude. \",\n",
")\n",
"\n",
"audio_data = io.BytesIO()\n",
"for chunk in audio:\n",
" audio_data.write(chunk)\n",
"\n",
"Audio(audio_data.getvalue())"
]
},
{
"cell_type": "markdown",
"id": "60c8bd1e",
"metadata": {},
"source": [
"## Speech Transcription\n",
"\n",
"Transcribe the audio input using ElevenLabs' speech-to-text model. We'll measure the transcription latency:"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "3ba9ff5c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Transcribed text: Hello, Claude.\n",
"Transcription time: 0.54 seconds\n"
]
}
],
"source": [
"audio_data.seek(0)\n",
"\n",
"start_time = time.time()\n",
"\n",
"transcription = elevenlabs_client.speech_to_text.convert(file=audio_data, model_id=\"scribe_v1\")\n",
"\n",
"end_time = time.time()\n",
"transcription_time = end_time - start_time\n",
"\n",
"print(f\"Transcribed text: {transcription.text}\")\n",
"print(f\"Transcription time: {transcription_time:.2f} seconds\")"
]
},
{
"cell_type": "markdown",
"id": "ad94cc69",
"metadata": {},
"source": [
"## Get a Response from Claude\n",
"\n",
"Send the transcribed text to Claude and measure the response time. We're using `claude-haiku-4-5` for fast, high-quality responses:"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "3b515831",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Hello! It's nice to meet you. How can I help you today?\n",
"\n",
"Response time: 1.03 seconds\n"
]
}
],
"source": [
"start_time = time.time()\n",
"\n",
"message = anthropic_client.messages.create(\n",
" model=\"claude-haiku-4-5\",\n",
" max_tokens=1000,\n",
" temperature=0,\n",
" messages=[{\"role\": \"user\", \"content\": transcription.text}],\n",
")\n",
"\n",
"end_time = time.time()\n",
"non_streaming_response_time = end_time - start_time\n",
"\n",
"print(message.content[0].text)\n",
"print(f\"\\nResponse time: {non_streaming_response_time:.2f} seconds\")"
]
},
{
"cell_type": "markdown",
"id": "9716dbf8",
"metadata": {},
"source": [
"## Optimize with Streaming\n",
"\n",
"Improve response latency by using Claude's streaming API. This allows us to receive the first tokens much faster, significantly reducing perceived latency:"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "fc747c00",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Hello! It's nice to meet you. How can I help you today?\n",
"\n",
"Streaming time to first token: 0.71 seconds - reducing perceived latency by 30.71%\n"
]
}
],
"source": [
"start_time = time.time()\n",
"first_token_time = None\n",
"\n",
"claude_full_response = \"\"\n",
"\n",
"with anthropic_client.messages.stream(\n",
" model=\"claude-haiku-4-5\",\n",
" max_tokens=1000,\n",
" temperature=0,\n",
" messages=[{\"role\": \"user\", \"content\": transcription.text}],\n",
") as stream:\n",
" for text in stream.text_stream:\n",
" claude_full_response += text\n",
" print(text, end=\"\", flush=True)\n",
" if first_token_time is None:\n",
" first_token_time = time.time()\n",
"\n",
"streaming_time_to_first_token = first_token_time - start_time\n",
"print(\n",
" f\"\\n\\nStreaming time to first token: {streaming_time_to_first_token:.2f} seconds - reducing perceived latency by {(non_streaming_response_time - streaming_time_to_first_token) * 100 / non_streaming_response_time:.2f}%\"\n",
")"
]
},
{
"cell_type": "markdown",
"id": "0bbecfb4",
"metadata": {},
"source": [
"Text to speech. Similar to above, we can stream the response to reduce the silence."
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "e5ec455c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Streaming TTS time to first audio chunk: 0.39 seconds\n"
]
},
{
"data": {
"text/html": [
"\n",
" \n",
" "
],
"text/plain": [
""
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"start_time = time.time()\n",
"first_audio_chunk_time = None\n",
"\n",
"audio_buffer = io.BytesIO()\n",
"\n",
"audio_generator = elevenlabs_client.text_to_speech.stream(\n",
" voice_id=VOICE_ID,\n",
" output_format=\"mp3_44100_128\",\n",
" text=claude_full_response,\n",
" model_id=\"eleven_turbo_v2_5\",\n",
")\n",
"\n",
"for chunk in audio_generator:\n",
" if first_audio_chunk_time is None:\n",
" first_audio_chunk_time = time.time()\n",
" audio_buffer.write(chunk)\n",
"\n",
"streaming_tts_time_to_first_chunk = first_audio_chunk_time - start_time\n",
"print(f\"Streaming TTS time to first audio chunk: {streaming_tts_time_to_first_chunk:.2f} seconds\")\n",
"\n",
"Audio(audio_buffer.getvalue())"
]
},
{
"cell_type": "markdown",
"id": "927ab0e0",
"metadata": {},
"source": [
"## Streaming Claude Directly to TTS (Sentence-by-Sentence)\n",
"\n",
"We've optimized Claude's streaming and TTS separately, but can we combine them? Let's stream Claude's response and synthesize audio as soon as we have complete sentences.\n",
"\n",
"This approach detects sentence boundaries (using punctuation like `.`, `!`, `?`) and immediately sends each sentence to TTS, further reducing latency."
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "0d063ed9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Hello! It's nice to meet you. How can I help you today?\n",
"\n",
"Time to first audio: 1.48 seconds\n"
]
},
{
"data": {
"text/html": [
"\n",
" \n",
" "
],
"text/plain": [
""
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import re\n",
"\n",
"sentence_pattern = re.compile(r\"[.!?]+\")\n",
"sentence_buffer = \"\"\n",
"audio_chunks = []\n",
"\n",
"start_time = time.time()\n",
"first_audio_time = None\n",
"\n",
"with anthropic_client.messages.stream(\n",
" model=\"claude-haiku-4-5\",\n",
" max_tokens=1000,\n",
" temperature=0,\n",
" messages=[{\"role\": \"user\", \"content\": transcription.text}],\n",
") as stream:\n",
" for text in stream.text_stream:\n",
" print(text, end=\"\", flush=True)\n",
" sentence_buffer += text\n",
"\n",
" if sentence_pattern.search(sentence_buffer):\n",
" sentences = sentence_pattern.split(sentence_buffer)\n",
"\n",
" # Process all complete sentences (all but the last element)\n",
" for i in range(len(sentences) - 1):\n",
" complete_sentence = sentences[i].strip()\n",
" if complete_sentence:\n",
" audio_gen = elevenlabs_client.text_to_speech.stream(\n",
" voice_id=VOICE_ID,\n",
" output_format=\"mp3_44100_128\", # Free tier format\n",
" text=complete_sentence,\n",
" model_id=\"eleven_turbo_v2_5\",\n",
" )\n",
"\n",
" sentence_audio = io.BytesIO()\n",
" for chunk in audio_gen:\n",
" if first_audio_time is None:\n",
" first_audio_time = time.time()\n",
" sentence_audio.write(chunk)\n",
"\n",
" audio_chunks.append(sentence_audio.getvalue())\n",
"\n",
" sentence_buffer = sentences[-1]\n",
"\n",
"if sentence_buffer.strip():\n",
" audio_gen = elevenlabs_client.text_to_speech.stream(\n",
" voice_id=VOICE_ID,\n",
" output_format=\"mp3_44100_128\",\n",
" text=sentence_buffer.strip(),\n",
" model_id=\"eleven_turbo_v2_5\",\n",
" )\n",
" sentence_audio = io.BytesIO()\n",
" for chunk in audio_gen:\n",
" sentence_audio.write(chunk)\n",
" audio_chunks.append(sentence_audio.getvalue())\n",
"\n",
"sentence_streaming_time_to_first_audio = first_audio_time - start_time\n",
"print(f\"\\n\\nTime to first audio: {sentence_streaming_time_to_first_audio:.2f} seconds\")\n",
"\n",
"combined_pcm = b\"\".join(audio_chunks)\n",
"Audio(combined_pcm)"
]
},
{
"cell_type": "markdown",
"id": "dho5t9gbaie",
"metadata": {},
"source": [
"### The Problem: Disconnected Audio\n",
"\n",
"While this approach achieves excellent latency, there's a quality issue. Each sentence is synthesized independently, which causes the audio to sound disconnected and unnatural. The prosody (rhythm, stress, intonation) doesn't flow smoothly between sentences.\n",
"\n",
"This happens because the TTS model doesn't have context about what comes next, so each sentence is treated as a standalone utterance."
]
},
{
"cell_type": "markdown",
"id": "bmyu8apfy7o",
"metadata": {},
"source": [
"## WebSocket Streaming: The Best of Both Worlds\n",
"\n",
"ElevenLabs offers a WebSocket API that solves this problem perfectly. Instead of waiting for complete sentences, we can stream text chunks directly to the TTS engine as they arrive from Claude.\n",
"\n",
"The WebSocket API:\n",
"- Accepts streaming text input (no sentence buffering needed)\n",
"- Maintains context across chunks for natural prosody\n",
"- Returns audio chunks as soon as they're ready\n",
"- Achieves the lowest possible latency with the best audio quality\n",
"\n",
"Let's implement this ultimate optimization:"
]
},
{
"cell_type": "markdown",
"id": "9e4d0f39",
"metadata": {},
"source": [
"## Building a Production Voice Assistant\n",
"\n",
"The techniques demonstrated in this notebook provide the foundation for building a real-time voice assistant. The WebSocket streaming approach minimizes latency while maintaining natural audio quality.\n",
"\n",
"### Key Implementation Challenges\n",
"\n",
"When building a production system, you'll need to solve several additional challenges beyond the basic streaming:\n",
"\n",
"1. **Continuous Audio Playback**: Audio chunks must play seamlessly without gaps or crackling. This requires careful buffer management and pre-buffering strategies.\n",
"\n",
"2. **Microphone Input**: Real-time recording from the microphone with proper handling of audio formats and sample rates.\n",
"\n",
"3. **Conversation State**: Maintaining conversation history across turns so Claude can reference previous context.\n",
"\n",
"4. **Audio Quality**: Converting between different audio formats (PCM, WAV) and avoiding artifacts from encoding.\n",
"\n",
"### Complete Implementation\n",
"\n",
"We've built a complete voice assistant script that demonstrates all these techniques:\n",
"\n",
"**`stream_voice_assistant_websocket.py`** - A production-ready conversational voice assistant featuring:\n",
"- Microphone recording with Enter-to-stop control\n",
"- ElevenLabs speech-to-text transcription\n",
"- Claude streaming with conversation history\n",
"- WebSocket-based TTS with minimal latency\n",
"- Custom audio queue for gapless playback\n",
"- Continuous conversation loop\n",
"\n",
"Run the script to experience a fully functional voice assistant:\n",
"\n",
"```bash\n",
"python stream_voice_assistant_websocket.py\n",
"```\n",
"\n",
"This demonstrates how the streaming optimizations from this notebook translate into a real-world application with production-quality audio handling."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "base",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.11"
}
},
"nbformat": 4,
"nbformat_minor": 5
}