# Muse AI - Conversational Voice Co-Producer & Studio Agent > Build an intelligent, real-time conversational Voice AI music studio assistant and melody ideation co-producer using Agora RTC and Agora Conversational AI. [![Agora RTC](https://img.shields.io/badge/Agora-RTC%20SDK%20v4.24-blue.svg)](https://docs.agora.io/en/voice-calling/overview/product-overview) [![Agora Conversational AI](https://img.shields.io/badge/Agora-Conversational%20AI-orange.svg)](https://docs.agora.io/en/conversational-ai/overview/product-overview) [![TypeScript](https://img.shields.io/badge/TypeScript-6.0-blue.svg)](https://www.typescriptlang.org/) [![React](https://img.shields.io/badge/React-19.2-cyan.svg)](https://react.dev/) [![Vite](https://img.shields.io/badge/Vite-8.2-purple.svg)](https://vitejs.dev/) --- ## Overview **Muse AI** is a real-time conversational Voice AI studio co-producer designed for musicians, songwriters, producers, and creative builders. When musical inspiration strikes — while humming a tune on the go, brainstorming lyrics, or crafting a chord progression — creators often struggle with the friction of configuring traditional Digital Audio Workstations (DAWs). Muse AI bridges this gap with Agora's sub-second Voice AI capabilities: - **Speak or Sing Directly**: Users hum melodies or converse naturally with specialized AI agent personas. - **Ultra-Low Latency Audio**: Two-way conversational audio stream powered by the Agora RTC Web SDK (`agora-rtc-sdk-ng`). - **Live Pitch Telemetry & Sing-to-Create**: Real-time pitch tracking, fundamental frequency extraction, and automatic scale/key identification. - **Dynamic Prompt Injection**: Live steering of AI agent personality, speed, pitch, and domain context during active voice sessions. - **In-Browser Web Audio Sandbox**: Built-in polyphonic synthesizer and beat engine for rapid playback and acoustic prototyping. --- ## Key Features 1. **Real-Time Voice AI Streaming with Agora RTC** - High-fidelity stereo audio (`high_quality_stereo` profile). - Built-in Acoustic Echo Cancellation (AEC), Automatic Gain Control (AGC), and Acoustic Noise Suppression (ANS). - Real-time `volume-indicator` telemetry for dual-channel speaking feedback (user vs. AI agent). 2. **Agora Conversational AI REST Gateway** - Direct session lifecycle management (`/start`, `/stop`). - Dynamic prompt injection via `/sessions/{agentId}/inject-prompt`. - Custom voice profile configuration (`voice_id`, `temperature`, `speech_rate`, `pitch`). 3. **Multi-Agent Creative Archetypes** - **Nova** *(Executive Strategist)*: Fast-paced decision matrices and release roadmaps. - **Devon** *(Systems & Code Architect)*: Technical architecture, Web Audio DSP, and API integration. - **Aria** *(Creative Director)*: Songwriting hooks, emotional arc planning, and lyric generation. - **Sora** *(Multilingual Polyglot)*: Cross-lingual lyrics, phonetic coaching in 30+ languages. - **Zenith** *(Vocal & Mindfulness Coach)*: Vocal warmups, breath telemetry, and performance coaching. 4. **Sing-to-Create & Audio Spectrum Analysis** - High-speed pitch detection via Autocorrelation algorithm. - Live holographic visualizer and audio frequency spectrum. - Automatic scale and note mapping (e.g., C Major, A Minor). 5. **Lyrics Studio & Multitrack Export** - Verse, chorus, and bridge structuring with real-time AI suggestions. - Session telemetry, transcription logs, and audio export (JSON / WAV / MIDI-compatible notes). --- ## Architecture & Data Flow ``` +-------------------------------------------------------------------------+ | Browser Client | | | | +-------------------+ +------------------+ +---------------+ | | | User Microphone | ---> | Agora RTC Track | --> | Agora Channel | | | +-------------------+ +------------------+ +-------+-------+ | | | | | | v v | | +-------------------+ +-----------------+ | | | Web Audio Analyzer| | Agora Speaker | | | | (Pitch & Spectrum)| | Audio Track | | | +-------------------+ +-----------------+ | +--------------------------------------------------------------+----------+ | Agora Voice Network | v +---------------------------------+ | Agora Conversational AI Agent | | (LLM + TTS + STT Pipeline) | +---------------------------------+ ``` --- ## Prerequisites - **Node.js**: Version 18.0 or higher - **npm**: Version 9.0 or higher - **Agora Developer Account**: - Sign up at [Agora Console](https://console.agora.io/). - Create a project to obtain an **App ID** and **App Certificate / Token**. - Enable **Conversational AI** in your Agora Console project settings. --- ## Quickstart Guide ### 1. Clone the Repository ```bash git clone https://github.com/Molly-Arora02/MUSE-AI-.git cd MUSE-AI- ``` ### 2. Install Dependencies ```bash npm install ``` ### 3. Start Development Server ```bash npm run dev ``` Open `http://localhost:5173` in your browser. ### 4. Connect with Agora Voice AI 1. Click **"Start Creating"** to enter the Studio. 2. Click the **Agora Settings** gear icon in the top header. 3. Enter your Agora credentials: - **Agora App ID**: `your_agora_app_id` - **Channel Name**: `muse-studio-01` - **Token** *(Optional for testing, required for secured channels)*: `your_rtc_token` - **Agent ID**: `agora-agent-vocalis-01` 4. Click **"Save Settings"**, then toggle **"Connect Voice"** to start real-time conversation! > **Demo / Offline Mode**: If you do not have Agora credentials immediately available, click **"Try Interactive Demo"** on the landing page. Muse AI will run in client-side Web Audio synthesis mode with full interactive voice and pitch simulation. --- ## Code Implementation Walkthrough ### 1. Initializing Agora RTC Client & Microphone Audio Track In `src/services/agoraService.ts`: ```typescript import AgoraRTC from 'agora-rtc-sdk-ng'; import type { IAgoraRTCClient, IMicrophoneAudioTrack, IRemoteAudioTrack } from 'agora-rtc-sdk-ng'; export class AgoraRTCService { private client: IAgoraRTCClient | null = null; private localAudioTrack: IMicrophoneAudioTrack | null = null; private remoteAudioTrack: IRemoteAudioTrack | null = null; public async initializeAndJoin(config: AgoraConfig, callbacks: AgoraCallbacks): Promise { this.client = AgoraRTC.createClient({ mode: 'rtc', codec: 'vp8' }); // Handle remote AI agent voice track this.client.on('user-published', async (user, mediaType) => { if (mediaType === 'audio') { await this.client?.subscribe(user, mediaType); this.remoteAudioTrack = user.audioTrack || null; if (this.remoteAudioTrack) { this.remoteAudioTrack.play(); callbacks.onAgentJoined?.(user.uid); } } }); // Volume level indicator for dual speaking detection this.client.enableAudioVolumeIndicator(); this.client.on('volume-indicator', (volumes) => { volumes.forEach((vol) => { const level = Math.min(100, Math.round((vol.level / 100) * 100)); if (vol.uid === 0 || vol.uid === config.uid) { callbacks.onUserAudioLevel?.(level); callbacks.onUserSpeaking?.(level > 15); } else { callbacks.onAgentAudioLevel?.(level); callbacks.onAgentSpeaking?.(level > 15); } }); }); // Create microphone track with noise suppression this.localAudioTrack = await AgoraRTC.createMicrophoneAudioTrack({ AEC: true, ANS: config.enableNoiseSuppression, AGC: true, }); // Join RTC channel and publish microphone audio await this.client.join(config.appId, config.channel, config.token || null, config.uid); await this.client.publish([this.localAudioTrack]); return true; } } ``` --- ### 2. Starting Conversational AI Agent via Gateway In `src/services/agentApi.ts`: ```typescript export class AgoraAgentGateway { public async startAgent(config: AgoraConfig, agent: VoiceAgent) { const endpoint = config.gatewayUrl.replace('{appId}', config.appId) + '/start'; const payload = { name: agent.name, properties: { channel_name: config.channel, agent_rtc_uid: 'agent_' + config.agentId, remote_rtc_uids: [config.uid], voice_id: agent.voiceId, system_instruction: agent.systemPrompt, temperature: agent.temperature, speech_rate: agent.speed, pitch: agent.pitch, tools: agent.tools } }; const res = await fetch(endpoint, { method: 'POST', headers: { 'Content-Type': 'application/json', 'Authorization': `Bearer ${config.token || 'demo-token'}` }, body: JSON.stringify(payload) }); return await res.json(); } public async injectPrompt(config: AgoraConfig, agentId: string, prompt: string) { const endpoint = config.gatewayUrl.replace('{appId}', config.appId) + `/sessions/${agentId}/inject-prompt`; return await fetch(endpoint, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ prompt, timestamp: Date.now() }) }); } } ``` --- ## Configuration Reference | Option | Type | Default | Description | | :--- | :--- | :--- | :--- | | `appId` | `string` | `""` | Agora App ID from Agora Console | | `channel` | `string` | `"vocalis-demo-room"` | Target RTC channel name | | `token` | `string` | `""` | Temporary or server-generated RTC token | | `uid` | `number` | `Random 6-digit` | Unique client numeric ID | | `enableNoiseSuppression`| `boolean` | `true` | Enables Agora hardware/software noise filtering | | `audioProfile` | `string` | `"high_quality_stereo"` | Agora RTC audio profile | | `vadSensitivity` | `number` | `75` | Voice Activity Detection sensitivity | --- ## Tech Stack - **Frontend**: React 19, TypeScript, Tailwind CSS - **Voice & Real-Time**: Agora RTC SDK (`agora-rtc-sdk-ng` v4.24+) - **Audio DSP**: Web Audio API (AnalyserNode, BiquadFilter, Custom Oscillators) - **Icons & Visuals**: Lucide React, Canvas Confetti - **Build Tool**: Vite 8.2 --- ## License This project is licensed under the MIT License. See [LICENSE](file:///Users/mollyarora/Desktop/thoughtworks/LICENSE) for details.