# Using the MiniCPM-o Realtime Frontend

qwen-audio-agent connects to
[MiniCPM-o 4.5](https://github.com/OpenBMB/MiniCPM-o-Demo) through ModelBest's public Realtime
protocol. The endpoint may be a user-managed local deployment or a compatible hosted service;
the Gateway does not install the model or manage the inference process.

The integration targets the official audio full-duplex WebSocket protocol:

```text
ws://127.0.0.1:8006/v1/realtime?mode=audio
```

Set the provider and endpoint after the service is ready:

```bash
QWEN_AUDIO_REALTIME_PROVIDER=minicpm-o
MINICPM_O_REALTIME_URL=ws://127.0.0.1:8006/v1/realtime?mode=audio
```

The default assumes the upstream Gateway was started with `--http` on the loopback interface.
For ModelBest cloud or another TLS deployment, set its corresponding `wss://` endpoint instead.
`MINICPM_O_AUTH_TOKEN` is sent as a Bearer token when the endpoint requires one. The same fields
are available in Desktop Settings under **Voice frontend → ModelBest**, where the model is shown
as the fixed `MiniCPM-o 4.5`.

The adapter converts the clients' 16-bit PCM stream to the protocol's 16 kHz mono float32 input,
converts its 24 kHz mono float32 output back to 16-bit PCM, and maps MiniCPM-o session and response
events into the shared realtime runtime.

## Live vision

Use MiniCPM-o's video full-duplex endpoint to enable the negotiated WebUI
visual-frame stream:

```bash
MINICPM_O_REALTIME_URL=ws://127.0.0.1:8006/v1/realtime?mode=video
```

The public GCP shape remains `input_image_buffer.append`. The MiniCPM-o adapter
holds only the latest JPEG frame and adds it to the next one-second audio
`input.append` as `video_frames`. With the default `mode=audio` URL, the Gateway
does not negotiate `input.image_buffer`, so clients cannot accidentally present
an unsupported camera control.

MiniCPM-o's public Realtime protocol does not currently define conversation items, structured
function-call events, client-triggered responses, or input transcription events. This integration
therefore focuses on realtime voice conversation. Typed input, restored transcript history,
proactive announcements, memory writes, and backend-Agent tools remain unavailable with this
provider. UI chat history is still retained locally where the client has content to display.