# dsh-voice-mic

[中文](README.md)

Voice input plugin for the DeepSeek Harness Web GUI. Records from the microphone, transcribes in real time, and writes the text into the input box without auto-sending.

## Installation

Requirements: dsh `>=0.1.0-rc.6`, Node.js `>=18`. Browser recognition requires Chrome/Edge; the local backend additionally requires Python `3.9+`.

```bash
dsh plugin --profile web add github:Zachary7456/dsh-voice-mic
# or (once published to npm): dsh plugin --profile web add dsh-voice-mic
```

Restart `dsh web` after installation. The plugin ships a bundle patch, so it is mounted into the profile automatically — no manual `cordis.patch.yml` edits needed.

Verify:

```bash
dsh --profile web --dump-config | grep dsh-voice-mic
curl -s http://127.0.0.1:3080/plugins/dsh-voice-mic/client.js | head -c 100
```

## Usage

- Microphone button on the left of the input box toggles recording; default hotkey `Alt+V` (configurable in settings)
- Results stream into the draft while recording; they are committed on stop, never auto-sent
- The hotkey only works while the page has focus (browser restriction)
- Live writes use tail replacement: each update only replaces the previously appended suffix, so manual typing is preserved; on failure or empty result the temporary text is rolled back

## Recognition engines

Settings → Voice input → Engine, three options.

### Browser built-in (default)

Zero configuration, uses the Web Speech API. Quality and latency depend on the browser and network. When the browser ends the recognition session on a pause, the plugin restarts it automatically and keeps listening until stopped manually.

### Local offline backend

One-click deployment from the settings page: detect Python → pip install dependencies → download model (resumable, cancellable) → start `asr_server.py` → poll until ready. Downloaded models are cached; switching models costs no re-download.

Models:

| Model | Use case | Size |
|---|---|---|
| SenseVoiceSmall int8 | zh/en/ja/ko/yue, everyday use | ~158MB |
| Paraformer | Mandarin only, best accuracy and punctuation | ~223MB |

- Model directory: `~/.dsh/voice/models/<model>` (override with `DSH_VOICE_MIC_MODEL_DIR`); download cache: `~/.dsh/voice/cache`
- Server port defaults to `7860`, bound to `127.0.0.1` only
- The backend process is managed by dsh: redeploy after a dsh restart (or set `autoStart: true`)
- Audio never leaves the machine during transcription

### Cloud API

OpenAI-compatible `POST /v1/audio/transcriptions` (multipart `file` + `model`, response `{text}`). Configure base URL, API key, and model name; presets for OpenAI / Groq / SiliconFlow are included. The key is stored in browser localStorage and the browser calls the provider directly — it never passes through dsh.

The "Test API connection" button in settings sends a silent sample to validate the config: `401/403` means a bad key, `404` usually means you entered the full endpoint instead of the base URL.

To exercise the full pipeline offline, use the mock server in this repo:

```bash
python server/mock_api.py   # 127.0.0.1:7861/v1, any key
```

Live transcription calls the API roughly once per second during recording, which incurs corresponding charges.

## How it works

Two halves:

```
lib/index.js   host half: config endpoint (GET /config), backend deploy manager (/backend/status|deploy|cancel|stop)
lib/client.js  client half: input-box button and hotkey, recording, transcription, settings page
server/         local ASR service (sherpa-onnx + SenseVoice/Paraformer) and API mock
```

Recording and transcription:

- Browser engine: Web Speech interim results go straight to the draft; final segments accumulate. The session only ends on explicit stop, timeout, or a fatal error
- Backend/API engine: PCM captured directly via AudioContext → 16kHz WAV; while recording, the accumulated audio is transcribed every second (partial results stream to the draft, stale responses dropped by sequence number); on stop, the full audio is transcribed once and committed
- All three engines share the same capture and live-write logic; only the transcription function differs

Deployment flow (host half): find Python → dependency check → model check/download → spawn `asr_server.py` (with `DSH_VOICE_MIC_MODEL_DIR/TYPE` injected) → poll `/health` until ready. Download and install steps can be interrupted via `/backend/cancel`.

## Configuration

All user-facing settings are editable in the settings page (stored in browser localStorage). Server-side configuration is also supported:

```yaml
# ~/.dsh/profiles/web/cordis.patch.yml
- insert:
    - id: dsh-voice-mic
      name: dsh-voice-mic
      config:
        hotkey: alt+v
        port: 7860
        autoStart: false
```

Environment variables:

| Variable | Purpose | Default |
|---|---|---|
| `DSH_VOICE_MIC_HOTKEY` | Hotkey | `alt+v` |
| `DSH_VOICE_MIC_LANG` | Browser recognition language | `zh-CN` |
| `DSH_VOICE_MIC_BACKEND` | Local backend URL | empty |
| `DSH_VOICE_MIC_PORT` | Backend port | `7860` |
| `DSH_VOICE_MIC_MODEL_TYPE` | Default model type | `sensevoice-small-int8` |
| `DSH_VOICE_MIC_MODEL_DIR` | Model directory | `~/.dsh/voice/models/<type>` |
| `DSH_VOICE_MIC_AUTOSTART` | Auto-deploy backend on startup | `false` |
| `DSH_VOICE_MIC_API_BASE` / `_MODEL` / `_KEY` | Cloud API config | empty |

Precedence: settings page (localStorage) > environment variables > cordis config > defaults.

## Tests

```bash
node test/smoke.mjs            # load both halves, routes, config merge
node test/verify-install.mjs   # activation check for an installed instance
node test/deploy-test.mjs      # one-click deployment end-to-end (uses port 7862)
node test/e2e-backend.mjs <wav> [url]   # local backend transcription
node test/e2e-api.mjs [base] [key] [model]  # cloud API protocol (with server/mock_api.py)
```

## Development

No build step — edit `lib/*.js` directly. Client-half changes take effect on page refresh; host-half changes require a `dsh web` restart.

## License

MIT