# dsh-eyes

<div align="center">

[中文](./README.md) · **English**

</div>

Give a **text-only LLM** (e.g. DeepSeek) in DeepSeek Harness "eyes on demand": pasted/attached images stay in the backend, and the model calls the `view_image` tool whenever it wants to look at one (backed by **any OpenAI-compatible vision endpoint**; DashScope Qwen by default) — as if the model were natively multimodal.

## Try it: paste an image, and it just works

No toggle, no manual tool call — **paste a screenshot with `Ctrl+V`** and ask like any normal message. DeepSeek decides in its reasoning that it needs to look, calls `view_image` on its own, and answers directly, in one smooth flow:

![dsh-eyes demo: paste screenshot → think → call view_image → structured answer](./assets/demo.png)

> Real capture above: paste an MSN screenshot and ask "describe the layout of this page". Notice the `Think → Tool call · view_image → Think` chain — recognition is never forced into the first step; the model decides to look exactly when it needs to, and vision extraction flows seamlessly into the final answer, just as if DeepSeek were natively multimodal.

## Problem

DeepSeek is a text-only model, so the Harness refuses to send image-bearing messages when "the current model does not support images". This plugin:

- lets image messages pass the send-admission check and get persisted;
- strips images into a reference note **before** the request reaches the main model;
- registers a `view_image` tool that the main model calls on demand to "see" an image; the vision model returns the description/OCR text and the main model continues answering.

## Install

```bash
# 1) Install the plugin (github form; a bare npm package name also works)
dsh plugin --profile web add github:Leeminjing/dsh-eyes

# 2) Configure the API key (Windows; use your vision provider's key)
setx VISION_API_KEY "sk-your-key"

# 3) Configure the vision model (required — use a model your account can call)
setx VISION_MODEL "qwen-vl-plus"

# 4) Configure the endpoint (defaults to DashScope; change it for other OpenAI-compatible providers)
setx VISION_ENDPOINT "https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions"

# 5) Restart dsh so the environment variables take effect
```

> `setx` only affects **newly started processes**, so restart dsh after configuring.
> The model id and endpoint are provider-specific: DashScope uses `qwen-vl-plus`/`qwen-vl-max`, OpenAI uses `gpt-4o`, OpenRouter uses `qwen/qwen2.5-vl-72b-instruct`, etc. See the table below.

## Usage

1. **Paste an image** (Ctrl+V), or drag/attach one;
2. Send a question, e.g. "What does this image say?";
3. The main model receives an "image reference note"; when it needs to look, it calls `view_image(attachment_id=…)`;
4. `view_image` extracts the image content via the vision model, and the main model answers from that text.

You can keep asking about the same image in later turns ("what's the number in the second row again?") — the image stays in the backend and can be viewed repeatedly.

## How it works

```
paste image + question
   │
   ▼
send admission ──(1) mark target model as image-capable── persist image (attachment_id)
   │
   ▼
before dispatch ──(2) strip image blocks → 【图片N attachment_id=…】note, register in the session's shard
   │
   ▼
main model receives text (note + your question)
   │   model decides to look
   ▼
call view_image(attachment_id) ──(3) read bytes → base64 → POST to vision endpoint
   │
   ▼
vision model returns description/OCR text
   │
   ▼
main model answers from the text
```

Three parts:

1. **Admission bypass**: wrap `llm.resolveModelInfo` so the target main model declares `inputModalities: ['text','image']`, letting image messages pass the host's send-admission check and be persisted.
2. **Image stripping**: wrap `llm.streamWithRegistration` to swap each `image` block (including those nested inside `tool-result`) for a reference note right before dispatch, and register `attachment_id → ImageAttachmentRef` sharded by `sessionId`. A **new options object** is built because the incoming one may be frozen.
3. **`view_image` tool**: reads image bytes by a single `attachment_id`, or an `attachment_ids` array to view several at once (or a local `image_path`), turns them into `data:` URLs, POSTs to the configured vision endpoint (Chat Completions or Responses protocol per `VISION_API_STYLE`), and returns text (each image labeled 【图片N】 when batching).

## Configuration

| Item | Environment variable | Default |
| --- | --- | --- |
| API key | `VISION_API_KEY` | (required) |
| Vision model | `VISION_MODEL` | (required, no default) |
| Endpoint | `VISION_ENDPOINT` | `https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions` |
| API style | `VISION_API_STYLE` | `auto` (derived from the endpoint URL) |
| Target text model | `targetProvider` (in code) | `deepseek-official` |
| Max local image bytes (`image_path`) | `maxImageBytes` (in code) | 15 MB |

> Pasted/attached images are capped by the Harness attachment store (5 MB by default), independent of the local-file limit above.

> **Chat Completions and Responses API are both supported**: `VISION_API_STYLE` is `auto` (default), `chat`, or `responses`.
> - `auto`: derived from the endpoint URL — `.../chat/completions` → Chat Completions, `.../responses` → Responses API; a bare base URL defaults to Chat Completions and gets the path appended automatically.
> - `chat` / `responses`: forced; the plugin normalizes the endpoint path to the matching protocol.
> Both the request body and the response parsing switch with the style (`messages`/`image_url` ↔ `input`/`input_image`), so switching is transparent to how you use the plugin.

**Common OpenAI-compatible vision providers:**

| Provider | Endpoint | Example models |
| --- | --- | --- |
| Alibaba Cloud Model Studio | `https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions` | `qwen-vl-plus` / `qwen-vl-max` |
| OpenAI | `https://api.openai.com/v1/chat/completions` | `gpt-4o` / `gpt-4o-mini` |
| Moonshot | `https://api.moonshot.cn/v1/chat/completions` | `moonshot-v1-8k-vision-preview` |
| OpenRouter | `https://openrouter.ai/api/v1/chat/completions` | `qwen/qwen2.5-vl-72b-instruct` |

You can also pass a `config` block for the row in `cordis.patch.yml` (overrides defaults / env vars):

```yaml
- insert:
    - id: dsh-eyes
      name: dsh-eyes
      config:
        apiKey: sk-xxx          # same as VISION_API_KEY
        model: qwen-vl-plus     # same as VISION_MODEL
        # endpoint, targetProvider, maxImageBytes likewise
```

## Known limitations

- **Two internal wraps**: the Harness currently exposes no public extension point for "image-capability during send admission" or "strip images before dispatch", so this plugin wraps `llm.resolveModelInfo` and `llm.streamWithRegistration` directly. Side effect: text-only main models show as image-capable in the model picker (intentional — required to admit image messages).
- **Main model auto-detection**: any text-only main model (any provider) is protected automatically; a natively multimodal main model is left untouched and images pass through natively.
- **Vision endpoint**: must be OpenAI-compatible (Chat Completions or Responses API); Anthropic / Gemini native APIs are unsupported unless accessed through their OpenAI-compatible gateways.
- **Session isolation & persistence**: the image-reference index is sharded by `sessionId` (a session can only view its own attachments) and persisted to `.dsh/attachments/v1/dsh-eyes-index.json` (loaded on startup, written on new images). So historical images remain viewable by `attachment_id` even after context compaction or a process restart.
- Keep the API key in env/credential storage — **never commit it**.

## License

[MIT](./LICENSE)