# anydoc + vision pipeline (`anydoc_vlm.py`)

The conversion engine behind the **dsh-anydoc-markdown** host plugin. It turns
any supported document into clean GitHub-Flavored Markdown with the Rust
[`firecrawl-anydoc`](https://github.com/firecrawl/anydoc) crate, then describes
every embedded image with an OpenAI-compatible vision model (VLM).

This is a pure local, dependency-free converter. anydoc's Rust core has **no
network calls and no `llm_client` parameter** — but it exposes
`anydoc.to_document()`, which returns the full document model *plus* the embedded
binary assets (images, MIME types, and alt-text markers) on `Document.assets`.
That is the seam this wrapper uses, replicating the **MarkItDown
`llm_client` + `llm_model`** pattern: intercept the image assets, pass them to a
VLM, and substitute the image placeholders in the output Markdown.

## Install

```sh
pip install -r requirements.txt      # firecrawl-anydoc
```

(The host plugin runs this by invoking `python`, which already has the crate.)

## How image description is batched (the hard requirement)

Images are **never all sent in one API request** — that degrades vision
performance. `describe_many()` slices the images into groups of
`max_images_per_request` (default **10**) and issues one chat-completions request
per group. Verify it for the bundled test doc (23 images, max 10):

```sh
python anydoc_vlm.py multi.docx --mock-vision --max-images 10 --json
# vision_batch_sizes: [10, 10, 3], vision_requests: 3
python anydoc_vlm.py multi.docx --mock-vision --max-images 4 --json
# vision_batch_sizes: [4, 4, 4, 4, 4, 3], vision_requests: 6
```

## CLI

```sh
python anydoc_vlm.py <path> [options]

# Offline metadata description (testable, no HTTP, still batches):
python anydoc_vlm.py report.docx --mock-vision

# Real VLM (OpenAI-compatible):
python anydoc_vlm.py report.docx \
  --vision-endpoint https://api.openai.com/v1 --vision-model gpt-4o

# Machine-readable envelope (markdown + meta):
python anydoc_vlm.py report.docx --json
```

| Option | Default | Description |
| --- | --- | --- |
| `--format` | auto | Explicit format name (only needed for signature-less formats like CSV). |
| `--ocr` | `reject` | `hosted` routes scanned PDF pages to Firecrawl Parse. |
| `--max-images` | `10` | Max images per vision request. |
| `--vision-endpoint` | env `ANYDOC_VISION_ENDPOINT` | OpenAI-compatible endpoint. |
| `--vision-model` | env `ANYDOC_VISION_MODEL` | Model id (e.g. `gpt-4o`). |
| `--api-key` | env `OPENAI_API_KEY` | Vision API key. |
| `--mock-vision` | off | Force the offline description (for testing). |
| `--save-images-dir` | unset | Directory to persist every extracted embedded image as a PNG file (re-encoded); the Markdown image references are rewritten to point at those files. |
| `--json` | off | Emit `{"markdown", "meta"}` instead of plain Markdown. |

## How it works

1. `anydoc.to_markdown_bytes(data)` produces the base Markdown (headings, tables,
   lists, equations, links, footnotes — all through one serializer).
2. `anydoc.to_document(data)` returns the model. `extract_images()` walks
   `Document.blocks` and recovers every `Inline(kind='image')` **in document
   order** with its `ImageSource` (which carries `asset_id` into
   `Document.assets`), the author alt text, media type, source part, and bytes.
3. `VisionClient.describe_many()` batches the embeddable images (≤
   `max_images_per_request`) and asks the VLM for one description each; it returns
   descriptions in order and parses the JSON array the model is told to emit.
4. `substitute()` replaces each image's rendered alt-text paragraph with
   `![<description>](asset://<id>)`. Images anydoc dropped (empty alt text) are
   appended as a described reference list so nothing is lost.
5. Formats with **no document model** — PDFs specifically — previously skipped
   the asset step (the wrapper caught `anydoc.UnsupportedError` from
   `to_document` and returned Markdown with no image descriptions). Now the
   wrapper recovers the embedded raster images directly from the PDF bytes with
   **pdfplumber**, re-encodes each to PNG/JPEG, and feeds them to the same
   kind of VLM pass. The descriptions are appended to the Markdown as a
   described reference list (PDFs carry no per-image alt text to substitute). If
   `pdfplumber`/`Pillow` are unavailable or the PDF can't be parsed, the wrapper
   fails closed and still returns the text Markdown.

## Persisting extracted images as PNG

By default the Markdown references an extracted image as `![<desc>](asset://<id>)`
— a pointer a caller can resolve, but not a real file. Pass
`--save-images-dir <dir>` to instead write every embedded raster image to that
directory as a PNG file and rewrite the Markdown image references to point at the
on-disk files (relative names like `image_0001.png`). This lets a caller keep the
converted Markdown and the images together in one folder:

```sh
python anydoc_vlm.py report.docx --mock-vision --save-images-dir out/
# out/report.md + out/image_0001.png ... out/image_000N.png
```

Each embeddable (has bytes) image is re-encoded to a plain RGB PNG (transparent
images are composited onto white). A non-decodable image fails closed and keeps
its `asset://` reference, so nothing is silently dropped. URL-kind images are
never persisted (they already render as ordinary Markdown images). The `meta`
envelope reports `image_dir`, `images_saved` and `saved_images` (the relative
file names, in document order).

## Model note

Images appear in Markdown as `![<description>](asset://<id>)`. The alt text holds
the VLM description, so a **text-only** model reading the Markdown still reasons
about the image content; the `asset://<id>` reference lets a caller (or a future
client half) fetch the original bytes.

Descriptions come from the real VLM. When the model is **not configured**, the
alt text is an honest offline metadata description
(`[Offline metadata description - no vision model was configured.]`).

A transient **fail-closed** fallback exists so a flaky API doesn't break the
pipeline: each batch is retried up to `attempts` (default 3) on an HTTP error,
timeout, or a response that can't be parsed as a JSON array of exactly N
strings. Only if all retries fail does the batch degrade to an offline metadata
description, and the note then says why (`vision request failed after retries`
or `vision unavailable after retries`) rather than claiming the model isn't
configured. Responses wrapped in prose or a ```json fence are still parsed.