# Experimental EmbeddingGemma 2 and local media

This optional backend uses [Google's `embeddinggemma-2`](https://huggingface.co/google/embeddinggemma-2) through Sentence Transformers. It supports asymmetric text retrieval, native code queries and local image/audio/video assets in one embedding space. Existing MiniLM, EmbeddingGemma ONNX and OpenAI-compatible backends remain available; the default model and drawer-only search are unchanged.

## Install and isolate

The new extras require Python 3.10 or newer. Use a separate environment, configuration directory and palace for the experiment:

```bash
uv venv --python 3.12 .venv/embeddinggemma2
uv pip install --python .venv/embeddinggemma2/bin/python -e '.[dev,multimodal]'
source .venv/embeddinggemma2/bin/activate

export MEMPALACE_CONFIG_DIR="$HOME/.config/mempalace-eg2"
export MEMPALACE_PALACE_PATH="$HOME/.local/share/mempalace/palace-eg2"
export MEMPALACE_EMBEDDING_MODEL=embeddinggemma2
export MEMPALACE_EMBEDDINGGEMMA2_MODALITIES=all
```

Use `.[embeddinggemma2]` for text/image inference, or `.[multimodal]` for audio/video decoding too. TorchCodec needs compatible FFmpeg libraries. Install FFmpeg using your platform's package manager before using audio/video file inputs.

On macOS, the required `torch>=2.14.0` has official wheels for Apple Silicon,
not Intel Macs. PyTorch [ended official Intel Mac support after 2.2.x](https://pytorch.org/blog/pytorch2-2/).
Choosing `embedding_device=cpu` cannot bypass this installation limit. Intel
Macs can use the ONNX backends with compatible dependencies or an
`openai-compat` endpoint. On Apple Silicon, the installed packages must support
the macOS/Python versions; `auto` uses MPS only when PyTorch reports it available.

The extras pull `torch` from PyPI. On Windows that wheel is CPU-only. To run on an NVIDIA GPU (Windows or Linux), install torch from the PyTorch CUDA index first, choosing the CUDA build for your driver at [pytorch.org](https://pytorch.org/get-started/locally/). For a smaller CPU-only install on Linux, use the CPU index:

```bash
uv pip install --python .venv/embeddinggemma2/bin/python torch --index-url https://download.pytorch.org/whl/cu132  # NVIDIA GPU
uv pip install --python .venv/embeddinggemma2/bin/python torch --index-url https://download.pytorch.org/whl/cpu    # CPU only
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
```

Model loading is lazy. First inference downloads the pinned model to the Hugging Face cache; subsequent inference is local. After the model is cached, set `HF_HUB_OFFLINE=1` to disable further model downloads. This does not configure other MemPalace services or providers.

## Configuration

Environment values override the corresponding `config.json` values:

| Config key | Environment variable | Values/default |
| --- | --- | --- |
| `embedding_model` | `MEMPALACE_EMBEDDING_MODEL` | `embeddinggemma2` to opt in |
| `embeddinggemma2_dimension` | `MEMPALACE_EMBEDDINGGEMMA2_DIMENSION` | `768` (default), `512`, `256`, `128` |
| `embeddinggemma2_modalities` | `MEMPALACE_EMBEDDINGGEMMA2_MODALITIES` | `text` (default), `text+vision`, `text+audio`, `all` |
| `embedding_device` | `MEMPALACE_EMBEDDING_DEVICE` | `auto` (default), `cuda`, `mps`, `cpu` |
| `embeddinggemma2_batch_size` | `MEMPALACE_EMBEDDINGGEMMA2_BATCH_SIZE` | Unset: 32 on CUDA, 4 on CPU/MPS. A positive integer applies on every device |
| `embeddinggemma2_revision` | `MEMPALACE_EMBEDDINGGEMMA2_REVISION` | Immutable 40-character model commit SHA |

The default revision is [`914f7f89142e33e77833254d9c9b90c3cef7303b`](https://huggingface.co/google/embeddinggemma-2/tree/914f7f89142e33e77833254d9c9b90c3cef7303b). Model, dimension, modalities, revision and prompt version identify the vector space. Changing these values on a populated palace requires rebuilding, not relabelling old vectors.

`auto` uses CUDA when PyTorch reports a GPU, then Apple MPS, then CPU; `mempalace mine` prints the device it resolved. Weights and inference use float32. An explicit `cuda` or `mps` that PyTorch cannot use warns and runs on CPU, as the ONNX models do for an unavailable accelerator. `embedding_device` is shared with the ONNX models, so the ONNX-only `dml` and `coreml` read as `auto` here, with a warning. The device is not part of the vector space: a palace mined on CUDA opens and searches on a CPU-only machine. MPS runs a repeatability witness on first use; initialization, MPS/Metal inference and invalid-vector failures warn and retry on CPU. Unrelated decoder errors fail the item without moving a healthy model to CPU. Inputs are truncated at the model's 8,192-token context. A CUDA out-of-memory error halves the batch and retries (freeing PyTorch's CUDA cache between attempts) down to batch size 1, for that call only; the next call starts at the configured size again. If batch size 1 still runs out of memory, the error names `MEMPALACE_EMBEDDINGGEMMA2_BATCH_SIZE`.

On Apple Silicon, use `MEMPALACE_EMBEDDING_DEVICE=mps` to request Metal explicitly
in the isolated environment above, or leave it at `auto`. This requires the
`embeddinggemma2` backend: setting `mps` for MiniLM or EmbeddingGemma ONNX does
not move those models to PyTorch. See the [model/device support
table](../website/guide/configuration.md#embedding-models-and-devices) for the
ONNX/CoreML restrictions. Check inference warnings for CPU fallback; a device
label shown before model loading is the planned device, and later status calls
in the same process reflect the loaded backend.

Documents use `Document`, with a real title or source filename when present. Queries use `SearchQuery`, or `CodeRetrieval` for explicit code search. All output vectors are dimension-checked, finite and L2-normalized.

## Mine and search

With the environment above:

```bash
# Replace the example paths with your own directories.
mempalace mine /path/to/text-or-code --wing ExampleProject
mempalace mine /path/to/media --source media --wing ExampleProject --dry-run
mempalace mine /path/to/media --source media --wing ExampleProject

mempalace search "a screenshot showing an authentication error" --include-media
mempalace search "function that checks bearer token headers" --query-task code
```

A one-command palace override goes before the command: `mempalace --palace /path/to/isolated-palace search "query" --include-media`.

The media adapter supports PNG/JPG/JPEG/WebP images, WAV/MP3/M4A/FLAC audio, and MP4/MOV/M4V/WebM video. It skips hidden/cache/build directories and symlinks. `--wing` supplies the project label. Add `--related-drawer-id EXISTING_DRAWER_ID` to explicitly associate assets with an already indexed drawer; invalid links are rejected.

Assets live in the fixed `mempalace_assets` collection, separate from verbatim drawers. Metadata includes stable path-derived ID, source path, type, MIME type, filename title, project/wing, modification time, size, optional drawer link and embedding identity. Supported audio headers also supply duration. Media bytes remain in their original local files, and embeddings come from the native media encoder without generated captions or descriptor-text substitutions.

`--include-media` merges text/code and assets by cosine distance in the verified shared vector space. It requires compatible cosine collections; default drawer-only search retains its existing hybrid ranking. MCP `mempalace_search` accepts optional `include_media: true` and `query_task: "code"`, returning typed asset references with path, type, project, drawer link and live `available` status. Existing clients keep the default result behavior when these options are omitted. Experimental CLI routes run locally rather than forwarding unsupported options to an older hub.

Corrupt items are reported individually while healthy files continue. All discovered assets failing exits nonzero; missing dependencies or incompatible provider settings are fatal setup errors. Repeated ingestion upserts the same path ID and retains its original filing time.

## Rebuild safely

Keep a backup and stop writers on the source before rebuilding. Prefer a separate destination:

```bash
# With the new backend settings above, replace source/destination paths.
mempalace --palace /path/to/new-palace repair --mode from-sqlite \
  --source /path/to/source-palace --dry-run
mempalace --palace /path/to/new-palace repair --mode from-sqlite \
  --source /path/to/source-palace
```

This Chroma-specific recovery preserves drawer/closet IDs, verbatim content and metadata, reconstructs text vectors with metadata-aware document prompts, and re-embeds assets from their local files. It records verified new collection identities. Native assets require the corresponding modality to be enabled. Compare IDs, documents, metadata, counts and representative queries before switching clients to the destination.

`mempalace --palace /path/to/selected-palace repair rebuild-index --yes` is the in-place full-palace recovery alias. It archives the original palace before rebuilding, so retain the archive. Missing or invalid asset references are refused before archival. A missing source file remains a searchable reference with `available: false` until restored; it cannot be re-embedded from its descriptor. Do not use `palace set-embedder` to bypass the identity gate.

## Validation and limits

Normal tests mock model inference and require no downloaded weights:

```bash
python -m pytest tests/ -q --ignore=tests/benchmarks \
  --cov=mempalace --cov-report=term
ruff check .
ruff format --check .
```

Real-model integration tests are explicitly opt in:

```bash
MEMPALACE_RUN_EG2_INTEGRATION=1 MEMPALACE_RUN_EG2_MULTIMODAL=1 \
  python -m pytest tests/test_embeddinggemma2_integration.py -q
```

The generated speech/video codec test needs macOS `say` and FFmpeg; other platforms skip that fixture. These integration tests may download weights unless `HF_HUB_OFFLINE=1` is set after caching them.

A local Mac acceptance trial used ten existing text/code files and ten existing media assets, with twenty locked known-answer questions. Both 768-dimensional and rebuilt 512-dimensional palaces retrieved all twenty expected files within five results. Repeated ingestion, restarted CLI/MCP, corrupt files, missing references and full record/identity comparisons passed. This hand-picked corpus is an acceptance test, not a general retrieval benchmark or cross-platform validation.

Assets are file-level references. Renames/moves produce a new path ID; old references need manual cleanup. There is no automatic relinking, garbage collection or segment retrieval. Video follows the upstream processor's 1-fps sampling, maximum 32 frames and uniform subsampling on overflow. Audio normalizes to mono 16 kHz. Arbitrary codecs, long recordings, sustained concurrency and live remote database deployments need separate validation. Native asset source/document edits require re-ingestion or explicit vectors; descriptive title-only edits retain native vectors.