### [oMLX](https://github.com/jundot/omlx)

> Handle: `omlx`<br/>
> URL: [http://localhost:34940](http://localhost:34940)

oMLX is an Apple Silicon inference server built on MLX with OpenAI-compatible and Anthropic-compatible APIs, continuous batching, multi-model serving, and paged SSD KV caching. Harbor runs the oMLX server natively on the macOS host for Metal access, then exposes it to Harbor containers through a Caddy proxy.

This service is intended for Apple Silicon systems running macOS 15 or newer. On other platforms the proxy can still point at an externally managed compatible oMLX endpoint, but Harbor cannot provide MLX acceleration inside a Linux container.

#### Starting

```bash
harbor up omlx
```

When `HARBOR_OMLX_MANAGE_HOST=true`, Harbor automatically starts the oMLX server before the proxy comes up. The server is launched via `uv run omlx serve` from the `services/omlx/` workspace, using the configured model directory and cache directory.

Start a frontend against oMLX:

```bash
harbor up webui omlx
```

Use it with host tools:

```bash
harbor launch --backend omlx --model Qwen3.5-4B-4bit codex
```

The oMLX admin dashboard is available through the proxy:

```bash
http://localhost:34940/admin
```

#### Configuration

##### Environment Variables

Following options can be set via [`harbor config`](./3.-Harbor-CLI-Reference.md#harbor-config):

```bash
# Harbor proxy port
HARBOR_OMLX_HOST_PORT              34940

# Proxy image
HARBOR_OMLX_IMAGE                  caddy
HARBOR_OMLX_VERSION                2-alpine

# Host workspace and data paths
HARBOR_OMLX_WORKSPACE              ./services/omlx
HARBOR_OMLX_BASE_PATH              ./services/omlx
HARBOR_OMLX_MODEL_DIR              ./services/omlx/models
HARBOR_OMLX_CACHE_DIR              ./services/omlx/cache

# Host endpoint
HARBOR_OMLX_UPSTREAM_URL           http://host.docker.internal:8096
HARBOR_OMLX_RUNNER_PORT            8096

# Default model and pull source
HARBOR_OMLX_MODEL                  Qwen3.5-4B-4bit
HARBOR_OMLX_HF_PATH                mlx-community/Qwen3.5-4B-4bit

# Host lifecycle and server options
HARBOR_OMLX_API_KEY                sk-omlx
HARBOR_OMLX_EXTRA_ARGS
HARBOR_OMLX_MANAGE_HOST            true
```

##### Volumes

The Harbor `omlx` service mounts only `services/omlx/Caddyfile` into the proxy container. The host runner uses:

- `services/omlx/pyproject.toml` - project file declaring the pinned oMLX dependency
- `services/omlx/.venv/` - virtual environment managed by `uv`
- `services/omlx/models/` - local MLX-format model directories
- `services/omlx/cache/` - paged SSD KV cache
- `services/omlx/logs/` - host runner log files

#### Model Management

```bash
harbor models ls --source omlx
harbor models pull --source omlx mlx-community/Qwen3.5-4B-4bit
harbor models rm --source omlx mlx-community/Qwen3.5-4B-4bit
```

The equivalent source-subcommand form is also supported:

```bash
harbor models omlx pull mlx-community/Qwen3.5-4B-4bit
```

`harbor omlx pull` downloads Hugging Face model repositories into `HARBOR_OMLX_MODEL_DIR`. oMLX discovers MLX-format model subdirectories automatically, including two-level paths such as `mlx-community/Qwen3.5-4B-4bit`.

To change Harbor's default model selection:

```bash
harbor config set omlx.model Qwen3.5-4B-4bit
```

#### API

Harbor exposes oMLX at:

```bash
http://localhost:34940/v1
```

Containers use:

```bash
http://omlx:8080/v1
```

oMLX provides OpenAI-compatible endpoints such as `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/rerank`, and `/v1/models`. It also exposes an Anthropic-compatible `/v1/messages` endpoint at the same proxy.

#### Integrations

`omlx` is a host-native OpenAI-compatible inference server (listening on port 8080 inside the Harbor proxy container) that many Harbor frontends and satellite tools auto-configure when started together. Wiring is performed through cross-compose overlay files (`compose.x.*.omlx.yml` and a few always-on config mergers). Containers reach the backend at `http://omlx:8080/v1` with key `sk-omlx` unless noted otherwise.

**Frontends and tools that auto-configure for omlx**

- `webui` — mounts `./services/webui/configs/config.omlx.json` (registers OpenAI provider at `http://omlx:8080/v1` with key `sk-omlx`).
- `chatui` — mounts `./services/chatui/configs/chatui.omlx.yml`.
- `aider` — mounts `./services/aider/configs/aider.omlx.yml`.
- `boost` — injects `HARBOR_BOOST_OPENAI_URL_OMLX` and `HARBOR_BOOST_OPENAI_KEY_OMLX`.
- `litellm` — mounts `./services/litellm/litellm.omlx.yaml`.
- `bifrost` — bootstrap sidecar (`compose.x.bifrost.omlx.yml`) registers omlx as a custom OpenAI provider using `HARBOR_BIFROST_OMLX_*`.
- `opint` (Open Interpreter) — overrides the entrypoint to pass `--api_base http://omlx:8080/v1 --api_key sk-omlx`.
- `optillm` — sets `OPTILLM_BASE_URL=http://omlx:8080/v1` and `OPENAI_API_KEY=sk-omlx`.
- `astrbot` — sets `HARBOR_ASTRBOT_OMLX_ENDPOINT` (also configurable in the AstrBot WebUI).
- `cognee` — `depends_on: [omlx]`, sets `LLM_PROVIDER=openai`, `LLM_ENDPOINT`, `LLM_API_KEY`, and `LLM_MODEL=${HARBOR_OMLX_MODEL}`.
- `mindsdb` — sets `LLM_FUNCTION_BASE_URL=http://omlx:8080/v1` and `OPENAI_API_KEY=sk-omlx`.
- `mi` — `depends_on: [omlx]`, sets `OPENAI_BASE_URL=http://omlx:8080` (no `/v1`) plus `MODEL=${HARBOR_OMLX_MODEL}`; `compose.x.mi.harbor.omlx.yml` supplies the harbor variant defaults.
- `ml-intern` — `depends_on: [omlx]`, sets `OMLX_BASE_URL` / `LOCAL_LLM_*` pairs and `ML_INTERN_MODEL=omlx/${HARBOR_OMLX_MODEL}`.
- `npcsh` — `depends_on: [omlx]`, configures chat/vision/reasoning models and OpenAI-compatible base URLs via `HARBOR_OMLX_MODEL`.
- `open-design` — `depends_on: [omlx]`, sets protocol, base URL, key, and `HARBOR_OPEN_DESIGN_DEFAULT_MODEL=${HARBOR_OMLX_MODEL}`.
- `opennotebook` — sets `OPENAI_COMPATIBLE_BASE_URL_LLM` and matching API key.
- `openterminal` — `depends_on: [omlx]`, injects `HARBOR_OMLX_URL`, `HARBOR_OMLX_OPENAI_URL`, `HARBOR_OMLX_API_KEY`, and `HARBOR_OMLX_MODEL`.
- `anythingllm` — sets `GENERIC_OPEN_AI_BASE_PATH=http://omlx:8080/v1`.
- `cmdh` — sets `OPENAI_BASE_URL=http://omlx:8080/v1`.
- `hermes` — `depends_on: [omlx]`, sets `OPENAI_BASE_URL=http://omlx:8080/v1`.
- `plandex` — sets `OPENAI_API_BASE=http://omlx:8080/v1`.
- `sillytavern` — `depends_on: [omlx]`, sets `SILLYTAVERN_OMLX_URL=http://omlx:8080`.

**Reverse proxy / external access**

- When `traefik` is also running, `compose.x.traefik.omlx.yml` exposes the proxy at `https://omlx.${HARBOR_TRAEFIK_DOMAIN}` on the `websecure` entrypoint and joins `traefik-public` in addition to `harbor-network`.

#### Troubleshooting

```bash
harbor omlx status
harbor omlx logs
harbor logs omlx
```

##### oMLX fails to start

Check the host runner log:

```bash
harbor omlx logs
```

Ensure `uv` is installed and that the machine is an Apple Silicon Mac with macOS 15 or newer. On first run, `uv` creates a virtual environment and installs oMLX into `services/omlx/.venv/`.

##### Proxy is unhealthy

Check the host runner:

```bash
harbor omlx status
curl http://localhost:8096/v1/models
```

If you manage oMLX yourself, set:

```bash
harbor config set omlx.manage.host false
harbor config set omlx.upstream.url http://host.docker.internal:8000
```

#### Links

- [oMLX](https://github.com/jundot/omlx)
- [oMLX website](https://omlx.ai/)
- [MLX](https://ml-explore.github.io/mlx/)