# LLM Providers > Provider configuration for consolidation and embeddings. Moved verbatim from the README front page. ai-memory runs without an LLM: hooks still capture sessions, search uses FTS5 + declared entities + graph neighbors, and summaries fall back to rule-based output. Add an LLM provider when you want LLM consolidation (on PreCompact / PostCompaction, on demand via `memory_consolidate`, or opt-in at session end with `AI_MEMORY_CONSOLIDATE_ON_SESSION_END`), richer linting, and bootstrap. Substantive session ends always write a rule-based summary page + handoff either way. A session containing only `SessionStart` / `SessionEnd` boundaries is closed without a page, handoff, or provider job. When that empty session had accepted startup context, its session-bound handoff is returned to the open pool for the next receiver instead of being lost. PreCompact / PostCompaction write a rule-based checkpoint on the hook request. Provider work for those checkpoints, and for opt-in SessionEnd, is durably queued after those deterministic writes and handled by one bounded server worker, so hook drain latency does not cancel it. The worker starts whenever a consolidator is configured; SessionEnd enqueue stays behind the opt-in flag. A checkpoint rewrite still queued when the session ends runs once anyway, even with the flag off. One already running when the session ends finishes after the SessionEnd summary and replaces it, so with the flag off the latest page can leave out the end of the session; the summary stays reachable as the superseded version. Every queued job runs as the session owner recorded at session start, not anonymously. Failed jobs retry with backoff and survive a server restart. A resumed native session is ended again only after its observation generation advances; the persisted generation watermark makes duplicate SessionEnd delivery and system clock skew converge without repeated provider work. The end watermark and automatic handoff commit atomically, and an interrupted keyed replay finishes the wiki commit, queue insert, and key completion without duplicating that handoff. On the next SessionStart, the newest cwd-eligible automatic handoff wins; accepting it expires older eligible automatic handoffs without consuming manual or sibling-directory work. A new automatic handoff also expires prior open automatic handoffs from its exact cwd, so repeated SessionEnds cannot accumulate there before a receiver starts. To keep consolidation style project-specific, write `_prompts/consolidation.md` in that project's wiki. Its body can express preferences such as "prefer Portuguese titles" or "omit routine CI noise". Automatic, single-page, and multi-page consolidation use the page; a manual `memory_consolidate` call can pass `instructions` to override it once. ai-memory sanitizes and caps the value at 2,000 characters, JSON-encodes it in the user message, and treats it as untrusted advisory data. It cannot supply facts, request tool use or disclosure, or override the consolidation schema and faithfulness rules. TTL-expired preference pages are ignored. With no active page or argument, no preference block is appended. Recommended defaults: | Provider | Default | Use when | |---|---|---| | `anthropic` | `claude-haiku-4-5` | Best default for consolidation quality and rule classification. | | `anthropic-oauth` | `claude-sonnet-4-6` | Use a Claude Pro/Max subscription via `claude setup-token`, no API key. Currently refused by Anthropic for third-party apps (usually `429`); see below. | | `openai` | `gpt-5.4-mini` | Cheaper and faster hosted option. | | `openai-oauth` | `gpt-5.5` | ChatGPT Pro/Plus/Codex backend via `ai-memory auth login openai-oauth`; no Platform API key. | | `codex` | `gpt-5.6-luna` | Reuse the Codex CLI-owned `auth.json`; access-token refresh remains owned by `codex app-server`. | | `cursor` | `cursor-grok-4.6-high` | Cursor subscription via the logged-in `agent` CLI. Runs `agent --print --mode ask` (read-only). Does not pass `--yolo`. The default id is the one `agent --list-models` labels "Grok 4.6". | | `copilot` | `gpt-5.5` | GitHub Copilot Chat backend via `ai-memory auth login copilot` or `COPILOT_GITHUB_TOKEN`; requires a Copilot subscription. | | `gemini` | `gemini-3.5-flash` | Google-hosted option with a generous free tier. | | `vertex` | `gemini-3.5-flash` | The same Gemini models through Vertex AI, billed to a Google Cloud project (Cloud credits apply). Uses `gcloud` Application Default Credentials; no API key. | | `opencode` | `mimo-v2.6-flash` | OpenCode Go or Zen via `OPENCODE_API_KEY`. Go is the default endpoint; `AI_MEMORY_LLM_BASE_URL` selects Zen. Set `AI_MEMORY_LLM_MODEL` to an id the chosen endpoint serves. | | `openai-compat` | no default | OpenRouter, Atlas Cloud, OrcaRouter, Cheaper Inference, API Route, FutureInfra, Standard Compute, Ollama, vLLM, LM Studio, and other compatible endpoints. | | `openai-compat` + `AI_MEMORY_LLM_BASE_URL=https://openrouter.ai/api/v1` | no default (recommended: `anthropic/claude-haiku-4.5`) | Hosted access to a large model catalogue through one key. See [OpenRouter](#openrouter) below and the empirical comparison in [`llm-provider-comparison.md`](llm-provider-comparison.md). | `openai-oauth` stores a refresh token in `/auth.json` and talks to the ChatGPT/Codex Responses backend, not `api.openai.com`. For Docker quick starts, run `ai-memory auth login openai-oauth` with the wrapper so the token lands in the same `ai-memory-data` volume as the server. `codex` is independent from `openai-oauth`: it never copies credentials into ai-memory's data directory and never reads a refresh token or ID token. It reads only `tokens.access_token` and `tokens.account_id` from `$CODEX_HOME/auth.json`, or from the platform home's `.codex/auth.json` when `CODEX_HOME` is unset or empty. Each request reloads the file. A first 401 may trigger one serialized `codex app-server --stdio` recovery followed by one retry; further 401 responses fail with a reauthentication hint. Override the binary with `AI_MEMORY_CODEX_EXECUTABLE` when `codex` is not on `PATH`. ```bash export AI_MEMORY_LLM_PROVIDER=codex export AI_MEMORY_LLM_MODEL=gpt-5.6-luna export AI_MEMORY_LLM_REASONING_EFFORT=medium ai-memory llm-test --provider codex --model gpt-5.6-luna --prompt "Reply with OK" ai-memory llm-test --provider codex --model gpt-5.6-luna --structured --prompt "Return a short answer" ``` `cursor` uses the Cursor Agent already logged in on this machine. Set `AI_MEMORY_LLM_PROVIDER=cursor`. The default model is `cursor-grok-4.6-high` ("Grok 4.6" in `agent --list-models`); any other id from that list works. `AI_MEMORY_CURSOR_AGENT` overrides the binary. The provider writes the prompt into a temporary workspace and calls `agent --print --mode ask --trust`. It does not pass `--yolo` or `--force`: consolidation text is untrusted captured content, and those flags would allow tool execution. The Cursor CLI has no JSON-schema mode, so the schema travels in the prompt. The answer is parsed leniently (code fences and surrounding prose are stripped) and then deserialised into the expected type; nothing checks it against the schema itself, so a value that deserialises is accepted even if it breaks a constraint serde does not enforce. The `agent` process does not inherit the server's environment. It gets `PATH`, `HOME`, user and locale variables (`LANG`, `LC_*`), temp and XDG directories, `DBUS_SESSION_BUS_ADDRESS`, proxy and CA settings, the Windows process basics, and every `CURSOR_*` variable, so `AI_MEMORY_AUTH_TOKEN` and other providers' keys never reach it. The per-call workspace is created under the system temp directory captured at startup. ```bash export AI_MEMORY_LLM_PROVIDER=cursor export AI_MEMORY_LLM_MODEL=cursor-grok-4.6-high ai-memory llm-test --provider cursor --model cursor-grok-4.6-high --prompt "Reply with OK" ``` `vertex` calls the same Gemini models as `gemini`, through Vertex AI instead of AI Studio. Usage is billed to a Google Cloud project, so Google Cloud credits (free trial, startup and promotional credits) pay for it; they do not pay for AI Studio. There is no API key. The provider uses Google Application Default Credentials, read from the same places Google's SDKs read them: - `GOOGLE_APPLICATION_CREDENTIALS`, the path to a service-account key file; - otherwise the file `gcloud auth application-default login` writes: `$CLOUDSDK_CONFIG/application_default_credentials.json`, else `~/.config/gcloud/` (`%APPDATA%\gcloud\` on Windows). `GOOGLE_CLOUD_PROJECT` names the project to bill. Without it, ai-memory uses the project the credentials file names (`project_id` in a service-account key, `quota_project_id` in a `gcloud` login). `GOOGLE_CLOUD_LOCATION` picks the region and defaults to `global`, which serves the newest Gemini models. The access token is refreshed before it expires and re-read after a `401`, so a new `gcloud` login is picked up without a restart. Workload identity federation, service-account impersonation and the GCE metadata server are not supported. The project needs the Vertex AI API enabled (`gcloud services enable aiplatform.googleapis.com`), and the account needs the `roles/aiplatform.user` role. Embeddings are unchanged: `vertex` covers the chat provider only. ```bash gcloud auth application-default login export GOOGLE_CLOUD_PROJECT=my-project export AI_MEMORY_LLM_PROVIDER=vertex ai-memory llm-test --provider vertex --model gemini-3.5-flash --prompt "Reply with OK" ``` Codex credential storage mode `file` is supported. `auto` works only when its effective credential is present in `auth.json`; keyring-only and ephemeral credentials are not read and produce an actionable missing-auth-file error. Docker is not configured automatically: the Codex executable, `CODEX_HOME`, and its credential file must all exist in the same container/environment as ai-memory. `opencode` talks to OpenCode's gateway with the `sk-...` key from `opencode.ai/auth`, read from `OPENCODE_API_KEY` only; it does not fall back to `LLM_API_KEY`. It defaults to the **Go** endpoint, `https://opencode.ai/zen/go/v1`: a subscription with a per-model monthly allowance rather than per-token billing. Set `AI_MEMORY_LLM_BASE_URL=https://opencode.ai/zen/v1` (or `llm_base_url` in `config.toml`) for **Zen**'s pay-per-token catalogue. Model ids are per catalogue and written plainly (`mimo-v2.5`, `glm-5.3-flash`), not in the `opencode-go/` form OpenCode's own client config uses. The built-in default is `mimo-v2.6-flash`, served by Go. When selecting Zen, set `AI_MEMORY_LLM_MODEL` to a model from its catalogue, such as `claude-sonnet-5-5`. The provider preserves the chosen catalogue and does not switch billing products automatically. Published Responses models, including `gpt-5.6-luna` and `gpt-6-luna`, use `/responses`. The wire API follows the selected catalogue: `minimax-m3`, `minimax-m2.7`, and `qwen3.8-max` use `/messages` on Go and Chat Completions on Zen. Go's `muse-spark-1.3-contributor` and `muse-spark-1.2-contributor` use `/responses`. Supported Claude and the other mapped Qwen models use `/messages`; remaining models retain Chat Completions. Changing the base URL reselects the transport and preserves timeout, reasoning effort, and operator headers. Go is identified by a URL path ending in `/zen/go` or `/zen/go/v1`, optionally including a complete endpoint; custom proxy URLs with other paths use the Zen mapping. See the official [Go endpoints](https://opencode.ai/docs/go/#endpoints) and [Zen endpoints](https://opencode.ai/docs/zen/#endpoints), whose source tables are [go.mdx](https://github.com/anomalyco/opencode/blob/dev/packages/web/src/content/docs/go.mdx) and [zen.mdx](https://github.com/anomalyco/opencode/blob/dev/packages/web/src/content/docs/zen.mdx). The provider sends the `x-opencode-session` correlation header OpenCode asks for, one id per logical operation, and its own `User-Agent`; `AI_MEMORY_LLM_HEADERS` overrides either. `AI_MEMORY_LLM_REASONING_EFFORT` and `AI_MEMORY_LLM_TIMEOUT_SECS` apply as for every other provider. `AI_MEMORY_LLM_PROVIDER` / `llm_provider` also accept the historical aliases `opencode-zen` and `opencode_zen` for the same provider (`llm-test --provider` takes only `opencode`); the endpoint is chosen by the base URL, not the alias. ```bash export AI_MEMORY_LLM_PROVIDER=opencode export AI_MEMORY_LLM_MODEL=mimo-v2.6-flash ai-memory llm-test --provider opencode --model mimo-v2.6-flash --prompt "Reply with OK" ``` For Claude through Zen: ```bash export AI_MEMORY_LLM_PROVIDER=opencode export AI_MEMORY_LLM_BASE_URL=https://opencode.ai/zen/v1 export AI_MEMORY_LLM_MODEL=claude-sonnet-5-5 ai-memory llm-test --provider opencode --model claude-sonnet-5-5 --prompt "Reply with OK" ``` [OpenRouter](https://openrouter.ai) is used through the generic `openai-compat` provider, the same wiring as Atlas Cloud, OrcaRouter, Ollama, vLLM, and LM Studio. There is no dedicated `openrouter` provider name, and setting `AI_MEMORY_LLM_PROVIDER=openrouter` fails at startup with `AI_MEMORY_LLM_PROVIDER=openrouter is not one of anthropic|openai|gemini|openai-compat|...`. Select it by pointing the compat base URL at OpenRouter and supplying its `sk-or-v1-...` key through `LLM_API_KEY`: A minimal working env-file (drop into `~/.config/ai-memory/openrouter.env` or the Docker `--env-file` argument): ```env LLM_API_KEY=sk-or-v1-... AI_MEMORY_LLM_PROVIDER=openai-compat AI_MEMORY_LLM_BASE_URL=https://openrouter.ai/api/v1 AI_MEMORY_LLM_MODEL=deepseek/deepseek-v4-flash AI_MEMORY_LLM_COMPAT_STRICT=true # Set only when the endpoint defaults an omitted field to SSE: AI_MEMORY_LLM_COMPAT_STREAM=false ``` Then verify: ```bash ai-memory llm-test --provider openai-compat \ --model deepseek/deepseek-v4-flash --prompt "Reply with OK" ``` `AI_MEMORY_LLM_COMPAT_STRICT=true` sends OpenAI-style `response_format=json_schema` for every structured call. Modern hosted gateways honour it; some model backends (DeepSeek among them) otherwise wrap their JSON output in markdown fences that ai-memory's tolerant parser has to strip, which occasionally drops a field. Setting it to `true` at the env-file level is safer for consolidation reliability. Turn it off (`AI_MEMORY_LLM_COMPAT_STRICT=false`) only for a specific model backend that explicitly rejects the schema-constrained shape. `AI_MEMORY_LLM_COMPAT_STREAM` controls the `stream` field on `openai-compat` Chat Completions requests. Leave it unset to omit the field, preserving the existing request shape. Set it to `false` when an endpoint defaults an omitted field to Server-Sent Events. Setting it to `true` is rejected before any request because this transport expects one JSON response and does not decode SSE. Use `false` or leave the setting unset. The `docker/.env.production.example` file in the repo ships with an OpenRouter setup pre-filled, with `moonshotai/kimi-k2.6` as the sample model (see the gotchas below for why reasoning models are ineligible). Model ids follow OpenRouter's `provider/model` convention. There is no built-in default: `AI_MEMORY_LLM_MODEL` is required, as it is for every `openai-compat` endpoint. ### Choosing a model For LLM selection guidance, see [`llm-provider-comparison.md`](llm-provider-comparison.md), where the project keeps its own A/B tests of consolidation quality, latency, and cost. That comparison uses English consolidation fixtures and covers the recommended default (`anthropic/claude-haiku-4.5`), the cheaper alternative (`openai/gpt-5.4-mini`), and the reasoning-model ineligibility note; the OpenRouter subsection here defers to it rather than duplicating rankings. For workloads dominated by non-English content, community reports suggest that a higher-recall multilingual embedder (for example `baai/bge-m3` via the `openai-compat` embedder) noticeably improves retrieval quality relative to the built-in English-only local embedder. Configuration is in the embedding paragraph below. ai-memory automatically layers OpenRouter's app-attribution headers (`HTTP-Referer` and `X-Title`) whenever the base URL points at `openrouter.ai`, so ai-memory traffic is credited on OpenRouter's app leaderboard. `AI_MEMORY_LLM_HEADERS=HTTP-Referer=https://example.com,X-Title=my-app` overrides them per operator. `AI_MEMORY_LLM_REASONING_EFFORT` is honoured on this path: OpenRouter hosts receive `reasoning: { effort, exclude: true }`. OpenRouter's LLM/completions API is what ai-memory uses above. It does **not** reliably expose an OpenAI-compatible `/v1/embeddings` endpoint (historically it has not), so do not assume an OpenRouter base URL works for the embedder. If a given deployment of OpenRouter does offer an embeddings endpoint, verify it answers the standard `/v1/embeddings` shape (`curl` the endpoint directly) before relying on it. When it does, configure the embedder through the openai-compat path rather than the openai path, which has an OpenAI-specific key precedence chain: ```bash export AI_MEMORY_EMBEDDING_PROVIDER=openai-compat export AI_MEMORY_EMBEDDING_BASE_URL=https://openrouter.ai/api/v1 export AI_MEMORY_EMBEDDING_MODEL=baai/bge-m3 export AI_MEMORY_EMBEDDING_DIM=1024 # LLM_API_KEY is reused automatically when EMBEDDING_API_KEY is unset on # the openai-compat embedder path; no second key is required. ``` `AI_MEMORY_EMBEDDING_DIM` is mandatory for openai-compat embeddings and must match the model. For English-only content the built-in local sentence-transformer (`AI_MEMORY_EMBEDDING_PROVIDER=local`, see [`local-embeddings.md`](local-embeddings.md)) is a valid alternative that avoids hosted egress; for non-English content, a multilingual hosted model (`baai/bge-m3`, `intfloat/multilingual-e5-large`, or similar) served by any provider that exposes a real `/v1/embeddings` endpoint is the practical choice. The two endpoints are still independent, so setting `AI_MEMORY_LLM_BASE_URL` alone does not redirect embeddings; set `AI_MEMORY_EMBEDDING_BASE_URL` too. > [!TIP] > **`openai-compat` sends `X-Request-Id` on every chat attempt.** Each HTTP > attempt of one logical operation, including the strict-to-tolerant > fallback, carries the same operation id (a 36-character UUID v7) in the > `X-Request-Id` header, so a gateway that records it (for example as > `req=`) can correlate every attempt of the same operation and forward > it to the engine (vLLM). The caller of the operation (the consolidation > invocation or the auto-improve review) generates the id fresh, and it > is never the agent's session id, so a gateway never sees two operations > of one session, or a crash-resumed run, as one. Crash re-entry is exactly > one new operation: a process restart or a queue re-claim mints a new id > for the calls it makes. The official `openai` provider and `opencode` do > not send this header; `opencode` keeps its own `x-opencode-session` > contract. Because ai-memory owns the header on the `openai-compat` path, a > static `x-request-id` entry in `AI_MEMORY_LLM_HEADERS` is refused at > startup for that provider; the dynamic value must not be shadowed or > duplicated. > [!TIP] > **`openai-compat` can also say what each operation is.** With > `AI_MEMORY_LLM_REQUEST_METADATA_HEADERS=true` (`llm_request_metadata_headers > = true`), every chat attempt carries, next to `X-Request-Id`: > > | Header | Value | Sent when | > | --- | --- | --- | > | `x-ai-memory-workload` | `consolidation`, `profile`, `dream`, `lint`, `auto-improve`, `experience`, `bootstrap`, `query-answer`, `rerank` or `unspecified` | always | > | `x-ai-memory-workspace` | workspace name | the operation reads one workspace | > | `x-ai-memory-project` | project name | the operation reads one project | > | `x-ai-memory-user` | username | the operation acts for one user | > > A gateway can use them to attribute cost or apply a limit per workload, > project or user. A header the operation cannot name with one value is left > out, never guessed: a profile classification batch mixing several > operators' sentences sends no user, a `memory_query` answer (whose > evidence spans projects) sends no scope, and the rerank call sends only the > workload. A value that is not a valid HTTP header value (a non-ASCII > project name, for example) is also left out. The values are identifiers, > never page content. The setting is off by default, only `openai-compat` > sends these headers (the official providers, `opencode`, and a fallback to > another provider never do), and the four names are refused in > `AI_MEMORY_LLM_HEADERS` for `openai-compat` whether or not the setting is > on. The setting is server-wide: an `openai-compat` entry in > `llm_fallbacks` sends the headers to its own endpoint too, so turn it on > only when every `openai-compat` endpoint you configure may see your > workspace, project and user names. ### Falling back when credit or quota runs out An ordered `[[llm_fallbacks]]` chain moves to the next provider on a transient failure and also when a provider says the account is out of credit, quota or billing: Anthropic's `400` "credit balance is too low", OpenAI's `429 insufficient_quota`, an OpenRouter `402`, a Gemini daily `RESOURCE_EXHAUSTED`, a Codex `usage_limit_reached` or a Copilot `quota_exceeded`. A plain invalid-key `401` does not fall back. The exhausted provider is skipped for `llm_quota_cooldown_secs` (default 900). The next call after that tries it again, so it takes over again once topped up. ```toml llm_provider = "anthropic" llm_model = "claude-sonnet-4-6" [[llm_fallbacks]] provider = "openai" model = "gpt-5.4-mini" api_key_env = "OPENAI_API_KEY" ``` `ai-memory status` then shows `primary anthropic/claude-sonnet-4-6 exhausted (credit/quota); using fallback openai/gpt-5.4-mini`. The full rule, with the per-provider signals, is in [`llm-provider-fallback.md`](llm-provider-fallback.md#credit-and-quota-exhaustion). ### Probing the server's own provider `ai-memory llm-test` builds a provider from the shell it runs in. That is the right check before a key goes into the config, and it works with the server down — but it is the wrong check when the server runs in a container or on another host, because the keys exist there and not in your shell. A failure like `building LLM provider` (no key) or a provider `400 Please pass a valid API key` says nothing about the server in that case. `ai-memory llm-test --server` asks the server to run the probe itself, with its own configured provider, fallback chain and keys, and prints the provider, model, latency and the error class: ```bash ai-memory llm-test --server ``` It is the same probe `status` and the health panel name when a provider call fails, it runs one fixed short prompt, and it takes no `--provider`, `--model`, `--prompt`, `--base-url` or `--api-key`: a client cannot hand the server a key nor point it at another URL. On a multi-user server the route is root-only, and a provider's error body is scrubbed before it is printed, so a key the provider quoted back never reaches your terminal. Without a configured provider the server answers `503` and says so. ### Security and gotchas - **Bind address vs. port publish.** When running inside a container, the process-level bind (`--bind 0.0.0.0:49374` or similar) does not decide who can reach the port; the container's publish spec does. `-p 127.0.0.1:49374:49374` restricts to loopback on the host; `-p 49374:49374` publishes on every host interface. If the port is ever reachable beyond loopback, generate a bearer token first: ```bash ai-memory generate-auth-token export AI_MEMORY_AUTH_TOKEN= export AI_MEMORY_ALLOWED_HOSTS= ``` The server logs a loud startup warning about non-loopback exposure without a token; treat it as blocking. `AI_MEMORY_ALLOWED_HOSTS` guards against DNS-rebinding. TLS termination is expected to sit in front of the process; see [`docs/https-via-proxy.md`](https-via-proxy.md). - **Secret hygiene.** The env file that holds `LLM_API_KEY` (and any provider-specific key you set) should be `chmod 600` and outside any directory a version-control tool sees. If you back up `~/.config/ai-memory/` (see [`docs/backup.md`](backup.md)), exclude the env file. The sample script and mirror `.gitignore` shipped there already exclude `*.env` by default. - **Cheap models can drift.** LLM consolidation writes durable wiki pages, so a model that occasionally fabricates a `kind: decision` page for a session that made no decision leaves a page in the wiki that ai-memory will treat as first-class evidence on subsequent reads. This is a known cost of running a smaller / cheaper model on the consolidation prompt. Spot-check `/wiki///decisions/` during the first week on a new model for pages you do not recognize; if any appear, either raise the provider tier for consolidation only (keep the cheaper model for retrieval reranking) or unset `AI_MEMORY_LLM_PROVIDER` to fall back to the deterministic rule-based path. - **`:free`-suffix models draw from a shared quota.** Any model id ending in `:free` on OpenRouter is served from a pool shared across all users of the free tier, not the caller's paid budget. When that pool is exhausted, requests fail with an upstream-shared-pool rate-limit error. They are fine for evaluation but unreliable for production; prefer a model listed with multiple upstream providers on its OpenRouter page. - **Reasoning-mode models are ineligible for consolidation.** Consolidation and lint use a strict-JSON output contract; reasoning models route most of their token budget through internal reasoning and either stall past the 300 s per-request ceiling or return empty strings. `docs/install.md#llm-provider-tiers` has the standing "what we don't recommend" note listing the pattern; the same applies to any OpenRouter model flagged as a reasoning/thinking variant. Prefer a non-reasoning peer for `AI_MEMORY_LLM_MODEL`. `anthropic-oauth` hits the same `/v1/messages` endpoint as `anthropic` but authenticates with an OAuth bearer token instead of an API key. Run `claude setup-token` once; it prints the token and does not save it anywhere, so export it yourself as `ANTHROPIC_OAUTH_TOKEN=` (or `CLAUDE_CODE_OAUTH_TOKEN`) and set `AI_MEMORY_LLM_PROVIDER=anthropic-oauth`. No `ANTHROPIC_API_KEY` is needed. The Docker wrappers forward either token by name to short-lived helper commands such as `llm-test`; configure the long-lived server container separately as shown in the installation guide. For both Anthropic providers, ai-memory omits `temperature` for Claude 4.7 and later models and Claude Mythos Preview because those models reject non-default sampling parameters. `llm-test` sends the same representative 0.2 value as the normal pipeline before the provider applies that compatibility rule. **⚠️ Unofficial and against Anthropic's usage policies. Use at your own risk; it may get your account rate-limited or banned. See [the warning in `docs/install.md`](install.md#anthropic-via-claude-subscription-oauth).** Anthropic currently refuses subscription tokens from third-party apps ([legal and compliance](https://code.claude.com/docs/en/legal-and-compliance)), so expect `anthropic-oauth` to fail on every request, typically with `provider error 429` and `rate_limit_error`, even though the same token works in the `claude` CLI. That 429 is a policy refusal, not a real rate limit. Use the `anthropic` provider with an Anthropic Console API key, or another provider from the table above. `copilot` stores a GitHub user token in the same auth file, exchanges it for a short-lived Copilot API token via GitHub's `/copilot_internal/v2/token`, and uses the Copilot Chat endpoint with `vscode-chat` integration headers. You can also set `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN` on the server. Structured Copilot Chat Completions stopped at the token budget (`finish_reason = "length"`) fail with `truncated-response`, even if the returned text parses as JSON. This is a terminal failure; increase the output budget or reduce the requested output before trying again. Plain text completions keep returning the generated text. > [!TIP] > **For the OAuth/subscription backends, prefer a small, fast model** via > `AI_MEMORY_LLM_MODEL` where the backend lets you choose one. ai-memory's LLM > work (consolidation, lint, explore) is summarisation, not hard reasoning, so a > Haiku/mini-class model is plenty and is much easier on subscription rate > limits. Save the high-effort thinking models for your coding agent. Per backend: > - `anthropic-oauth`: set `claude-haiku-4-5`. > - `openai-oauth` / `codex`: leave the provider default (`gpt-5.5`). The > Codex/ChatGPT backend only accepts a small server-defined set of model ids > and rejects others (e.g. `gpt-5-mini`) with a deterministic 400, so do not > override the model here. > - `copilot`: a mini-class id such as `gpt-5-mini` may work, but Copilot's > accepted model set is unverified. Check before relying on it, and fall back > to the default if the endpoint rejects your choice. > [!TIP] > **OpenAI-compatible structured output is schema-constrained by default.** > ai-memory sends each operation's JSON Schema through > `response_format=json_schema`, which recent Ollama, vLLM, LM Studio, and > llama.cpp releases honour. It falls back to the tolerant parser when an > endpoint explicitly rejects that field or returns a malformed shape. Set > `AI_MEMORY_LLM_COMPAT_STRICT=false` only for an incompatible endpoint. The tolerant parser removes ``, ``, `` and `` wrappers outside the JSON answer and unwraps Markdown fences. Literal tags inside JSON strings remain intact, so summaries can quote prompt templates or other tagged text. Anthropic's structured text replies use the same parsing rules when the provider uses automatic tool choice. > [!TIP] > **Thinking-capable local engines (vLLM / SGLang serving Qwen3-class > models).** Such engines spend the output budget on a reasoning pass before > the structured payload and can truncate the JSON mid-object > (`finish_reason = "length"`). Set > `AI_MEMORY_LLM_COMPAT_DISABLE_THINKING=true` to send > `chat_template_kwargs: {"enable_thinking": false}` on every openai-compat > request so the engine spends the budget on the payload instead. The knob is > opt-in, openai-compat only, and off by default: existing vLLM / Ollama / > LM Studio setups are unchanged unless it is set, and every other provider > ignores it. When the engine still truncates a structured response, or > returns HTTP 2xx with an empty `message.content`, the call fails fast > with the terminal errors `truncated-response` / `empty-content`. There is > no retry and no second HTTP call, and the error text carries no response > content. For small-context local models, configure both consolidation limits. The input target accounts for the complete rendered prompt, including bounded slot and current-page context plus the structured-output schema; the output limit is sent to the provider. Their sum must fit the model context window, with extra headroom because provider tokenizers differ: ```toml [consolidation] max_input_tokens = 10000 max_output_tokens = 1000 ``` The equivalent environment variables are `AI_MEMORY_CONSOLIDATION__MAX_INPUT_TOKENS` and `AI_MEMORY_CONSOLIDATION__MAX_OUTPUT_TOKENS`. Provider failures during an automatic PreCompact/PostCompaction checkpoint fall back to the deterministic rule-based page; admission, storage, and scope errors still fail closed. The validated minimums are 8,350 input and 1,000 output tokens at the default `input_token_safety_margin` of 0.8; a lower margin shrinks the character budget and needs a proportionally larger input target. Every chat provider bounds each completion request at 300 seconds (`AI_MEMORY_LLM_TIMEOUT_SECS` to override, or `llm_timeout_secs = 900` in config.toml; the quick openai-oauth token refresh keeps the default ceiling). The default tolerates a local engine cold-loading a large model; slow hosted gateways whose long completions exceed the ceiling fail every request with `http: error sending request`, so raise the value there instead of watching consolidation exhaust its retries. Reranking is optional and off by default. With an LLM provider configured, `AI_MEMORY_RERANKER=llm` makes project and explicit-scope `memory_query` calls over-fetch from the hybrid stage, fuse scopes, and make at most one LLM call to reorder the best candidates. This can promote a relevant page that RRF ranked below the requested cut, at the cost of LLM latency and usage. The request sends the query plus at most 30 bounded page titles and search snippets to the configured provider; all values are JSON-encoded and treated as untrusted data. A timeout, provider error, or incomplete/invalid score set preserves the normal order. The timeout is 20 seconds per query; raise it with `AI_MEMORY_RERANK_TIMEOUT_SECS` (or `rerank_timeout_secs` in `config.toml`) when a slow provider or a fallback chain needs longer to answer. `global=true` and supplemental global-preference hits keep their existing non-RRF ranking. Concurrent provider calls are capped at four; saturated queries keep their local ranking without waiting. ## Workload telemetry `ai-memory status --json` (the root-protected `GET /admin/status` response) includes `providers.llm_workloads` whenever an LLM provider is configured. Its closed labels are `unspecified`, `consolidation`, `rerank`, `auto-improve`, `bootstrap`, `query-answer`, `lint`, `profile`, `dream`, and `experience`. Each row reports process-lifetime calls, successes and errors, total/max latency, fixed-bucket p50/p95 upper bounds, last outcome, and token counts. The counters are process-local and reset on restart; the server constructs the configured provider wrapper once at startup and has no runtime provider reload. `providers.llm_workloads_started_ms` gives the Unix time in milliseconds when the counters started (null when no LLM provider is configured). A window is the difference between two snapshots with the same `llm_workloads_started_ms`. A restart, including one for a configuration change, starts a new value, so it closes a segment, and a window spanning restarts adds its segments. `estimated_input_tokens` and `estimated_output_tokens` use a deliberately coarse four-characters-per-token estimate so structured and plain calls remain comparable. `reported_input_tokens` and `reported_output_tokens` count only responses for which the provider supplied usage; `usage_reported_calls` shows that coverage. Structured-output provider adapters currently return decoded JSON without usage, so their reported counters can remain zero even when the estimate is nonzero. The telemetry stores no prompt, completion, scope, project, or caller-supplied label. A fallback chain is measured as one provider-interface invocation, not as one call per internal candidate attempt; an outer caller retry is another invocation. The workload label itself never selects a provider or model. A task uses its own provider and fallback chain only when the config declares an `[llm.]` profile for it (see `config.default.toml`); every other task uses the global provider and its fallback chain. Embeddings are optional and separate from the LLM provider. Set `AI_MEMORY_EMBEDDING_PROVIDER=openai`, `voyage`, `google`/`gemini`, `openai-compat`, or `copilot` when you want vector retrieval in addition to FTS5 + entity + graph-neighbor retrieval. `openai-compat` targets self-hosted engines (Ollama, LM Studio, vLLM): it needs no API key and requires explicit `AI_MEMORY_EMBEDDING_BASE_URL`, `AI_MEMORY_EMBEDDING_MODEL`, and `AI_MEMORY_EMBEDDING_DIM`. The optional `EMBEDDING_API_KEY` credentials the embedder alone and is checked before `OPENAI_API_KEY` and `LLM_API_KEY`, so embeddings can run on a different provider than the LLM. Because the two endpoints are independent, `AI_MEMORY_LLM_BASE_URL` redirects only the LLM; set `AI_MEMORY_EMBEDDING_BASE_URL` as well or embedding traffic still goes to the embedding provider's default endpoint. Both the FTS-only and hybrid paths apply the same bounded page-authority adjustment after candidate generation; embeddings improve relevance recall but do not decide which source is canonical. Asymmetric embedding models, trained with a different instruction for queries than for the text being indexed, need that instruction added to the text itself; the OpenAI-compatible `/v1/embeddings` wire format has no field for it. `AI_MEMORY_EMBEDDING_QUERY_PREFIX` and `AI_MEMORY_EMBEDDING_DOCUMENT_PREFIX` (only applied by the `openai` and `openai-compat` embedders) are prepended to query and document text respectively, before the existing truncation, so a long body is still truncated to the same overall input cap with the prefix included. Both are empty by default (no behaviour change when unset) and are not trimmed, so a publisher's trailing space is preserved exactly. For example, NVIDIA's [`nvidia/Nemotron-3-Embed-1B-BF16`](https://huggingface.co/nvidia/Nemotron-3-Embed-1B-BF16) (2048-dim, served locally through vLLM/`openai-compat`) specifies `"query: "` for queries and `"passage: "` for documents per its model card: ```toml embedding_provider = "openai-compat" embedding_model = "nvidia/Nemotron-3-Embed-1B-BF16" embedding_base_url = "http://localhost:8000/v1" embedding_dim = 2048 embedding_query_prefix = "query: " embedding_document_prefix = "passage: " ``` Base E5 models (`intfloat/e5-base-v2`, `e5-large-v2`, multilingual E5, …) use the same `"query: "` / `"passage: "` convention. Instruction-tuned E5 variants and Qwen3-Embedding instead need a task-instruction string on the **query side only** (their documents are embedded plain, with no document prefix), and the two use **different exact spacing**, confirmed against each model card: - `intfloat/e5-mistral-7b-instruct`: `embedding_query_prefix = "Instruct: {task description}\nQuery: "`, with a trailing space after `Query:`. - `Qwen/Qwen3-Embedding-0.6B` (and the other Qwen3-Embedding sizes): `embedding_query_prefix = "Instruct: {task description}\nQuery:"`, with **no** trailing space; the query text follows the colon directly. Fill in your own task description for `{task description}`, leave `embedding_document_prefix` unset for both, and don't copy one model's exact string for the other. The trailing-space difference is publisher-specified, not a typo. Changing `embedding_query_prefix` alone never requires re-embedding: only `embedding_document_prefix` is folded into the stored embedding identity (a non-empty value changes what `ai-memory` treats as the current `{provider, model, dim}` triple for that page's vectors), so a document prefix change makes `memory_query` stop matching the old vectors and the next `ai-memory embed` (scheduled or manual) pass re-embed them automatically, without `--force`. `--force` remains useful to re-embed everything immediately rather than waiting for scheduled backfill, or to force a re-embed for a reason the triple alone doesn't capture (e.g. touching up the vLLM server's tokenizer or quantization). `AI_MEMORY_EMBEDDING_PROVIDER=copilot` reuses the same Copilot OAuth login as the `copilot` LLM provider (`ai-memory auth login copilot` or `COPILOT_GITHUB_TOKEN`/`GITHUB_COPILOT_API_TOKEN`), with no separate API key. It defaults to model `text-embedding-3-small`, dim 1536, and calls Copilot's `/embeddings` endpoint following the same OpenAI-compatible contract Copilot documents for chat. That endpoint's exact shape is not covered by a live test against Copilot here; treat it as needing a real-Copilot smoke test before relying on it in production. `AI_MEMORY_EMBEDDING_PROVIDER=local` needs no key and no server at all: sentence embeddings run in-process (pure-Rust `all-MiniLM-L6-v2`, 384-dim), with the model fetched once into `/models/` under pinned checksums; see [`docs/local-embeddings.md`](local-embeddings.md). See [`docs/install.md#llm-provider-tiers`](install.md#llm-provider-tiers) for env vars and Ollama/OpenRouter/Atlas Cloud/OrcaRouter/Cheaper Inference/API Route/FutureInfra/Standard Compute examples, and [`docs/llm-provider-comparison.md`](llm-provider-comparison.md) for the empirical model comparison.