--- name: autorag-setup description: Install and configure AutoRAG, register its Lite MCP server for external agents, or repair its single search model, approved roots, indexes, datasources, and health checks without exposing credentials. Use when AutoRAG or MCP is missing, init/refresh/health fails, indexes are stale, or the user wants to add folders or datasources. license: Apache-2.0 --- # AutoRAG setup Use this skill when AutoRAG is unconfigured, the `autorag` CLI is missing, model resolution fails, indexes are missing or stale, or the user wants to change the document collection or datasources. For model-free external-agent search, use `autorag-lite-setup` to configure and register `autorag-mcp`; do not configure a search model or install a search skill for Lite. This skill's model setup and CLI search apply to the full librarian, not the Lite MCP server. ## Safety - Inspect only non-secret provider/model metadata and credential availability. - Never print, copy, migrate, compare, or persist credential values. Store only environment-variable names such as `apiKeyEnv`. - Do not scan the whole filesystem or home directory without explicit approval. - Setup, indexing, and refresh never move, rename, edit, or delete source documents. The librarian itself may organize files when the user asks it to. - Do not index system trees, app bundles, caches, credential stores, `node_modules`, `.git`, `dist`, `build`, `target`, `.cache`, `.autorag`, or `.jikji`. ## Install the CLI if needed The CLI is `@autorag/librarian` (`autorag`). Runtime is Node.js ≥ 24 or Bun. ```bash command -v autorag >/dev/null || bun install -g @autorag/librarian autorag --help ``` If Bun is unavailable, `npm install -g @autorag/librarian` is acceptable. ## Inspect existing configuration Check `--config`, `AUTORAG_CONFIG`, `$AUTORAG_HOME/config.json`, or `~/.autorag/config.json`. Relevant fields are: - `searchPaths`, `workspacePath`, and `memoryPath` - `model.provider`, `model.id`, `model.api`, `model.baseUrl`, `model.apiKeyEnv` - `bm25`, `minSync`, and `jikji` - `limits` (retrieval, baseline-prefetch, and model-facing candidate caps) - `datasources` and `ui` Preserve explicit user choices and a working config unless the user asks to replace them or health checks fail. ## Configure one search model AutoRAG is the specialized librarian agent. One configured model plans the search, calls retrieval and filesystem tools, reads sources, judges evidence, and curates the answer in one loop. There are no orchestrator/explorer roles. Prefer a model with reliable tool calling and structured output, enough context for source excerpts, high output TPS, and low first-token latency. Use a larger reasoning model only when difficult synthesis or domain judgment matters more than latency. Start from a provider/model the current runtime can actually call. A Pi-usable ChatGPT, Claude, Gemini, or other authenticated subscription is valid; an installed CLI or subscription that Pi cannot invoke is not. For custom OpenAI-compatible endpoints, record the real wire API, base URL, and credential environment-variable name. Allowed `api` values: - `openai-completions` - `openai-responses` - `anthropic-messages` - `openai-codex-responses` - `azure-openai-responses` If no callable setup can be established, ask the user for the provider, model id, API protocol, base URL when custom, and credential environment-variable name. Do not invent provider identities or model ids. ## Propose and approve document roots Explicit user paths win. Otherwise inspect only these likely document-dense candidates for existence and approximate supported-file counts, then ask for approval before indexing: | OS | Recommended | Optional | |---|---|---| | macOS | `~/Documents`, `~/Downloads`, `~/Desktop` | `~/Notes`, `~/Obsidian`, user-named project docs | | Linux | XDG Documents/Downloads/Desktop or their `~/` defaults | `~/Notes`, `~/Sync`, Nextcloud/Syncthing roots | | Windows | Documents, Downloads, Desktop shell folders | OneDrive document roots, user-named project docs | Supported parsed formats are `md`, `markdown`, `txt`, `text`, `pdf`, `docx`, `pptx`, `xlsx`, `xlsm`, `xlsb`, `xls`, `ods`, `csv`, `tsv`, `hwp`, `hwpx`, and `eml`. OCR for `jpg`, `jpeg`, `png`, `bmp`, and `tiff` is optional (`parserOptions.ocr.enabled`). Do not present legacy `.doc` as a supported parsed format. Keep the first-run set small, usually one to three roots. Present a concrete proposal and require `yes`, a narrowed keep-list, a custom list, or `skip` before running `refresh`. ## Initialize ```bash autorag init \ --search-paths "/path/to/documents,/path/to/notes" \ --workspace "/path/to/workspace" \ --model-provider PROVIDER \ --model-id MODEL ``` Model resolution runs through the pi runtime. A configured `provider`/`id` is resolved against the pi runtime catalog — the built-in providers plus any `~/.pi/agent/models.json`, custom, or extension providers — and credentials come from stored pi auth (`~/.pi/agent/auth.json`, API key or OAuth established with `pi /login`), then provider environment variables. When no `model` is configured, AutoRAG uses pi's `defaultProvider`/`defaultModel` when that provider has configured credentials; otherwise it falls back to the authenticated local codex runtime. So model flags may be omitted when either pi or the local runtime already supplies the intended model. Inspect what pi can resolve with `autorag models list` (`--available` shows only providers with configured credentials); it never prints credential values. For a custom endpoint, add `api`, `baseUrl`, and `apiKeyEnv` to the single `model` object in the trusted config: ```json { "searchPaths": ["/path/to/documents"], "model": { "provider": "openrouter", "id": "anthropic/claude-sonnet-5.5", "api": "openai-completions", "baseUrl": "https://openrouter.ai/api/v1", "apiKeyEnv": "OPENROUTER_API_KEY" } } ``` When `provider`/`id` names a pi runtime catalog model, the catalog entry stays the base: `baseUrl`, `api`, and any declared `reasoning`, `input`, `contextWindow`, or `maxTokens` override only those fields, and the catalog's reasoning, thinking, and compat settings are kept. Only an id outside the catalog (private proxy, Ollama, LiteLLM) gets a generic text model with a 128k context window unless those fields are declared. Use `--force` only when intentionally replacing an existing config, and target it with an explicit path (`--config` or `AUTORAG_CONFIG`); `--force` refuses to replace the implicit `~/.autorag/config.json`. Legacy cwd `autorag.config.json` is a migration source only and is never deleted by init. ### Retrieval defaults MinSync and Jikji are enabled by default. Leave them enabled unless the user explicitly asks otherwise. Indexing never happens while answering: a question only reads indexes that `autorag refresh` (or `autorag watch`) built, so run a refresh after setup and whenever documents change. A never-refreshed workspace still answers, but without MinSync/Jikji evidence. Refresh (never a query) auto-installs the binaries: MinSync installs a verified GitHub release into `/.autorag/bin` (`minSync.autoInstall` defaults to true), and Jikji installs the pinned `jikji-cli` 0.2.1 through cargo (`jikji.autoInstall` defaults to true; requires the Rust toolchain). A `jikji` older than 0.2.0 on PATH or in the cache is skipped and replaced by that build on refresh. Set `"autoInstall": false` only when managing the binary yourself. Refresh is incremental: MinSync syncs only changed parsed mirrors, and `jikji prepare` reuses unchanged documents. Jikji roots prepare one at a time (they share one central SQLite file that rejects concurrent writers). A search root nested inside, or duplicating, another root is skipped for Jikji because the outer root's index already covers it. Jikji keeps its prepared corpus (search index, parsed text, wiki, routes) in one central store, not inside your documents: `/jikji/index.sqlite` plus `/jikji/roots//` (macOS `~/Library/Application Support`, Linux `$XDG_DATA_HOME` or `~/.local/share`, Windows `%LOCALAPPDATA%`; override with the `JIKJI_DATA_DIR` environment variable). The only file `refresh` adds inside an approved document root is the `.jikji_agent_map.md` guide. Tell the user that before the first `refresh`, and never delete or edit Jikji's central store by hand; removing it only forces a full Jikji re-prepare on the next refresh. Jikji 0.1.x wrote a `/.jikji` directory instead; such a leftover is now unused and safe for the user to delete. Jikji 0.2.0 and newer refuse to index a root nested inside, or containing, an already indexed root (`prepare root overlaps existing indexed root`), which is why overlapping search roots are collapsed to the outermost one before Jikji runs. Exact duplicate exclusion is enabled by default. AutoRAG invokes the external `dupey` CLI before parsed-mirror indexing, keeps the newest filesystem copy for each exact canonical-text hash, and excludes older copies from the mirror. Install dupey during setup when it is missing (`command -v dupey || cargo install dupey --locked`) and tell the user the feature is available; when installation is impossible, refresh continues without this optimization and the user is told duplicate exclusion is off. Set `"excludeExactDuplicates": false` to index every copy. AutoRAG owns no embedding model. MinSync provisions the embedder (`Qwen/Qwen3-Embedding-0.6B`, 1024 dimensions) in its own cache (`~/.cache/minsync/models`); there is no local model download and no bundled runtime. One shared embedder serves everything that embeds inside AutoRAG: MinSync indexing/query, memory similar-question search, the chat-crawler sidecars, the GitHub Gist vector index, and the endpoint external CLIs are pointed at. The default needs no embedder flags and no API key. It resolves in order: 1. A configured remote endpoint (`minSync.embedder`), used for everything; no daemon is started and no local model is loaded. 2. A running `minsync serve` daemon (loopback only; MinSync 0.4.12+). AutoRAG never starts it: run `minsync serve --idle-timeout 0` yourself, as a launchd/systemd service (see `docs/indexing/embedding-backends.md`), or press Ctrl+Shift+E in `autorag tui`. `autorag setup` only reports whether it is alive. 3. MinSync's in-process engine, through the optional Node-API addon `@nomadamas/minsync-native` (same engine and weights, so bit-for-bit identical vectors; not bundled). Without a daemon and without the addon, embedding fails with an error naming the missing package. Override the embedder only when intentionally using a remote provider: ```bash autorag init \ --embedder-id "voyageai/voyage-4-lite" \ --embedder-base-url "https://openrouter.ai/api/v1" \ --embedder-api-key-env "OPENROUTER_API_KEY" \ --embedder-dimension 1024 \ --embedder-batch-size 64 ``` A remote `minSync.embedder` names an `id` (`openai:`, `tei:`, or any id together with a `baseUrl`) plus optional `apiKeyEnv` (the **name** of the environment variable holding the bearer token), `dimension`, `batchSize`, and `timeoutMs`. OpenAI-protocol endpoints POST `{baseUrl}/v1/embeddings`; a `tei:` endpoint POSTs `{baseUrl}/embed`; `openai:` with no `baseUrl` means `https://api.openai.com`. Only store the environment-variable name, never its value. `dimension` and `batchSize` must be positive integers, and `dimension` must match the endpoint (the local default is 1024). ### Jev decision model (required) and question decomposition (on by default) Jev is required: AutoRAG cannot answer without it. Decomposition is on by default; leave it enabled. Together they make simple questions fast and multi-part questions thorough. - **Jev** (`jev`, default `{ "backend": "dazzi" }` when signed in to Dazzi Models, else `{ "backend": "openrouter" }`; model `jev-1.13` on Dazzi, `typesafe/jev-1.13` on OpenRouter) runs before the fast answer. It routes each question to local search (everything needing more information), `organize` (moving, renaming, converting, or sorting files and folders), `extra` (a knowledge answer, or any other action that needs no retrieval, such as committing code: one turn with every tool), or the `config` branch, and decides whether to decompose it. A non-local branch only wins above a 0.75 confidence floor; anything weaker stays on local search. On local search it also decides, per registered datasource, whether to search it before the fast answer, using each datasource's `description` and where similar past questions were answered (retrieval memory). When setting up a datasource, always write a `description` from what it actually holds: channels or rooms, people, topics, time range (for example `"Team Slack, 2024-2026: #release and #on-call channels, dependabot notifications"`). Jev is told descriptions are short, non-exhaustive summaries, so list the main content and do not try to list everything. After the fast answer it decides whether verification is needed, so a complete, evidence-backed fast answer ends the run. - **Question decomposition** (`queryDecomposition`, default model `dazzi/openai/gpt-6-luna` when signed in to Dazzi Models, else `openrouter/qwen/qwen3.7-flash`) splits a multi-part question into at most five search queries that run in parallel. Both need a credential for the chosen backend: a Dazzi Models `/login` (check with `autorag models list --provider dazzi --available`), or the user's `OPENROUTER_API_KEY` (`test -n "$OPENROUTER_API_KEY"`, never print it). The first question of every session checks that Jev answers; without a working credential the search stops with `[decision-model-unavailable]` (`autorag search` exits 3, and `--json` carries `code`, `reason`, `backend`, `model`). Read `reason`: `missing-credential` means set the key or sign in, `unauthorized` means replace the key, `model-not-found` means fix `jev.model`, `payment-required` means the backend account is out of credit, `unreachable` and `rate-limited` mean retry. ```json { "jev": { "backend": "openrouter" }, "queryDecomposition": { "model": { "provider": "openrouter", "id": "qwen/qwen3.7-flash" } } } ``` `autorag init` writes these defaults into new configs. To change them: - Jev backend and model: any provider with decision models, e.g. `{ "backend": "openai", "model": "gpt-6-luna" }` (`OPENAI_API_KEY`), `{ "backend": "openrouter", "model": "openai/gpt-6-luna-decisions" }`, `"typesafe"` (`TYPESAFE_API_KEY`), `"vercel-ai-gateway"` (`AI_GATEWAY_API_KEY`), `"cloudflare-workers-ai"`, or `"opencode"`. List the choices with `autorag models list --decision --available`; never invent a model id. - sLLM fallback: when Jev is unavailable, the session runs on a small chat model (default: the decomposition model) and warns `decision-model-fallback`. Set the model with `"jev": { "sllm": { "model": { "provider": …, "id": … } } }`, or `"fallback": false` to fail instead. - sLLM as the decision model: `"jev": { "backend": "sllm" }` decides with the small chat model alone and never calls Jev (no Jev credential needed). - Decomposition model: any catalog `provider`/`id`, with the same fields as the top-level `model`. - `"queryDecomposition": false` decomposes with the search model itself. - Jev cannot be turned off: `"jev": false` and `jev.enabled` are config errors. When questions must never leave the machine, AutoRAG is not usable; say so instead of editing the config (Jev sends the question text to its backend). When a user asks the running agent itself to change its settings (switch the default model, add a provider, check that a provider works), Jev's `config` branch loads this whole skill into that turn. The agent edits only the active config file (and `models.json` for a custom provider), verifies with `autorag health --json` and `autorag models list --available`, and reports each change as old → new in its final message. It never prints a credential value and never uses `init --force`. ### Retrieval and ingest caps `limits` bounds retrieval, baseline prefetch, and the candidate lists handed to the model. Every field is optional — an omitted field keeps the shipped default, so add the section only to tighten or widen a specific cap. Values must be positive integers; unknown keys (and unknown `prefetch` keys) fail config resolution. | Field | Default | Controls | |---|---|---| | `mergedEvidenceCeiling` | 500 | `search_all_documents` / model-free merge ceiling when the model omits `topK` | | `singleDatasourceTopK` | 20 | `search_datasource_*` merge default when the model omits `topK` | | `minSyncTopK` | 50 | MinSync semantic default `topK` | | `minSyncScopedQueryTopK` | 100 | MinSync fetch cap applied when a scope narrows the query | | `toolDescriptionInstanceScopes` | 8 | Instance scopes listed in one datasource tool description | | `prefetch.jikjiTopK` | 10 (30 with a rerank model) | Jikji find candidate count | | `prefetch.minSyncTopK` | 10 (100 with a rerank model) | MinSync retrieve candidate count | | `prefetch.datasourceTopK` | 10 (20 with a rerank model) | Hits requested from each selected datasource | | `prefetch.jikjiPathLimit` | 100 | Max Jikji answer paths rendered into the baseline | | `prefetch.sectionLimit` | 100 | Max results rendered per baseline section | ```json { "limits": { "mergedEvidenceCeiling": 1000, "prefetch": { "jikjiTopK": 12, "sectionLimit": 30 } } } ``` The baseline renders each prefetched result's full chunk content (not a per-result excerpt) so the model can see where the hit came from. Bound baseline size with `prefetch.sectionLimit` / `prefetch.minSyncTopK` / `prefetch.datasourceTopK` / `prefetch.jikjiPathLimit`, not with a per-result content truncation. The `prefetch.*TopK` defaults depend on reranking. Reranking is opt-in and runs only when `rerank.model` names a model; then the pool is wide (Jikji 30, MinSync 100, each datasource 20) and the reranker picks the best. Without it (the default) each source contributes its own top 10 to the first answer. An explicit value always wins. ### Record cut (`limits.recordCut`) On by default. A search that keeps only the top N *chunks* lets one long document whose dozens of passages all match fill every slot while a document only the keyword leg found falls past the cut. The record cut spreads the top N across documents instead. Turn it off with `limits.recordCut.enabled: false`. When on: - MinSync and the merged search ask for `depth` times the requested count (3x, never past 300 hits), then keep the requested count spread across documents: the best half with at most `max(perRecordFloor, topK / perRecordDivisor)` chunks per document (8 at `topK` 50), then the best chunk of each document not shown yet (`newRecordShare`, default 0.5 of `topK`), then rank order. - A chunk a search tool already showed during the same run is listed by id without its text (`already shown earlier in this run`). A new run resends it. - Each search tool result is held to `resultChars` (60000). The budget is split max-min fairly across the chunks, so one huge chunk is cut with a note instead of starving the rest; chunks whose share would fall under `minReadChars` (2000) are listed as `not read` with their ids. | Field | Default | Controls | |---|---|---| | `enabled` | true | Set `false` to turn the record cut off | | `depth` | 3 | Fusion depth, 1-10 | | `newRecordShare` | 0.5 | Share of `topK` reserved for documents not shown yet, 0-1 | | `perRecordFloor` | 4 | Fewest chunks one document may keep in the head, 1-300 | | `perRecordDivisor` | 6 | One document keeps at most `topK / perRecordDivisor` in the head, 1-300 | | `resultChars` | 60000 | Characters one search tool result may hold | | `minReadChars` | 2000 | Smallest share worth reading | ```json { "limits": { "recordCut": { "enabled": false } } } ``` Measured on the maintainer's own corpus (297 sampled passages, a natural question and a keyword query each, file-level gold, hybrid mode, top 10): nDCG@10 0.646 → 0.656, MRR@10 0.588 → 0.600, Recall@10 0.825 → 0.830, and the top 10 holds 8.8 distinct files instead of 7.5. The Recall difference is within noise. Ingest caps are trusted connector options under `datasources..connector` and are never settable from model/tool arguments: `maxDocuments`, `maxItemsPerFeed`, and `maxContentChars` (RSS). `maxContentChars` defaults to 20000 for RSS. ### Refresh safety limits `refresh` guards the parsed-mirror refresh against deleting a corpus because a search root became unavailable. Every field is optional; unknown keys fail config resolution. | Field | Default | Controls | |---|---|---| | `maxDeleteRatio` | 0.3 | Largest share (0–1) of indexed documents one refresh may delete; a bigger deletion above 100 documents is suppressed (`mass-delete-suppressed`) unless the refresh is forced | | `maxScannedEntries` | 2000000 | Directory entries examined per search root before `walk-truncated` | | `maxDepth` | 64 | Deepest directory level walked below a root before `walk-truncated` | A root that is missing, unlistable, or empty while it previously held documents keeps its mirrors and reports `search-root-unavailable`; see `docs/indexing/embedding-backends.md`. ## Probe and configure datasource skills (setup wizard) Use `autorag setup` to probe the embedding backend, MinSync, and known datasources automatically: ```bash autorag setup --format json ``` A datasource setup UI is not shipped in this build — do not recommend it for datasource setup. Configure datasources directly in trusted config, wizard-style: 1. Probe every datasource for setup feasibility before asking the user anything: the backing CLI exists (`lazykatok`, `discrawl`, `slacrawl`, `wacrawl`, `telecrawl`, `notcrawl`, `mailcrawl`, `wombat`, `lark-cli`) and its local store or archive is present. CLI-backed datasources own their own archive, index, and authentication, so environment credentials (such as bot tokens) are never required or checked for them. Non-CLI connectors (such as `github`) require their credential environment variable (`GITHUB_TOKEN`). 2. Auto-configure every datasource that probes feasible — write its trusted `datasources` entries without asking. For example, when Slack (`slacrawl`) and Discord (`discrawl`) are installed with local stores present, set both up automatically. Discord uses discrawl's local desktop wiretap archive; no Discord bot token is configured or needed. 3. Skip every datasource that probes infeasible (for example Notion or Telegram when their CLIs or native stores are not present) and always report the skipped list to the user, with what is missing for each. 4. Set up a skipped datasource only when the user explicitly asks for it: install or authenticate the backing CLI first, then configure it. 5. E-mail (`mailcrawl`) matters to most users — always probe it and report its status, even when it ends up skipped. Datasource skills belong in trusted config. Builtin template names are `kakao`, `whatsapp`, `telegram`, `slack`, `discord`, `clawgallery`, `notion`, `github`, `github-gist`, `mailcrawl`, `obsidian`, `rss`, `lark`, `hister`, and `wombat`. Config keys may be connection aliases with `"type": "