---
name: ketch
description: "Research skill for ketch — a fast stateless CLI for web search, OSS code search, curated library docs, page scraping, and site crawling; an optional MCP server exists for operators who want it, but the CLI is the primary interface. Use when a question needs live sources: 'research X', 'what are people saying about Y', 'find docs or real-world examples for Z', 'scrape/crawl this site' — or when installing or configuring ketch backends. Routes search vs code vs docs vs scrape vs crawl, keeps every fetch inside a token budget, turns error prefixes into control flow, and produces cited syntheses. Not for local codebase search, private repos, or pages behind auth."
version: 0.1.0
---

# Ketch

Route every live-source question to one of ketch's five research surfaces — search, code, docs, scrape, crawl — over the transport the operator gave you, with a token budget on every fetch and a source URL on every claim. ketch is one stateless binary — call, result, exit — with web search, OSS code grep, curated library docs, and page/site extraction together, so a complete research pipeline needs no other tool and no daemon.

## Transport: stateless CLI by default, MCP when the operator wired it

The CLI is ketch's identity: call → result → exit, `--json` on every call, exit codes as control flow, zero daemon. That is the default transport and the zero-infrastructure path. The MCP server is a supported alternative for operators who want it — never a prerequisite.

Decide once per session, before the first call:

1. `which ketch` succeeds → the CLI is your transport: `--json` on every call, exit codes as control flow.
2. Also check for ketch's MCP tools in your tool list — `search`, `code`, `docs`, `scrape`, `crawl` and `tag` from a server named `ketch` (in Claude Code: `mcp__ketch__search`, …). Present → the operator wired them up on purpose, and using them for research calls is correct and good: structured output, per-URL errors, no shell round-trip. Do not shell out around tools the operator set up.
3. Both live → either transport serves research calls, but know the tradeoff: a running MCP server holds the single-process page-cache lock, so concurrent CLI scrapes silently run cache-disabled.
4. Neither CLI nor MCP tools → ketch is not installed. Offer `brew install ketch` or `go install github.com/1broseidon/ketch@latest` — an operator action: propose, wait for confirmation.

The rule: **use the transport the operator gave you** — when both are live, either is fine for research calls, and operator actions are always CLI.

`tag` is the one exception to the operator-action rule below: it changes local state but the agent is both writer and reader, so it is published over MCP as well as the CLI.

Config discovery is CLI regardless of transport: `ketch config` prints effective settings and available backends as JSON; there is no config tool over MCP. Operator actions — `config set`, `cache`, `browser install`, `crawl --background`/`status`/`stop`, `doctor` — are deliberately not in MCP. They are always CLI.

## Glossary

Use only these terms in ketch output.

| Term | Meaning |
| --- | --- |
| **surface** | One of the five research operations: `search`, `code`, `docs`, `scrape`, `crawl` |
| **tag** | A label filed across surfaces: search hits, code hits, docs chunks, scraped pages, crawled pages. Not a research surface — it answers "what did I already find?", not "what is out there?" |
| **transport** | How a surface is called: the CLI binary (default) or the optional MCP tools |
| **backend** | The provider behind a surface: auto (default; a fallback chain, not a provider)/brave/ddg/searxng/exa/firecrawl/keenable/tavily/parallel/serpbase/degoog/serply/youcom (search), grepapp/sourcegraph/github (code), context7 (docs) |
| **operator action** | A system-managing or diagnostic command — `config set`, `cache`, `browser install`, background crawls, `doctor` — CLI-only by design |
| **error prefix** | The stable class on every ketch error: CLI exit codes 2–6, mirrored as the bracketed prefix opening every MCP tool error — `[validation]`, `[not_found]`, `[upstream]`, `[precondition]`, `[cancelled]` |
| **fan-out** | How many queries are searched and URLs scraped under one plan |
| **token budget** | The per-call output bound: `max_chars`/`trim` on scrapes, `tokens` on docs, `limit`/`--minimal` on lists |
| **probe** | One cheap read-only call that tests whether a surface is configured and reachable |

## How to use this skill

- Default: answer one question with one or two routed calls. Use the surface routing table, token budgets, and error control flow below.
- `ketch research <question>`: deep multi-source research — search fan-out → scrape top hits → optional code/docs corroboration → synthesized, cited answer. Read `references/verbs/ketch-research.md`.
- `ketch setup`: configure backends with the operator — probe current state, propose exact commands, mutate only on confirmation. Read `references/verbs/setup.md`. Enter this verb whenever any call returns `[precondition]` / exit 5.

One question = one plan. Escalate a default run into `ketch research` when the first search shows the answer is contested, multi-part, or needs corroboration.

## Non-negotiable disciplines

1. **Use the transport the operator gave you.** The CLI is the default; MCP tools in your list mean the operator opted in — use them for research rather than shelling out around them. When both are live, either serves research calls (a running MCP server holds the page-cache lock, so concurrent CLI scrapes run uncached); operator actions — config, cache, browser, background crawls, doctor — are always CLI.
2. **Bound every fetch.** `max_chars` 4000–8000 plus `trim` on any scrape of a page you have not seen — an unguarded page can cost ~25k tokens. Skipping the cap requires a stated one-line reason ("known ~200-word page").
3. **Cite every claim.** A research synthesis without source URLs is not a deliverable.
4. **Error prefixes are control flow.** Classify before reacting. Never retry `[validation]` or `[not_found]` unchanged.
5. **Propose, then mutate.** `config set`, `browser install`, docker runs, installs — only after the operator confirms the exact command. Never touch a value that is already configured and working.
6. **The binary outranks this file.** `ketch config` and `--help` are ground truth; where they disagree with a table here, trust the binary and flag the skill as drifted.

## Gold decision trace

```text
Request: "ketch research — do people actually use Go's iter.Seq in real projects, and what are the gotchas?"

Transport: operator wired mcp__ketch__* into this session → honor it; research calls go over MCP.
Plan: 2 queries · scrape top 3 · max_chars 6000 + trim · ≤8 calls

search {query: "Go iter.Seq real-world experience gotchas", limit: 5}
  → "[upstream] ddg rate limited" → an explicit backend failed; rotate to another
    usable provider from available_backends, retry once:
    search {query: ..., backend: "brave", limit: 5} → ok
search {query: "Go range-over-func adoption production", backend: "brave", limit: 5}
  → 10 results, 8 unique hosts → picked 3: official blog post, one experience
    report, one issue thread (primary sources over aggregators)

scrape {urls: [u1, u2, u3], max_chars: 6000, trim: true}
  → isError=false; checked results[] one by one: u1, u2 ok;
    u3.error = "[upstream] … 503" → dropped, will be named in synthesis

code {query: "iter.Seq", lang: "go", limit: 3}      # corroborate real usage
  → 3 repos with file/line URLs

Synthesis: five claims, each cited to its URL; u3 listed as unretrieved;
one conflict between u1 and u2 stated and attributed, not averaged.
Budget: 5 of 8 calls (the rate-limited attempt counts).
```

## Surface routing

First match wins:

| The question needs | Surface | Not |
| --- | --- | --- |
| Current web pages, opinions, news, comparisons | `search` | `docs` — that is curated library docs only |
| How real projects call an API | `code` | `search` — blogs talk *about* code; `code` greps public OSS repos via grep.app |
| A library's own documentation, version-aware | `docs` | `scrape` of the docs site — `docs` is already extracted and token-budgeted |
| The content of a URL you already hold | `scrape` | `search` — never re-find a known URL |
| Many pages from one site | `crawl` | looped `scrape` — crawl dedupes, bounds, and streams |
| Anything you already found earlier in this project | `tag` (`operation: show`) | `search` again — you already paid for these once |

In reverse: `search` finds URLs; `scrape` reads them; `crawl` reads a site; `code` reads public source; `docs` reads library docs. `search` with `scrape: true` fuses the first two when you will want full content from every hit — budget it like a scrape.

### Keeping a working set with `tag`

On work that spans more than one session or more than a handful of sources,
pass `--tag <name>` (CLI) or `tag` (MCP) on the calls you will want again,
naming the project or scope of work — `remote-access`, not `results-3`.

It works on every surface, and one tag holds them all: `search`, `code`,
`docs`, `scrape`, `crawl`. So a project tag ends up holding the vendor's
documentation, the code that calls it, and the write-up that explained the
undocumented flag, in one list. Later, `tag show` returns that as an
llms.txt-shaped index — titles, URLs, descriptions — limited to the newest 50,
and you re-read one source instead of re-running the searches that found them.
Tag the sources you actually used, not every hit.

Use `--limit N` / MCP `limit` for a smaller view; `0` explicitly requests all.
`entries` counts the whole tag and `shown` counts returned pages. Read the index
before searching again, but do not request all of a large tag by default.

The index is durable and outlives the cached page bodies, so it still answers
days later. `cached: false` means no fresh body was confirmed, not a dead link.
When `cache_status` is `unavailable`, the page cache could not be checked;
bookmarks still work and sources can still be fetched. Entries from `code`, `docs` and
unscraped `search` hits start that way by nature, since those calls return
snippets rather than fetched pages; `scrape` the URL when you want the whole
thing. Read the index first, then fetch only what you need: assembling the
whole tag defeats the point.

Check bookmark diagnostics separately from research success: CLI warnings go to
stderr (structured under `--json`), and MCP returns `warnings`. A failed write
does not discard the useful research result. Durable bookmarks use a separate
file; test labs must set `KETCH_TAGS_PATH` as well as isolating the page cache.

## Token budgets

| Call | Bound with | Measured cost |
| --- | --- | --- |
| `search`, limit 5 | `limit` | ~1.4 KB |
| `code`, limit 3 | `limit` | ~0.7 KB |
| `docs`, default budget | `tokens` (default 4000) | ~3.3 KB |
| `scrape`, unknown page | `max_chars` 4000–8000 + `trim` | unguarded: up to ~100 KB (~25k tokens) |
| `crawl` (MCP) | `max_pages` + per-page `max_chars` | 30 pages default, 100 cap, 3-min wall clock |
| `tag show` / MCP `tag` show | `limit` (default 50; 0 all) | bounded source metadata; no page bodies |
| Any CLI list | `--minimal` | roughly halves output |

## Error control flow

| Exit (CLI) | Prefix (MCP) | Meaning | Do |
| --- | --- | --- | --- |
| 2 | `[validation]` | Bad input | Fix the call; retrying unchanged can never succeed |
| 3 | `[not_found]` | Nothing matched | Change the query or selector; not an outage |
| 4 | `[upstream]` | Backend or network failure | Explicit backend: rotate to another provider (`available_backends` in `ketch config`) or retry once. `auto` already fell through every usable provider: retry once, then report the outage |
| 5 | `[precondition]` | Operator config missing | Stop researching; enter `ketch setup` |
| 6 | `[cancelled]` | Cancelled or timed out | Rerun with smaller scope |

Situations → class: unknown backend, `regexp` on github → `[validation]`. Selector matched nothing, a `repo` the code backend does not have → `[not_found]` (for `repo`, change the backend). ddg rate limit (it rate-limits readily under fan-out), DNS failure, grepapp's intermittent 504 → `[upstream]`, rotate or retry once. Missing API key, docs backend `local` (planned, unimplemented), `force_browser` with no browser configured → `[precondition]`. One asymmetry: a CLI `crawl` interrupted by SIGINT exits **0** with partial results, by design.

## Gotchas

Detail for each lives in `references/surfaces.md`.

- Scraping a **bare domain** auto-probes `/llms.txt` and may silently return that instead of the homepage — the `title` field reveals the swap; `no_llms_txt` opts out.
- `docs` is a two-step: `resolve` the name → **vet the matches** → fetch by `library` ID. Resolve never returns empty — garbage in gets confident fuzzy matches out, so check the name, not just the trust score.
- Batch scrape reports per-URL failures inside a successful call: `isError=false` with `results[].error` set. Check every entry.
- `regexp` works on grepapp and sourcegraph only; github rejects it with a pointer to those backends.
- Background crawls (`--background`, `status`, `stop`) are CLI-only; the MCP `crawl` is synchronous and capped.
- The page cache (bbolt, 72h default TTL) is single-process: a long-running MCP server holds the lock, so concurrent CLI scrapes silently run cache-disabled — `ketch doctor` reports the cache as locked by another process. Running the server degrades the CLI; prefer CLI-only when both would run long-term.

## BAD/GOOD contrasts

**BAD:** `scrape {url: "https://docs.example.com"}` — no bound; you get llms.txt or ~25k tokens, whichever is worse.
**GOOD:** `scrape {url: "https://docs.example.com/quickstart", max_chars: 6000, trim: true}` — plus `no_llms_txt: true` when you want the page itself, not the site's llms.txt.

**BAD:** Telling a user they must run an MCP server to use ketch with agents — the CLI plus a prompt block is the zero-infrastructure path, and a long-running server holds the page-cache lock against every CLI call.
**GOOD:** CLI by default; MCP when the operator wired it — and when `mcp__ketch__*` tools are in your list, use them for research instead of shelling out around the operator's setup.

**BAD:** `[upstream] ddg rate limited` → retry the identical call three times.
**GOOD:** Rotate — `backend: "brave"` (or another provider from `available_backends`; `auto` is a chain, not a rotation target) — retry once, and note the swap. When `auto` itself failed, it already tried every usable provider: retry once, then report the outage.

**BAD:** Fetch docs from resolve's first match because its trust score is high, even though its name is not the library you asked about.
**GOOD:** Vet name + snippet count + trust; if no match names the intended library, say so instead of fetching junk docs.

## Reference loading

- `ketch research …` → read `references/verbs/ketch-research.md` before starting.
- `ketch setup`, any `[precondition]`/exit 5, or an install → read `references/verbs/setup.md`.
- Full flag/param tables, CLI↔MCP name mapping, backend/key matrix, or a surface behaving oddly → read `references/surfaces.md`.

## Scope

In scope: the five research surfaces over both transports, the research and setup verbs, token budgets, error-prefix control flow, backend configuration. Out of scope: local or private codebase search (use repo tools), pages behind auth or paywalls, bulk archival crawling beyond the caps, browser automation beyond ketch's headless-rendering fallback.

Bound every fetch; cite every claim.