__`, so `DONSETCH_FETCH__PDF_MAX_MB=25`. Sections: `transport`, `mcp`, `paths`, `state`, `proxy`, `tls`, `persona`, `cli`, `fetch`, `bypass`, `search`, `browser`, `debug`, `rules`. ```toml [fetch] h3 = true # opt into the h3 lane shadow_fetch = "never" # or "auto" / "always" pdf_max_mb = 100 dns_cache_ttl_secs = 30 # seconds a resolved name is reused in-process, 0 disables [bypass] max_daily = 100 ``` Booleans take `true`/`false`, enums take their listed spellings. - `donsetch config show` prints every knob with its value **and its origin** (default, legacy, file, env). - `donsetch config show --markdown` prints the full reference table, one row per knob. - `donsetch config show --legacy` maps every old env name to its config key. - `donsetch doctor` warns about the legacy vars active in your shell, mapped to their keys. ### Local rules: deny a site, or pin its tier Some sites treat any automated traffic as hostile, however well it is disguised. The block lands on your IP, sometimes on your whole network, outlasts the session that caused it, and also locks you out in your own browser. A note in the agent's instructions file ("never fetch site X") helps only if the model reads it and obeys it every time, and one forgotten turn is enough to get banned. Other sites need a particular tier every time, and the agent has no way to know that in advance. [Pick the right tool](#-pick-the-right-tool) covers which task shapes draw these blocks. Local rules enforce your answer inside DonSeTch, whatever the model decides. Per-host rules in `donsetch.toml` can refuse a URL before any request, handing the agent your own instructions in place of the page, or pin the fetch tier for a host. DonSeTch ships none; the table is empty until you write one. ```toml [rules.url."example.org"] # the host and every subdomain, http and https action = "deny" reason = "ip_ban" # error code policy.denied.ip_ban message = "do not fetch this site with curl, scripts or another HTTP client: it bans our IP for automated access. Find file elsewhere, or ask the human operator to download it" ``` `web_fetch`, `web_crawl` and `web_screenshot` then refuse that site, redirects and cached copies included, and the agent reads `message` as its `Next action:`, so lead with what to do instead, and say why when the reason rules out workarounds. The most specific matching rule wins in full: a narrower rule that only sets `tier` also cancels a broader deny. `donsetch rules test ` shows which rule a URL gets and why, offline. Rules are a guardrail, not a security boundary: no MCP argument can switch them off, but an agent with a shell can run `donsetch fetch --ignore-rules`, set `DONSETCH_RULES__MODE=off`, or edit `donsetch.toml`. Pattern grammar, load errors, precedence, tier pins, what the agent sees and the known limits: [local rules](docs/rules.md). ## 💻 CLI Thin adapter over the same engine the MCP server uses. | Command | What it does | |---|---| | `donsetch fetch ` | Fetch as clean markdown (`--focus`, `--max-chars`, `--json`) | | `donsetch search ` | Search, keyless + BYOK (`--intent`, `--max-results`) | | `donsetch crawl ` | Crawl (`--mode map\|full\|content`, `--topic`, `--max-pages`) | | `donsetch screenshot ` | Render to PNG (`--out`) | | `donsetch mcp` | MCP server (stdio, or `--http --port N`) | | `donsetch doctor` | Health check and auto-fix (`--deep`, `--json`, `--fix`) | | `donsetch status` | Version, keys, proxies, cache, health overview (`--json` for scripts and agents) | | `donsetch keys` | BYOK providers and plugins (`add`, `list`, `default`, `export`) | | `donsetch proxy` | Proxy management (`add`, `list`, `check`, `test`, `import`, `export`, `remove`, `clear`, `fetch on\|off`, `crawl on\|off`) | | `donsetch login` | Authenticated sessions for walled sites (`--list`, `--status`, `--logout`, `--import`) | | `donsetch config` | `show`, `--markdown`, `--legacy` | | `donsetch rules test ` | Explain what the local rules do with a URL, offline | | `donsetch tools` | Tool schemas as JSON, same as MCP `tools/list` | | `donsetch update` / `rollback` | Self-update from GitHub Releases, and revert | Every management command validates its flags: an unknown one is refused with the nearest known match and exits 1. A usage error is never a silent run. The `proxy` help and its empty-pool state carry one labelled sponsor line (Proxy-Seller); `DONSETCH_NO_SPONSOR=1` removes it, and it never appears in `--json`, `proxy export`, or any MCP result or tool definition. ## 🔀 Proxies Adding a proxy is one command. Nothing else is required. ```bash donsetch proxy add socks5://user:pass@1.2.3.4:1080 # validated, probed, then saved donsetch proxy list # what is configured (credentials masked) donsetch proxy check # live probe: latency + real exit IP ``` - **Formats**: `http://host:port` (HTTP CONNECT, credentials optional as `user:pass@`), `socks5://host:port` and its `socks5h://` alias (SOCKS5 sends the target as a name, so the proxy resolves DNS and your resolver never sees the host). A bare `host:port` is HTTP CONNECT. Anything else is refused with the reason. - **Many at once**: `donsetch proxy add url1 url2 url3`, or a whole file with `donsetch proxy import proxies.txt` (one URL per line, `#` comments) — `donsetch proxy export` writes that file back out. `--no-check` skips the probe when you are adding hundreds. - **Try before you add**: `donsetch proxy test ` probes one proxy without saving it. - **Where they live**: `donsetch proxy add` appends `/proxies.txt` (mode 0600, credentials included). `DONSEEK_PROXIES="url1,url2"` in the environment and `[proxy] pool = ["url1", "url2"]` in `donsetch.toml` are the other two doors; all three merge, and an environment/config entry replaces a file entry with the same `host:port`. The pool is read at startup, so restart the MCP server to pick up a change. - **Removing**: `donsetch proxy remove 1` (the index `proxy list` prints, or `host:port`) and `donsetch proxy clear`. - **Checking is a tunnel, not a TCP handshake**: the probe connects through the proxy to `api.ipify.org` and prints the exit IP, so a proxy that accepts the connection and then cannot carry traffic is reported dead. `donsetch status` shows the count, `donsetch doctor` shows every lane and its health. ### What uses the pool, and what does not A proxy you configure is a proxy that sees your traffic, so nothing uses one by accident. | Tool | Default | Off switch | |---|---|---| | `web_search` | always rotates the pool: one lane per engine, sticky per host, so a fan-out search does not hit nine engines from one IP | — | | `web_crawl` | each host starts on the direct lane; its later requests spread across the pool lanes as their pacing allows, and a host that blocks or throttles direct moves off it | `donsetch proxy crawl off` | | `web_fetch` | your own IP | `donsetch proxy fetch on` | `donsetch proxy fetch on` makes `web_fetch` (and its browser escalation) sticky to a pool lane per host, rotating on 429/407/dead/timeout. A one-shot read does not need a shared exit, which is why fetch is opt-in while search and crawl are not. Crawl is where the first request decides things: a host that bans on contact has already seen your own IP before a lane can take over. When a site must never see your address at all, keep it off entirely with a **[local rule](#local-rules-deny-a-site-or-pin-its-tier)** rather than leaning on the pool. ### Riding your own proxy network Set the environment and every fetch follows the curl convention: - `HTTPS_PROXY` / `HTTP_PROXY` / `ALL_PROXY` (any case) route fetches through that proxy, and `NO_PROXY` exempts hosts (suffix match, `*`, `host:port`, CIDR). `DONSETCH_NO_ENV_PROXY=1` disables the convention entirely. `[proxy] http` / `https` / `all` in `donsetch.toml` do the same thing without environment variables. - A configured proxy is a promise: if it cannot be dialled the fetch fails and says so, instead of quietly leaving on your own IP. Same for a pool lane: a dead lane is benched, not replaced by a direct read of a host it was carrying. - In an intercepting network (a TLS-terminating middlebox), HTTP CONNECT gets an interception-safe handshake (no GREASE/ALPS/ECH/compress-cert, which some middleboxes reset on) and `SSL_CERT_FILE` / `SSL_CERT_DIR` load into the trust store, exactly like curl. SOCKS5 keeps the Chrome-true handshake with TLS end to end. `donsetch doctor` reports the whole posture and names the interception fix when a live check fails. - Authenticated lanes work in the browser tier too: Chrome cannot carry proxy credentials, so it is pointed at a local credential-free relay that authenticates upstream, instead of dying as `ERR_SOCKS_CONNECTION_FAILED`. ## 🔐 Logged-in sessions Pages behind a login (x.com, gated docs, internal tools) need a real session. `donsetch login` gives you one without ever seeing your credentials: ```bash donsetch login x.com # opens YOUR browser, you sign in, press Enter donsetch login --list # stored sessions (names and counts only) donsetch login --status x.com # one domain, in detail donsetch login --logout x.com # forget a domain donsetch login --import cookies.txt x.com # servers and CI, Netscape format ``` - Credentials never enter DonSeTch. It opens a real Chromium on your display in a dedicated profile, never the automation one. You type into the browser. No keystroke capture, no screenshots, no CDP attach until you press Enter. - Afterwards, cookies selected by your domain or explicit import are stored regardless of whether their names look like authentication tokens. Known analytics and clearance cookies, expired records and invalid values are excluded. Names and paths retain their identity, including opaque persistent names such as `c_user`, `xs` and `li_at`. Cookie values stay in the private vault; `Saved`/`Imported` lists names only. Tier-1 fetches and warm tier-2 browsers reload the session without a daemon restart. Login changes invalidate cached account documents; captures from older browsers cannot overwrite the replacement session. - A named-domain post-login probe checks reachability and reports login/access redirects as unverified. A public home-page HTTP 200 cannot prove authentication, so it never earns a verified checkmark. Multi-site saves avoid one probe per domain. Verify the actual protected page you need with `web_fetch`. - Failed writes and captures return an error instead of reporting success. EOF and Ctrl+C save nothing to the vault; `--fresh` waits for the exclusive profile lock before wiping it. The dedicated browser profile may retain its own cookies until cleared with `--fresh`. - The registry (`auth-state.json`) stores metadata only: names, counts, expiries, probe verdicts. Never values. - Multi-site: run bare `donsetch login`, sign into as many tabs as you like, press Enter once. New cookies and rotated values are captured, including opaque persistent cookies. ## 🐳 Docker ```bash docker build -t donsetch-mcp . docker compose up -d # loopback-only by default ``` Multi-stage build, non-root user, optional Chrome, resource limits in the compose file. An opt-in `http` profile serves the HTTP transport with a healthcheck, and `docker compose stop` gives in-flight tier-2 fetches a 45s grace period. ## 🧭 Pick the right tool DonSeTch is a **rapid-fire research tool**: search, read, verify. A search, a fetch or two, a docs page, one PDF. Its speed is the point, and that speed is its stealth for one-shot reads. It is NOT built for tasks where an agent "works through" a defended site the way a person would: - **Bulk document harvesting**: discovering and downloading many PDFs from one repository in a single run. - **Long sessions against one site**: page after page, at machine speed, same IP, no human pauses. - **Mass extraction**: mirroring a file library, collecting a dataset, systematic downloads. DonSeTch will *probably* work on those, and nothing stops it. But every request fires hundreds of times faster than a human, and a defended site reads that pattern itself as a bot, not just the fingerprint. The realistic risk is an IP-level block, sometimes on your whole network mid-run. When that happens it is the task shape, not a fetch-layer failure: the same page fetched once, as a research read, is fine. For that other shape of work, use **[Bladebro](https://github.com/dondai44423/bladebro)**: a real browser doing what a human does, page by page, download by download, at a human's pace. Slower than DonSeTch by design, because over a long session against a defended site, looking human beats being fast. **Rule of thumb: one-shot research = DonSeTch. Working a defended site like a person to collect things = Bladebro.** ## ⚠️ Gotchas & honest limits | Surprise | Why | |---|---| | First build ~2 min | BoringSSL compiles from source, cached after. Go is a build dependency too, BoringSSL's build system is Go-based. | | OCR and rerank are not in the default build | ONNX Runtime is heavy and optional: `--features ocr,rerank`. Prebuilts ship them on linux-x64, macOS-arm64, macOS-x64 and Windows-x64. | | First OCR/rerank use downloads models | ~24MB reranker, ~37MB OCR, cached forever. | | Captchas need an unlocker key | hCaptcha, reCAPTCHA and Turnstile cannot be solved locally, by design. With `donsetch keys add unlocker [::zone]` they come through rendered; without one you get a clear honest error, never a hang. | | robots.txt is ON for crawl | `respect_robots=true` for crawl. `fetch` does not check robots. | | Keyless search rate-limits without a proxy | It hits engines from your IP. Set `DONSEEK_PROXIES` for heavy use. | | Rerank in a CPU-limited container | Auto-clamped to cgroup parallelism on Linux, `DONSEEK_RERANK_THREADS` to override. | | It cannot | Why | |---|---| | Solve interactive captchas locally | hCaptcha, reCAPTCHA, Turnstile: an honest dead end, no solving service by design. | | Search with every engine down | An error with per-engine status. Honest, never fake. | | Replace a human working a defended site | See "Pick the right tool" above. | ## 🆚 How it compares The honest summary: nothing else in this space combines a real Chrome TLS stack, browser-free keyless search, PDF pixel-fusion and local-only operation in one binary. | | **DonSeTch** | Hound | Crawl4AI | Jina Reader | Firecrawl | |---|---|---|---|---|---| | Language | **Rust, one binary** | Python | Python | Python (API) | TypeScript | | TLS fingerprint | **Chrome's own BoringSSL** | curl-impersonate | requests | their servers | their servers | | Own HTTP/2 + temporal stealth | **yes** | no | no | no | no | | Browser tier | **solve-and-bounce** | browser fetches all | browser fetches all | n/a | n/a | | Search | **keyless, 10+ engines, local rerank** | keyed | no | yes | no | | Crawl | **yes, resume tokens** | yes | yes | no | cloud only | | PDF + OCR | **pixel-fusion + PP-OCR** | yes | partial | yes | cloud, paid | | Self-improving routing | **yes, local state** | no | no | no | no | | Runs locally, no account | **yes** | yes | yes | no | self-host or paid | | MCP server | **first-class** | yes | community | yes | build it | | Tool schema | **~2.4k tokens** | ~2.7k | varies | n/a | varies | | License | AGPL v3 | MIT | Apache 2.0 | proprietary | MIT | **Against Firecrawl, head to head, same tasks, live.** Firecrawl is a paid cloud API, DonSeTch is free, local and keyless: - **Fetch**: Wikipedia comes back 16x smaller (16KB against 267KB). An arXiv PDF is 22x faster and 4.4x smaller (1.4s/16KB against 32.6s/71KB). Reddit: Firecrawl refuses the site, DonSeTch returns the real feed. - **Crawl**: 2.5x faster on the same docs target (18.7s against 47.6s), with `--topic` ranking the frontier instead of firehosing, and an honest small failure when the topic has no match instead of verbose unrelated content. - **Search**: Firecrawl is genuinely faster (1-2s against 5-7s) and leans mainstream authority. DonSeTch leans technical specificity, matched or beat it on exact GitHub issues, and costs far fewer tokens per query. ## 🤝 Contributing PRs welcome, see [CONTRIBUTING.md](CONTRIBUTING.md). Before submitting: `just check`, `just t ` for the area you touched, and `just lint` (clippy with `-Dwarnings`). CI runs the full matrix on three platforms. AGPL v3: all contributions land under the same license. ## 📄 License Copyright (c) 2026 Bishesh Bhandari. AGPL-3.0, see [LICENSE](LICENSE). ---

### If DonSeTch saves you time, ⭐ the repo [![Stars](https://img.shields.io/github/stars/dondai44423/donsetch?color=ff9f43&style=flat-square)](https://github.com/dondai44423/donsetch) **AGPL v3** · [Changelog](CHANGELOG.md) · [Issues](https://github.com/dondai44423/donsetch/issues) · [Releases](https://github.com/dondai44423/donsetch/releases)