# Install AgentCrawl For An Agent Client

This file is the canonical installation procedure for coding agents. When a user asks you to inspect this repository, install AgentCrawl, connect it to your tools, or make it the default scraper, execute the procedure. Do not only explain commands.

## Goal

Install AgentCrawl, verify direct scraping, register its standards-based stdio MCP server in your own client configuration, reload when required, and verify `scrape_url`.

## Safety Rules

- Never print API keys or copy secrets into chat.
- Do not overwrite unrelated MCP servers or client settings.
- Back up a configuration file before modifying it.
- Prefer the client own supported MCP command or structured configuration format.
- Do not install Playwright browsers unless the user needs JavaScript-rendered pages. HTTP extraction works without a browser runtime.

## 1. Install

Install from PyPI (the `mcp` extra is required for the MCP server):

```bash
python -m pip install "agentcrawl-ai[mcp]"
```

Add capabilities only when needed:

```bash
# JavaScript-rendered pages, browser_actions and screenshots
python -m pip install "agentcrawl-ai[mcp,browser]"
python -m playwright install chromium
# sites behind Cloudflare: Patchright retry for challenged pages
python -m pip install "agentcrawl-ai[stealth]"
python -m patchright install chromium
# local PDF ingestion
python -m pip install "agentcrawl-ai[docs]"
```

From a repository checkout use `python -m pip install -e ".[mcp]"` instead.

Inspect the installation:

```bash
agentcrawl doctor
```

`doctor` reports installed extras, Python/command discovery, local scrape health,
whether the browser fallback can actually start (with the fix when it cannot),
and optional remote API health when `AGENTCRAWL_BASE_URL` is set. It only reports
whether an API key is configured and never prints secret values.

## 2. Verify Direct Scraping

```bash
agentcrawl scrape https://docs.python.org/3/library/json.html
```

Success is a JSON document whose `markdown` contains `json.dumps` and whose
`metadata` has no `error_type`. When a site answers with a bot challenge,
AgentCrawl returns `error_type: "client_challenge"` with the signals it saw in
`metadata.challenge_signals` instead of returning the challenge page as content.

## 3. Register The MCP Server

For Claude Code, Codex, Gemini CLI, Cursor or VS Code, register it with one command, naming your own client (`claude_code`, `codex`, `gemini_cli`, `cursor`, `vscode`):

```bash
agentcrawl setup --client claude_code --yes
agentcrawl doctor --client claude_code   # "state": "aligned" means done
```

It goes through the client's own `mcp add` command (Claude Code, Codex, Gemini CLI) or writes its JSON file with a backup (Cursor, VS Code); other servers and settings are kept, and an existing `agentcrawl` entry keeps its `env`. A config file with comments is refused, never overwritten: add the printed line by hand. Run without `--yes` to only print the plan. For any other client, register it by hand as below.

Current stdio launcher:

```text
command: agentcrawl
args: ["mcp"]
```

Equivalent generic MCP configuration:

```json
{
  "mcpServers": {
    "agentcrawl": {
      "command": "agentcrawl",
      "args": ["mcp"]
    }
  }
}
```

Use your own client supported registration mechanism. Inspect its existing configuration or CLI help instead of guessing a path. Preserve all unrelated settings.

Without `AGENTCRAWL_BASE_URL`, the MCP server runs the local HTTP scraper directly and needs no separate API process. If AgentCrawl is a remote HTTP service, set environment variables on the MCP server process:

```json
{
  "mcpServers": {
    "agentcrawl": {
      "command": "agentcrawl",
      "args": ["mcp"],
      "env": {
        "AGENTCRAWL_BASE_URL": "https://agentcrawl.example.com",
        "AGENTCRAWL_API_KEY": "<secret>"
      }
    }
  }
}
```

Store real secrets using the client credential mechanism or protected environment files. Never commit them.

**Local files are off on the MCP.** An agent's tool input can be steered by the pages it reads, so the local MCP engine refuses file paths (`error_type: "local_files_disabled"`) and only fetches URLs. To let it read a document folder, opt in and confine it to that folder:

```json
"env": {
  "AGENTCRAWL_ALLOW_LOCAL_FILES": "true",
  "AGENTCRAWL_LOCAL_FILES_ROOT": "/home/me/docs"
}
```

Paths outside the root, including `..` and symlinks that escape it, return `error_type: "local_file_outside_root"`. Do not enable local files without a root.

## 4. Reload And Verify

Reload or restart your client if it does not hot-reload MCP configuration. By
default the server exposes the core tools an agent needs:

```text
scrape_url
scrape_many
map_site
crawl_site
extract_structured
```

`search_web` appears when `AGENTCRAWL_SEARCH_ENGINE` or `AGENTCRAWL_BASE_URL` is set, and `get_job` when
`AGENTCRAWL_BASE_URL` points at a server. Operator tools (`check_changes`,
`job_events`, `cancel_job`, `inspect_failures`, `retry_failures`, `usage`,
`cache_stats`, `clear_cache`) need `AGENTCRAWL_MCP_PROFILE=full` in the server's
environment; they are hidden by default because every tool schema costs context
on every turn.

Call `scrape_url` with:

```json
{
  "url": "https://docs.python.org/3/library/json.html",
  "formats": ["markdown", "metadata"]
}
```

Success requires clean Markdown containing `json.dumps`.

## 5. Normal Tool Selection

After registration:

- Use `scrape_url` for one known URL, `scrape_many` for several.
- For a long page, call `scrape_url` with `formats=["outline"]` first, then `section=<id>` for the part you need (and `max_tokens` to cap it); the page is downloaded once.
- On an error, follow `metadata.next_step` instead of retrying as is.
- Use `extract_structured` with `describe` when you have no schema (needs `AGENTCRAWL_LLM_MODEL`); keep the returned schema and pass it as `schema` next time.
- Use `search_web` when there is a question but no URL (needs `AGENTCRAWL_SEARCH_ENGINE`).
- Use `map_site` to discover site URLs without scraping all pages.
- Use `crawl_site` for bounded multi-page extraction; pass `query` to read the most relevant pages first and stop when pages stop matching.
- With the full profile, use `check_changes` to see whether a page changed since you last read it (pass the previous Markdown or its `markdown_sha256`, and the `etag`/`last_modified` if you have them).
- Use `scrape_url` with `browser_actions` only when content appears after a click, typing or scrolling, and add `"screenshot"` to `formats` when you need to see the page.
- For asynchronous crawl jobs, provide a stable idempotency key, keep the returned `job_id`, and poll `get_job`; do not start duplicates.
- A queued job with a future `available_at` is waiting for persisted backoff, not stuck.
- Read large completed crawls page by page with `offset` and `limit` until `has_more` is false.
- Use browser automation only for interactive actions, not ordinary page reading.
- Use another extractor only after AgentCrawl returns a definitive error.

## Operator Backup

Before deploying over an existing server database, run:

```bash
agentcrawl backup --db /path/to/agentcrawl.db --output-dir /path/to/backups
```

Use `--env-file /path/to/agentcrawl.env` only when an environment file should be
copied into the backup directory. Do not print or paste its contents.

Restore only with the service stopped and only from a verified backup:

```bash
agentcrawl restore --backup-db /path/to/backup.db --db /path/to/agentcrawl.db --force
```

## Completion Report

Report only:

- installation method and version;
- whether direct scraping succeeded;
- whether MCP registration succeeded;
- number of AgentCrawl tools discovered;
- whether the functional `scrape_url` test succeeded.

Do not include credentials.