█████╗ ██████╗██████╗ █████╗ ██╗ ██╗██╗
██╔══██╗██╔════╝██╔══██╗██╔══██╗██║ ██║██║
███████║██║ ██████╔╝███████║██║ █╗ ██║██║
██╔══██║██║ ██╔══██╗██╔══██║██║███╗██║██║
██║ ██║╚██████╗██║ ██║██║ ██║╚███╔███╔╝███████╗
╚═╝ ╚═╝ ╚═════╝╚═╝ ╚═╝╚═╝ ╚═╝ ╚══╝╚══╝ ╚══════╝
Browser automation agent that acts and observes. Navigate, click, and extract — or inspect network traffic, debug console errors, profile performance, and audit accessibility. One Rust binary.
Single Rust binary. Full DevTools observability. 43 tools. 25 LLM providers. MCP server built-in.
---
## Why acrawl?
Most browser agents stop at navigate and click. acrawl goes further: the agent can also inspect network requests, analyze console errors, measure page performance, audit accessibility, and intercept network calls — the full DevTools surface, available as first-class agent tools.
It ships as a single Rust binary. No Python runtime, no Node runtime, no Docker. Drop it into any server or CI pipeline and describe a goal; the agent figures out what to visit, what to click, what to inspect, and when it's done.
- **One binary, zero runtimes.** `cargo build --release` produces a self-contained executable. No Python, no Node runtime — just Rust and a Chromium download for browser automation.
- **Acts and observes.** The agent has the full DevTools surface as first-class tools: inspect network requests with timing, analyze and deduplicate console logs, stream WebSocket messages, measure page performance (TTFB, resource breakdown), audit cookies and browser storage, measure JS/CSS coverage, run axe-core WCAG accessibility audits, and intercept or mock network calls. No other agent framework exposes this.
- **Deterministic where you can, AI where you must.** Define loops, conditionals, and parallel branches as JSON scripts — executed without any LLM calls. Fall back to the agent when pages behave unexpectedly. Best of both worlds.
- **No code required.** Describe the goal in plain English. The agent plans, navigates, and extracts.
- **Smart fetching.** Static pages are served over HTTP (fast). When JavaScript or interaction is needed, acrawl detects JS framework markers (`__next_data__`, `__nuxt`, `__vue`, `ng-app`, React roots), auth redirects, and short `` bodies — then transparently escalates to a headless browser.
- **Sub-agent parallelism.** Fork child agents onto separate browser tabs with independent state and step budgets. A URL-claiming registry prevents siblings from crawling the same page twice.
- **MCP client and server.** Extend the agent with custom tools via [Model Context Protocol](https://modelcontextprotocol.io) servers. Or flip it: `acrawl mcp` exposes 38 browser and DevTools tools plus `run_goal` to Claude Code, Cursor, VS Code, Zed, and 13 other clients.
- **25 LLM providers.** Anthropic, OpenAI, Google Gemini, DeepSeek, AWS Bedrock, Azure OpenAI, Vertex AI, GitHub Copilot, Groq, Mistral, xAI, Cohere, Alibaba DashScope, OpenRouter, and more. Or bring your own via any OpenAI-compatible endpoint.
### How does it compare?
#### vs. AI web agents and scraping tools
| | acrawl | browser-use | Stagehand | Skyvern | Firecrawl | Playwright MCP | Scrapy | Playwright scripts |
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| No code needed | Yes | No | No | Partial | No | No | No | No |
| Single binary | Yes | No | No | No | No | No | No | No |
| JS rendering | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes |
| LLM-powered navigation | Yes | Yes | Yes | Yes | Limited | No | No | No |
| No Python / Node needed | Yes | No | No | No | No | No | No | No |
| Form filling / interaction | Yes | Yes | Yes | Yes | No | Yes | No | Yes |
| Sub-agent parallelism | Yes | No | No | Partial | Partial | No | Partial | No |
| 25 LLM providers | Yes | Via LiteLLM | Partial | Partial | N/A | N/A | N/A | N/A |
| MCP client (use tools) | Yes | No | No | No | No | No | No | No |
| MCP server (expose as tools) | Yes | No | No | No | Yes | Yes | No | No |
| Stealth browser built-in | Yes | Cloud only | Via Browserbase | Cloud only | No | No | No | No |
| DevTools observability (network, console, perf, a11y) | Yes | No | No | No | No | No | No | No |
| Deterministic script layer (zero LLM calls) | Yes | No | Partial | No | No | No | Yes | Yes |
| Open source | Yes | Yes (MIT) | Yes (MIT) | Yes (Apache) | Engine only | Yes (MIT) | Yes (BSD) | Yes (Apache) |
Notes:
- **browser-use** (85k+ GitHub stars): Python + Playwright, DOM + screenshots, supports GPT/Claude/Gemini/Ollama via LiteLLM, 89.1% WebVoyager. No single binary — requires Python and `pip install`. Every action calls an LLM: 2-5s/step, ~$0.02-0.30/task. Cloud tier adds stealth; self-hosted is bare Playwright.
- **Stagehand** (Browserbase, 21k+ stars): TypeScript + CDP (v3), mixes deterministic Playwright with AI primitives (`act()`, `extract()`, `observe()`). Action caching reuses successful clicks without re-calling the LLM. Requires Node and, for production, Browserbase cloud hosting.
- **Skyvern** (21k+ stars, Apache 2.0): vision-first (screenshot-only, no DOM), handles legacy portals and government forms that DOM tools struggle with. No-code cloud UI available. Each step costs vision-model tokens — ~$0.10-0.50/task. 85.85% WebVoyager.
- **Firecrawl** (82k+ stars): managed scraping API. Returns LLM-ready Markdown, JSON extraction, site-wide crawl. Not an agentic tool — minimal multi-step interaction. Ships an official MCP server. Per-page pricing from $19/month.
- **Playwright MCP** (Microsoft, 29k+ stars): MCP server that exposes browser control via the accessibility tree. Sub-100ms actions, zero vision tokens. Drives an LLM client's browser rather than having its own reasoning — no autonomous goal navigation. Used in GitHub Copilot Agent.
#### vs. native LLM provider browsing
Most AI providers offer some form of browsing, but it is designed for **conversational information retrieval**, not programmatic web automation. Key constraints:
| | acrawl | ChatGPT Agent | Claude Computer Use | Claude in Chrome | Gemini Deep Research | Copilot / Edge |
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| Real JS-rendered browser | Yes | Yes (sandboxed cloud VM) | Indirect (dev provides env) | Yes (your Chrome) | No (search API only) | Limited (Bing retrieval) |
| Click / fill forms | Yes | Yes (requires user confirmation) | Yes | Yes | No | Limited |
| Programmable / scriptable | Yes | No | Yes (API beta) | No | No | No |
| Sub-agent parallelism | Yes | No | No | No | No | No |
| MCP server (expose as tools) | Yes | No | No | No | No | No |
| Returns structured data | Yes | No (text summaries) | No (screenshots) | No | No | No |
| Stealth / anti-bot | Yes | No | No | No | No | No |
| No vendor lock-in | Yes (25 providers) | OpenAI only | Anthropic only | Anthropic only | Google only | OpenAI / Bing only |
| Runs without paid subscription | Yes (OSS; LLM key needed) | No (Plus/Pro/Business) | No (API cost) | No (Max plan) | Partial | Yes (free tier) |
Notes:
- **ChatGPT Agent** (OpenAI, July 2025): runs in a sandboxed cloud virtual machine with its own Chromium instance. Can browse, click, and fill forms but pauses for user confirmation on sensitive actions (purchases, logins). Uses two modes: a fast text browser for research queries and a visual browser for interaction. Cannot run code in the browser, install extensions, or access your local file system. Susceptible to prompt injection. Available to Plus/Pro/Business subscribers.
- **ChatGPT Atlas** (OpenAI, October 2025): a full Chromium browser with ChatGPT integrated as a sidebar + agent. Agent mode drives the same sandboxed cloud VM as ChatGPT Agent; core limitations are identical.
- **Claude Computer Use** (Anthropic API, beta since October 2024): screenshot + mouse/keyboard API for any desktop application, not just browsers. Vision-only — no DOM access. Developers must provide and manage the entire computing environment (typically a Docker container with Xvfb + Firefox). Not a ready-to-use binary. Requires significant infrastructure to operate in production.
- **Claude in Chrome** (Anthropic Chrome extension, beta November 2025+): lets Claude operate within your existing Chrome session using your real cookies and logins. Available to Max plan subscribers. Not an open API — no programmatic control. Good for interactive personal tasks; not suitable for batch automation.
- **Gemini / Deep Research** (Google): browsing is grounded via Google Search API calls, not a live browser session. Deep Research synthesizes across many searches but cannot interact with pages (click, fill forms, navigate dynamically). Project Mariner (experimental computer use) is a separate, limited research preview.
- **Copilot / Edge** (Microsoft): Edge's Copilot Mode uses Bing retrieval with some ability to navigate pages. Real-world tests show high latency (6+ minutes for multi-page comparison tasks) and frequent interruptions for user confirmation. Not a developer API.
## Quick Start
### Install
**Linux / macOS (x64 / ARM64):**
```bash
curl -fsSL https://raw.githubusercontent.com/Mingye-Lu/AgenticCrawler/main/install.sh | bash
```
**Windows (x64, PowerShell):**
```powershell
irm https://raw.githubusercontent.com/Mingye-Lu/AgenticCrawler/main/install.ps1 | iex
```
This downloads the latest binary, verifies its SHA256 checksum, and sets up CloakBrowser for stealth browser automation. Requires Node.js 20+ for browser features.
acrawl checks for updates on startup and shows a notification when a new version is available.
Build from source
```bash
git clone https://github.com/Mingye-Lu/AgenticCrawler.git
cd AgenticCrawler
cargo build --release
# Install CloakBrowser (required for browser automation — binary auto-downloads on first use)
npm install
```
### Browser Extension (optional)
The acrawl Bridge extension lets acrawl control your real browser (with your sessions, cookies, and existing extensions) instead of a headless CloakBrowser instance.
1. Install **acrawl Bridge** from the [Chrome Web Store](https://chromewebstore.google.com/detail/acrawl-bridge/nbkpmfofkdjcgkncmpicbdkjmddllcep) and click **Add to Chrome**. The same listing works in Edge, Brave, Arc, Vivaldi, and Opera.
2. Run `/extension` in the acrawl REPL to start the bridge and show a pairing code, then type the code into the extension popup. This is a one-time step.
To install without the store (for example, to run an unreleased build), see [extension/README.md](extension/README.md#manual-install-from-release). It has the full setup details.
### Agent Skill (optional)
If you drive acrawl from a coding agent (via [`acrawl mcp`](#mcp-server-expose-acrawl-as-a-tool)), install the `acrawl-mcp` skill so the agent knows when to use direct tools, scripts, or `run_goal`, and how to debug with the network/console/cookie tools:
```bash
npx skills add Mingye-Lu/AgenticCrawler
```
This uses the community [`skills`](https://github.com/vercel-labs/skills) CLI, which detects your installed agents (Claude Code, Cursor, Codex, Gemini CLI, OpenCode, and many more) and copies the skill into each one's skills directory. Add `-g` for a global (user-level) install, or `-a claude-code` to target a single agent.
Prefer to skip Node? `acrawl mcp install --client claude-code --skill` configures the MCP server and installs the same skill in one step (see [Quick install](#quick-install)).
### Configure
```bash
# Set up your LLM provider (interactive prompt)
./target/release/acrawl auth anthropic # or: openai, other
```
Credentials are stored in `~/.acrawl/credentials.json`. Override the config directory with `ACRAWL_CONFIG_HOME`.
### Run
```bash
# Interactive REPL
./target/release/acrawl
# One-shot mode
./target/release/acrawl prompt "scrape all book titles and prices from books.toscrape.com"
# Resume a saved session
./target/release/acrawl --resume session.json /status /compact
```
## Examples
**Scrape a product catalog:**
```
acrawl > scrape all book titles, prices, and ratings from books.toscrape.com
```
The agent navigates to the site, reads the page, extracts the data, paginates through all 50 pages, and returns structured JSON.
**Fill and submit a form:**
```
acrawl > go to example.com/contact, fill in name "Jane Doe", email "jane@example.com",
message "Hello", and submit the form
```
The agent locates form fields, fills them in, clicks submit, and confirms the result.
**Monitor a price:**
```
acrawl > check the current price of "Rust in Action" on books.toscrape.com
```
Single-page extraction — the agent fetches, reads, and returns the price without unnecessary navigation.
**Extract from JS-rendered pages:**
```
acrawl > get all repository names and star counts from github.com/trending
```
Static HTTP won't work here. acrawl detects React/Next.js markers and automatically escalates to a headless browser to render the JavaScript.
**Parallel multi-page crawl:**
```
acrawl > scrape the title, author, and price of every book across all 50 pages on books.toscrape.com.
Fork a sub-agent for each page to speed this up.
```
The agent spawns up to 5 concurrent sub-agents, each on its own browser tab, to crawl pages in parallel. Results are merged when all sub-agents finish.
## Features
### 43-Tool Toolbox
#### Navigation
| Tool | Description |
|------|-------------|
| `navigate` | Go to a URL (supports `format`: markdown/text/html/fit_markdown). Uses HTTP first, auto-escalates to browser when JS is detected. Returns structured content with a `page_map`. `fit_markdown` prunes boilerplate DOM nodes before conversion, saving tokens. |
| `go_back` | Browser back button. Returns `page_state` with the resulting page structure. |
| `get_page_info` | Cheap read of the current page's `url`, `title` and `ready_state` (no DOM walk, no refetch). `all_tabs: true` also lists every tab you can use as `{index, url, title, active}` without switching. Read-only: never advances `seq`. |
| `scroll` | Scroll up or down by pixel amount (`pixels`, default: 500). Returns `page_state` after scrolling. |
| `switch_tab` | Switch to a different browser tab by index. Returns `page_state` of the new tab. |
| `wait` | Wait for a CSS selector to reach a given state (`visible`, `hidden`, `attached`, `detached`) or a fixed timeout (up to 300s). Set `silent: true` on a time-only wait to skip the `page_state` diff and save tokens. Returns `page_state` after the condition is met. |
| `refresh` | Reload the current page. Returns `page_state` after reload. Use after setting intercept rules to replay the page load with rules active. Seq counter increments for temporal observation queries. |
#### Content Formats
The `navigate` tool's `format` parameter controls how the page is returned:
| Format | Description |
|--------|-------------|
| `markdown` | Full HTML → markdown conversion. All content preserved. |
| `fit_markdown` | **Recommended.** Prunes boilerplate before conversion, saving 30-60% tokens on typical pages. |
| `text` | Plain text, no markdown. |
| `html` | Raw HTML. |
`fit_markdown` works in two passes:
1. **Hard-block removal**, elements whose `class` or `id` attribute contains any of these strings are removed immediately: `nav`, `footer`, `header`, `sidebar`, `ads`, `comment`, `promo`, `advert`, `social`, `share`.
2. **Score-based pruning**, remaining elements are scored; anything below 0.48 is removed. The score is:
```
0.4 × text_density
+ 0.2 × (1 − link_density)
+ 0.2 × tag_weight
+ 0.1 × class_id_score
+ 0.1 × ln(text_length + 1)
```
Tag weights: `article` = 1.5 · `h1` = 1.2 · `h2` = 1.1 · `h3/p/section` = 1.0 · `h4` = 0.9 · `h5/table` = 0.8 · `h6` = 0.7 · `span` = 0.3 · `div/li/ul/ol` = 0.5.
**Use `markdown` instead of `fit_markdown` when:** the page has important content inside elements named `sidebar`, `nav`, or similar, for example, metadata panels, related-article links, or author info stored in a sidebar div.
If `fit_markdown` prunes all content (empty result), the tool automatically falls back to plain text.
#### Interaction
| Tool | Description |
|------|-------------|
| `click` | Click an element by CSS selector, `@eN` ref, or visible label text. Use `text` (with optional `role`/`region`) to activate a button, tab, or link by its label — handy for SPA admin UIs and modals where CSS paths are fragile. Returns `page_state` after the click. |
| `click_at` | Click at specific viewport coordinates (x, y). Use for canvas, maps, or SVGs. Returns `page_state`. |
| `fill_form` | Fill form fields by selector, name, `@eN` ref, or visible label text — labels resolve page-wide, so fields in modals and div-based UIs without a ` 
