31 web scraping, crawling, deep-research & autonomous-extraction tools for Claude, Cursor & any MCP client.
Clean Markdown & structured JSON from any site. Get started with 1,000 free credits — no credit card required.

⭐ Star us on GitHub to follow along — it genuinely helps others discover the project.

## Table of Contents - [Why CrawlForge?](#-why-crawlforge) - [CrawlForge vs. alternatives](#-crawlforge-vs-alternatives) - [Quick Start (2 Minutes)](#-quick-start-2-minutes) - [Available Tools](#-available-tools) - [Pricing](#-pricing) - [Advanced Configuration](#-advanced-configuration) - [Usage Examples](#-usage-examples) - [Security & Privacy](#-security--privacy) - [Support](#-support) - [Contributing](#-contributing) ## 🎯 Why CrawlForge? - **31 MCP-native tools** — scraping, crawling, search, real Google SERP rank tracking, deep research, an autonomous `agent`, a unified multi-format `scrape`, document processing, stealth browsing, stateful browser sessions, and more, callable directly from your AI assistant. - **Generous free tier** — 1,000 credits to start instantly, no credit card. The grant is one-time rather than monthly, and the credits never expire. - **Local-LLM by default** — `extract_with_llm` runs against a local **Ollama** model out of the box: no LLM API key, no per-token cost, and your data never leaves your machine. Cloud (OpenAI/Anthropic) is opt-in. - **LLM-ready output** — clean Markdown, structured JSON (schema-driven), screenshots, links, and metadata from a single fetch. - **Autonomous `agent`** — describe what you need in natural language; it plans, gathers, and shapes an answer under orchestrator-enforced hard stops (max steps/URLs/wall-clock) — no URLs required. - **Security-hardened** — SSRF protection on every request, a fail-closed backend allow-list, a vetted action allowlist for browser automation, and per-tool credit gating. - **Works everywhere MCP does** — Claude Desktop, Claude Code, Cursor, and any other MCP-enabled client, configured in one command. ## 📊 CrawlForge vs. alternatives | | **CrawlForge MCP** | Firecrawl | Raw scraping API | |---|:---:|:---:|:---:| | Native MCP server | ✅ 31 tools | ✅ | ❌ | | Free tier | ✅ 1,000 credits, one-time, never expire | Limited | Varies | | Self-hosted / local LLM extraction (Ollama) | ✅ default, $0/token | ❌ | ❌ | | Autonomous agent (no URLs needed) | ✅ `agent` | ✅ | ❌ | | Deep research with source verification | ✅ `deep_research` | Partial | ❌ | | Browser automation / actions | ✅ `scrape_with_actions` | ✅ | Varies | | Stealth / anti-detection engines | ✅ Chromium + Camoufox | ✅ | Add-on | | Pre-built site templates | ✅ 10 sites | ❌ | ❌ | | License | MIT | AGPL-3.0 | Proprietary | > Comparison reflects publicly documented capabilities at time of writing. CrawlForge is MIT-licensed and MCP-first — built to plug straight into AI coding assistants. ## 🚀 Quick Start (2 Minutes) ### 1. Install from NPM ```bash npm install -g crawlforge-mcp-server ``` ### 2. Setup Your API Key (required) Every tool requires a CrawlForge API key — new accounts get 1,000 free trial credits to start. The recommended path signs you in through the browser, so the key is never pasted into a terminal (a coding agent can run this for you and relay the URL): ```bash crawlforge login ``` It prints an approval URL; open it, approve, and the key is stored in `~/.crawlforge/config.json`. Then run `crawlforge init` to register the MCP server with your client. Or use the interactive wizard, which also configures your clients: ```bash npx crawlforge-setup ``` This will: - Guide you through getting your free API key - Configure your credentials securely - **Auto-configure Claude Code and Cursor** (if installed) - Verify your setup is working **Don't have an API key?** Get one free at [https://www.crawlforge.dev/signup](https://www.crawlforge.dev/signup) > **One-step setup (v4.6.0+):** `crawlforge init` detects your API key, installs the agent skill, and idempotently merges the MCP config stanza into Claude Code, Claude Desktop, and Cursor. Use `crawlforge init --all --yes` to configure every detected client non-interactively. ### 3. Configure Your IDE (if not auto-configured)

🤖 For Claude Desktop

Add to `claude_desktop_config.json`: ```json { "mcpServers": { "crawlforge": { "command": "npx", "args": ["-y", "crawlforge-mcp-server"] } } } ``` **Location:** - macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` - Windows: `%APPDATA%/Claude/claude_desktop_config.json` - Linux: `~/.config/Claude/claude_desktop_config.json` Restart Claude Desktop to activate.

🖥️ For Claude Code CLI (Auto-configured)

The setup wizard automatically configures Claude Code by adding to `~/.claude.json`: ```json { "mcpServers": { "crawlforge": { "type": "stdio", "command": "crawlforge-mcp" } } } ``` After setup, restart Claude Code to activate.

💻 For Cursor IDE (Auto-configured)

The setup wizard automatically configures Cursor by adding to `~/.cursor/mcp.json`: ```json { "mcpServers": { "crawlforge": { "type": "stdio", "command": "crawlforge-mcp" } } } ``` Restart Cursor to activate.

🔁 For n8n (workflow automation)

n8n's built-in **MCP Client Tool** node connects over Streamable HTTP (works on n8n Cloud and self-hosted). Run the server in HTTP mode: ```bash export CRAWLFORGE_API_KEY=your_api_key npm run start:http # Streamable HTTP endpoint at http://localhost:10000/mcp ``` Then point the MCP Client Tool node at `http://:10000/mcp` with transport **HTTP Streamable** and a **Bearer** credential set to the same API key. On self-hosted n8n you can instead use the community `n8n-nodes-mcp` node over STDIO (`npx -y crawlforge-mcp-server`). Full guide: [docs/n8n-integration.md](docs/n8n-integration.md)

> **Which launch command?** `npx -y crawlforge-mcp-server` needs no global install and always runs the published version (recommended for Claude Desktop). For a global install (`npm i -g crawlforge-mcp-server`), use the dedicated `crawlforge-mcp` bin — it resolves on your `PATH`, so it survives Node/nvm version switches. The bare `crawlforge` command still launches the server when an MCP client spawns it over stdio (backward compatibility for configs created before v4.2.5); interactively it's the CLI — run `crawlforge mcp` to start the server by hand. ## 📊 Available Tools CrawlForge requires a CrawlForge API key — **every tool is metered and consumes credits**. New accounts get **1,000 free trial credits** to start. Get a key at [crawlforge.dev/signup](https://www.crawlforge.dev/signup). **All Tools** (API key required) | Tool | Credits | What it does | |------|---------|--------------| | `fetch_url` | 1 | Fetch content from any URL | | `extract_text` | 1 | Extract clean text from web pages | | `extract_links` | 1 | Get all links from a page | | `extract_metadata` | 1 | Extract page metadata (title, OG tags, schema.org) | | `scrape_template` | 1 | Structured data from well-known sites (Amazon, GitHub, LinkedIn, YouTube, Reddit, Hacker News, npm, and more) without writing selectors | | `list_ollama_models` | 1 | List the Ollama models installed locally (helps you pick a `model` for `extract_with_llm`) | | `get_batch_results` | 1 | Retrieve paginated results for a `batch_scrape` job by `batchId` | | `read_result` | 1 | Search, slice, read lines or a JSON path from a result a tool returned with `truncated: true` and a `result_handle` (kept 1 hour on your own machine) — never fetch the page again | | `scrape` | 2 | **Unified single-fetch, multi-format extraction.** Pass a `formats` array (markdown/html/rawHtml/text/links/metadata/screenshot/json-schema, plus `{type:"highlights",query}` and `{type:"question",question}` for only the matching sentences, table rows and code blocks, verbatim with offsets into the markdown: +1 credit once per call, `mode:"model"` +3) plus `onlyMainContent`; one fetch serves every requested format with per-format partial-success warnings. `escalate:true` retries a blocked plain fetch once in the stealth browser and returns the page instead of the block (+5 as the projected ceiling; the charge drops back to 2 when the plain fetch succeeded and no escalation ran) | | `scrape_structured` | 2 | Extract structured data with CSS selectors | | `extract_embedded_state` | 2 | Read a page's embedded JavaScript state — `__NEXT_DATA__`, React Server Component payloads, Nuxt, Apollo, Redux, `