servo-fetch

A self-contained browser engine that fetches, renders, and extracts web content as Markdown, JSON, or screenshots — no Chromium, no API key, no setup.

CI crates.io MSRV MIT OR Apache-2.0

servo-fetch demo
servo-fetch embeds the [Servo](https://servo.org/) browser engine. It executes JavaScript, computes CSS layout, captures screenshots with a software renderer, and extracts clean content — available as a CLI, a Rust library, a Python SDK, and a Node.js SDK. ```bash # CLI servo-fetch "https://example.com" # clean Markdown servo-fetch "https://example.com" --format png -o page.png # PNG screenshot ``` ```rust // Rust let md = servo_fetch::markdown("https://example.com").await?; ``` ```python # Python page = servo_fetch.fetch("https://example.com") print(page.markdown) ``` ```ts // Node.js import { fetch } from "servo-fetch"; const md = await fetch("https://example.com"); ``` ## Why servo-fetch - **Zero dependencies** — single binary, no Chromium, no API key - **Real JS execution** — SpiderMonkey runs JavaScript, parallel CSS engine computes layout - **Layout- and visibility-aware extraction** — strips navbars, sidebars, footers by rendered position, plus cookie banners, modals, and CSS-hidden content (`opacity:0`, `aria-hidden`, sr-only) - **Schema-driven JSON** — declarative CSS-selector schema pulls structured data - **Parallel batch fetch** — multiple URLs fetched concurrently - **Isolated browser sessions** — one-use worker process per session keeps cookies and storage fully separated - **Site crawling** — BFS link traversal with robots.txt, same-site scope, and rate limiting - **URL discovery** — sitemap-based URL mapping without rendering (fast, lightweight) - **Screenshots without GPU** — software renderer captures PNG/full-page screenshots anywhere - **Accessibility tree** — AccessKit integration with roles, names, and bounding boxes - **Agent-ready** — drop-in web tool for AI agents: a built-in MCP server, or wrap the Python API as a tool in any agent framework ## Performance and quality Apple M3 Pro, versus Playwright (the typical AI-agent stack): | Benchmark | servo-fetch | playwright:optimized | | ------------------- | ----------: | -------------------: | | Time — static-small | ~231 ms | ~645 ms | | Time — spa-heavy | ~331 ms | ~798 ms | | Memory (peak RSS) | 51–64 MB | 300–328 MB | Extraction quality: mean word-F1 0.819 vs Readability's 0.728 across eight page-type fixtures, with `without[]` boilerplate removal at 95.0% vs 78.6%. Direct-binary engine peers (chrome-headless-shell, Lightpanda, curl) are opt-in. Methodology, three-axis breakdown, per-fixture F1, and raw JSON: [`benchmarks/README.md`](benchmarks/README.md) + [`benchmarks/results/`](benchmarks/results/). ## Install | Interface | Install | Docs | | --------- | ------- | ---- | | **CLI** | `curl -fsSL https://raw.githubusercontent.com/konippi/servo-fetch/main/install.sh \| sh` | [CLI docs](crates/servo-fetch-cli/README.md) | | **Rust** | `cargo add servo-fetch` | [Library docs](crates/servo-fetch/README.md) | | **Python** | `pip install servo-fetch` | [Python docs](bindings/python/README.md) | | **Node.js** | `npm install servo-fetch` | [Node docs](bindings/node/README.md) |
CLI install alternatives ```bash cargo binstall servo-fetch-cli # prebuilt binary cargo install servo-fetch-cli # build from source ``` Or download from [GitHub Releases](https://github.com/konippi/servo-fetch/releases). **Linux** — install runtime deps and use `xvfb-run` on headless servers: ```bash sudo apt install -y libegl1 libfontconfig1 libfreetype6 xvfb-run --auto-servernum servo-fetch "https://example.com" ``` **Windows** — `cargo binstall` does not copy sidecar files ([cargo-binstall#353](https://github.com/cargo-bins/cargo-binstall/issues/353)), so the installed `servo-fetch.exe` fails at startup with a missing `libEGL.dll`. Download the `.zip` from [Releases](https://github.com/konippi/servo-fetch/releases) instead — it bundles `libEGL.dll` and `libGLESv2.dll`. **macOS** — no extra setup needed.
## Quick Start ### CLI ```bash servo-fetch "https://example.com" # Markdown (default) servo-fetch "https://example.com" --format json # Structured JSON servo-fetch "https://example.com" --format png -o page.png # PNG screenshot servo-fetch "https://example.com" --js "document.title" # Run JavaScript servo-fetch "https://example.com" --schema schema.json # Schema-driven JSON servo-fetch "https://example.com" --cookies cookies.txt # Send session cookies servo-fetch "https://example.com" --cookie-jar cookies.txt # Save cookies (Netscape format) servo-fetch "https://example.com" -H "X-Api-Key: KEY" # Custom request header servo-fetch URL1 URL2 URL3 # Parallel batch servo-fetch "https://example.com" --output page.md # Save to a single file servo-fetch URL1 URL2 --output-dir ./out/ # Save each URL to its own file servo-fetch crawl "https://docs.example.com" --limit 20 # Crawl a site servo-fetch crawl URL --output-dir ./pages/ # Save each crawled page to its own file servo-fetch map "https://example.com" # Discover URLs via sitemap servo-fetch mcp # MCP server (stdio) servo-fetch serve # HTTP API server ``` Full CLI reference → [`servo-fetch-cli`](crates/servo-fetch-cli/README.md) ### Exit status Batch fetch attempts every URL and exits non-zero for any fetch failure servo-fetch detects. Crawl exits non-zero for page failures only when every attempted page fails. Command-level failures remain fatal; a closed output pipe is treated as success. See the [CLI exit-code reference](crates/servo-fetch-cli/README.md#exit-codes) for exact codes. ### Rust ```bash cargo add servo-fetch ``` ```rust // URL → Markdown in one line (async by default; use `blocking::*` for sync) let md = servo_fetch::markdown("https://example.com").await?; // Fetch with options use servo_fetch::{fetch, FetchOptions}; use std::time::Duration; let page = fetch(&FetchOptions::new("https://example.com").timeout(Duration::from_secs(60))).await?; println!("{}", page.url); // Final URL after redirects or script navigation println!("{}", page.html); let md = page.markdown()?; // Crawl a site servo_fetch::crawl_each( &servo_fetch::CrawlOptions::new("https://docs.example.com") .limit(100) .user_agent("MyBot/1.0"), |result| match &result.outcome { Ok(page) => println!("{}: {} chars", result.url, page.content.len()), Err(e) => eprintln!("{}: {e}", result.url), }, ).await?; // Discover URLs via sitemap (no rendering) let urls = servo_fetch::map( &servo_fetch::MapOptions::new("https://example.com").limit(1000), ).await?; for u in &urls { println!("{}", u.url); } ``` Full API reference → [`servo-fetch`](crates/servo-fetch/README.md) ### Python Requires Python 3.11 or later. ```bash pip install servo-fetch ``` ```python import servo_fetch page = servo_fetch.fetch("https://example.com") print(page.markdown) # Schema extraction from servo_fetch import Schema, Field schema = Schema( base_selector=".product", fields=[ Field(name="title", selector="h2", type="text"), Field(name="price", selector=".price", type="text"), ], ) page = servo_fetch.fetch("https://shop.example.com", schema=schema) print(page.extracted) ``` Full API reference → [`bindings/python`](bindings/python/README.md) ### Node.js ```bash npm install servo-fetch ``` ```ts import { fetch, crawl } from "servo-fetch"; const md = await fetch("https://example.com"); for await (const page of crawl("https://docs.example.com", { limit: 50 })) { if (page.ok) console.log(page.url, page.title); } ``` Or run the bundled CLI without installing: ```bash npx servo-fetch "https://example.com" ``` Full API reference → [`bindings/node`](bindings/node/README.md) ## MCP Server Built-in [Model Context Protocol](https://modelcontextprotocol.io/) server with six tools: `fetch`, `batch_fetch`, `crawl`, `map`, `screenshot`, and `execute_js`. ```json { "mcpServers": { "servo-fetch": { "command": "servo-fetch", "args": ["mcp"] } } } ``` Streamable HTTP: `servo-fetch mcp --port 8080` Full MCP tool reference → [`servo-fetch-cli` README](crates/servo-fetch-cli/README.md) Prefer in-process tools? Wrap the Python API as agent tools — see [`bindings/python/examples/strands_agent.py`](bindings/python/examples/strands_agent.py). ## HTTP API REST endpoints for containerized deployments and HTTP clients: ```bash servo-fetch serve # 127.0.0.1:3000 servo-fetch serve --host 0.0.0.0 --port 80 # expose to network curl -X POST http://127.0.0.1:3000/v1/fetch \ -H 'content-type: application/json' \ -d '{"url":"https://example.com"}' ``` Endpoints: `GET /health`, `GET /version`, `POST /v1/fetch`, `POST /v1/batch_fetch`, `POST /v1/screenshot`, `POST /v1/execute_js`, `POST /v1/crawl`, `POST /v1/map`. Full HTTP API reference → [`servo-fetch-cli` README](crates/servo-fetch-cli/README.md#http-api-server) ## Docker Multi-arch image on GitHub Container Registry (`linux/amd64`, `linux/arm64`): ```bash docker run --rm -p 3000:3000 ghcr.io/konippi/servo-fetch:latest curl -X POST http://127.0.0.1:3000/v1/fetch \ -H 'content-type: application/json' \ -d '{"url":"https://example.com"}' ``` Runs as non-root (UID 1001). Images are signed with [cosign](https://github.com/sigstore/cosign) (keyless) and published with SLSA provenance and SBOM attestations. ## Agent Skills servo-fetch ships with an [Agent Skills](https://agentskills.io/) package for AI coding agents: ```bash npx skills add https://github.com/konippi/servo-fetch/tree/main/skills/servo-fetch ``` ## Security servo-fetch blocks all private and reserved IP ranges ([RFC 6890](https://datatracker.ietf.org/doc/html/rfc6890)), strips credentials from URLs, disables HTTP redirects to prevent SSRF bypass, and sanitizes all output against terminal escape injection ([CVE-2021-42574](https://www.cve.org/CVERecord?id=CVE-2021-42574)). See [SECURITY.md](./SECURITY.md) for details. ## Limitations - Sites behind CAPTCHAs are not supported. ## Contributing See [CONTRIBUTING.md](./CONTRIBUTING.md) for development setup, commit conventions, and PR guidelines. ## License [MIT](./LICENSE-MIT) OR [Apache-2.0](./LICENSE-APACHE)