# Roadmap

AgentCrawl focuses on one job first: help AI agents read and crawl web content reliably from infrastructure you control.

This roadmap is **directional**. Public performance or quality claims need reproducible tests before they appear in docs or marketing: benchmark numbers appear only with a public, reproducible run (see [docs/BENCHMARKS.md](docs/BENCHMARKS.md)), and a gap that is not significant is reported as no significant difference, never as a win.

## Current community status

AgentCrawl Community is a serious self-hosted alpha surface. The core paths work:

- **CLI, Python library, HTTP API, Docker/GHCR image, and MCP server** — pick whichever interface fits the agent.
- **HTTP-first scraping** with optional browser/Camofox fallback when a page needs it.
- **Durable crawl jobs** with checkpoints, retries, cancellation, pagination, event history, and selective failure retries.
- **SQLite-backed local state** for cache, usage, jobs, events, failures, and extracted documents — plus a read-only local dashboard.
- **Safer server defaults**: bearer auth, per-key job isolation, `robots.txt` support, SSRF protections, unsafe redirect blocking, and private-network controls.
- **Quality extraction baseline**: checked-in fixtures, quality report, provenance metadata, JSON-LD/Product extraction, Markdown table + code preservation, noisy-layout handling.
- **Distribution readiness**: wheel/sdist checks, clean install smoke tests, CI, lightweight Docker image, and GHCR publication.

## Completed

### Safety & privacy
- Text normalization, IDNA domain normalization, safer URL validation, unsafe redirect blocking.
- Sliding-window per-key API rate limiting.
- SSRF and private-network controls.
- PDF safety limits (size + page count).
- Clearer scrape error behavior on partial / malformed pages.

### Extraction quality
- Main-content extraction for semantic containers and text-rich fallback blocks.
- Boilerplate reduction: page chrome, cookie banners, sidebars, related posts, hidden content, unsafe tags.
- Cookie-consent node-level filter (0.1.1) — drops text-only consent blocks inside generic containers, preserves documentation that mentions cookies as a feature.
- Markdown table + fenced-code-block preservation with language tags.
- Quality extraction hardening: checked-in fixtures (23 today, see [docs/QUALITY_BENCHMARKS.md](docs/QUALITY_BENCHMARKS.md)), browser-rendered SPA coverage, score thresholds, JSON report, richer provenance, JSON-LD/Product schema + rating extraction, hidden-class filtering.

### Crawler & ops
- Sitemap index expansion and `robots.txt` discovery.
- Local document ingestion: Markdown, text, JSON, XML/RSS/Atom, PDF-to-Markdown through the optional `docs` extra.
- Opt-in browser fallback on 200-OK challenge pages (0.1.1) — `client_challenge` is honest failure, not a disguised content scrape.
- Pagination, job events, and failure inspection end-to-end.

### Observability & packaging
- Local SQLite-backed cache, usage, jobs, events, failures, and extracted documents.
- Read-only local dashboard: `agentcrawl dashboard` + FastAPI `GET /dashboard` and `GET /api/dashboard/summary`.
- Failure alert hook: `agentcrawl crawl ... --alert-on-failure --cmd "..."` runs a local command with failures JSON on stdin.
- Audit trail end-to-end into `ScrapeDocument.metadata` when `audit=True`.
- Token Efficiency pillar: `estimated_tokens`, `raw_html_tokens_estimate`, `raw_html_bytes` on every document + `--token-stats` CLI flag.

### Reliability fixes (2026-06-28 audit, shipped in v0.1.3)
- SQLite-backed scheduling lease cross-process — multi-worker uvicorn no longer enqueues the same job twice.
- Audit trail now reaches the document on terminal fetch failure, not just on success.
- Blocked-page heuristic strips HTML chrome before matching — fewer false negatives on `nginx default 403` and friends.
- `CrawlConfig.__post_init__` warns when `llm` is a dict rather than an import path, instead of silently accepting it.
- `list_crawl_failures` domain filter uses 3-pattern LIKE in SQL (no Python post-filter).
- `_pop_ready_item` no busy-spins when nothing is ready yet.
- `SQLiteStore._migrated_paths` per-process cache, `:memory:`-aware.
- `_export_failures_csv` skips `mkdir` when there are no rows to write.

### Release hygiene
- `agentcrawl doctor`, `agentcrawl --version`, package build verification, clean install smoke tests.
- Lightweight Docker image on GHCR (`ghcr.io/jorg18/agentcrawl:latest`).
- PyPI publication pipeline (`pip install agentcrawl-ai`).
- CI badge in README.

### Security & correctness fixes (2026-09 audits, shipped in v0.2.0)
- SSRF guard rails extended to `robots.txt` and sitemap discovery; sitemap-index recursion bounded (depth + entry budget).
- Optional local-file jail (`AGENTCRAWL_LOCAL_FILES_ROOT`) for servers with `ALLOW_LOCAL_FILES=true`.
- Crawl jobs scoped to the creating API key; owner keys are the only ones that see every job. Foreign jobs answer `404`.
- The HTTP dashboard follows `AGENTCRAWL_AUTH_ENABLED` instead of leaking job/cache/usage state unauthenticated (`AGENTCRAWL_DASHBOARD_PUBLIC` restores the open local view).
- A missing browser backend no longer masks the honest HTTP error; deterministic refusals (SSRF, airgap) are not retried.
- Unsupported `config` overrides are rejected with `400` instead of being silently dropped.
- `doctor` and the OpenAPI schema report the real package version; the source-checkout version fallback reads `pyproject.toml` instead of a hardcoded literal.
- Playwright sessions close their context and browser exactly once, including on navigation failure.
- Airgap and audit cover robots.txt/sitemap discovery and every request a local Playwright browser makes (redirect hops, iframes, sub-resources, XHR, WebSockets); the HTTP path pins DNS against rebinding.
- Per-key scoping reaches the cache, every aggregate and the dashboard; usage is billed to the job owner and LLM calls are metered separately.
- CLI local mode reads the same `AGENTCRAWL_*` configuration as MCP, and exits non-zero on failed results.

### New capabilities (v0.2.0)
- `scrape_many` on the library, API, MCP and CLI.
- Deterministic CSS-schema extraction (`extract-css`, `/v1/extract_css`, MCP `extract_structured`) — no LLM, zero tokens.
- Query-aware (BM25) selection when a page exceeds the output or chunk budget.
- CSV/TSV ingestion for local files and `text/csv` URLs.
- `benchmarks/compare.py`: reproducible offline comparison lane (published for 0.6.0 in [docs/BENCHMARKS.md](docs/BENCHMARKS.md)).

**Audit context:** each audit pass is recorded, finding by finding, in [CHANGELOG.md](CHANGELOG.md) (the 2026-06 audit under 0.1.3, the 2026-09 passes under 0.2.0).

### 0.5.0: Community competes on the open web
- The browser reaches Cloudflare-protected pages that clear on their own: real Chrome identity, native browser TLS, full Chromium in headless mode, optional Patchright retry, proxy rotation.
- Browsers stay open between pages (four at a time); one time budget per page.
- Long pages are read in parts: `outline`, `section`, `max_tokens`, `max_age`.
- Firecrawl v2-compatible API; LLM-written CSS schemas; summaries; `next_step` on every error.

## Enhanced (hosted, planned)

Managed residential and mobile proxies, geolocation, a managed browser fleet, CAPTCHA solving, schedules, webhooks, retained datasets, teams and billing, behind the same Firecrawl-compatible API. First as a hosted API, to learn from real traffic.

## Next community priorities

We're working on, in roughly this order:

1. **Keep the public benchmark lane honest.** The 400-page random web sample (`benchmarks/web`) against Crawl4AI, Scrapling and Firecrawl, run from GitHub's datacenter network, rerun before each release and published in [docs/BENCHMARKS.md](docs/BENCHMARKS.md). Numbers of hosted tools are published only where their terms allow it.
2. **Maintained release smoke targets.** Smoke-tested paths stay tested. They don't grow into unsupported "vs X" comparison copy.
3. **Lighter document ingestion.** More file types only when the dependency cost stays small enough to remain an optional extra.
4. **Verified examples.** Every example in `examples/` runs against a real public site we actually tested. Anything else gets removed.

These are scope keepers, not feature promises. Each only lands when its verification criteria are met.

## Public launch readiness

Public announcements only go out after:

1. The `agentcrawl-ai` package installs cleanly in a fresh venv and reports the version we tagged.
2. `pytest` + `ruff check` + `ruff format --check` are clean on the release commit.
3. The GitHub Release for the tagged version has explicit release notes (not auto-generated).
4. The GHCR workflow for the tag is green.
5. No proprietary or non-Community code exists in any public source file.

Comparative claims follow [docs/BENCHMARKS.md](docs/BENCHMARKS.md): no numbers without a reproducible public run, with quality scored on pages drawn at random.