# Agent Scrape Kit 🕷️

**Systematic web scraping for AI agents.** Two LLM-driven skills that teach agents to do senior-level scraping: discover how a site works, then extract data deterministically.

```
scraper-discover  →  blueprint.json  →  scraper-run  →  structured_data
   (learn once)                           (run forever)
```

No browser in the hot path. No LLM tokens on repeat runs. Just clean data extraction.

---

## Why This Exists

Most scraping is done ad-hoc: open browser, find the element, extract, repeat for the next site. This works but doesn't scale. Every run pays the same discovery cost.

This kit flips the model:

1. **scraper-discover** — spends LLM time once to reverse-engineer the site. Produces a **blueprint** describing the exact API endpoints, auth flow, anti-bot measures, and pagination.

2. **scraper-run** — reads the blueprint and executes deterministically. No browser. No LLM. Just HTTP calls + data extraction + rate limiting + checkpointing.

The result: you pay for exploration once and get reliable extraction forever.

---

## The Skills

### `scraper-discover` — Site Reconnaissance

Systematically classifies a website's architecture and discovers the fastest data extraction path.

**What it does:**
- Classifies the site into 6 archetypes (SSR, SPA REST, SPA GraphQL, static commerce, anti-bot shielded)
- Probes API endpoints directly (no browser when possible)
- Captures network traffic via Playwright when needed
- Detects auth mechanisms, CSRF tokens, session cookies
- Fingerprints anti-bot protection (Cloudflare, Datadome, PerimeterX, Akamai)
- Identifies pagination patterns (cursor, offset, page number, infinite scroll)
- Validates the blueprint by extracting at least 5 records
- Saves a reusable blueprint JSON

**Scripts:**
- `scripts/recon_capture.py` — Playwright-based network capture
- `scripts/detect_antibot.py` — Anti-bot fingerprinting

### `scraper-run` — Deterministic Extraction

Executes a blueprint to extract all data from a site. No browser. No LLM. Just extraction.

**What it does:**
- Loads the blueprint, validates it
- Executes pagination automatically (cursor, offset, page number)
- Handles auth refresh (session cookies, JWT, CSRF)
- Respects rate limits (built-in throttling + backoff)
- Checkpoints progress every 500 records
- Self-heals: detects stale endpoints, response shape changes, anti-bot escalation
- Produces clean JSON + CSV output

---

## Usage

### 1. Discover a site

```bash
# Scraper-discover handles this conversationally.
# Give it a URL and what you want to extract:
"scraper-discover: scrape all products from target.com, get name, price, SKU"
```

### 2. Run the blueprint

```bash
# Once the blueprint exists, scraper-run extracts everything:
"scraper-run: extract all products using the target.com blueprint"
```

### 3. Monitor changes (optional)

```bash
# Cron job setup:
"schedule scraper-run on target.com every 24h and notify me of changes"
```

---

## Blueprint Format

Blueprints are saved to `~/.hermes/scraper/blueprints/<domain>.json`.

```json
{
  "site": "target.com",
  "archetype": "spa_with_api",
  "tech_stack": {
    "framework": "nextjs",
    "api": "rest",
    "auth": "session_cookie + csrf_token",
    "anti_bot": "none"
  },
  "endpoints": [{
    "name": "list_products",
    "method": "GET",
    "url": "https://target.com/api/v2/products",
    "pagination": {
      "type": "cursor",
      "request_param": "cursor",
      "response_path": "$.meta.next_cursor"
    },
    "extraction": {
      "data_path": "$.data.items",
      "fields": ["id", "name", "price", "image_url"]
    },
    "rate_limit": {
      "max_rps": 2.0,
      "concurrency": 1
    }
  }],
  "quality": {
    "proven_working": true,
    "validated_records": 5
  }
}
```

Full schema: `schemas/blueprint-v1.json`

---

## Architecture Anti-Patterns This Avoids

| Anti-Pattern | How This Kit Fixes It |
|---|---|
| Re-extracting via browser every time | Replays the API directly — browser is discovery only |
| LLM in the hot path | Zero tokens during extraction |
| Hardcoded selectors that break | Blueprints are self-healing — failures trigger re-discovery |
| No rate limiting | Blueprint captures limits, runner enforces them |
| No checkpointing | Every 500 records saved, resume from failure |
| Mixing discovery and execution | Separate skills with clear boundaries |

---

## Directory Layout

```
~/.hermes/scraper/
├── blueprints/          # Reusable site blueprints (JSON)
│   └── target.com.json
├── data/                # Extracted data (JSON/CSV)
│   └── target.com_20250520.json
├── checkpoints/         # Checkpoints for resume
│   └── target.com_latest.json
└── skills/
    └── web-scraping/
        ├── scraper-discover/  # Discovery logic
        └── scraper-run/       # Execution logic
```

---

## License

MIT — use it, modify it, ship it.