# FetchGate

**A 200 is not a read.**

When an AI agent fetches a page and it quietly comes back empty (a JavaScript app,
a 403, a Cloudflare wall), the agent often does not stop. It answers from memory
and never tells you it did not actually read the page. FetchGate is a small gate
at the fetch boundary that refuses to let that happen: the agent gets the content
only when the page was really read, and an honest refusal otherwise.

It is deterministic (no model, no API keys), stdlib-only at its core, and free.

## Quickstart

```bash
git clone https://github.com/MoAz06/FetchGate
cd FetchGate
python demo.py https://example.com https://httpbin.org/status/403
```

```
https://example.com
  verdict     : RETRIEVED  (clean)
  answerable  : ALLOW
  bytes       : raw 559 / extracted 127
  tiers       : static
  gate-confirmed content (127 chars): Example Domain This domain is for use ...

https://httpbin.org/status/403
  verdict     : FAILED  (transport.http_403)
  answerable  : STOP
  refused: the model gets no content and cannot answer as if it read this page.
```

Run the tests (no network, no model):

```bash
pip install -e .
python -m unittest discover -s tests
```

## Easiest: one double-click (Claude Desktop)

Download `fetchgate.mcpb` from the [Releases](https://github.com/MoAz06/FetchGate/releases)
page and double-click it. Claude Desktop shows an Install button, no JSON editing.
The bundle source is in [bundle/](bundle); rebuild it with
`npx @anthropic-ai/mcpb pack bundle fetchgate.mcpb`.

## Vibecoders: add it from inside your editor

**Cursor, one click** (needs `uv`, which most setups have):

[Add FetchGate to Cursor](cursor://anysphere.cursor-deeplink/mcp/install?name=fetchgate&config=eyJjb21tYW5kIjogInV2eCIsICJhcmdzIjogWyItLWZyb20iLCAiZmV0Y2hnYXRlW21jcF0iLCAiZmV0Y2hnYXRlLW1jcCJdfQ==)

**Or just ask your AI.** Paste this into Cursor, Claude Code, or Windsurf:

> Add the fetchgate MCP server to this setup. Run `pip install "fetchgate[mcp]"`
> and register it as an MCP server named `fetchgate` with the command
> `fetchgate-mcp`. It stops an agent from answering as if it read a page it could
> not actually fetch.

Your assistant runs the install and wires up the config for you.

## Use it in Claude Desktop or Cursor (manual)

```bash
pip install ".[mcp,http,extract]"    # the fetch tool, real redirect chain, better extraction
pip install ".[browser]"             # optional: render JavaScript pages
python -m playwright install chromium
```

Add to your MCP config (details in [docs/mcp.md](docs/mcp.md)):

```json
{ "mcpServers": { "fetchgate": { "command": "fetchgate-mcp" } } }
```

Your agent now has a `fetch_url` tool that returns content only for a confirmed
read, and a refusal otherwise.

This repo is also a Claude Code plugin. After `pip install "fetchgate[mcp]"`:

```bash
/plugin marketplace add MoAz06/FetchGate
/plugin install fetchgate@fetchgate
```

See [docs/claude-code-plugin.md](docs/claude-code-plugin.md).

## Use it in your own agent

```python
from fetchgate import default_gate

gate = default_gate()   # renders JS pages if the [browser] extra is installed

def read_url(url: str) -> str:
    out = gate.guarded_fetch(url)
    content = gate.content_for(out.handle)
    if content is None:
        return f"NOT READ: {url} ({out.result.reason}). Do not answer as if you read it."
    return content
```

## How it decides

Every fetch gets one verdict:

- **RETRIEVED**: a real read. The agent may answer and cite it.
- **FAILED**: a definite non-read (a 4xx or 5xx, a timeout, an https to http
  downgrade, an empty body). Stop.
- **UNKNOWN**: fetched, but not enough readable content to confirm a read (often a
  JavaScript page). Stop, or render and try again.

Three layers feed the verdict, and the result is the strictest of them:

- **transport**: status and redirect chain.
- **extraction**: readable text kept separate from raw bytes. A 17 byte price JSON
  is a read, a 400 KB article at 2 percent text is a read, a JS shell with no text
  is not.
- **content-validity**: a best-effort fingerprint for Cloudflare, consent,
  paywall, and soft-404 walls. It can only downgrade to UNKNOWN, never fake a read.

When a page looks thin or empty and the render tier is on, FetchGate escalates to
a real headless browser and re-checks. It renders, it does not evade: a page still
walled after an honest render stays a non-read.

## Stated limits

- `content_sha256` is a receipt anchor, not tamper-evidence. A cloaked page hashes
  cleanly.
- Provenance is not authenticity. The gate closes fail-silent-on-empty, not
  fetch-succeeds-on-wrong-content.
- A read is not an answer. RETRIEVED means enough content arrived, not that it held
  the fact you need. Claim support is a downstream job.
- Fail-closed has an availability cost. Anyone who can make your fetch fail can
  force a refusal. That is the trade for correctness.

## What is built

The deterministic core, the offline evaluation with a signed manifest, the
build-breaking cheat-test, the MCP server, and the Playwright render tier. Optional
extras: `[mcp]`, `[http]` (httpx redirect chain), `[extract]` (trafilatura),
`[browser]` (Playwright), `[sign]` (Ed25519). When installed they are used
automatically; the core stays stdlib-only.

MIT licensed.