# GovData MCP

A Model Context Protocol (MCP) server that lets AI agents **search, explore, and fetch open datasets from official government data portals**. One unified tool interface across the United States, United Kingdom, Canada, and Australia — with per-portal quirks (API migrations, auth walls, bilingual metadata, stale DataStore flags) absorbed behind a common adapter layer.

Built for the typical agent workflow: *"Is there a government dataset for X?"* → find candidates → inspect the files → preview rows or grab a download link and move on.

### Key Features

- **Four national portals, one interface.** US (data.gov v4 / DCAT), UK (data.gov.uk / CKAN), Canada (open.canada.ca / CKAN), Australia (data.gov.au / CKAN + DataStore).
- **Token-frugal by design.** Descriptions are HTML-stripped and truncated, search results are compact summaries, and large files are never inlined by accident.
- **Row-level preview.** Uses CKAN DataStore where available (Australia), and transparently falls back to reading the head of CSV/TSV files everywhere else.
- **Agent-friendly failure modes.** Errors are returned as actionable text ("use `get_download_link` instead", "retry with `dataset_id`", "available portals: …") — never raw tracebacks.
- **Extensible registry.** Adding another CKAN-based portal is a single config entry; non-CKAN portals plug in via a small adapter class.
- **Dual transport.** `stdio` for local clients (Claude Desktop etc.), stateless `streamable-http` for serverless deployment (Google Cloud Run).

### Supported Portals

| Code | Portal | API | Search | Explore | Row Preview | Notes |
|------|--------|-----|:------:|:-------:|:-----------:|-------|
| `us` | [data.gov](https://catalog.data.gov) | data.gov v4 (DCAT) | ✅ | ✅ | CSV fallback | Requires free [api.data.gov](https://api.data.gov/signup/) key |
| `uk` | [data.gov.uk](https://www.data.gov.uk) | CKAN | ✅ | ✅ | CSV fallback | Resource lookups need the parent `dataset_id` |
| `ca` | [open.canada.ca](https://open.canada.ca/en/open-data) | CKAN | ✅ | ✅ | CSV fallback | Bilingual titles normalized to English |
| `au` | [data.gov.au](https://data.gov.au) | CKAN + DataStore | ✅ | ✅ | ✅ DataStore | Full row queries with text filtering |

### Requirements

- Python 3.11+
- An [api.data.gov](https://api.data.gov/signup/) API key for the US portal (free; other portals need no credentials)

### Installation

```bash
git clone https://github.com/YOUR_ORG/Gov-Stat-MCP-Server.git
cd Gov-Stat-MCP-Server

# pip
pip install -e ".[dev]"

# or conda
conda env create -f environment.yml
conda activate opendata-mcp
```

Copy `.env.example` to `.env` and set your key:

```bash
DATAGOV_API_KEY=your-api-data-gov-key
```

### Getting Started

**Standard config** for most MCP clients (stdio transport):

```json
{
  "mcpServers": {
    "govdata": {
      "command": "govdata-mcp",
      "env": {
        "DATAGOV_API_KEY": "your-api-data-gov-key"
      }
    }
  }
}
```

<details>
<summary>Claude Code</summary>

```bash
claude mcp add govdata -e DATAGOV_API_KEY=your-key -- govdata-mcp
```

</details>

<details>
<summary>Claude Desktop</summary>

Follow the MCP install [guide](https://modelcontextprotocol.io/quickstart/user) and use the standard config above. Note: `govdata-mcp` must be on the PATH Claude Desktop uses — provide an absolute path to the executable inside your virtualenv/conda env if needed, e.g. `/path/to/envs/mcp-env/bin/govdata-mcp`.

</details>

<details>
<summary>Cursor / Windsurf / other JSON-config clients</summary>

Use the standard config above in the client's MCP settings file.

</details>

<details>
<summary>HTTP transport (remote / long-running server)</summary>

Run the server in HTTP mode:

```bash
MCP_TRANSPORT=http govdata-mcp
```

Then point your client at the endpoint:

```json
{
  "mcpServers": {
    "govdata": {
      "url": "http://localhost:8080/mcp"
    }
  }
}
```

</details>

### Tools

The intended agent flow: `list_portals` → `search_datasets` → `get_dataset` → `preview_resource` / `fetch_resource` / `get_download_link`.

<details>
<summary><b>Discovery</b></summary>

- **list_portals**
  - Description: List available government open-data portals and their capabilities.
  - Parameters: None
  - Read-only: **true**

- **search_datasets**
  - Description: Full-text search for datasets on a portal. Returns compact summaries with dataset ids.
  - Parameters:
    - `portal` (string): Portal code — `us`, `uk`, `ca`, `au`
    - `query` (string): Search terms, e.g. `"air quality monitoring"`
    - `limit` (number, optional): Max results (default 10, max 50)
    - `org` (string, optional): Publisher/organization slug filter (CKAN portals)
    - `format` (string, optional): Resource format filter, e.g. `"CSV"` (CKAN portals)
  - Read-only: **true**

</details>

<details>
<summary><b>Exploration</b></summary>

- **get_dataset**
  - Description: Full dataset metadata — organization, license, tags, description, and the list of downloadable resources with their ids, formats, and sizes.
  - Parameters:
    - `portal` (string): Portal code
    - `dataset_id` (string): Dataset id/slug from `search_datasets`
  - Read-only: **true**

- **preview_resource**
  - Description: Preview rows from a tabular resource. Tries the portal DataStore first (row counts + text filtering); falls back to reading the head of CSV/TSV files. Non-tabular formats can't be previewed.
  - Parameters:
    - `portal` (string): Portal code
    - `resource_id` (string): Resource id from `get_dataset`
    - `rows` (number, optional): Rows to return (default 20, max 100)
    - `query` (string, optional): Full-text row filter (DataStore-backed resources only)
    - `dataset_id` (string, optional): Parent dataset id — recommended; required on `uk`
  - Read-only: **true**

</details>

<details>
<summary><b>Retrieval</b></summary>

- **fetch_resource**
  - Description: Download a small text resource (CSV/JSON/XML/…) and return its content inline, truncated at a size cap on a clean line boundary. Binary content is refused with a pointer to the download URL.
  - Parameters:
    - `portal` (string): Portal code
    - `resource_id` (string): Resource id from `get_dataset`
    - `max_kb` (number, optional): Max kilobytes to inline (default 256, cap 512)
    - `dataset_id` (string, optional): Parent dataset id — recommended; required on `uk`
  - Read-only: **true**

- **get_download_link**
  - Description: Return the direct download URL, format, and size for a resource so the agent can proceed on its own. Works for any format, including large/binary files.
  - Parameters:
    - `portal` (string): Portal code
    - `resource_id` (string): Resource id from `get_dataset`
    - `dataset_id` (string, optional): Parent dataset id — recommended; required on `uk`
  - Read-only: **true**

</details>

### Configuration

All settings are environment variables (or a local `.env` file):

| Variable | Default | Description |
|----------|---------|-------------|
| `MCP_TRANSPORT` | `stdio` | `stdio` (local clients) or `http` (streamable-http server) |
| `PORT` | `8080` | Listen port for HTTP transport (injected automatically on Cloud Run) |
| `DATAGOV_API_KEY` | — | api.data.gov key for the US portal; falls back to `DEMO_KEY` (~30 req/hr) |
| `LOG_LEVEL` | `INFO` | Logging verbosity (always written to stderr) |
| `HTTP_TIMEOUT_SECONDS` | `30` | Timeout for outbound portal requests |
| `HTTP_MAX_RETRIES` | `2` | Retries on transient portal errors (5xx / transport), exponential backoff |
| `SEARCH_DEFAULT_LIMIT` | `10` | Default number of search results |
| `NOTES_TRUNCATE_CHARS` | `200` | Description truncation length in search summaries |
| `FETCH_MAX_BYTES` | `524288` | Hard cap for inlined resource content |

### Docker

```bash
make docker-build          # build image
make docker-run            # run on :8080, exactly as Cloud Run would
```

The image runs the HTTP transport in stateless mode and is ready for serverless platforms (Google Cloud Run deployment script included under `deploy/`).

### Development

```bash
make run          # stdio mode
make run-http     # HTTP mode on :8080
make inspect      # MCP Inspector interactive UI (requires Node.js)
make test         # offline unit tests
make test-live    # live end-to-end tests against all four real portals
make lint         # ruff check + format check
```

The live suite (`pytest -m live`) verifies the full search → explore → preview → fetch chain per country plus portal-specific behavior (Canada's bilingual normalization, UK's dataset-scoped resource lookup, Australia's DataStore paths). Run it before deploying — government portals change without notice.

### Architecture

```text
src/govdata_mcp/
├── server.py            # FastMCP instance + tool registration
├── config.py            # env-driven settings
├── ckan/                # generic async CKAN client, models, typed errors
├── portals/
│   ├── registry.py      # portal registry — single source of truth
│   ├── base.py          # Portal config + default (vanilla CKAN) adapter
│   ├── us.py            # data.gov v4 DCAT adapter (auth, cursor paging)
│   ├── uk.py            # dataset-scoped resource lookup
│   └── ca.py            # bilingual metadata normalization
├── tools/               # MCP tools: search, explore, fetch
└── utils/               # formatting (token economy), CSV-head preview
```

**Adding a portal:** for a standard CKAN portal, add one `Portal(...)` entry to `portals/registry.py` — done. For portals with quirks, subclass `PortalAdapter` and override the narrow hooks (`normalize_dataset`, `build_search_params`) or, for non-CKAN APIs, the operation methods themselves (see `us.py` for a full custom adapter).

### TODO / Roadmap

- [ ] **🇨🇳 China portal support** — no unified national open-data API exists; planned as a custom adapter targeting National Bureau of Statistics data (via an existing stats package) with graceful degradation.
- [ ] **🇯🇵 Japan portal support** — e-Gov Data Portal ([data.e-gov.go.jp](https://data.e-gov.go.jp)) is CKAN-compatible and should slot into the existing registry; the richer e-Stat statistics API (app-ID auth, Japanese-language metadata) planned as a dedicated adapter.
- [ ] Offline unit test suite with recorded portal fixtures (respx) for CI
- [ ] Google Cloud Run deployment walkthrough (service is container-ready; script in `deploy/`)
- [ ] US v4 publisher/format search filters (pending API documentation)
- [ ] More CKAN portals (New Zealand [data.govt.nz](https://data.govt.nz) is a near-free addition)
- [ ] Content-type sniffing to rescue mislabeled resource formats

### Security Notes

- This server is **read-only** toward the portals — no tool can create, modify, or delete anything.
- `fetch_resource` and `preview_resource` download from URLs contained in portal metadata; content is size-capped and binary-checked before being returned to the model, but treat fetched content as untrusted input.
- The server itself has no authentication. For remote deployment, put it behind an authenticating proxy or platform-level auth (e.g. Cloud Run with `--no-allow-unauthenticated`).
- Keep `DATAGOV_API_KEY` in `.env` / secret manager — never commit it.

### License

MIT