# Hosted Octocrawl: the API and the remote MCP endpoint

Phase 1 of the roadmap's [PH · Hosted Octocrawl](../ROADMAP.md#ph--hosted-octocrawl): one Cloud Run service, `octocrawl-api`, serves `https://api.octocrawl.dev` (REST) and `https://mcp.octocrawl.dev/mcp` (MCP over Streamable HTTP) from the same process, `packages/mcp/src/hostedApiCli.ts`. It is the API's hosted mode (`packages/api/src/listen.ts`: private addresses, robots overrides, saved logins, handoff, local proxies and non-HTTPS webhooks refused) behind a gate that serves scrape and map to every caller and batch and crawl jobs to accounts ([Batch and crawl jobs](#batch-and-crawl-jobs)), counts every caller, and pins callers without a key to the HTTP lane (`packages/mcp/src/hostedApi.ts`).

| | Keyless | With a key | Signed in (`/mcp-oauth`) |
| --- | --- | --- | --- |
| How | No `Authorization` header | `Authorization: Bearer oc_…` | OAuth sign-in; the client sends `Authorization: Bearer oca_…` to `POST /mcp-oauth` |
| Identity | The address Cloudflare reports, hashed | The key's HMAC | The connection's own key record (`kind: "oauth"`), found from the token's HMAC |
| Pages a day | `W2L_KEYLESS_DAILY` (20) an address, and `W2L_KEYLESS_TOTAL_DAILY` (500) every keyless caller together | the key's `dailyLimit` | the account's credits, at most `W2L_ACCOUNT_DAILY` (500) starts an account, and `W2L_ACCOUNTS_DAILY` (3000) for every account together |
| Lane | HTTP only (`fastMode` pinned, and an `agentHints` line says so when a page needed the browser or the call sent `fastMode: false`; a scrape's `screenshot` or `waitFor` is HTTP 400 naming what the caller lacks) | HTTP, then the browser when the key's `browser` is true | HTTP, then the browser when the connection's record has `browser` and the account has credits for a browser page |
| Tools and routes | `scrape`, `map`; `POST /v1/scrape`, `POST /v1/map`, `GET /v1/scrapes/:id`, `GET /v1/maps/:id` | the same, plus `scrape_product`, and for a key tied to an account the job routes (`POST /v1/crawl`, `POST /v1/batches` and their reads) and the job tools | MCP only: `scrape`, `map`, `scrape_product` with the browser lane, `get_scrape`, `list_active_crawls` and the twelve job tools (`crawl`, `batch_scrape`, their get, wait, cancel and error tools: [Batch and crawl jobs](#batch-and-crawl-jobs)); REST `/v1/*` refuses `oca_` tokens |
| Starts a minute | 10 | 60 | 60 |

Keyless calls from Claude's published egress range (`160.79.104.0/21`, https://platform.claude.com/docs/en/api/ip-addresses; claude.ai, the desktop and mobile apps, the API's MCP connector) share one pool, `W2L_CLAUDE_DAILY` (200) pages a day at 60 starts a minute, under the quota id `<day>-keyless-claude`, since they come from Anthropic's servers and would otherwise split one address's allowance among every Claude user. Claude Code connects from the user's own machine and keeps the per-address allowance. The range is `CLAUDE_EGRESS` in `packages/mcp/src/hostedApi.ts`; check the published list when Anthropic announces a change. The pool is first come, first served: one caller of the API's MCP connector can spend it for everyone until 00:00 UTC, and the per-address limit (120 requests a minute, counted before the caller is known, and twice for an MCP tool call) still applies to each Anthropic address. A key, which the API's MCP connector can send, keeps its own allowance.

MCP directories read `GET /.well-known/mcp/server-card.json` ([Smithery's server card](https://smithery.ai/docs/build/publish)): the server's name and version, `authentication.required: false`, and the tools `/mcp` lists to a caller without a key, built from the same definitions. `tools/list` names only what the caller can use (`scrape_product` only for a key with the browser lane), and every hosted tool carries a `title` and `readOnlyHint: true`, since hosted Octocrawl refuses page actions and logins. Their input schemas list only what hosted serves (`HOSTED_SCRAPE_KEYS` in `packages/mcp/src/hostedApi.ts`): no `actions`, `robotsOverride`, `lane`, `handoff`, `skipTlsVerification`, `access`, `mode: authed` or cache options, no regular-expression checks unless the operator switched them on ([The regular-expression checks](#the-regular-expression-checks)), and without the browser lane no `screenshot` or `waitFor`. A hosted MCP scrape stops at 120 s when it names no `timeout`, as a local one does, and at 180 s at most (REST keeps 300 s; Cloud Run's request timeout is 300 s), and hosted MCP results leave out the call's telemetry and internal ids (`usage`, `channelsTried`, `metadata.scrapeId` and the concurrency fields) and the campaign tags in `agentHints` links, unless `debug` is set, though a connection that lists `get_scrape` keeps `metadata.scrapeId` to read the record back by, and `get_scrape` answers the record without the same telemetry and the client that made it (`origin`); REST keeps them, and the Evidence Record stays whole; the same `scrape` on a server on your computer is not marked read-only, as its actions can click and type; `/.well-known/oauth-protected-resource/mcp-oauth` is the Protected Resource Metadata (RFC 9728) for `/mcp-oauth`, which names `https://octocrawl.dev` as its authorization server; any other `/.well-known` file is 404 `not_found`, so a client on `/mcp` finds no OAuth metadata and connects without signing in. `crawl` and `batch` without an account's key answer 403 `jobs_need_account`; an account's connection lists the twelve job tools, `list_active_crawls` and `get_scrape` (`HOSTED_JOB_TOOLS`, H2 M1–M5 and stage H W0; a scrape's record is kept 10 minutes, on the instance that ran the scrape, so a read that reaches another instance answers `not_found`), with their start options cut to `HOSTED_BATCH_KEYS` and `HOSTED_CRAWL_KEYS` (no `webhook`, which is REST only; `maxCredits` is listed on the hosted start tools only, H2 M6) and waits of at most 120 s, as locally. Everything else (Monitors, logins, `/fc`) answers 403 `hosted_unavailable` with an `agentHints` entry that says to run Octocrawl on your computer. Callers without an account (keyless, and keys issued without one) share `W2L_SITE_DAILY` (1500) pages a day, under the quota id `<day>-site`, and keyless callers together get `W2L_KEYLESS_TOTAL_DAILY` (500) of it, under `<day>-keyless`. Accounts have a day apart from it, so a rush of keyless calls refuses no one who signed in: `W2L_ACCOUNTS_DAILY` (3000) pages for every account together, under `<day>-accounts`, and `W2L_ACCOUNT_DAILY` (500) for each, under `<day>-account-<uid>`. Over an allowance: 429 `quota_exhausted` with `Retry-After` until 00:00 UTC; a caller without an account is told first that signing in is free and gives the month's credits (`/mcp-oauth` for an MCP client, a key from `/signin/` for the API). The use that takes a shared pool (`site`, `keyless`, `keyless-claude`, `accounts`, and the worker's `jobs`) to 70% of its day logs one `hosted_quota_threshold` line; [Alerts](#alerts) mails it. Every request, of any kind, is limited to 120 a minute per address before any key lookup. Nothing is retained: `storeInCache` is pinned off, files are capped at 5 MiB, and scrape and map records and saved files are swept from the task root (Cloud Run's in-memory `/tmp`) after 10 minutes.

## Connect

```sh
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp
codex mcp add octocrawl --url https://mcp.octocrawl.dev/mcp
curl -s -X POST https://api.octocrawl.dev/v1/scrape -H 'content-type: application/json' -d '{"url":"https://example.com","formats":["markdown"]}'
```

With a key, add `--header "Authorization: Bearer oc_…"` (Claude Code), or `"headers": { "Authorization": "Bearer oc_…" }` in a JSON config; the key is never put in the URL. The published packages work against it too: `npx -y @octocrawl/mcp --base-url https://api.octocrawl.dev --token oc_…`, the SDK's `baseUrl` and `token`, and the Python client's `W2L_API_URL` and `W2L_API_TOKEN`.

## Deploy

The runbook for the owner, in the public preview's project (`docs/public-preview.md` created the project, the Artifact Registry repository, the runtime service account `w2l-preview` with `roles/datastore.user`, and the secrets `w2l-quota-hash-key` and `w2l-proxy-secret`); this service reuses all of them. Pass `--project` on every `gcloud` call.

```sh
export W2L_PROJECT_ID='octopus-w2l-20260924-7r7'
export W2L_REGION='asia-southeast1'
export W2L_REPOSITORY='w2l-public-preview'
export W2L_RUNTIME_SA="w2l-preview@${W2L_PROJECT_ID}.iam.gserviceaccount.com"
```

Once per project: the TTL policy that deletes a day's counters two days on (`expireAt`, `packages/mcp/src/hostedApi.ts`), and a budget alert at $40 a month (the instance cap below is the real bound; an alert only tells).

```sh
gcloud firestore fields ttls update expireAt --collection-group=hostedQuotas --enable-ttl --project="$W2L_PROJECT_ID"
gcloud firestore fields ttls update expireAt --collection-group=creditUsage --enable-ttl --project="$W2L_PROJECT_ID"
gcloud firestore fields ttls update expireAt --collection-group=requests --enable-ttl --project="$W2L_PROJECT_ID"
gcloud firestore fields ttls update expireAt --collection-group=hostedLeases --enable-ttl --project="$W2L_PROJECT_ID"
```

Build the image from a clean commit and deploy it without traffic, then move all traffic to it under a new tag; keep the previous tag as the rollback (as the preview's releases):

```sh
test -z "$(git status --porcelain)" || { echo 'Commit and verify a clean source tree before deployment' >&2; exit 1; }
export W2L_SOURCE_SHA="$(git rev-parse HEAD)"
export W2L_IMAGE="${W2L_REGION}-docker.pkg.dev/${W2L_PROJECT_ID}/${W2L_REPOSITORY}/hosted-api:${W2L_SOURCE_SHA}"
gcloud builds submit . --config=cloudbuild.hosted-api.yaml --region="$W2L_REGION" --timeout=20m --substitutions="_IMAGE=${W2L_IMAGE}" --project="$W2L_PROJECT_ID"
```

The first deploy creates the service and sets every variable; gcloud refuses `--no-traffic` on a service that does not exist yet, so the first revision takes the traffic itself and is then tagged:

```sh
export W2L_REVISION_SUFFIX=phase1     # names the revision octocrawl-api-phase1
export W2L_TAG=h1phase1               # one tag per release; the previous tag stays as the rollback
gcloud run deploy octocrawl-api --image="$W2L_IMAGE" --region="$W2L_REGION" --project="$W2L_PROJECT_ID" \
  --service-account="$W2L_RUNTIME_SA" --allow-unauthenticated \
  --cpu=1 --memory=2Gi --concurrency=2 --min-instances=0 --max-instances=3 --timeout=300s --use-http2 \
  --revision-suffix="$W2L_REVISION_SUFFIX" \
  --set-env-vars="W2L_FIRESTORE_PROJECT_ID=${W2L_PROJECT_ID},W2L_PUBLIC_API_ORIGIN=https://api.octocrawl.dev,W2L_KEYLESS_DAILY=20,W2L_SITE_DAILY=1500,W2L_SOURCE_COMMIT=${W2L_SOURCE_SHA}" \
  --update-secrets='W2L_QUOTA_HASH_KEY=w2l-quota-hash-key:latest,W2L_PROXY_SECRET=w2l-proxy-secret:latest'
gcloud run services update-traffic octocrawl-api --region="$W2L_REGION" --project="$W2L_PROJECT_ID" \
  --update-tags="${W2L_TAG}=octocrawl-api-${W2L_REVISION_SUFFIX}"
gcloud run services describe octocrawl-api --region="$W2L_REGION" --project="$W2L_PROJECT_ID" --format='value(status.url)'
```

A later release deploys the new image without traffic, with `--update-env-vars=W2L_SOURCE_COMMIT=…` alone (`--set-env-vars` replaces them all) and the request timeout the code assumes (`HOSTED_REQUEST_TIMEOUT_MS`, 300 s since stage H W0; a revision keeps the last one set), then moves all traffic to it under a new tag:

```sh
gcloud run deploy octocrawl-api --image="$W2L_IMAGE" --region="$W2L_REGION" --project="$W2L_PROJECT_ID" \
  --revision-suffix="$W2L_REVISION_SUFFIX" --no-traffic --timeout=300s --use-http2 --update-env-vars="W2L_SOURCE_COMMIT=${W2L_SOURCE_SHA}"
gcloud run services update-traffic octocrawl-api --region="$W2L_REGION" --project="$W2L_PROJECT_ID" \
  --to-revisions="octocrawl-api-${W2L_REVISION_SUFFIX}=100" --update-tags="${W2L_TAG}=octocrawl-api-${W2L_REVISION_SUFFIX}"
```

`--use-http2` has Cloud Run speak HTTP/2 cleartext to the container, the only way it passes a client's disconnect on ("When you use HTTP/1.1 on Cloud Run, client disconnect events are not propagated to the Cloud Run container", [troubleshooting](https://docs.cloud.google.com/run/docs/troubleshooting)): a scrape whose client went away then stops, frees its browser lease and is not charged. Without it a cancelled scrape ran to its end and was charged (2026-10-28, revision `octocrawl-api-w0-4414d88`). The service answers HTTP/1 and h2c on the same port (`packages/mcp/src/h2c.ts`), so a revision with or without the flag serves; the Worker passes its client's disconnect on as well (`request.signal`, the `enable_request_signal` flag in its `wrangler.toml`).

`--concurrency=2` and `--max-instances=3` bound the spend: at most six pages are read at once, and a browser-lane page holds one slot for its whole run. Raise them only from the recorded cost per page (below). A key's scrapes also run one at a time in the browser (ROADMAP PH item 1; a job's pages take no such lease): just before a browser rung starts, the call takes a lease, `hostedLeases/browser-{key HMAC}`, in one commit with an update-time precondition, holds it to the end of the call and deletes it after, so the bound holds across instances. A page HTTP reads takes no lease, nor does a scrape that sends `fastMode: true`, nor any map. If another scrape holds the lease when a page needs it, the rung does not run: the answer is what HTTP read, with a `browser_busy` warning and hints that name a batch job first where the account can run one, since jobs take no such lease. It is priced as any such answer: 1 credit for a page HTTP read, nothing for one that failed or was blocked. A lease store that fails leaves the rung out the same way, with a `browser_unavailable` warning and a `hosted_lease_failed` line in the log. A scrape that starts in the browser, with `waitFor` above 0 or a screenshot, has no HTTP read to keep, so it takes the lease before it runs, and a held lease is `429 browser_busy` with `Retry-After` (at most 10 s: when to ask again, not when the running call ends, which can take the whole 300 s). A Playground run and a call for the json model fallback take the lease before they run too. A lease an instance never released (a crash) is taken over once its `expireAt` passes, 310 s after it was taken (the request timeout and 10 s).

The Worker in `cloudflare/hosted-api-proxy/` answers on `api.octocrawl.dev` and `mcp.octocrawl.dev` (Workers custom domains, no DNS record for Cloud Run) and forwards to the run.app address with the shared secret, so the service can believe `CF-Connecting-IP`. Set `ORIGIN_URL` in its `wrangler.toml` to the URL the describe command printed, then:

```sh
cd cloudflare/hosted-api-proxy
npx wrangler secret put PROXY_SECRET   # the value of w2l-proxy-secret
npx wrangler deploy
```

Verify from outside:

```sh
curl -s https://api.octocrawl.dev/health
curl -s -X POST https://api.octocrawl.dev/v1/scrape -H 'content-type: application/json' -d '{"url":"https://example.com"}' | head -c 300
curl -s -o /dev/null -w '%{http_code}\n' -X POST https://api.octocrawl.dev/v1/crawl -H 'content-type: application/json' -d '{"url":"https://example.com"}'   # 403 without an account's key
curl -s -o /dev/null -w '%{http_code}\n' -X POST https://api.octocrawl.dev/v1/scrape -H 'authorization: Bearer nope' -H 'content-type: application/json' -d '{}'   # 401
claude mcp add --transport http octocrawl https://mcp.octocrawl.dev/mcp && claude mcp list
```

To roll back, `gcloud run services update-traffic octocrawl-api --to-tags=<previous tag>=100`.

## Batch and crawl jobs

Hosted batch (1–1,000 URLs) and crawl (at most 500 pages) jobs (ROADMAP PH stage H1, [plan](launch/2026-10-19-h1-hosted-jobs-plan.md), [task list](h1-hosted-jobs-tasks.md)) run for accounts' keys only, a slice at a time: the API (`hostedJobRoutes.ts`) writes the job to Firestore (`hostedJobs`) and queues its first slice on Cloud Tasks; the worker service `octocrawl-jobs` (`hostedJobWorker.ts`, on the same image) runs each slice and writes its pages to GCS. Both services turn jobs on when all five variables are set (`hostedJobsCloud.ts`):

| Variable | Value |
| --- | --- |
| `W2L_FIRESTORE_PROJECT_ID` | the project |
| `W2L_JOBS_BUCKET` | `octocrawl-jobs-${W2L_PROJECT_ID}` |
| `W2L_JOBS_QUEUE` | `asia-southeast1/octocrawl-jobs` |
| `W2L_JOBS_WORKER_URL` | `https://octocrawl-jobs-c7fpmz73tq-as.a.run.app` | A revision's tag URL (`https://<tag>---octocrawl-jobs-…`) works too, on a tagged API and tagged worker that both carry it, to exercise a worker revision before traffic moves; the task's token still names the service URL as its audience (`workerAudience`), since Cloud Run refuses a tag URL there with 401. The template keeps the value, so the next deploy sets it back. |
| `W2L_JOBS_TASKS_ACCOUNT` | `octocrawl-jobs-tasks@${W2L_PROJECT_ID}.iam.gserviceaccount.com` |

The global switch is the Firestore document `hostedConfig/jobs`, field `enabled`; absent means on, and both services read it at most every 30 s.

The same document's `allowedUids`, an array of account uids, makes jobs invite only (H2 R2). It held Howard's account alone from 2026-10-21 and was removed on 2026-10-22, so every account may start jobs now; the rest of this paragraph describes the switch for when it is set again. When the field is absent, every account may start jobs. When it exists, only the listed accounts may; an empty array lets no one. It is read together with `enabled`; a failed read keeps the last value, and with nothing read yet lets no one in.
- **Starts.** A start from another account answers 403 `jobs_invite_only`, on REST and through MCP, which calls the same gate.
- **MCP tool lists.** An MCP connection lists the job tools only to an allowed account. Another account gets a hint that hosted jobs are not open to it right now (since 2026-10-22 every account is allowed, so the hint shows only while the switch is off).
- **Reads and cancels.** These stay open on REST, so an account taken off the list can still read and cancel its jobs there. Over MCP the tools are gone for it.
- **Jobs already queued.** A job an account queued before it left the list still runs when its turn comes.
- **Replays.** A start replayed by its idempotency key is refused like a new one.

```sh
# Invite only, for the accounts named (each is users/{uid}); remove the field to open jobs to every account.
T=$(gcloud auth print-access-token); curl -s -X PATCH -H "authorization: Bearer $T" -H "x-goog-user-project: $W2L_PROJECT_ID" -H 'content-type: application/json' \
  "https://firestore.googleapis.com/v1/projects/$W2L_PROJECT_ID/databases/(default)/documents/hostedConfig/jobs?updateMask.fieldPaths=allowedUids" \
  -d '{"fields":{"allowedUids":{"arrayValue":{"values":[{"stringValue":"<uid>"}]}}}}'
``` Off, a start answers 503 `jobs_paused` and a running job's slice stops at its next page and waits five minutes.

The resources, made once on 2026-10-20:

```sh
gcloud services enable cloudtasks.googleapis.com iamcredentials.googleapis.com --project="$W2L_PROJECT_ID"
gcloud iam service-accounts create octocrawl-jobs-tasks --project="$W2L_PROJECT_ID"
# The runtime account (w2l-preview) queues tasks, delivers them as octocrawl-jobs-tasks, and signs export links as itself.
gcloud projects add-iam-policy-binding "$W2L_PROJECT_ID" --member="serviceAccount:${W2L_RUNTIME_SA}" --role=roles/cloudtasks.enqueuer
gcloud iam service-accounts add-iam-policy-binding "octocrawl-jobs-tasks@${W2L_PROJECT_ID}.iam.gserviceaccount.com" --member="serviceAccount:${W2L_RUNTIME_SA}" --role=roles/iam.serviceAccountUser --project="$W2L_PROJECT_ID"
gcloud iam service-accounts add-iam-policy-binding "$W2L_RUNTIME_SA" --member="serviceAccount:${W2L_RUNTIME_SA}" --role=roles/iam.serviceAccountTokenCreator --project="$W2L_PROJECT_ID"
# Files: private, deleted 30 days after they were written as a backstop (the worker deletes a job's files when it expires).
gcloud storage buckets create "gs://octocrawl-jobs-${W2L_PROJECT_ID}" --location=asia-southeast1 --uniform-bucket-level-access --public-access-prevention --project="$W2L_PROJECT_ID"
gcloud storage buckets update "gs://octocrawl-jobs-${W2L_PROJECT_ID}" --lifecycle-file=lifecycle.json --project="$W2L_PROJECT_ID"   # {"rule":[{"action":{"type":"Delete"},"condition":{"age":30}}]}
gcloud storage buckets add-iam-policy-binding "gs://octocrawl-jobs-${W2L_PROJECT_ID}" --member="serviceAccount:${W2L_RUNTIME_SA}" --role=roles/storage.objectAdmin --project="$W2L_PROJECT_ID"
# Retries without a limit, backing off from 10 s to 10 min: a held lease answers 503 until it expires (16 min), and a job
# with no delivery left would stay running. A slice that fails every time is retried every 10 min (hosted_job_delivery_failed).
gcloud tasks queues create octocrawl-jobs --location=asia-southeast1 --max-concurrent-dispatches=3 --max-dispatches-per-second=1 --max-attempts=-1 --min-backoff=10s --max-backoff=600s --max-doublings=5 --project="$W2L_PROJECT_ID"
for spec in hostedJobs:deleteAt hostedJobIdempotency:expireAt creditSettles:expireAt; do gcloud firestore fields ttls update "${spec#*:}" --collection-group="${spec%%:*}" --enable-ttl --async --project="$W2L_PROJECT_ID"; done
```

The worker is deployed from the hosted API's image, not public, one slice an instance; the queue's account may invoke it:

```sh
gcloud run deploy octocrawl-jobs --image="$W2L_IMAGE" --region="$W2L_REGION" --project="$W2L_PROJECT_ID" \
  --service-account="$W2L_RUNTIME_SA" --no-allow-unauthenticated --ingress=all \
  --cpu=1 --memory=4Gi --concurrency=1 --min-instances=0 --max-instances=3 --timeout=900s \
  --command=node --args=packages/mcp/dist/hostedJobsCli.js --revision-suffix="$W2L_REVISION_SUFFIX" --env-vars-file=jobs-env.yaml \
  --update-env-vars=NODE_OPTIONS=--max-old-space-size=3072
gcloud run services add-iam-policy-binding octocrawl-jobs --region="$W2L_REGION" --project="$W2L_PROJECT_ID" \
  --member="serviceAccount:octocrawl-jobs-tasks@${W2L_PROJECT_ID}.iam.gserviceaccount.com" --role=roles/run.invoker
```

4 GiB and a 3 GB JavaScript heap since 2026-10-20 (`octocrawl-jobs-w2-mem4g`): at 2 GiB a 30-page crawl of docs.python.org/3/ ran the heap out of memory (`FATAL ERROR: Ineffective mark-compacts near heap limit`, about 1 GB) on every delivery, until the change.

The shape of a slice since 2026-11-07 (`hostedJobRoutes.ts` `JOB_WORKER_COUNT`, `JOB_BROWSER_PAGES`; `hostedJobWorker.ts` `WORKER_LIMITS`): its engine reads 8 pages at once and renders at most 2 of them in a browser (`browserPagesAtOnce`: a worker whose page needs the browser waits for one of the two slots while HTTP pages flow on the others), it takes up to 100 hosts, leases asked for ten at a time, and the instance keeps one Chromium across the slices it runs; a page waiting its turn for a render slot does not spend its own `timeout` waiting (its deadline stands still, never past the slice's), and its trace opens with a `slot_wait` event saying how long it waited (`browserKeeper`, closed after ten minutes without a page), so a slice's first browser page no longer pays for a launch. Until then a slice read 2 pages at once, took 20 hosts, and launched Chromium for each slice. A slice reserves its pages and a headroom of 7 (the pages a drain may store past the limit, one per worker but one) on the day's pools and the account's credits; a pool with no room for the headroom gives the pages alone, and the slice then runs as many workers as the headroom it got allows, so it never stores a page it did not reserve. Whether the new shape is faster on real lists is to be measured on a tagged revision before traffic moves (the 124-site Amedcon batch of 2026-11-05 took 4 min 53 s at the old shape).

The API takes the same variables with `--update-env-vars` on its next release. Verified on 2026-10-20 (worker `octocrawl-jobs-w1-5141c6d`, API `octocrawl-api-jobs-5141c6d`, tag `h28jobs`, rollback `h27shares`), with a key issued for the purpose and revoked after: a 6-URL batch and a 5-page crawl through the real queue, worker and bucket each completed in one slice in about 11 s, paged back, exported by a signed link, were charged 6 and 4 credits once each (`creditSettles`), and had their expiry queued 7 days on. A request to the worker without a token is 403 from Cloud Run, and with another account's token 401 from the worker. The switch was left off.

`GET /v1/jobs?limit=&before=` lists an account's jobs, newest first, with the fields of a job's status (H2 P1, for the dashboard and MCP). `limit` is 1–50 (default 20); `before` is the `nextBefore` of the previous page, a job id, and `hasMore` says whether another page follows. A call without an account's key is 403. The list is the last 200 jobs the account started, kept in its slots document (`hostedJobWorkspaces/{workspaceId}.slots.recent`), so it needs no Firestore index; a job leaves it when its record is deleted (7 days of results and 30 more of record after it finishes). Jobs started before the list existed are not in it.

`GET /v1/jobs/:id/export` answers, for a job that has ended, `{ url, expiresInSeconds: 900 }`: a V4-signed GCS link to the job's export, every page as one gzip JSONL file (a page a line, as the items route lists them), written by the worker when the job ends (`finish` in `hostedJobWorker.ts`); 409 `job_running` before that. A finished job's status (`GET /v1/batches/:id`, `GET /v1/crawl/:id`, and the MCP tools `get_batch`, `wait_batch`, `get_crawl`, `wait_crawl`) carries the same link as `export: { url, expiresInSeconds: 86400 }`, signed for a day so a person who comes back to the chat later still finds it alive; the status read again gives a fresh one. The link is a capability: whoever holds it can download the file until it expires. The list `GET /v1/jobs` signs none. The MCP answer adds an `agentHints` line that says what the link is and that `get_batch_items` reads a few pages at a time; a local server writes `results.jsonl` and `results.csv` beside the task instead and names them in `resultFiles`. A finished batch's status also carries, before that line, one `agentHints` line a reason for its failed pages (pages `fastMode` kept from the browser lane, robots.txt refusals, challenges, timeouts, and one line for the rest), built from `failures` on the job document, which each slice's commit folds its charges into (`hostedJobWorker.ts`), so no status read opens a segment; a job started before this change has no `failures` and no such lines.

`GET /v1/crawl/active` lists the account's crawl jobs that are `queued` or `running`, oldest first, as `{ crawls: [{ id, url, status, startedAt, pagesFetched }] }`: a local server's list of the crawls it runs, in a job's statuses and without the options each was started with (stage H W0; the MCP tool `list_active_crawls`). It reads the same list, and a call without an account's key is 403.

A start checks its request on an engine set up like the worker's (`jobEngineOptions` in `hostedApi.ts`; the engine's `checkCrawlStart` and `checkBatchStart`). It refuses with 400 whatever the job's first slice would refuse, and names the problem: a `maxFileBytes` above 5 MiB, a selector Octocrawl does not match in `includeTags`, `excludeTags`, an attributes format or a list format, or `fastMode` beside a `screenshot` or a `waitFor` above 0. It also refuses a check whose format is missing (`tableRowsMin` without `tables`, a Markdown check without `markdown`) and, unless the operator switched them on ([The regular-expression checks](#the-regular-expression-checks)), a regular-expression check. Such a start used to answer 202, and the job then failed at its first slice. If the worker's engine still refuses a stored request at the first slice, the job fails at once with `failureReason: request_refused`, and the worker logs the message as `hosted_job_request_refused`. Before, the slice was retried until the job ended `delivery_failed`.

### Job webhooks

A start may name a webhook for the job's end (H2 W1–W4): `"webhook": "https://…"` or `"webhook": { "url": "https://…", "events": ["completed", "failed", "cancelled"], "metadata": { … } }`.

- **What it accepts.** The url must be https to a public address. `events` may be any of the three end events, and all three is the default. `headers` and `secretEnv`, which a local server takes, are refused: a hosted job signs with a secret of its own instead.
- **The secret.** The start's answer carries `webhook.secret`, `whsec_` and 43 characters. A start replayed by its idempotency key carries the same secret again, so a client that lost the first answer can still verify deliveries. Status and the list show the webhook's `url`, `events` and `delivery`, never the secret. Keep it the way you keep a key.
- **When it is sent.** When the job ends, the worker queues one Cloud Tasks delivery, `{jobId}-webhook`, to `/internal/jobs/:id/webhook`. That delivery posts the event once.
- **The body.** It is a local job's terminal event (`w2l.job-event/v1`): `eventId` `{jobId}:{status}`, `sequence` (the pages read plus one), `jobId`, `jobKind`, `event`, `at` (when the job finished), `metadata`, `report` (the job's status as `GET /v1/batches/:id` or `/v1/crawl/:id` shows it), and `error` on a failure. Hosted sends no `started` or `page` events.
- **Headers.** `x-w2l-event-id`, `x-w2l-event-version` (the sequence), `x-w2l-delivery-id`, `x-w2l-timestamp` (milliseconds since the epoch), and `x-w2l-signature`. The signature is `sha256=` followed by the hex HMAC-SHA256, under the secret, of `{x-w2l-timestamp}.{raw body}`.
- **Retries.**
  - A 2xx is delivered.
  - A 408, 425, 429, 5xx or no answer within 30 s is retried on the queue's backoff (10 s doubling to 10 min), up to 8 attempts, about 20 minutes. After the eighth it is recorded as failed.
  - Any other answer, or an address the egress policy refuses, is failed at once.
  - Each outcome is recorded on the job as `webhook.delivery`, `{ state, attempts, httpStatus, error, at }`, and logged as `hosted_job_webhook`.
- **Over MCP.** The MCP tools do not offer a webhook, and a call that passes one anyway is refused, because the secret would land in the agent's transcript.
- **A job never queued.** A job whose first slice could not be queued fails with `not_queued`, and how its end is handled depends on when that happened.
  - **At its own start:** the start answered 503, so nothing is posted, and `webhook.delivery` says why. A start replayed by its idempotency key gets this job and its secret back.
  - **After it waited in the queue:** it is finished at once by a delivery of its own (`{jobId}-finish`), which posts its end. If the queue refuses that delivery too, `webhook.delivery` says it was not sent.
  - A "not sent" record never replaces a delivery under way or done.
- **Cancel.** A job cancelled while no slice holds its lease is finished at once by `{jobId}-finish`, so that a slice waiting in the queue with a delay (until the next UTC day, for example) does not hold back its export and webhook. A slice that holds the lease finishes the job when it commits, with its pages counted. If the finish delivery is refused, the waiting slice still finishes the job later.

To verify a delivery, check the signature over the raw body before parsing it, compare in constant time, and reject an old timestamp:

```js
import { createHmac, timingSafeEqual } from 'node:crypto'

export function verifyOctocrawl(rawBody, headers, secret, toleranceMs = 5 * 60_000) {
  const timestamp = headers['x-w2l-timestamp']
  const expected = Buffer.from(`sha256=${createHmac('sha256', secret).update(`${timestamp}.${rawBody}`).digest('hex')}`)
  const given = Buffer.from(String(headers['x-w2l-signature'] ?? ''))
  return given.length === expected.length && timingSafeEqual(given, expected) && Math.abs(Date.now() - Number(timestamp)) <= toleranceMs
}
```

```python
import hashlib, hmac, time

def verify_octocrawl(raw_body: bytes, headers, secret: str, tolerance_ms: int = 300_000) -> bool:
    timestamp = headers.get("x-w2l-timestamp", "")
    expected = "sha256=" + hmac.new(secret.encode(), timestamp.encode() + b"." + raw_body, hashlib.sha256).hexdigest()
    fresh = timestamp.isdigit() and abs(time.time() * 1000 - int(timestamp)) <= tolerance_ms
    return fresh and hmac.compare_digest(expected, headers.get("x-w2l-signature", ""))
```

Deliveries can repeat (Cloud Tasks delivers at least once), so a receiver keys on `x-w2l-event-id`.

## Keys

Keys are made on the dashboard after sign-in (Accounts and credits, below); this section is the operator's way to issue one by hand. A key is printed once; only its HMAC under `w2l-quota-hash-key` is stored, as the id of a document in Firestore's `hostedApiKeys` collection with `enabled`, `plan`, `dailyLimit`, `browser` and `label`. A start (`POST /v1/scrape`, `POST /v1/map`, and an MCP tool call, which goes through the same gate) reads the key's document fresh, so a key revoked by the script stops at its next start (ROADMAP PH item 2). What starts no work (the MCP tool list, reading a result by id) uses a lookup cached for a minute.

```sh
export W2L_PROJECT_ID='octopus-w2l-20260924-7r7'
node scripts/hosted/issue-key.mjs issue --label 'ada@example.org' --daily 1000 --browser
node scripts/hosted/issue-key.mjs list
node scripts/hosted/issue-key.mjs revoke <digest>
```

The script calls Firestore with Node's `fetch`, which ignores `HTTPS_PROXY`. On a machine that reaches Google only through a proxy, set `NODE_USE_ENV_PROXY=1` (Node 24 and later) so `fetch` uses it; `gcloud` honours the proxy variables on its own. Without it the call ends in `ConnectTimeoutError` against `firestore.googleapis.com` (seen 2026-10-06).

The key is printed once. Keep it somewhere ignored by git, such as `.w2l/hosted/`, with owner-only permissions, until it has been sent.

## Accounts and credits

A key that names an account (`uid` on its `hostedApiKeys` document) spends that account's credits instead of a daily limit. Decided 2026-10-11 (ROADMAP PH); `packages/mcp/src/hostedCredits.ts` holds the rules.

- **The grant.** Every account gets `W2L_MONTHLY_CREDITS` (1,000) credits each month. The grant resets at 00:00 UTC on the 1st and nothing rolls over. It is spent before bought credits.
- **What a call costs**, settled from its result:

  | Result | Credits |
  | --- | --- |
  | A page read over HTTP | 1 |
  | A page read on a browser lane | 5 |
  | A page that does not exist (HTTP 404 or 410) | 1 |
  | A `map` that returned URLs | 1 |
  | The json model fallback, on a page whose json called the model ([below](#the-json-model-fallback)) | +2 |
  | Anything else: blocked by a check, a challenge or robots.txt, nothing read, an invalid request, a server error | 0 |

- **How it is charged.** A call reserves 1 credit before it runs, and a scrape that reaches the browser reserves the other 4 of a browser page when its rung has the lease, so a reservation of a browser page always comes with the lease. A scrape that starts in the browser reserves all 5 at once, and 2 more for the json model fallback. An account short of the 4 does not run the rung: the answer is what HTTP read, with a `browser_credits` warning. What its other calls have reserved counts against it until they end, so on an account with few credits, parallel scrapes can each go without the browser, and a retry alone reads the page. An account with nothing to reserve while its own key's browser holds the lease is told `429 browser_busy`, not that its credits are used up. When a call ends, it gives back what its result did not cost.
- **Running short.** An account with at least 1 credit but fewer than 5 reads over HTTP alone for that call, where a screenshot or a `waitFor` is refused. With none left it is refused: `429 quota_exhausted`, with `Retry-After` counting down to the 1st, and a hint naming hello@octocrawl.dev for more. A job whose credits run out mid-way is not failed: it waits (`paused` with `reason: "credits"` on its status, the credits a page needs as `limit`, the 1st as `resumesAt`), asks again every 6 hours, and goes on by itself once credits are there (a job started before 2026-11-07 failed with `credits_exhausted` instead). A job waiting this way keeps one of the account's two running slots.
- **More credits by hand.** `CreditsStore.topUp(uid, amount, id)` adds bought credits once for an id (`creditTopUps/{id}` records it; `credits/{uid}.purchased` is incremented server-side). From a shell, the same two writes: `T=$(gcloud auth print-access-token); curl -s -X POST -H "authorization: Bearer $T" -H "x-goog-user-project: $W2L_PROJECT_ID" -H 'content-type: application/json' "https://firestore.googleapis.com/v1/projects/$W2L_PROJECT_ID/databases/(default)/documents:commit" -d '{"writes":[{"update":{"name":"projects/'$W2L_PROJECT_ID'/databases/(default)/documents/creditTopUps/<id>","fields":{"uid":{"stringValue":"<uid>"},"amount":{"integerValue":"<n>"}}},"currentDocument":{"exists":false}},{"update":{"name":"projects/'$W2L_PROJECT_ID'/databases/(default)/documents/credits/<uid>","fields":{}},"updateMask":{"fieldPaths":[]},"updateTransforms":[{"fieldPath":"purchased","increment":{"integerValue":"<n>"}}]}]}'`. A paused job picks them up at its next try. `W2L_TRIAL_CREDITS` (API and worker; default 0, off) adds that many credits once to an account the first time one of its calls or jobs finds the credits used up, as `trial-{uid}`; the amount is Howard's call.
- **What the response says.** A scrape or map the gate answered says `X-Credits-Used` and `X-Credits-Remaining`. An error raised inside (an invalid request, a server error) costs nothing but carries neither header. An MCP tool call spends the same way, through the same gate.
- **Firestore.**
  - `credits/{uid}` holds `month`, `grant` and `purchased`. An account's first balance is written only while `users/{uid}` exists (the same commit checks it): a deleted account (public-preview auth.ts) cannot start a new one through a key the API still has cached, and such a call is `401 unauthorized`. A key issued by hand with `--uid` needs that account to have signed in once.
  - A reservation updates that document and the accounts' daily counters (`hostedQuotas/{day}-accounts` and `hostedQuotas/{day}-account-{uid}`) in one commit with update-time preconditions, so two instances cannot overspend.
  - The refund and the day's usage (`creditUsage/{uid}-{day}`: `http`, `browser`, `not_found`, `map`, `unread`, `model`, `job_pages`, `credits`; `model` counts the pages whose json called the model, and `job_pages` the pages a job read, each beside their own kind) are server-side increments. The usage documents expire after 90 days (the TTL policy above).
  - A hosted job leaves one request line when it ends, not one a page, so that a 1,000-page job is one write and one row on Activity: `endpoint` `batch` or `crawl`, the key, its start URL or first URL, its status and reason, how long it ran and its credits (`kind` `job`), and `job`: its id, the URLs asked for, the pages read, the errors among them and its pages by outcome. The line's id comes from the job's end and id, so a later delivery writes the same line again. It is written apart from the credits, which the job's slices settled; a line that cannot be written logs `hosted_job_line_failed` and holds nothing up. `job_pages` on the day's usage counts the pages read by jobs, beside their kinds, and the dashboard takes them off before it checks that every call has a line.
  - The same commit writes the call's request line, `requestLog/{uid}/requests/{id}`: endpoint, key HMAC, the URL asked for and the final URL, lane, status, HTTP status, reason, robots.txt decision, the scrape's readiness state (`readiness`, since 2026-11-05: a `success` the site answered with another page is `not_served`, which the dashboard shows as "another page" rather than as a read), the Evidence Record's `rawSha256`, a map's link count, the format names a scrape asked for (`formats`, empty when it named none, absent on a map), how long the call took (`ms`), what it was counted as and its credits, with `modelCredits` (the model's part of them) when the scrape asked for the json model fallback. Never the page's content. The id is `9999999999999 − epoch ms` and a random tail, so Firestore's default order lists the newest first without an index. Lines expire 30 days on (`expireAt`, the TTL policy above), and the dashboard reads them through `GET /api/account/activity` (packages/public-preview/src/account.ts).
- **Logs.** Every key's start that passed its allowance or reserved its credits logs `hosted_usage` with the first 16 characters of the key's HMAC (never the key; `playground:` and the uid's first characters for the dashboard's Playground, below), its plan, the account id for an account's key, the endpoint and the UTC day: the per-key, per-day count for keys issued by hand as well (ROADMAP PH item 2). Every call that reserved credits logs `hosted_credits` with the account id, what it was counted as, and what it reserved and spent. A reservation or settle that fails logs `hosted_credits_reserve_failed` or `hosted_credits_settle_failed`. A commit that timed out may still have applied, so check `credits/{uid}` and `creditUsage` before giving anything back by hand. Every keyless start logs `hosted_keyless_usage` with the first 16 characters of the address's HMAC (the daily counter's; `null` and `pool: "claude"` for Claude's connectors), the endpoint and the UTC day, and every MCP `initialize` logs `hosted_mcp_initialize` with the endpoint, the caller's kind (`keyless`, `key`, `oauth`) and the `clientInfo` name and version the client gave, as printable text of at most 64 characters. `scripts/hosted/funnel.mjs` counts active keyless addresses and connections by client from these lines, and sign-ups by `users/{uid}.firstSource`, which the site writes from its `w2l_src` cookie on an account's first sign-in (packages/public-preview/src/site.ts).
- **Edge.** A call that runs across 00:00 UTC on the 1st gives its refund to the new month's grant, at most 4 credits over it.
- **The dashboard's Playground.** `/dashboard/scrape/` and `/dashboard/map/` on octocrawl.dev run a scrape or a map for the signed-in account without a key. The site's server (`POST /api/account/run`, packages/public-preview/src/account.ts) calls `/v1/scrape` or `/v1/map` with `X-W2L-Site-Account: {uid}.{expires ms}.{HMAC}`, signed with `W2L_QUOTA_HASH_KEY` and good for two minutes. The gate checks it without a store (`verifySiteAccount`), so it is not counted per address, and treats the call as the account's: same price, same reservation, same `X-Credits-*` headers, same request line, under the key id `playground:{uid}`. A header that does not verify is `401 unauthorized`. A run does not count against the account's five keys. Its scrape takes a browser-lane lease of its own, `browser-playground:{uid}`, taken before the run starts, so a second run started before the first ends is `429 browser_busy`, with the wait as its only hint: the dashboard shows a refusal's first hint, not a result's, and offers no way past.

Keys for an account are made on octocrawl.dev's dashboard (`/dashboard/keys/`) after sign-in (docs/public-preview.md, Sign-in). One can also be issued by hand, for a test account:

```sh
node scripts/hosted/issue-key.mjs issue --label 'test account' --uid test-account-1 --browser
```

## The json model fallback

`modelFallback: true` on a json format calls a model for the required fields a page's own data left empty ([JSON extraction](json-extraction.md)). On hosted Octocrawl it runs on Octocrawl's Anthropic account, for accounts only. Decided by Howard on 2026-10-24 (ROADMAP P5); `packages/mcp/src/hostedModel.ts` holds the rules.

- **The model.** Claude Haiku 5.5 (`claude-haiku-5-5`) through the native provider (`W2L_EXTRACT_PROVIDER=anthropic`), at the code's default effort (`low`, `CLAUDE_DEFAULT_EFFORT` in `packages/api/src/structured.ts`; no `W2L_EXTRACT_EFFORT`).
- **The price.** 2 credits more for each page on which the model was called, as `json.modelUsage.attempts` shows it: one call or two (a repair) cost the same. Nothing more when `modelFallback` is off, the json was complete without the model, or a refusal below kept the model from being called. A scrape reserves the 2 with the page, before it runs, and gives them back when no call was made. An account without the 2 on top of the page's own price reads the page without the model; a job's slice does the same. A keyless caller and a key issued by hand have no credits, so their pages never call the model.
- **The daily cap.** Every hosted call together may spend `dailyCapUsd` (default $5) a UTC day on the model. Before a page's first call its gate reserves the most the page's calls can cost: both attempts, one input token per byte sent (the page's Markdown, the schema, the prompt and the values already read, plus 8 KiB for the system prompt, the format's framing and a repair, whose text a gated page cuts to 1,000 characters; this assumes a token covers at least one byte of UTF-8) and 16,000 output tokens each, at Claude Haiku 5.5's published prices. For a 30 KB page that is about $0.024 against a measured $0.0011 a call (19 calls cost $0.0208 in `research/accuracy/runs/2026-10-24-f2-live-v2-453b2105.md`): it is held only while the call runs. The reservation is a commit to `hostedModelSpend/{day}` (`microUsd`, micro-USD) with an update-time precondition, so two instances cannot pass the cap together. After the call it is replaced by what the calls cost (`modelUsage.externalCostUsd`), or given back when none was made. A cost that is not known (a call that failed after it was sent) keeps the whole reservation. The day's document expires 90 days on (`expireAt`).
- **The switch.** The Firestore document `hostedConfig/model`: `enabled` (boolean) and `dailyCapUsd` (number, optional; 0 admits nothing). Both services read it at most every 30 s. While the document is absent, or `enabled` is anything but `true`, the fallback is off, so a deploy alone turns nothing on. A failed read keeps what was read last; with nothing read yet it is off.
- **A refusal.** A switch that is off, a cap that is reached, a caller without credits for it, or a budget that could not be read calls no model and charges nothing more. The json stays as deterministic extraction read it, `incomplete`, with a `model_unavailable` issue that says which.
- **What the response says.** A scrape that asked for the fallback carries `X-Credits-Model` (2 or 0), within `X-Credits-Used`. Its json's `modelUsage` gives the model, the calls, the tokens and their price in USD. A job's `credits.used` counts the 2 with each page's own price, and its start's `estimatedCredits` counts 2 more a page.
- **Logs.** `hosted_model_call` gives the day, the calls, the tokens, the cost and the reservation; `hosted_model_cap_reached` the day's spend and the cap; `hosted_model_budget_failed` and `hosted_model_settle_failed` a store error (a failed settle keeps the reservation, so the day counts more than was spent, never less). `hosted_credits` adds `model`, the credits the model's part cost. No line carries the page, the prompt, the schema, the model's answer or the key.
- **Local and self-hosted Octocrawl** do not change: the model is called whenever `W2L_EXTRACT_*` configure one, with no gate, no cap and no credits.

### Turning it on

Steps for Howard to approve; nothing here has been run. Howard makes the Anthropic API key in the Anthropic Console on Octocrawl's organization, with its payment method bound, and keeps it in a git-ignored file with owner-only permissions (for example `.w2l/hosted/anthropic-api-key`, no trailing newline).

```sh
# 1. The secret, readable by the runtime account both services run as.
gcloud secrets create w2l-anthropic-api-key --replication-policy=automatic --data-file=.w2l/hosted/anthropic-api-key --project="$W2L_PROJECT_ID"
gcloud secrets add-iam-policy-binding w2l-anthropic-api-key --member="serviceAccount:${W2L_RUNTIME_SA}" --role='roles/secretmanager.secretAccessor' --project="$W2L_PROJECT_ID"

# 2. The day's spend expires 90 days on.
gcloud firestore fields ttls update expireAt --collection-group=hostedModelSpend --enable-ttl --async --project="$W2L_PROJECT_ID"

# 3. The switch, written off first: the deploy below then enables nothing.
T=$(gcloud auth print-access-token); curl -s -X PATCH -H "authorization: Bearer $T" -H "x-goog-user-project: $W2L_PROJECT_ID" -H 'content-type: application/json' \
  "https://firestore.googleapis.com/v1/projects/$W2L_PROJECT_ID/databases/(default)/documents/hostedConfig/model" \
  -d '{"fields":{"enabled":{"booleanValue":false},"dailyCapUsd":{"doubleValue":5}}}'

# 4. A release of both services from a commit that has this code, as any release (Deploy above; Batch and crawl jobs for
#    octocrawl-jobs), with the model's variables and the key added to what each has:
gcloud run deploy octocrawl-api --image="$W2L_IMAGE" --region="$W2L_REGION" --project="$W2L_PROJECT_ID" \
  --revision-suffix="$W2L_REVISION_SUFFIX" --no-traffic --timeout=300s --use-http2 \
  --update-env-vars="W2L_SOURCE_COMMIT=${W2L_SOURCE_SHA},W2L_EXTRACT_PROVIDER=anthropic,W2L_EXTRACT_MODEL=claude-haiku-5-5" \
  --update-secrets='W2L_EXTRACT_API_KEY=w2l-anthropic-api-key:latest'
gcloud run deploy octocrawl-jobs --image="$W2L_IMAGE" --region="$W2L_REGION" --project="$W2L_PROJECT_ID" \
  --revision-suffix="$W2L_REVISION_SUFFIX" --no-traffic \
  --update-env-vars="W2L_SOURCE_COMMIT=${W2L_SOURCE_SHA},W2L_EXTRACT_PROVIDER=anthropic,W2L_EXTRACT_MODEL=claude-haiku-5-5" \
  --update-secrets='W2L_EXTRACT_API_KEY=w2l-anthropic-api-key:latest'
#    Then update-traffic for each to its new revision under a new tag, keep the previous tags as the rollback, and check
#    that each startup line names "extractModel":"claude-haiku-5-5" and that a scrape with modelFallback answers
#    model_unavailable "switched off" with X-Credits-Model: 0.

# 5. On: one test page with an account's key, then the switch.
curl -s -X PATCH -H "authorization: Bearer $T" -H "x-goog-user-project: $W2L_PROJECT_ID" -H 'content-type: application/json' \
  "https://firestore.googleapis.com/v1/projects/$W2L_PROJECT_ID/databases/(default)/documents/hostedConfig/model?updateMask.fieldPaths=enabled" \
  -d '{"fields":{"enabled":{"booleanValue":true}}}'
```

`W2L_EXTRACT_BASE_URL` stays unset, so the SDK calls `https://api.anthropic.com`. The services pass no proxy to the SDK: Cloud Run reaches Anthropic directly.

To turn it off, write `enabled: false` (or delete the document): within 30 s no new call is admitted, and a call already admitted finishes. To change the cap, write `dailyCapUsd`. To take the model away altogether, a release with `--remove-env-vars=W2L_EXTRACT_PROVIDER,W2L_EXTRACT_MODEL --remove-secrets=W2L_EXTRACT_API_KEY` on both services; rolling back to the previous tags does the same, since those revisions have no key. Read a day's spend from `hostedModelSpend/{day}.microUsd`, and the bill from the Anthropic Console.

## The compatible transport

ADR 0005's `compatible_transport` sends a page's requests with [impit](https://github.com/apify/impit) (0.14.5, profile `chrome142`), whose TLS and HTTP/2 handshakes and headers are a real Chrome's, instead of undici's ([access](access.md), `W2L_COMPAT_HOSTS`). It is the first step of ROADMAP stage H item 2. Hosted Octocrawl reads no access grant, so the transport is granted per request, by an operator switch that is off by default. `packages/mcp/src/hostedAccess.ts` holds the rules.

- **The switch.** The Firestore document `hostedConfig/compatibleTransport`:
  - `enabled` (boolean). While the document is absent, or `enabled` is anything but `true`, no call holds the transport, so a deploy alone turns nothing on.
  - `allowedUids` (an array of account uids, optional). When it exists, only those accounts' calls hold the transport: a keyless caller, a key issued without an account and every other account do not. An empty array grants it to no one. When it is absent, every caller holds it.
  - Both services read the document at most every 30 s. A failed read keeps what was read last; with nothing read yet the switch is off.
- **Who asks.** The gate asks for each scrape, REST and MCP alike. The job worker asks as each slice starts, for the job's account, so a change reaches a running job at its next slice. The answer goes to the engine with the call (`ScrapeContext.capabilities`, `SliceLimits.capabilities`); the engine grants a call nothing beyond `compatible_transport` and `hosted_chrome` (`HOSTED_CALL_CAPABILITIES`; the second has its own switch, below), and the other capabilities stay refused here: `installed_chrome`, the CAPTCHA solver, vendors, egress proxies, Patchright.
- **Which pages.** As on a server on your computer whose grant names the transport and sets no `W2L_COMPAT_HOSTS`:
  - only the five hosts its acceptance showed it helps (`research/access/benefit-hosts.v1.json`), with their subdomains;
  - their `http` rung becomes `http_compat`;
  - a request with custom `headers` or `mobile` keeps `http`, and a map's start page never uses the transport;
  - every other page is read as before.
- **The address guard.** impit resolves names itself and takes no resolver, so a hosted engine's transport sends every request through `CompatAddressGuard` (`packages/bench/src/compatGuard.ts`):
  - a SOCKS5 server on `127.0.0.1` in the same process, with a login it makes up when it starts and gives impit alone;
  - it resolves each name, refuses it when any address it resolves to is one hosted Octocrawl does not connect to (private, shared, loopback, link-local, multicast, unspecified, the metadata service's), and connects only to an address it checked;
  - impit's TLS handshake and HTTP/2 pass through it unchanged;
  - impit follows no redirect itself: the lane checks each hop, and the hop's connection goes through the guard again;
  - a refused address is `failed` / `policy_denied`, a name that does not resolve `failed` / `dns_error`, as on the undici lane.
- **The price and the record.** A page read over the transport is an HTTP page: 1 credit, as `lane: "http"` shows. Its Evidence Record says how it was read, as a local server's does: `access.route: "http_compat"`, `executor: "impit"`, `executorVersion`, `profile`, and `egress.source: "direct"` (the guard is not a proxy). The trace has the lane's `transport` event, and a scrape's ladder audit says `http` was set aside (`ladder_channels_filtered`, reason `compatible_transport`).

### Turning it on

Steps for Howard to approve; nothing here has been run.

1. A release of both services from a commit that has this code (Deploy above; Batch and crawl jobs for `octocrawl-jobs`). It needs no new variable or secret. Both services load impit as they start, so a revision whose image lacks it does not start.
2. Write the switch for one account first, so the parity board can measure the transport before every caller has it.
3. Measure it on the parity board (`research/hosted/parity-window.mjs`), switch on and switch off, in two windows. Turn it on for every caller only after that, by removing `allowedUids`.

```sh
# The switch for one account (users/{uid}); write enabled false to turn it off, within 30 s.
T=$(gcloud auth print-access-token); curl -s -X PATCH -H "authorization: Bearer $T" -H "x-goog-user-project: $W2L_PROJECT_ID" -H 'content-type: application/json' \
  "https://firestore.googleapis.com/v1/projects/$W2L_PROJECT_ID/databases/(default)/documents/hostedConfig/compatibleTransport" \
  -d '{"fields":{"enabled":{"booleanValue":true},"allowedUids":{"arrayValue":{"values":[{"stringValue":"<uid>"}]}}}}'
```

## The hosted_chrome switch

ADR 0005's `hosted_chrome` (revision of 2026-10-27) is Google Chrome on Octocrawl's own server, in the `octocrawl-chrome` service. Hosted Octocrawl uses it only when three things hold together: the operator's config, the account's switch, and the call's `access: "enhanced"`. The account's switch is the Firestore document `hostedConfig/hostedChrome`, read the way `hostedConfig/compatibleTransport` is (at most every 30 s, a failed read keeping the last one) and granted by the same gate and job worker (`packages/mcp/src/hostedAccess.ts`):

- `enabled` (boolean): anything but `true` is off, and so is a missing document.
- `allowedUids` (an array of account uids): an allowlist and nothing else. Absent or empty grants it to no one. A keyless caller and a key issued without an account never hold it.
- `monthlyCapUsd` (number) and `dailyAttemptsPerAccount` (whole number): the operator's dollar cap a month for the rung, and the reads of the rung one account may start a UTC day. Their values wait for decision D5 and have no default: while either is absent or not positive, the switch grants it to no one.

**The operator's config** (`packages/mcp/src/hostedChromeOperator.ts`) is three variables, read at start by the hosted API and the job worker. Set all three or none:
- `W2L_HOSTED_CHROME_URL`: the `octocrawl-chrome` service's https URL, also the audience of the ID token the client sends;
- `W2L_HOSTED_CHROME_ATTESTATION`: the operator's statement, JSON `{ "principal", "at", "statement" }` (`at` in ISO 8601);
- `W2L_HOSTED_CHROME_EGRESS`: the service's exit and its price. It is `google` for Cloud Run's own egress, or JSON `{ "vendor", "tariff" }` with a `VendorTariff`. It must describe the exit the service is deployed with, since the counter prices each read by it: the datacenter tariff when the service mounts the proxy list, `google` when it does not.

With none set, nothing changes: no engine builds the rung, and `access: "enhanced"` is refused as before, whatever the switch says. With some set and not all, or a value it cannot use, the process does not start, and the message names the variable, never its value. With all three, the API needs the Firestore store, where the switch and the counters are. The startup line says `hostedChrome: null`, or names the exit's vendor.

**Who may ask.** With the config, the gate takes `access: "enhanced"` on a scrape, and on a batch or crawl start, only from a caller the switch grants `hosted_chrome` at that moment. Every other caller still gets a 400 `unsupported_parameter`, and a job start gets the key sets' refusal (`hosted batch jobs do not take access`), as a deploy without the rung answers them. A refused call reads nothing and is charged nothing. An admitted call reaches the rung only after the rungs before it were blocked, and only on a key with the browser lane (an HTTP-only call drops the browser rungs). A job's slices hold the rung while the switch still lists the job's account; a slice that starts after the account was taken off the list reads its pages with the earlier rungs (ADR decision two).

**The counting** (`packages/mcp/src/hostedChromeMeter.ts`). The operator config must carry the counter (`meter`): an engine given the rung without one refuses to start, so the rung can never run unmetered. The counter is asked before each read of the rung, for the call's account (a scrape's caller, a job's account):
- **The monthly cap.** The month (UTC) is a monthly ledger (`openMonthlySpendLedger`): it opens with the egress tariff's fixed rents (`perIpMonthUsd` for each of `rentedIps`) settled first, then what the month's reads settled, then this instance's reads still in flight. A read whose price ceiling does not fit what `monthlyCapUsd` leaves is not made. Settled reads only are counted, in `hostedChromeSpend/{YYYY-MM}` (`settledMicroUsd`, `calls`, and `expireAt` for TTL, 90 days after the month ends). So instances reserving at once can pass the cap by the ceilings of the reads in flight on the others.
- **The daily attempts.** Each read takes one from `hostedQuotas/{day}-hostedChrome-{uid}`, in a commit with an update-time precondition, as the hosted API's other daily counters are (TTL two days on). An attempt is taken once the cap admits the read, and is not given back when the read fails.
- **What a read is charged.** The egress tariff's price, its rent left out (`tariffCostUsd`). Bytes the exit did not report are charged at the ceiling's allowance, and a read that threw is charged its ceiling. With Cloud Run's own egress (no tariff) a read pays no third party: it is settled at $0 and still counted. Cloud Run's compute is the project's billing budget's, not this cap's.
- **What its record says.** A read through a priced exit is one of its Evidence Record's paid calls (`access.paidCalls[]`): `provider` is the tariff's vendor, `rung` and `capabilities` are `hosted_chrome`, `ceilingUsd` and `chargedUsd` are the counter's ceiling and charge (the rent left out), `reportedCostUsd` is null, and `fixedMonthlyUsd` is the month's rent (2.99 for the Webshare plan below). A read's share of the rent is not known, so the record's `externalCostUsd` is null, never 0 (ADR 0005, decision three), and so is the answer's `usage.externalCostUsd`; a batch or crawl that made such a read reports its cost as unknown. The call stays on the record when another rung's page is the answer, when the deadline cut the read, and, on a batch or crawl page, when the read threw (a single scrape whose read threw answers 500, with no record). With Cloud Run's own egress there is no paid call, and `externalCostUsd` stays 0.
- **A read over either limit** does not use the rung, and the scrape does not fail: the page is what the rungs before it read. The result carries a `hosted_chrome_not_admitted` warning saying which limit, and the ladder trace a `ladder_channel_skipped` event for `hosted_chrome` with its reason (`monthly_cap`, `daily_attempts`, `switched_off`, `budget_unread`). So does a counter that could not read the month's spend or take an attempt: unread is not zero.

**A robots.txt the hosted network cannot reach.** A call that holds the rung goes straight to it when robots.txt kept the page out only because it could not be reached (a timeout, or a connection that failed before the site answered), since the rung reads robots.txt from its own exit and obeys what it finds there (ADR 0005, revision of 2026-11-05). A site that looks to have no HTTPS is read over `http://` first, for one credit, and if `http://` does not reach it either, the `https://` address goes to the rung. A rule, a 5xx or a lookup the site answered (a redirect loop, a body too large) stops the ladder as before, and a call that did not choose `access: "enhanced"` never reaches the rung. The rung's own result is the answer, a rule or a failure at the exit included; when it throws, the answer stays the robots.txt refusal. No other hosted request leaves through a proxy.

**The egress (D2).** Howard chose datacenter proxies on 2026-10-29. The basis is the H2b comparison (PR #748): over the 19 robots-allowed tasks, datacenter verified 9, at $0.0013 per verified page; residential verified 8 and 5 (2 vCPU, in the two windows) and Google's own egress 4. Datacenter could not resolve `challenges.cloudflare.com`, so T047 and T048 did not pass.

The service leaves through the datacenter proxies when the proxy list is mounted (The octocrawl-chrome service, below), and from Cloud Run's own egress without it. The API's `W2L_HOSTED_CHROME_EGRESS` is then the plan's rent, which the monthly ledger counts first: the Webshare plan H2b bought is 100 IPs for $2.99 a month (`{"vendor":"webshare","tariff":{"perIpMonthUsd":0.0299,"rentedIps":100}}`), and the counter charges a read $0 past the rent (its record's cost is null, above). Check the plan's price before deploying; the tariff is the operator's, not this page's.

**The values set on 2026-10-29** (Howard, D5). They are recorded here only: this change writes nothing to Firestore. Howard sets them when the rung may serve:
- `hostedConfig/hostedChrome.enabled`: `true`;
- `allowedUids`: Howard's account only;
- `monthlyCapUsd`: `20`;
- `dailyAttemptsPerAccount`: `20`.

```sh
T=$(gcloud auth print-access-token); curl -s -X PATCH -H "authorization: Bearer $T" -H "x-goog-user-project: $W2L_PROJECT_ID" -H 'content-type: application/json' \
  "https://firestore.googleapis.com/v1/projects/$W2L_PROJECT_ID/databases/(default)/documents/hostedConfig/hostedChrome" \
  -d '{"fields":{"enabled":{"booleanValue":true},"allowedUids":{"arrayValue":{"values":[{"stringValue":"<Howard'"'"'s uid>"}]}},"monthlyCapUsd":{"integerValue":"20"},"dailyAttemptsPerAccount":{"integerValue":"20"}}}'
gcloud firestore fields ttls update expireAt --collection-group=hostedChromeSpend --enable-ttl --project="$W2L_PROJECT_ID"
```

## The octocrawl-chrome service

`hosted_chrome`'s Chrome runs in a Cloud Run service of its own, `octocrawl-chrome` (ROADMAP stage H, decision D1: minimum instances 0). Its image is `Dockerfile.hosted-chrome`:
- Google Chrome 155.0.8059.79, pinned by version and SHA-256, the build G4 measured on Cloud Run gen2;
- Xvfb as its display, and tini as PID 1;
- a non-root user, so Chrome's sandbox stays on;
- Node 22.23.4, the hosted API's, pinned by digest (The regular-expression checks, below). Node only drives Chrome through Playwright.

`deploy/hosted-chrome/start.sh` opens the display and starts `packages/mcp/src/hostedChromeCli.ts`. `cloudbuild.hosted-chrome.yaml` builds and pushes the image, with no deploy step; its `_IMAGE`, and so the region, is given at submit time. The size Howard set on 2026-10-29: 2 vCPU and 4 GiB in asia-southeast1, minimum instances 0, one request an instance (below). A hosted engine reaches it once the API or the worker is deployed with the operator's config (The hosted_chrome switch, above).

- **Its HTTP surface** (`packages/mcp/src/hostedChrome.ts`):
  - `POST /v1/read` with `{ url, options?, deadlineAt? }` answers the browser lane's result for that page.
  - 400 is a request the rung does not take: not http(s), an option `installed_chrome` also refuses, or a body over 256 KiB.
  - 503 means a page is being read: one window per instance.
  - `GET /health` answers 200.
- **How each page is read.** The service reads each page with the hosted browser lane on the `hosted_chrome` engine:
  - robots.txt is read and obeyed, as on every hosted rung, on the route the page is read on: with the proxy list mounted, it goes through the same address guard and the same exit, with 12 seconds to answer, since every redirect hop opens a tunnel of its own (a direct lookup has 5);
  - every connection goes through its address guard;
  - each call gets a fresh profile, deleted after;
  - its trace names the pinned version, the launch (`nodriver-0.50.6-no-allow-origins+webrtc-proxied-udp-only`) and the exit's kind, which the Evidence Record carries: `google` from Cloud Run's own egress, or `datacenter` with the proxy as `host:port` (`access.egress.proxy`, source `pool`).
- **Its exit** (`packages/mcp/src/hostedChromeEgress.ts`; D2) is the proxy list, a Secret Manager secret mounted as a file. Its path is `W2L_HOSTED_CHROME_PROXIES_FILE`, and the file holds one `host:port:user:pass` line per proxy.
  - Each read chooses one proxy before its first request and keeps it for every connection, whatever the site answers.
  - The proxy is the address guard's dial: Chrome's connections reach the guard, which resolves and checks the name, and the proxy is asked for a CONNECT tunnel to the checked address alone.
  - The login goes to the proxy and nowhere else. The startup line names the kind and the count of proxies, and a malformed line is named by its number.
  - Without the variable the service leaves from Cloud Run's own egress; a file named and not readable, or a list that does not parse, stops the start.
- **Authentication** is Cloud Run's IAM: the service takes no unauthenticated call. On the service itself, only the runtime account of the hosted API and the job worker (`w2l-preview`) holds `roles/run.invoker`. Roles granted on the project reach it too: a principal that holds any role with `run.routes.invoke` on the project, among them Owner, Editor, Cloud Run Admin and Cloud Run Developer, can call the service with its own ID token. The client (`httpHostedChromeClient`) sends a Google ID token for the service's URL from the metadata server (`metadataIdToken`). Cloud Run checks the token before a request reaches the container.
- **Its ingress is `all`, not internal.** Cloud Run counts a call from another Cloud Run service as internal only when it comes through a VPC network. The caller must use Direct VPC egress or a connector and route all its traffic through the VPC network, with Private Google Access on the subnet, or put Private Service Connect or an internal load balancer in front of the service ([private networking](https://docs.cloud.google.com/run/docs/securing/private-networking), [ingress](https://docs.cloud.google.com/run/docs/securing/ingress), read 2026-10-29). The hosted API and the worker have no VPC egress. Routing all their traffic through one would send their public page reads through a Cloud NAT, which is a new cost, a new exit address and a new part to run. So the service takes `ingress=all`, and IAM is the bound: a request without an ID token of an account allowed to invoke it is refused 403 by Cloud Run. The limit: the public preview runs as `w2l-preview` too, so its identity could also invoke the service, though its code never calls it. A runtime account for the API and the worker alone would close that.
- **Before it serves a call**, three things must happen:
  - the launch passes ADR 0005's gate: two challenge16 runs no worse than the baseline, which covers the WebRTC preference that sets this launch apart from `installed_chrome`'s;
  - Howard decides to move traffic: until then only a tagged API revision carries the operator's config;
  - the switch carries the values above.

All three held on 2026-10-31: Howard counted the gate as passed (PR #795) and moved the API's traffic to a revision with the config, and the job worker was released with it the same day. The switch has carried the values set on 2026-10-29 (above) since that day.

### Deploying hosted_chrome

These steps were run between 2026-10-28 and 2026-10-31: the service on 2026-10-28, its proxies and the API's tagged revision on 2026-10-29, the API's traffic and the job worker on 2026-10-31. The first revision's differences are listed below. Pass `--project` on every call. Build from a clean commit.

The service's own runtime account, with no role but reading its proxy list. Then the list: the provider's Direct connection list, one `host:port:user:pass` a line, from a file that is never printed or pasted into a shell:

```sh
export W2L_CHROME_SA="octocrawl-chrome@${W2L_PROJECT_ID}.iam.gserviceaccount.com"
gcloud iam service-accounts create octocrawl-chrome --display-name='octocrawl-chrome runtime' --project="$W2L_PROJECT_ID"
gcloud secrets create w2l-hosted-chrome-proxies --replication-policy=automatic --data-file=<the proxy list file> --project="$W2L_PROJECT_ID"
gcloud secrets add-iam-policy-binding w2l-hosted-chrome-proxies --member="serviceAccount:${W2L_CHROME_SA}" --role='roles/secretmanager.secretAccessor' --project="$W2L_PROJECT_ID"
```

Build the image and deploy the service: 2 vCPU, 4 GiB, minimum 0 instances, one request an instance (a second answers 503 and the rung fails over to the earlier page), gen2 as G4 measured, and the request timeout over the longest read (180 s). The maximum instances bound its compute and were not decided on 2026-10-29: set `W2L_CHROME_MAX_INSTANCES` to Howard's number.

```sh
export W2L_CHROME_IMAGE="${W2L_REGION}-docker.pkg.dev/${W2L_PROJECT_ID}/${W2L_REPOSITORY}/hosted-chrome:${W2L_SOURCE_SHA}"
gcloud builds submit . --config=cloudbuild.hosted-chrome.yaml --region="$W2L_REGION" --timeout=30m --substitutions="_IMAGE=${W2L_CHROME_IMAGE}" --project="$W2L_PROJECT_ID"
gcloud run deploy octocrawl-chrome --image="$W2L_CHROME_IMAGE" --region="$W2L_REGION" --project="$W2L_PROJECT_ID" \
  --service-account="$W2L_CHROME_SA" --no-allow-unauthenticated --ingress=all --execution-environment=gen2 \
  --cpu=2 --memory=4Gi --concurrency=1 --min-instances=0 --max-instances="$W2L_CHROME_MAX_INSTANCES" --timeout=300s \
  --revision-suffix="$W2L_REVISION_SUFFIX" \
  --set-env-vars="W2L_SOURCE_COMMIT=${W2L_SOURCE_SHA},W2L_HOSTED_CHROME_PROXIES_FILE=/secrets/hosted-chrome/proxies" \
  --update-secrets='/secrets/hosted-chrome/proxies=w2l-hosted-chrome-proxies:latest'
gcloud run services add-iam-policy-binding octocrawl-chrome --region="$W2L_REGION" --project="$W2L_PROJECT_ID" \
  --member="serviceAccount:${W2L_RUNTIME_SA}" --role=roles/run.invoker
export W2L_CHROME_URL="$(gcloud run services describe octocrawl-chrome --region="$W2L_REGION" --project="$W2L_PROJECT_ID" --format='value(status.url)')"
curl -s -o /dev/null -w '%{http_code}\n' "$W2L_CHROME_URL/health"   # 403: no unauthenticated call reaches it
```

The first revision, `octocrawl-chrome-c1-95be03e` (2026-10-28, image of `95be03ee`), differs from these commands in four ways, and the API cannot reach it until the first two change:
- it was deployed with `--ingress=internal`, which a caller without VPC egress cannot reach. From outside, `/health` answered 404 with and without an ID token. Change it with `gcloud run services update octocrawl-chrome --ingress=all`;
- it has no invoker binding: add the `add-iam-policy-binding` above for `w2l-preview`;
- it runs as `w2l-preview`, which also holds the project's Firestore role. The commands above give it `octocrawl-chrome`, which holds no role but reading its proxy list, so a page that got past Chrome reaches no account worth having. That needs a redeploy with `--service-account`;
- it has neither the proxy secret nor `W2L_HOSTED_CHROME_PROXIES_FILE`, so it leaves from Cloud Run's own egress (`google`), with `--max-instances=1`.

Its log's `hosted_chrome_started` line says `egressKind: "datacenter"` and the count of proxies. Without the `W2L_HOSTED_CHROME_PROXIES_FILE` variable and the secret, it says `google`.

The operator's statement and tariff, as secrets the runtime account reads:

```sh
printf '%s' '{"principal":"<who accepted it>","at":"2026-10-29T00:00:00Z","statement":"hosted_chrome on octocrawl-chrome, datacenter egress (D2), D5 caps"}' | gcloud secrets create w2l-hosted-chrome-attestation --replication-policy=automatic --data-file=- --project="$W2L_PROJECT_ID"
printf '%s' '{"vendor":"webshare","tariff":{"perIpMonthUsd":0.0299,"rentedIps":100}}' | gcloud secrets create w2l-hosted-chrome-egress --replication-policy=automatic --data-file=- --project="$W2L_PROJECT_ID"
for secret in w2l-hosted-chrome-attestation w2l-hosted-chrome-egress; do
  gcloud secrets add-iam-policy-binding "$secret" --member="serviceAccount:${W2L_RUNTIME_SA}" --role='roles/secretmanager.secretAccessor' --project="$W2L_PROJECT_ID"
done
```

An API revision with the config, without traffic, under a tag of its own. Traffic stays on the current tag until the gate has passed and Howard moves it, and the revision keeps `--timeout=300s` and `--use-http2`:

```sh
gcloud run deploy octocrawl-api --image="$W2L_IMAGE" --region="$W2L_REGION" --project="$W2L_PROJECT_ID" \
  --revision-suffix="$W2L_REVISION_SUFFIX" --no-traffic --timeout=300s --use-http2 \
  --update-env-vars="W2L_SOURCE_COMMIT=${W2L_SOURCE_SHA},W2L_HOSTED_CHROME_URL=${W2L_CHROME_URL}" \
  --update-secrets='W2L_HOSTED_CHROME_ATTESTATION=w2l-hosted-chrome-attestation:latest,W2L_HOSTED_CHROME_EGRESS=w2l-hosted-chrome-egress:latest'
gcloud run services update-traffic octocrawl-api --region="$W2L_REGION" --project="$W2L_PROJECT_ID" \
  --update-tags="${W2L_TAG}=octocrawl-api-${W2L_REVISION_SUFFIX}"
```

The tag's own URL reaches that revision alone, without the Worker, with an account's key. The job worker takes the same variables on its next release, with the same `--update-env-vars` and `--update-secrets`. To turn the rung off again, deploy without the three variables (`--remove-env-vars=W2L_HOSTED_CHROME_URL --remove-secrets=W2L_HOSTED_CHROME_ATTESTATION,W2L_HOSTED_CHROME_EGRESS`), or set `hostedConfig/hostedChrome.enabled` to false, which takes effect within 30 s.

## The regular-expression checks

A task contract's pattern checks, `markdownMatches` and `markdownCountMin`, run with a time limit on every server (ADR 0006, amendment of 2026-10-29). A scan stops at 250 ms. The checks of one final judgement share 1 s, and the partial judgements that steer one request share another 1 s. An MCP scrape result is judged again as it is shortened and delivered: its shortened versions share 1 s, and the version delivered gets 1 s of its own. A check stopped at a limit is undecided, and its verification fails. Hosted Octocrawl takes these checks only when its operator switched them on and its start-up self-test passed. Otherwise a scrape, crawl or batch that sends one is refused by name (400 `unsupported_parameter`), and the MCP schemas and the server card leave the two types out.

- **The switch** is `W2L_HOSTED_REGEX_CHECKS=1`, the same on `octocrawl-api` and `octocrawl-jobs`: the API admits a job that the worker then runs. Unset or `0` is off; any other value stops the start.
- **The self-test** (`regexBoundSelfTest`, `packages/runtime/src/regexSelfTest.ts`) takes about 0.1 s. A quadratic scan must stop near its limit, a scan past the backtrack stack must be an engine error, and counts and matches must equal `String.prototype.match`'s.
  - The API and the worker run it at their start when the switch is 1. `octocrawl-chrome` always runs it, and takes the checks when it passes, since its callers admitted them first.
  - A failed self-test leaves the checks off. The API and the worker log `regex_self_test_failed` at ERROR, naming each failure, and their startup lines say `"regexChecks":false`. `octocrawl-chrome` logs `hosted_chrome_started` at ERROR, with `regexFailures`.
  - Every startup line names its Node (`node`).
- **The images' build** (`cloudbuild.hosted-api.yaml`, `cloudbuild.hosted-chrome.yaml`) runs the same self-test in the image just built. It also judges every pattern of the task files (`research/access/tasks.v1.json`, `research/access/captcha-tasks.v1.json`, `research/parity/sites.v1.json`, which `.gcloudignore` uploads for it) on synthetic Markdown, with the image's `verify()` and with `RegExp`. A failure stops the build before the image is pushed. Both images run Node 22.23.4, pinned by digest: on Node 26.8.1 a scan stopped at its limit left every later match in the process about 3.7 times slower.
- **What they cost** is logged, never the page or the pattern. Each scrape, job page, chrome read or MCP scrape result that ran a pattern scan writes one `regex_checks` line:
  - `kind`: `scrape`, `page`, `chrome_read`, or `mcp_result` for the re-judgements of one MCP scrape result, which the scrape's own line does not include;
  - who it was for: a scrape's and an MCP result's line give `account`, the first 16 hex digits of an HMAC under `W2L_QUOTA_HASH_KEY`, null without an account. A job page's line names its `jobId` instead. A chrome read's line names neither, as the service is not told the account;
  - `scans`, `memoHits` (scans reused), `totalMs`, `maxMs` (the longest scan), `maxCallMs` (the longest one judgement held the process) and `limited` (the checks left undecided, counted in each judgement, so a check undecided in three judgements counts three).
- **The event loop.** The `hosted_scrape_phases` line of a scrape with a task contract adds `loopMaxMs` and `loopP99Ms`: the event loop's longest wait and its 99th percentile while the scrape ran, measured in the process, other calls' work included. Each is the part beyond the 10 ms between samples.
  - The loop is sampled from the scrape's start to the moment it answers or throws, so the final judgement counts.
  - What runs after that does not count: the answer's serialization, and an MCP result's re-judgements. The re-judgements run in one stretch, so the `totalMs` of their `mcp_result` line is about how long they held the loop.
  - The line is written as the scrape answers, after the scrape's `regex_checks` line.

### Turning them on

First run for the release from `4df8feed`: `octocrawl-chrome` and the API's tag `h45regex` on 2026-10-30, the API's traffic and `octocrawl-jobs` on 2026-10-31. Each service is released from the same commit, in this order:

1. `octocrawl-chrome`, as in Deploying hosted_chrome. Its `hosted_chrome_started` line says `"regexChecks":true` and `"node":"v22.23.4"`.
2. `octocrawl-jobs`, with `--update-env-vars=W2L_HOSTED_REGEX_CHECKS=1`. Its startup line says `"regexChecks":true`.
3. `octocrawl-api`, a revision without traffic, under a tag of its own:

```sh
gcloud run deploy octocrawl-api --image="$W2L_IMAGE" --region="$W2L_REGION" --project="$W2L_PROJECT_ID" \
  --revision-suffix="$W2L_REVISION_SUFFIX" --no-traffic --timeout=300s --use-http2 \
  --update-env-vars="W2L_SOURCE_COMMIT=${W2L_SOURCE_SHA},W2L_HOSTED_REGEX_CHECKS=1"
gcloud run services update-traffic octocrawl-api --region="$W2L_REGION" --project="$W2L_PROJECT_ID" \
  --update-tags="${W2L_TAG}=octocrawl-api-${W2L_REVISION_SUFFIX}"
```

The tag's URL is checked before traffic moves:
- a challenge16 run with its task contracts;
- the same run against a local API from the same commit;
- a page of about 4 million characters with one pathological pattern, then with 32.

Howard decides when traffic moves.

To turn the checks off, move the API's traffic back to the previous tag, or release it with `--remove-env-vars=W2L_HOSTED_REGEX_CHECKS`. Its scrapes and job starts then refuse them by name again. The worker keeps them until no queued job holds a pattern check; then release it without the variable. The chrome service takes them only from those two callers.

## Read the service

Every request is in Cloud Run's request log. The two-week cost record the roadmap asks for is read from Cloud Billing (the service's own SKU lines) against the request counts:

```sh
gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="octocrawl-api" AND httpRequest.requestUrl:"/v1/scrape"' --project="$W2L_PROJECT_ID" --freshness=7d --format='value(httpRequest.status)' | sort | uniq -c
```

Write the result to `research/hosted/` with the date, the pages per lane, the billed amount and the cost per 1,000 pages; the allowances and `--max-instances` are adjusted from that record, not before.

The service's own structured lines (`jsonPayload.event`), each one JSON on stdout, never a page's content:

- `hosted_usage`: a key's or account's start that passed its allowance or reserved its credits (the key's HMAC prefix, plan, `uid`, endpoint, day); `hosted_keyless_usage`: a keyless start (address prefix or the `claude` pool).
- `hosted_credits`: what an account's call cost once settled (`uid`, kind, used, reserved, the model's part).
- `hosted_scrape_phases`: one line per scrape with its rungs and waits, and since 2026-11-05 `caller` (`keyless`, `key`, `oauth` or `site`) and `uid`, so the phases can be split by who called.
- `hosted_mcp_initialize`: each MCP `initialize` with the client's name and version; `hosted_mcp_call` (since 2026-11-05): each tool call's end with `endpoint`, `caller`, `tool`, `ms`, `truncated` (the answer was cut to the size budget) and `error` (it ended as a tool error), and `fields`, as on `hosted_refused`, when the tool refused its own arguments before calling the API.
- `hosted_refused` (since 2026-11-05): every 4xx of the API routes, with `status`, `code`, `path`, `caller` (null before the caller is known); for a 400, `fields`: the top-level names of the request fields it refused (a key hosted does not take is named as sent, and what was sent in a field never is), null when the answer names none; and, for a limit, `pool`: `minute` (the per-minute rate), `browser` (browser_busy), `credits`, `account`, `accounts`, or the day's pool (`site`, `keyless`, `keyless-<address prefix>`, `keyless-claude`, `key-<prefix>`).
- `hosted_quota_threshold` (WARNING): a shared pool at 70% of its day; `hosted_api_error` and `hosted_mcp_error` (ERROR): a request that threw.

Not logged: what `/mcp` and `/mcp-oauth` answer themselves before a tool runs (a bad key or token, a body too large or not JSON, an address over its rate).

### Alerts

The API and the job worker log `{"event":"hosted_quota_threshold","pool":…,"day":…,"count":…,"limit":…}` once a pool a day, from the use that takes a shared pool to 70% of its limit. The `jobs` pool can log a few such lines around the mark, since a slice gives back the pages it did not read and a later one crosses again. A log-based alert mails it, and a billing budget mails spend. Set them up once, in this order:

```sh
# 1. An email channel; the command prints the channel's name (projects/…/notificationChannels/…).
gcloud beta monitoring channels create --display-name="Octocrawl ops" --type=email --channel-labels=email_address=<operator address> --project="$W2L_PROJECT_ID"

# 2. The pool alert: any hosted_quota_threshold line from either service, at most one mail an hour.
cat > quota-alert.json <<'EOF'
{"displayName": "Octocrawl: a shared pool reached 70% of its day", "combiner": "OR",
 "conditions": [{"displayName": "hosted_quota_threshold", "conditionMatchedLog": {
   "filter": "resource.type=\"cloud_run_revision\" AND jsonPayload.event=\"hosted_quota_threshold\"",
   "labelExtractors": {"pool": "EXTRACT(jsonPayload.pool)"}}}],
 "alertStrategy": {"notificationRateLimit": {"period": "3600s"}, "autoClose": "86400s"},
 "notificationChannels": ["<the channel name from step 1>"]}
EOF
gcloud alpha monitoring policies create --policy-from-file=quota-alert.json --project="$W2L_PROJECT_ID"

# 3. A monthly budget in US dollars, mailed to the billing account's admins at 50%, 90% and 100%.
gcloud billing budgets create --billing-account=<billing account id> --display-name="Octocrawl monthly" --budget-amount=<amount>USD \
  --threshold-rule=percent=0.5 --threshold-rule=percent=0.9 --threshold-rule=percent=1.0 --filter-projects="projects/${W2L_PROJECT_ID}" --billing-project="$W2L_PROJECT_ID"
```

A pool's limit is its environment variable on the service (`W2L_SITE_DAILY`, `W2L_KEYLESS_TOTAL_DAILY`, `W2L_CLAUDE_DAILY`, `W2L_ACCOUNTS_DAILY` on `octocrawl-api`; `W2L_JOBS_DAILY` and `W2L_JOBS_ACCOUNT_DAILY` on `octocrawl-jobs`); raising one is a new revision with `--update-env-vars`, moved to as any deploy is.

The two job pools count pages read. A slice takes its pages from them before it reads, and its settle gives back the pages it did not read, in the commit that gives back its credits; a slice that dies gives back all of them. A job a pool refuses waits for 00:00 UTC, and its status says so (`paused`, [How long it takes](batch-crawl-map.md)); the worker logs `hosted_job_daily_pool`. A job whose account's credits run out waits the same way (`paused.reason` `credits`, log `hosted_job_credits_paused`), asking again every 6 hours and at the 1st.

## Hosted and local, side by side

Where the hosted MCP endpoint stands against a server on your computer (`npx octocrawl serve` with `@octocrawl/mcp`): every local MCP tool, every option of `scrape`, `map`, `crawl` and `batch_scrape`, and the option values hosted leaves out. This is the parity board of the roadmap's current phase (stage H item 1). Supported is as locally; limited is listed with a lower cap, or to some connections only; planned names the stage H item that brings it, or says no item does yet; different on purpose gives the reason. "Sent anyway" says what hosted does with an option or value it does not list.

The table is generated from the manifest, `packages/mcp/src/hostedParity.ts`. `packages/mcp/test/hostedParity.test.ts` fails when an entry claims more or less than the hosted tool lists, key sets and engine serve, or when this table differs from the manifest; `node --import tsx scripts/hosted/parity-table.ts --write` rewrites it.

<!-- hosted-parity:start -->
**Tools.** Of the local server's 39 MCP tools, hosted serves 17: 2 as locally and 15 limited. 21 are planned and 1 is different on purpose.

**Options.** Of the 120 local options of scrape, map, crawl and batch_scrape, hosted lists 87: 75 as locally and 12 limited. 23 are planned and 10 different on purpose. Below the options, 7 option values that hosted leaves out or limits.

**Planned, by stage H item** (tools, options and values): item 2, hosted access: 3; item 3, actions and waitFor: 2; item 4, monitors and delivery: 18; item 5, identity-bound features: 5; item 7, the json model fallback: 1; item 8, beyond local: 13; no item yet: 5.

| Tool | Hosted | How |
| --- | --- | --- |
| `scrape` | Supported | Every connection. Results leave out usage and channelsTried unless debug is set, and metadata.scrapeId unless debug is set or the connection lists get_scrape (an account's); the Evidence Record stays whole. |
| `map` | Supported | Every connection. |
| `get_scrape` | Limited | An account's connection only (its key, or signing in at /mcp-oauth); hosted keeps a scrape's record 10 minutes, on the server instance that ran the scrape, so a read that reaches another instance answers not_found, and answers it without the call's telemetry, as a scrape's result (locally records are kept without a time limit and answered whole). |
| `crawl` | Limited | An account's connection only (its key, or signing in at /mcp-oauth): a job run a slice at a time, at most 500 pages, each page priced as a scrape, results kept 7 days. |
| `get_crawl` | Limited | An account's connection only (its key, or signing in at /mcp-oauth), for its own jobs; a finished job's status carries `export`, a link to every page as one gzip JSONL file, good for 24 hours (locally `resultFiles`, the files written beside the task). |
| `wait_crawl` | Limited | An account's connection only (its key, or signing in at /mcp-oauth). |
| `get_crawl_pages` | Limited | An account's connection only (its key, or signing in at /mcp-oauth); limit at most 100 (local 1,000), maxResults at most 100 (local 200). |
| `get_crawl_errors` | Limited | An account's connection only (its key, or signing in at /mcp-oauth); limit at most 100 (local 1,000). |
| `cancel_crawl` | Limited | An account's connection only (its key, or signing in at /mcp-oauth). |
| `resume_crawl` | Planned: no stage H item yet | A hosted job goes on slice by slice by itself; a failed or cancelled one is started again, not resumed. |
| `list_active_crawls` | Limited | An account's connection only (its key, or signing in at /mcp-oauth): its crawl jobs queued or running, in their job statuses and without the options each was started with (REST GET /v1/crawl/active). |
| `batch_scrape` | Limited | An account's connection only (its key, or signing in at /mcp-oauth): a job run a slice at a time, up to 1,000 URLs, each page priced as a scrape, results kept 7 days. |
| `get_batch` | Limited | An account's connection only (its key, or signing in at /mcp-oauth), for its own jobs; a finished job's status carries `export`, a link to every page as one gzip JSONL file, good for 24 hours (locally `resultFiles`, the files written beside the task). |
| `get_batch_items` | Limited | An account's connection only (its key, or signing in at /mcp-oauth); limit at most 50 (local 50), maxResults at most 100 (local 200). |
| `wait_batch` | Limited | An account's connection only (its key, or signing in at /mcp-oauth). |
| `cancel_batch` | Limited | An account's connection only (its key, or signing in at /mcp-oauth). |
| `get_batch_errors` | Limited | An account's connection only (its key, or signing in at /mcp-oauth); answers the failed items as { items, notServed, nextCursor, hasMore } (local { errors, robotsBlocked, notServed }), limit at most 100 (local 1,000). |
| `hand_off_batch` | Different on purpose | Reads or hands over the page in the person's own Chrome. Hosted never drives the person's Chrome (ADR 0005); stage H item 5 forwards such calls to the server on their own computer (decided 2026-11-04). |
| `scrape_product` | Limited | A key or a signed-in connection with the browser lane only. |
| `batch_products` | Planned: no stage H item yet | Not listed; more Amazon.sg work is paused (ROADMAP Paused). |
| `import_login` | Planned: stage H item 5, identity-bound features | Hosted keeps no saved logins and never will (decided 2026-11-04). Stage H item 5 forwards these calls to the server on the person's own computer, which holds their logins; a browser profile on Octocrawl's servers the person signed in to themselves is the fallback. |
| `list_logins` | Planned: stage H item 5, identity-bound features | Hosted keeps no saved logins and never will (decided 2026-11-04). Stage H item 5 forwards these calls to the server on the person's own computer, which holds their logins; a browser profile on Octocrawl's servers the person signed in to themselves is the fallback. |
| `remove_login` | Planned: stage H item 5, identity-bound features | Hosted keeps no saved logins and never will (decided 2026-11-04). Stage H item 5 forwards these calls to the server on the person's own computer, which holds their logins; a browser profile on Octocrawl's servers the person signed in to themselves is the fallback. |
| `preview_monitor` | Planned: stage H item 4, monitors and delivery | Monitors run on a server on the person's computer; stage H item 4 brings them to hosted. |
| `create_monitor` | Planned: stage H item 4, monitors and delivery | Monitors run on a server on the person's computer; stage H item 4 brings them to hosted. |
| `list_monitors` | Planned: stage H item 4, monitors and delivery | Monitors run on a server on the person's computer; stage H item 4 brings them to hosted. |
| `get_monitor` | Planned: stage H item 4, monitors and delivery | Monitors run on a server on the person's computer; stage H item 4 brings them to hosted. |
| `run_monitor` | Planned: stage H item 4, monitors and delivery | Monitors run on a server on the person's computer; stage H item 4 brings them to hosted. |
| `get_monitor_run` | Planned: stage H item 4, monitors and delivery | Monitors run on a server on the person's computer; stage H item 4 brings them to hosted. |
| `pause_monitor` | Planned: stage H item 4, monitors and delivery | Monitors run on a server on the person's computer; stage H item 4 brings them to hosted. |
| `resume_monitor` | Planned: stage H item 4, monitors and delivery | Monitors run on a server on the person's computer; stage H item 4 brings them to hosted. |
| `cancel_monitor_run` | Planned: stage H item 4, monitors and delivery | Monitors run on a server on the person's computer; stage H item 4 brings them to hosted. |
| `create_delivery_destination` | Planned: stage H item 4, monitors and delivery | Webhook destinations for Monitors and jobs are a local server's; stage H item 4 brings them to hosted. A hosted job's end can already be posted over REST (POST /v1/crawl, POST /v1/batches). |
| `list_delivery_destinations` | Planned: stage H item 4, monitors and delivery | Webhook destinations for Monitors and jobs are a local server's; stage H item 4 brings them to hosted. A hosted job's end can already be posted over REST (POST /v1/crawl, POST /v1/batches). |
| `pause_delivery_destination` | Planned: stage H item 4, monitors and delivery | Webhook destinations for Monitors and jobs are a local server's; stage H item 4 brings them to hosted. A hosted job's end can already be posted over REST (POST /v1/crawl, POST /v1/batches). |
| `resume_delivery_destination` | Planned: stage H item 4, monitors and delivery | Webhook destinations for Monitors and jobs are a local server's; stage H item 4 brings them to hosted. A hosted job's end can already be posted over REST (POST /v1/crawl, POST /v1/batches). |
| `list_deliveries` | Planned: stage H item 4, monitors and delivery | Webhook destinations for Monitors and jobs are a local server's; stage H item 4 brings them to hosted. A hosted job's end can already be posted over REST (POST /v1/crawl, POST /v1/batches). |
| `get_delivery` | Planned: stage H item 4, monitors and delivery | Webhook destinations for Monitors and jobs are a local server's; stage H item 4 brings them to hosted. A hosted job's end can already be posted over REST (POST /v1/crawl, POST /v1/batches). |
| `retry_dead_letter` | Planned: stage H item 4, monitors and delivery | Webhook destinations for Monitors and jobs are a local server's; stage H item 4 brings them to hosted. A hosted job's end can already be posted over REST (POST /v1/crawl, POST /v1/batches). |

| Option | Hosted | How |
| --- | --- | --- |
| `scrape` `url` | Supported |  |
| `scrape` `mode` | Supported |  |
| `scrape` `handoff` | Different on purpose | Reads or hands over the page in the person's own Chrome. Hosted never drives the person's Chrome (ADR 0005); stage H item 5 forwards such calls to the server on their own computer (decided 2026-11-04). Sent anyway: refused by name. |
| `scrape` `lane` | Different on purpose | Reads or hands over the page in the person's own Chrome. Hosted never drives the person's Chrome (ADR 0005); stage H item 5 forwards such calls to the server on their own computer (decided 2026-11-04). Sent anyway: refused by name. |
| `scrape` `access` | Planned: stage H item 2, hosted access | Hosted reads no access grant: its ladder is HTTP, then the Playwright browser for a connection with the browser lane. compatible_transport is granted per request by an operator switch, off by default (hostedConfig/compatibleTransport), on the hosts a local server uses it for; the live service's state is not checked here. Stage H item 2 adds hosted_chrome, a real Chrome on Octocrawl's servers: a deploy that carries its operator config (W2L_HOSTED_CHROME_*) takes access enhanced from an account its switch lists (hostedConfig/hostedChrome) and refuses it from any other caller; a deploy without it refuses it from everyone. CAPTCHA solving behind a switch comes after. access standard, the default, is taken; my-browser is refused by name, and so is enhanced from a caller the hosted_chrome switch does not list. Sent anyway: refused by name. |
| `scrape` `allowlistedDomains` | Supported |  |
| `scrape` `formats` | Supported |  |
| `scrape` `includeLinks` | Supported |  |
| `scrape` `debug` | Supported |  |
| `scrape` `markdownOffset` | Supported | MCP only, here and locally: where the answer's Markdown starts, to read on from a cut answer. REST answers the whole Markdown. |
| `scrape` `onlyMainContent` | Supported |  |
| `scrape` `waitFor` | Limited | A connection with the browser lane only, at 5 credits a page; one without does not list it, and one that sends it anyway, or an account with fewer credits, is refused: a page is never read without the wait. |
| `scrape` `timeout` | Limited | At most 180 s, and 120 s when left out (local 300 s, and 120 s when left out): 60 s inside the 240 s claude.ai gives a tool call, under Cloud Run's 300 s request timeout. |
| `scrape` `maxFileBytes` | Limited | At most 5 MiB (local 500 MiB). A file is read and hashed, and not saved: `file.path` is `null`. |
| `scrape` `includeTags` | Supported |  |
| `scrape` `excludeTags` | Supported |  |
| `scrape` `headers` | Supported |  |
| `scrape` `mobile` | Supported |  |
| `scrape` `skipTlsVerification` | Different on purpose | A hosted engine never loosens certificate checks for a caller (engine.ts checkHostedOptions). Sent anyway: refused by name. |
| `scrape` `fastMode` | Supported |  |
| `scrape` `blockAds` | Supported |  |
| `scrape` `removeBase64Images` | Supported |  |
| `scrape` `verify` | Supported | Every check type, markdownMatches and markdownCountMin included: pattern checks run with the same time limits on every server (ADR 0006, amendment of 2026-10-29). Hosted takes those two behind its operator's switch, W2L_HOSTED_REGEX_CHECKS=1, on api.octocrawl.dev and the job worker since 2026-10-31; an instance whose start-up self-test failed refuses them by name. |
| `scrape` `maxAge` | Planned: stage H item 8, beyond local | Hosted keeps no page cache (storeInCache is pinned off); stage H item 8 plans one shared across users. Taken, and always read live. Sent anyway: taken, not honoured. |
| `scrape` `minAge` | Planned: stage H item 8, beyond local | Hosted keeps no page cache (storeInCache is pinned off); stage H item 8 plans one shared across users. Taken, and always read live. Sent anyway: taken, not honoured. |
| `scrape` `storeInCache` | Planned: stage H item 8, beyond local | Hosted keeps no page cache (storeInCache is pinned off); stage H item 8 plans one shared across users. Taken, and nothing is stored. Sent anyway: taken, not honoured. |
| `scrape` `lockdown` | Planned: stage H item 8, beyond local | Hosted keeps no page cache (storeInCache is pinned off); stage H item 8 plans one shared across users. Taken, and always failed/cache_miss. Sent anyway: taken, not honoured. |
| `scrape` `actions` | Planned: stage H item 3, actions and waitFor | A step runs the caller's clicks and scripts in the operator's browser; hosted takes none until that isolation is reviewed, and stage H item 3 brings actions to keyed callers. Sent anyway: refused by name. |
| `scrape` `robotsOverride` | Different on purpose | Hosted fetches from the operator's addresses and obeys robots.txt for every URL (ROADMAP Direction). Sent anyway: refused by name. |
| `scrape` `integration` | Supported |  |
| `map` `url` | Supported |  |
| `map` `search` | Supported |  |
| `map` `sitemap` | Supported |  |
| `map` `includeSubdomains` | Supported |  |
| `map` `ignoreQueryParameters` | Supported |  |
| `map` `limit` | Limited | At most 5,000 links (local 100,000). Stage H item 6, limits and capacity raises it. |
| `map` `timeout` | Limited | At most 60 s (local 300 s). Stage H item 6, limits and capacity raises it. |
| `map` `includePaths` | Supported |  |
| `map` `excludePaths` | Supported |  |
| `map` `regexOnFullURL` | Supported |  |
| `map` `crawlEntireDomain` | Supported |  |
| `map` `deduplicateSimilarURLs` | Supported |  |
| `map` `ignoreRobotsTxt` | Different on purpose | Hosted fetches from the operator's addresses and obeys robots.txt for every URL (ROADMAP Direction). Sent anyway: refused by name. |
| `map` `mode` | Supported |  |
| `map` `debug` | Supported |  |
| `map` `integration` | Supported |  |
| `crawl` `url` | Supported |  |
| `crawl` `mode` | Supported |  |
| `crawl` `access` | Planned: stage H item 2, hosted access | Hosted reads no access grant: its ladder is HTTP, then the Playwright browser for a connection with the browser lane. compatible_transport is granted per request by an operator switch, off by default (hostedConfig/compatibleTransport), on the hosts a local server uses it for; the live service's state is not checked here. Stage H item 2 adds hosted_chrome, a real Chrome on Octocrawl's servers: a deploy that carries its operator config (W2L_HOSTED_CHROME_*) takes access enhanced from an account its switch lists (hostedConfig/hostedChrome) and refuses it from any other caller; a deploy without it refuses it from everyone. CAPTCHA solving behind a switch comes after. Sent anyway: refused by name. |
| `crawl` `maxPages` | Limited | At most 500, and 100 when left out or null (locally null is no cap). Stage H item 6, limits and capacity raises it. |
| `crawl` `maxDepth` | Supported |  |
| `crawl` `useCached` | Planned: stage H item 8, beyond local | Hosted keeps no page cache (storeInCache is pinned off); stage H item 8 plans one shared across users. Sent anyway: refused by name. |
| `crawl` `allowlistedDomains` | Supported |  |
| `crawl` `formats` | Supported |  |
| `crawl` `includeLinks` | Supported |  |
| `crawl` `includePaths` | Supported |  |
| `crawl` `excludePaths` | Supported |  |
| `crawl` `regexOnFullURL` | Supported |  |
| `crawl` `ignoreQueryParameters` | Supported |  |
| `crawl` `deduplicateSimilarURLs` | Supported |  |
| `crawl` `crawlEntireDomain` | Supported |  |
| `crawl` `allowSubdomains` | Supported |  |
| `crawl` `allowExternalLinks` | Supported |  |
| `crawl` `sitemap` | Supported |  |
| `crawl` `ignoreRobotsTxt` | Different on purpose | Hosted fetches from the operator's addresses and obeys robots.txt for every URL (ROADMAP Direction). Sent anyway: refused by name. |
| `crawl` `maxConcurrency` | Limited | At most 8, the pages a job's slice reads at once (local: up to the server's worker count, W2L_WORKER_COUNT); 2 of them rendered in a browser at once; the start lowers a larger value to 8. Stage H item 6, limits and capacity raises it. |
| `crawl` `idempotencyKey` | Supported |  |
| `crawl` `webhook` | Planned: stage H item 4, monitors and delivery | REST takes it for the job's end (https, no headers or secretEnv), signed with a secret the start returns; the MCP tool refuses it so the secret stays out of the transcript. Sent anyway: served over REST only. |
| `crawl` `onlyMainContent` | Supported |  |
| `crawl` `waitFor` | Limited | A connection with the browser lane only, at 5 credits a page; one without does not list it, and one that sends it anyway, or an account with fewer credits, is refused: a page is never read without the wait. |
| `crawl` `timeout` | Supported |  |
| `crawl` `maxFileBytes` | Limited | At most 5 MiB (local 500 MiB). A file is read and hashed, and not saved: `file.path` is `null`. REST takes a larger value and the start refuses it by name. |
| `crawl` `includeTags` | Supported |  |
| `crawl` `excludeTags` | Supported |  |
| `crawl` `headers` | Supported |  |
| `crawl` `mobile` | Supported |  |
| `crawl` `skipTlsVerification` | Different on purpose | A hosted engine never loosens certificate checks for a caller (engine.ts checkHostedOptions). Sent anyway: refused by name. |
| `crawl` `fastMode` | Supported |  |
| `crawl` `blockAds` | Supported |  |
| `crawl` `removeBase64Images` | Supported |  |
| `crawl` `verify` | Supported | Every check type, markdownMatches and markdownCountMin included: pattern checks run with the same time limits on every server (ADR 0006, amendment of 2026-10-29). Hosted takes those two behind its operator's switch, W2L_HOSTED_REGEX_CHECKS=1, on api.octocrawl.dev and the job worker since 2026-10-31; an instance whose start-up self-test failed refuses them by name. |
| `crawl` `maxAge` | Planned: stage H item 8, beyond local | Hosted keeps no page cache (storeInCache is pinned off); stage H item 8 plans one shared across users. Sent anyway: refused by name. |
| `crawl` `minAge` | Planned: stage H item 8, beyond local | Hosted keeps no page cache (storeInCache is pinned off); stage H item 8 plans one shared across users. Sent anyway: refused by name. |
| `crawl` `storeInCache` | Planned: stage H item 8, beyond local | Hosted keeps no page cache (storeInCache is pinned off); stage H item 8 plans one shared across users. Sent anyway: refused by name. |
| `crawl` `lockdown` | Planned: stage H item 8, beyond local | Hosted keeps no page cache (storeInCache is pinned off); stage H item 8 plans one shared across users. Sent anyway: refused by name. |
| `crawl` `integration` | Supported |  |
| `batch_scrape` `urls` | Supported |  |
| `batch_scrape` `mode` | Supported |  |
| `batch_scrape` `lane` | Different on purpose | Reads or hands over the page in the person's own Chrome. Hosted never drives the person's Chrome (ADR 0005); stage H item 5 forwards such calls to the server on their own computer (decided 2026-11-04). Sent anyway: refused by name. |
| `batch_scrape` `access` | Planned: stage H item 2, hosted access | Hosted reads no access grant: its ladder is HTTP, then the Playwright browser for a connection with the browser lane. compatible_transport is granted per request by an operator switch, off by default (hostedConfig/compatibleTransport), on the hosts a local server uses it for; the live service's state is not checked here. Stage H item 2 adds hosted_chrome, a real Chrome on Octocrawl's servers: a deploy that carries its operator config (W2L_HOSTED_CHROME_*) takes access enhanced from an account its switch lists (hostedConfig/hostedChrome) and refuses it from any other caller; a deploy without it refuses it from everyone. CAPTCHA solving behind a switch comes after. Sent anyway: refused by name. |
| `batch_scrape` `formats` | Supported |  |
| `batch_scrape` `includeLinks` | Supported |  |
| `batch_scrape` `onlyMainContent` | Supported |  |
| `batch_scrape` `waitFor` | Limited | A connection with the browser lane only, at 5 credits a page; one without does not list it, and one that sends it anyway, or an account with fewer credits, is refused: a page is never read without the wait. |
| `batch_scrape` `timeout` | Supported |  |
| `batch_scrape` `maxFileBytes` | Limited | At most 5 MiB (local 500 MiB). A file is read and hashed, and not saved: `file.path` is `null`. REST takes a larger value and the start refuses it by name. |
| `batch_scrape` `includeTags` | Supported |  |
| `batch_scrape` `excludeTags` | Supported |  |
| `batch_scrape` `headers` | Supported |  |
| `batch_scrape` `mobile` | Supported |  |
| `batch_scrape` `skipTlsVerification` | Different on purpose | A hosted engine never loosens certificate checks for a caller (engine.ts checkHostedOptions). Sent anyway: refused by name. |
| `batch_scrape` `fastMode` | Supported |  |
| `batch_scrape` `blockAds` | Supported |  |
| `batch_scrape` `removeBase64Images` | Supported |  |
| `batch_scrape` `verify` | Supported | Every check type, markdownMatches and markdownCountMin included: pattern checks run with the same time limits on every server (ADR 0006, amendment of 2026-10-29). Hosted takes those two behind its operator's switch, W2L_HOSTED_REGEX_CHECKS=1, on api.octocrawl.dev and the job worker since 2026-10-31; an instance whose start-up self-test failed refuses them by name. |
| `batch_scrape` `maxAge` | Planned: stage H item 8, beyond local | Hosted keeps no page cache (storeInCache is pinned off); stage H item 8 plans one shared across users. Sent anyway: refused by name. |
| `batch_scrape` `minAge` | Planned: stage H item 8, beyond local | Hosted keeps no page cache (storeInCache is pinned off); stage H item 8 plans one shared across users. Sent anyway: refused by name. |
| `batch_scrape` `storeInCache` | Planned: stage H item 8, beyond local | Hosted keeps no page cache (storeInCache is pinned off); stage H item 8 plans one shared across users. Sent anyway: refused by name. |
| `batch_scrape` `lockdown` | Planned: stage H item 8, beyond local | Hosted keeps no page cache (storeInCache is pinned off); stage H item 8 plans one shared across users. Sent anyway: refused by name. |
| `batch_scrape` `actions` | Planned: stage H item 3, actions and waitFor | Hosted takes no page actions yet; stage H item 3 brings them to keyed callers. Sent anyway: refused by name. |
| `batch_scrape` `robotsOverrides` | Different on purpose | Hosted fetches from the operator's addresses and obeys robots.txt for every URL (ROADMAP Direction). Sent anyway: refused by name. |
| `batch_scrape` `maxConcurrency` | Limited | At most 8, the pages a job's slice reads at once (local: up to the server's worker count, W2L_WORKER_COUNT, with no ceiling in the schema); 2 of them rendered in a browser at once; the start lowers a larger value to 8, as a crawl's. Stage H item 6, limits and capacity raises it. |
| `batch_scrape` `retry` | Supported |  |
| `batch_scrape` `ignoreInvalidURLs` | Supported |  |
| `batch_scrape` `allowExternalLinks` | Planned: no stage H item yet | Locally only false is taken, since a batch follows no link; hosted refuses the field by name. Sent anyway: refused by name. |
| `batch_scrape` `includeSubdomains` | Planned: no stage H item yet | Locally only false is taken, since a batch follows no link; hosted refuses the field by name. Sent anyway: refused by name. |
| `batch_scrape` `idempotencyKey` | Supported |  |
| `batch_scrape` `appendToId` | Planned: no stage H item yet | A hosted batch's URLs are fixed when it starts. Sent anyway: refused by name. |
| `batch_scrape` `webhook` | Planned: stage H item 4, monitors and delivery | REST takes it for the job's end (https, no headers or secretEnv), signed with a secret the start returns; the MCP tool refuses it so the secret stays out of the transcript. Sent anyway: served over REST only. |
| `batch_scrape` `integration` | Supported |  |
| `scrape` `mode: authed` | Planned: stage H item 5, identity-bound features | Hosted holds no saved logins: the schema leaves the value out, and REST reads such a page anonymously. Sent anyway: taken, not honoured. |
| `scrape` `access: my-browser` | Different on purpose | Reads or hands over the page in the person's own Chrome. Hosted never drives the person's Chrome (ADR 0005); stage H item 5 forwards such calls to the server on their own computer (decided 2026-11-04). Sent anyway: refused by name. |
| `scrape` `formats: screenshot` | Limited | A connection with the browser lane only, at 5 credits a page; an account with fewer credits reads over HTTP alone, where a screenshot is refused. The same on crawl and batch_scrape. |
| `scrape` `formats: screenshot@fullPage` | Limited | A connection with the browser lane only, at 5 credits a page; an account with fewer credits reads over HTTP alone, where a screenshot is refused. The same on crawl and batch_scrape. |
| `scrape` `formats: json.modelFallback` | Planned: stage H item 7, the json model fallback | Listed, and answered as switched off until the model fallback is deployed with a model and switched on (hostedConfig/model); the live service's state is not checked here. Sent anyway: taken, not honoured. The same on crawl and batch_scrape. |
| `batch_scrape` `mode: authed` | Planned: stage H item 5, identity-bound features | Hosted holds no saved logins; the start refuses the value by name. Sent anyway: refused by name. |
| `batch_scrape` `access: my-browser` | Different on purpose | Reads or hands over the page in the person's own Chrome. Hosted never drives the person's Chrome (ADR 0005); stage H item 5 forwards such calls to the server on their own computer (decided 2026-11-04). Sent anyway: refused by name. |
<!-- hosted-parity:end -->