--- name: botify-mcp description: > Call Botify MCP tools correctly, and fan a tool out with batch_run so lists of URLs or rows never enter the conversation. Use whenever the user is on the Botify MCP, asks to run a tool on many URLs, pages or rows, mentions batch_run, batch_status, batch_sample, batch_cancel, or wants to keep I/O out of the LLM. Also use as soon as there are more than about ten items to process. --- # Botify MCP Every Botify MCP tool processes **exactly one item**. Fanning a tool out is `batch_run`'s job: the records are read from a SQL query, a table or a file, and the results are written to a sink. They do not pass through this conversation. For SQL over the catalog (`tables_list`, `tables_query`, `catalog.` prefix), also read `../botify-tables-sql/SKILL.md`. Tools may be deferred: call `tool_search` (e.g. `tool_search(query="batch_run html_code_executor")`) before the first use. Every call except `list_projects` needs `organization` and `project`. If the slugs are unknown, call `list_projects` rather than guessing. Retry a timed-out connector once or twice before treating it as a failure. ## One item per call ``` tool_name organization: acme project: www items: - { …one item matching the tool's schema… } config: { …optional, shared settings… } ``` `items` is an array of length 1. A second element is refused with an error that names `batch_run`. Do not loop the same tool tens of times as a workaround. Standalone tools (no `items`): `list_projects`, `batch_status`, `batch_sample`, `batch_cancel`. ## When to use batch_run | Situation | What to do | |---|---| | One URL / one question | Call the tool directly | | 2–10 small results you will read now | `batch_run` is fine; the run stays **inline** and the results come back | | **More than ~10 records**, or any list you do not want in context | `batch_run` with a `sql`, `table` or `file` source | | Large HTML, a crawl URL set, an uploaded CSV | Same: `batch_run`. Never paste the rows into the prompt | The threshold is 10. At 11 records the run becomes a **job**: you get a `job_id`, not the data. That is the point. **The failure to avoid:** `tables_query` a few thousand URLs, then call `html_fetch` / `html_code_executor` / `html_grep` once per row. The list has already entered the context, each page would again, and you will hit the one-item cap. Explore with aggregates and a `LIMIT` of 5–20, then hand the **same SQL** to `batch_run` as a `sql` source. `rows` is the only source whose content travels through the model. It is capped at 256 KB and refused above that. Prefer `sql` or `file`. ## batch_run Read the target tool's item schema first (`tool_documentation` when it has extended docs — `html_code_executor` does). `item_map` must match that schema. `batch_run` cannot target itself. ``` batch_run organization: acme project: www items: - tool: html_code_executor source: type: sql query: | SELECT url FROM catalog.crawl_pages_20260816 WHERE segments__pagetype__value = 'product' AND compliant__is_compliant LIMIT 500 item_map: url: "{{row.url}}" tool_config: function: "function() { return {h1: document.querySelector('h1')?.innerText}; }" dry_run: true config: on_error: skip ``` Then the same call with `dry_run: false`. | Field | Role | |---|---| | `tool` | Target tool id (`html_code_executor`, `html_grep`, `html_fetch`, `ai_gateway`, …) | | `source` | Where records come from (`sql`, `table`, `file`, `rows`) | | `item_map` | Projection onto the tool's item. Omit when each record already has that shape | | `tool_config` | The target's **config**, shared by every record (JS function, grep queries, LLM schema) | | `sink` | Where results go. Default: a gzipped JSONL file whose URL is a valid `file` source | | `dry_run` | First 5 records only, results in full, nothing launched, no sink | | `config.on_error` | `skip` (default): record the failure and continue. `fail_job`: stop on the first error | ### Sources | `type` | Use when | Notes | |---|---|---| | `sql` | The list already lives in the catalog | BigQuery Standard SQL, `catalog.` prefix. Optional `limit` | | `table` | A whole catalog table | Bare name or `catalog.name`. Optional `limit` | | `file` | A CSV/TSV the user uploaded, or the previous batch's JSONL | URL from `https://app.botify.com/:organization/:project/o/storage/tmp`, a previous `sink.url`, or a public HTTP(S) URL. CSV, TSV, JSONL; gzip allowed. `format` defaults to `auto` | | `rows` | A handful of objects you already have | Last resort. This **is** I/O through the LLM | Do not `tables_query` the full set and copy it into `rows`. A `COUNT(*)` plus `LIMIT 20` is enough to understand the population; `batch_run` re-runs the query itself. Ask the user to upload a local file rather than pasting it. ### item_map A projection, not an expression language: `{{row.}}` and dotted paths. A path that does not resolve is refused on the sample **before** the job starts. - `{"url": "{{row.page_url}}"}` — whole placeholder: type is preserved (number stays a number) - `"https://{{row.domain}}/{{row.path}}"` — interpolation: always a string - `{{row.blocks.0.text}}` — walk objects and list indexes - Namespace is `row` only (`{{rows.url}}` is rejected) Omit `item_map` when the source columns already are the tool's item (a `rows` source written in the item shape, or a SQL `SELECT url` against `html_code_executor`). ### Sinks and chaining Each written record is `{item, response, error}`. `item` is kept so a tool that does not echo its input stays attributable. **File (default).** Gzipped JSONL. The returned `sink.url` is the `file` source of the next batch. A previous batch's lines become `row.item.*` and `row.response.*`: ``` item_map: user_prompt: "Classify this title: {{row.response.result.h1}}" ``` **Table.** Editable catalog, then query it with `tables_query` instead of pulling the file back. ``` sink: type: table table: product_h1s replace_existing: false row_map: url: "{{item.url}}" h1: "{{response.result.h1}}" ``` `row_map` sees `{{item.*}}`, `{{response.*}}` and `{{row.*}}`. It is not strict: a failed record has no `response`. The default (no `row_map`) stores raw `item` and `response` JSON columns. Never write into crawl/logs/GSC tables; pick a new name. ### What comes back | `mode` | When | In the tool result | |---|---|---| | `dry_run` | `dry_run: true` | 5 `{item, response, error}` objects. No job, no sink | | `inline` | ≤ 10 records | All results. A sink is still written if you asked for one | | `job` | > 10 records | `job_id`, `rows_total`, `estimated_credits`, `sink`. **`results` is null** | Do not try to fetch a job's full output into the conversation. ## Following a job ``` batch_status organization, project, job_id batch_sample organization, project, job_id, limit (default 5, hard cap 20) batch_cancel organization, project, job_id ``` `batch_status` states: `pending`, `running`, `completed`, `failed`, `cancelled`. Poll it; do not start a second job for the same work. `rows_ok` / `rows_error` and `sample_errors` tell you whether to cancel. `batch_sample` is a peek so you can check the mapping before trusting the sink. The cap of 20 exists because a batch exists to keep its data out of the chat. Do not raise `limit` to "download" the output. `batch_cancel` stops at the next chunk. Records already written stay in the sink; the rest is not paid for. ## Typical pipelines **1. Extract structured fields from many crawled URLs** 1. `tables_list` / `tables_query` — latest SiteCrawler crawl, `COUNT(*)` and a 10-row sample of the population. 2. `tool_documentation` `html_code_executor` — confirm the JS DOM API. 3. `batch_run` `dry_run: true` — `sql` source, `item_map: {url: "{{row.url}}"}`, JS in `tool_config.function`. 4. Same call, `dry_run: false`. If `mode` is `job`, poll `batch_status`. 5. `batch_sample` a few records. Chain the `sink.url` or query the table sink. **2. Grep a string or regex across a URL set** Same as above with `tool: html_grep`. `url` is the item; `queries` go in `tool_config` (shared by every record). Prefer this to fetching HTML and searching in the prompt. **3. Fetch HTML without pouring it into the chat** Do not batch `html_fetch` if you would then read `html`. Either: - set `save_to_tmp_storage: true` on the item (via `item_map` / source columns) so each result is a temporary URL, then batch `html_code_executor` / `html_grep` on those URLs, or - skip the fetch and batch `html_code_executor` / `html_grep` on the live URLs (they load crawl HTML themselves). **4. User-supplied CSV** Upload at `https://app.botify.com/:organization/:project/o/storage/tmp`, then: ``` source: type: file url: item_map: url: "{{row.url}}" ``` **5. Second tool over the first batch's output** ``` source: type: file url: item_map: user_prompt: "{{row.response.result.h1}}" tool_config: # ai_gateway system_prompt, structured_output_json_schema, model_settings ``` ## Do not - Call a tool in a loop because `items` only takes one element. - `tables_query` the full URL list, then feed it as `rows` or as N tool calls. - Use a `rows` source for more than a handful of small objects. - Raise `batch_sample.limit` (or `tables_query.max_results`) to pull a batch back into the context. - Batch `html_fetch` and then read the HTML. - Target `batch_run` with `batch_run`. - Skip `dry_run` on a new `item_map` or a new JS/LLM config. Credits: a job reports `estimated_credits` at launch. `dry_run` is how you check a mapping without paying for the source.