# Toolkit workflow Concepts for driving the AI Browser Toolkit well. Read this before touching any site. These are the lessons that took the longest to learn. ## The mental model A long-running server owns a real Chrome window, and **`abt` drives it**. You do not write WebDriver code; you run small commands and read JSON back. The browser stays up between commands, so tabs, logins and focus persist. ```bash # lifecycle -- these are the only subcommands abt status # a server up? usually yes. URL and tabs abt up # start one if not; returns immediately abt browser start # a SEPARATE step from starting the server abt ops # every op and its exact parameters abt guidelines show toolkit-workflow # this document abt guidelines search # a playbook for the site you are on? # every page action goes through command-list, and it takes a LIST. # Send everything you already know in one round trip: abt command-list '[{"op":"goto","url":"https://example.com/login"}, {"op":"find","css":"input"}]' # find gave you levels. Act on all of them in the NEXT single call -- # this is the shape almost every task has, and the one most often # paid for as three round trips instead of one: abt command-list '[{"op":"input","level":"AEBCa","value":"me@example.com"}, {"op":"input","level":"AEBCb","value":"hunter2"}, {"op":"click","level":"AEBCc"}]' # a lone op is just a list of one, and is the exception, not the habit abt command-list '{"op":"reload"}' ``` **`abt command-list` is how you drive the page -- one command, taking an op or a list of them.** The subcommands are lifecycle only — start the server, start the browser, look at what is recorded. Every op in the tables below goes through `command-list`, spelled exactly as the table spells it, so there is no second vocabulary to learn and nothing that can disagree with `abt ops`. On PowerShell, pipe JSON rather than quoting it inline: `'{"op":"press","key":"Enter"}' | abt command-list -`. **Never run `abt serve` from a tool call.** That is the command loop itself: it never returns, so whatever launched it hangs until killed. `abt up` is the one you want — safe at any time, no-ops when a server already answers, and spawns the server outside your job object so your call returns while it keeps running. The server usually *is* already running, holding tabs and logins that must not be thrown away, so check before starting anything. **Batch what you already know — this is the single biggest thing you can do here.** `abt command-list` runs a list in order and stops at the first error, naming it, so a batch is never a blind leap. Measured on a real benchmark, agents sent one op per call 64% of the time and paid a full model round trip for each: typing a value and pressing Enter, billed twice, in a pair that was never in doubt. The rule is simple: **before you send, ask what else you already know.** You know the whole form once you have its levels. You know `press Enter` follows `input`. You know the click that follows the select. Those go in the same call. What you cannot know yet — which product the search returns — is where a batch legitimately ends. **The corollary, and it costs turns when missed: never put a read behind an action you are unsure of.** A list stops at the first failure, so if the click you guessed does not land, the `get_text` you queued behind it never runs and you have spent a turn learning one thing instead of two. A watched agent did exactly this twice in a row on a grid header it could not hit. Send the uncertain action alone, or pass `continue_on_error` when you genuinely want the rest to run regardless. **The viewer.** `/viewer` in a browser tab replays every command and response as it happens — the best debugging tool here. Open it beside your work. **Underneath is HTTP**, and that surface is supported: `POST /command-list`, `GET /status`, `GET /ops`. Every `abt` subcommand except `serve` is one of those requests. Use it directly when you are already making HTTP calls and want to avoid a process launch per command; otherwise prefer `abt`. ## The op groups | Group | Ops | |---|---| | Navigate | `goto` `back` `forward` `reload` `current_url` | | Read | `find` `find_full` `get_text` `get_html` `run_js` `screenshot` | | Inspect | `read_console` `read_network` | | Interact | `click` `input` `press` `select` `hover` `scroll` `wait_for` | | Tabs | `tab_new` `tab_switch` `tab_close` `tab_list` `tab_claim` `tab_release` | | Control | `diff` `status` `shutdown` `files` | ### Sessions Every command runs in a session — a profile, its own tabs, its own log. With none named it is `default`, which behaves as the server always has. Whoever launches you picks your session; do not pick one yourself. - `--session` is **not** a security boundary. Any process can name any open session; it only keeps cooperating agents out of each other's way. - **Sealed** sessions need a token only their launcher holds. `session_sealed` means it is not yours to use. - A tab belongs to the session that opened it, and popups follow it. Another session's tab shows in `tab_list` as `locked` and refuses every action with `tab_locked` — open your own with `tab_new` rather than retrying. - `tab_claim` takes an `unowned` tab; `tab_release` gives one up. - **Files:** a page's file input takes only files from the profile's uploads folder (`Documents/AI Browser Toolkit//uploads`); anything else is `file_blocked`. Call `files` to see what is there and what was downloaded — names and paths, never contents — and `{"op": "files", "open": true}` to open the folder so the person can drop the file in. Ask them to; do not guess paths. - Sessions on different profiles, and on the same profile, run in parallel. A few sites have sequences that always run together, and those are packaged as shortcuts of their own. They are shortcuts over these same ops, never a replacement: when one does not fit what you need, drive the page with the ops directly. Which sites have them, and what they do, belongs in that site's playbook — `abt guidelines search `. ## Targeting - **`near` qualifies a selector that matches too much.** A table whose every row has an `Edit` button is the case: `text` alone matches all of them, and an `index` is a guess that breaks the moment a row is added. ```json {"op": "click", "text": "Edit", "near": "Medication"} ``` It picks the match sharing the *smallest* container with that text — the row on a table, the section on a card. **Do not reach for `run_js` to stamp an attribute on the right row and click that.** An agent doing exactly that spent 14 of its 33 `run_js` calls on it, and when one stamp failed it left a selector matching nothing and retried the same click ten times. A `near` that matches nothing lists the qualifiers that do exist, so a miss tells you what to ask for instead. - Ops that touch an element take exactly **one** of `css`, `xpath`, `text` (exact visible text), or `level`. An `index` picks the Nth match. - `find` returns compact element **shells** (tag + attributes, no children) — perfect for surveying a page cheaply. `find_full` returns inner content. - Every match carries its `level`. Act on it directly: `find` then `click {"level": "AEDBa"}` — no re-selecting. - A level names a position, and it is checked before it acts: if the page changed and something else now sits there, the op fails rather than clicking the wrong thing. - Levels describe one page. A navigation renumbers them, so use a level from the result you were most recently given. - **A control the page just revealed already carries its level in the diff.** Reaching for `find` right after a click is usually a wasted round trip. ## Diffs: your primary feedback loop > **Trust the diff. Do not re-read the page to check what a command did.** > > This is the most common and most expensive mistake made with this toolkit. A > click succeeds, returns a `dom_diff` saying exactly what changed — and the > agent fires `get_text {"css": "body"}` to "see what happened". You already > have what happened. That follow-up costs a round trip and can cost tens of > thousands of tokens, to learn what was in the response you just received. > > The diff is snapshotted from the live DOM either side of the command. It is > not a hint to be confirmed. **Read it and act.** (It was once worth > distrusting — navigations snapshotted before an SPA had rendered. Navigation > now waits for network idle and a still DOM. A `Loading…` in a diff today is a > bug to report, not a reason to re-read.) Interactive ops (`click input press select hover scroll wait_for run_js`) and navigation ops (`goto back forward reload`) snapshot before/after and return `dom_diff`. Three tracks: **`text` — always on, no budget.** Strings that appeared on screen, one entry per element, plus form-control values. Read this first; on most pages it is the whole answer. ```json "text": {"added": ["ACBa Widgets", "ACBb Gadgets"], "removed_count": 1, "truncated": false} ``` ### Every string says where it sits The letters in front of each string are the element's **position in the page**, one letter per level down from the body: ``` AEDBAAAB a 000000192 b Sep 3, 2022 c $109.00 AEDBAAAC a 000000174 b May 2, 2022 ``` Read it this way: - **A longer prefix is deeper.** `AEDB` sits inside `AED`, which sits inside `AE`. - **A shared prefix means a shared container.** `AEDBAAABa` and `AEDBAAABb` are siblings — the same table row, the same form group, the same list item. This is how you tell which cells belong together, and it is the thing a flat list of strings cannot tell you. Counting rows, matching a value to its label and reading a table all depend on it. - **Past the 52nd sibling the level is a number** (`AB53`), and a dot only ever separates two numbers (`AB100.200`). Nothing else uses dots. **A `#` line is interactable, and the same address acts on it.** It is also an edge: everything inside it is on that one line. ``` AEDBa#btn Save changes AEDBb#lnk Issues → /dashboard/issues?assignee_username=byteblaze AEDBc#inp-q laptop AEDBd#sel-country United Kingdom AEDBe#lnk → /notifications ``` `#btn` button · `#lnk` link, target after the arrow · `#inp` text field · `#sel` select · `#chk` `#rad` `#opt` `#file`. On `#inp` and `#sel` the **name is in the mark** and the line's text is the **current value** — so a field you typed into shows its new value on the next diff. ```json {"op": "click", "level": "AEDBa"} {"op": "input", "level": "AEDBc", "value": "laptop pro"} ``` This is the cheapest way to act on something you have just read: no `find` in between, because the line already carries the address. Use a selector when you are looking for something by what it is; use a level when you are acting on something you can already see. **The prefix is an address.** Give it back to read that part of the page and nothing else: ```json {"op": "get_text", "level": "AEDBAAAB"} ``` That returns one subtree. It is how you re-read a table after acting on it, and how you look at something a navigation reported as unchanged — the page told you where it was, and it is still there. Prefer it over `get_text {"css": "body"}`, which pulls the whole document to answer a question about one part of it. Levels describe **one page**. A navigation renumbers them, so use a level from the result you were most recently given, not one you remember from earlier. ### What a navigation leaves out Landing on a page hands you its text. Strings identical to the page you came from are **not repeated** — the nav, header, footer and other furniture you have already read. A line at the end says how many were held back and names a level that holds some of them, so nothing is hidden from you: ask for that level if you need it. What is shown is what changed. Only *rendered* text counts, so a hover that reveals a menu reads as its items appearing, and an attribute-only change (`aria-expanded`, a class flip) shows as an empty diff. That is the trade: text stays clean because it drops that churn. Removals are counted, not listed — on a page that rewrites itself they are the whole old document. Add `"include_removed": true` when the count says it matters. **After a navigation, `added` is the whole destination page.** `goto` `back` `forward` `reload`, and any click that redirected, return the text of the page they landed on. So you never need a `find` or `get_text` merely to see what is on a page you just opened. The element track is skipped there. **Controls arrive on the text track itself.** A line whose address carries `#role` is something you can operate, and that same address acts on it — click, read what appeared, click what appeared, with no `find` in between: ```json {"op": "click", "css": "#insert-menu"} → "text": {"added": [ "AEDBa#btn Chart", "AEDBb#btn Pivot table", "AEDBc#btn Macro"]} {"op": "click", "level": "AEDBa"} ``` `#btn` button · `#lnk` link, target after the arrow · `#inp` text field · `#sel` select · `#chk` `#rad` `#opt` `#file`. On `#inp` and `#sel` the name is in the mark and the line's text is the current value, so a field you typed into shows its new value on the next diff. **A hidden file input is still reported**, because sites hide the real `` behind a custom uploader. Look for `#file` and write the path straight to it — `input` handles the hiding: ```json {"op": "input", "level": "AEDBd", "value": "C:/shots/page.png"} ``` **`elements` — pass `element_diff: true`.** A line-per-element unified diff with tags, ids, classes and attributes. For a change with no visible text, or when you need a selector for something the text track reported. Budget it with `diff_max_tokens`; passing a budget implies `element_diff: true`. - Navigation and tab ops reset the baseline automatically. - Suppress noise on one command with `"diff": false`; disable entirely with `--no-diff`. - Set a manual baseline with `{"op": "diff", "reset": true}`, then re-check with `{"op": "diff"}` to catch async SPA updates. The manual `diff` is explicit, so it returns everything: both tracks, removals listed. ### When the diff looks empty The one case that tempts you back into re-reading the page. Don't — there are three causes and each has a cheaper answer: 1. **No visible text changed** — a class flip, `aria-expanded`, a `data-` attribute. Ask for the element track: `{"op": "click", "css": "…", "element_diff": true}`. 2. **It has not landed yet** — an SPA updating after the command returned. Take a second look with `{"op": "diff"}`, which compares against the state the last command left. That is what it is for. 3. **Nothing happened** — a real outcome, and worth knowing. `click` hit-tests first and raises `not_interactable` naming what covered it, so `ok: true` with an empty diff genuinely means the click landed and the page did not react. `get_text {"css": "body"}` answers none of these better, and costs more than all three together. There used to be a fourth cause — a change inside an iframe, invisible to every selector. All three tracks now read into frames and refs from inside them act normally; you never switch frames yourself. **Verify effects with the diff, not with screenshots or downloads.** Downloads and exports lag; the diff is live. **The loop, in one line:** act, read `dom_diff.text.added` to see what appeared, act on the level the diff reported. A `find` between those steps is usually a round trip you did not need. ## Searching for something that should be there The rule above is about *verifying what a command did*. This one is about *looking for something*, and its failure mode is the opposite: a search comes back empty, the agent reads that as "my selector was wrong", and starts guessing. On a live LinkedIn profile an agent searched for `input[type=file]`, got `count: 0`, and spent **fifteen commands and six minutes** widening `run_js` scans. There was no file input — the page creates it when you engage the drop zone. **A search that finds nothing is an answer.** Climb this ladder once, then believe it: | | step | what it covers | |---|---|---| | 1 | **Re-read the response you have** | A navigation already returned what changed (or, off-site, the page) in `text.added` | | 2 | **`find`** — `css`, then `text` | The document and every frame | | 3 | **`get_text`** | Rendered text, *including open shadow roots* | | 4 | **`find` with `"shadow": true`** | Turns shadow content into a level you can act on | | 5 | **Stop** | It is not there | Step 3 follows the composed tree, so a component's internals are in it with no flag — discovery is free. Step 4 exists because reading a label is not the same as being able to click it. **You are told when step 4 is worth taking.** Shadow roots are counted on every snapshot but never walked: ```json {"op": "find", "css": "#resume"} → {"count": 0, "shadow_hosts": 2, "note": "nothing matched, but this page has 2 shadow root(s) … retry with \"shadow\": true"} ``` No hosts, no note — so on most pages this costs and says nothing. **Do not `run_js` a `querySelectorAll` to "look harder".** Steps 2–4 already searched the document, its frames and its open shadow roots. JavaScript finds the same nothing, a round trip and a few thousand tokens later. **The usual real cause:** the page has not created the control yet. Hidden file inputs are the classic — mounted when you click the visible upload button or drop zone. Act on what the page is showing you, then look again. **The one honest limit:** a `mode: "closed"` shadow root returns `null` from `.shadowRoot`, so no JavaScript can read it or prove it exists, and neither can this. "Not there" means *nothing reachable has it*. Closed roots are rare outside browser internals like `