---
name: vellum-browser-use
description: Browse the web using `assistant browser` CLI commands
compatibility: "Designed for Vellum personal assistants"
metadata:
  emoji: "🌐"
  vellum:
    category: "browsing"
    display-name: "Browser"
    activation-hints:
      - "Load first if you need to browse the web (navigating, clicking, extracting web content) via `assistant browser` commands"
---

Use this skill to browse the web. All browser operations are executed through the `assistant browser` CLI, invoked via `bash` or `host_bash`. Each operation is a subcommand:

| Command                               | Description                                        |
| ------------------------------------- | -------------------------------------------------- |
| `assistant browser navigate`          | Navigate to a URL                                  |
| `assistant browser snapshot`          | List interactive elements on the current page      |
| `assistant browser screenshot`        | Take a visual screenshot                           |
| `assistant browser click`             | Click an element                                   |
| `assistant browser type`              | Type text into an input                            |
| `assistant browser press-key`         | Press a keyboard key                               |
| `assistant browser scroll`            | Scroll the page or a specific element              |
| `assistant browser select-option`     | Select an option from a native `<select>` element  |
| `assistant browser hover`             | Hover over an element to reveal menus/tooltips     |
| `assistant browser wait-for`          | Wait for a condition                               |
| `assistant browser extract`           | Extract page text content                          |
| `assistant browser wait-for-download` | Wait for a file download to complete               |
| `assistant browser fill-credential`   | Fill a stored credential into a form field         |
| `assistant browser attach`            | Attach the Chrome debugger to the active tab       |
| `assistant browser detach`            | Detach the Chrome debugger from the active tab     |
| `assistant browser close`             | Close the browser page                             |
| `assistant browser status`            | Diagnose browser backend readiness and setup steps |

## Getting Started — Check Browser Readiness

Before using any browser commands, run `assistant browser --json status` first to check which browser backends are available. The status command returns JSON with readiness information for each backend mode:

```bash
assistant browser --json status
```

The response includes:

- `recommendedMode` — the best available backend (use this)
- `modes[]` — per-mode status with `available`, `summary`, and `userActions` (remediation steps)

For `--virtual-desktop`, Chrome and desktop components are included in the assistant image. Browser and computer use start the existing desktop without installing dependencies. If components are missing, report the image problem; do not install packages or launch a separate Chrome process.

## Browser Modes

Use `--browser-mode <mode>` on the `assistant browser` parent command to pin the browser backend:

| Value         | Backend          | Description                                                           |
| ------------- | ---------------- | --------------------------------------------------------------------- |
| `auto`        | Automatic        | Default. Picks the best available backend based on context.           |
| `extension`   | Chrome extension | Routes through the user's Chrome browser via the extension debugger.  |
| `cdp-inspect` | CDP inspect      | Connects to an already-running Chrome instance via DevTools Protocol. |
| `local`       | Playwright       | Drives a dedicated Playwright-managed Chromium instance.              |

```bash
assistant browser --browser-mode extension navigate --url http://www.example.com
```

### Prefer the Chrome Extension

The **Chrome extension** (`extension` mode) is the preferred browser backend. It is:

- More secure than Chrome's native remote debugging
- Uses the user's real browser profile (cookies, sessions, saved logins)
- Best experience for interacting with the user's actual browsing context

If the status check shows the extension is **not available**, encourage the user to install and pair it:

1. Install the **Vellum Assistant Chrome Extension** from the Chrome Web Store: https://chromewebstore.google.com/detail/vellum-assistant-browser/hphbdmpffeigpcdjkckleobjmhhokpne
2. Open the extension in Chrome and pair it with the assistant.

The status response's `userActions` array for the `extension` mode provides these same steps when the extension is not connected.

## When a Page Cannot Be Reached

If navigate, curl, or any fetch times out, hits an auth wall, or cannot reach a host (VPN, company login, internal dashboard):

1. Tell the user a connected desktop app or Chrome extension can open the page in a browser where they are already logged in.
2. Give the install links:
   - Desktop app: https://www.vellum.ai/downloads
   - Chrome extension: https://chromewebstore.google.com/detail/vellum-assistant-browser/hphbdmpffeigpcdjkckleobjmhhokpne
3. Offer those first. Only ask for a screenshot or pasted page content if they cannot install either.
4. On iOS or Android there is no in-app browser and no extension to install on the phone. Offer the desktop app or Chrome extension on a computer. Do not describe a browser panel.

### Fallback Modes

If the user declines to install the extension:

- **`cdp-inspect`** — Connects to an already-running Chrome instance via DevTools Protocol (Chrome 146+). Requires enabling remote debugging in Chrome settings.
- **`local`** — Drives a dedicated Playwright-managed Chromium instance. Last resort — does not use the user's browser profile.

Only fall back to these if the user explicitly indicates they do not want to install the extension. Prefer `cdp-inspect` over `local`.

## Targeting a Specific Client

When multiple clients support `host_browser` (e.g. two Chrome profiles, a macOS client and a Chrome extension), use `--target-client-id <id>` on the `assistant browser` parent command to pin all operations in the invocation to one specific client:

```bash
assistant browser --target-client-id <client-id> navigate --url https://example.com
```

Obtain client IDs from:

```bash
assistant clients list --capability host_browser
```

Omit `--target-client-id` when only one client is connected — the default interface-preference order (`chrome-extension` first, then `macos`) picks the best available client automatically.

## Tab Handling (Chrome extension)

On the Chrome extension backend, `navigate` opens a dedicated tab the first time it runs in a conversation and pins subsequent operations to it, so browsing never disturbs the tab the user is on (often the tab they're chatting with the assistant from). Later navigates reuse that pinned tab.

- `--new-tab` — force a brand-new tab even when one is already pinned.
- `--use-active-tab` — navigate the user's currently-active tab instead of a dedicated one.

Both flags are ignored on the `local` and `cdp-inspect` backends, which manage their own browser context.

## Session Management

Use `--session <id>` on the `assistant browser` parent command to group sequential operations so they share browser state (same page, cookies, etc.). Different session IDs create independent browser contexts.

```bash
assistant browser --session myflow navigate --url https://example.com
assistant browser --session myflow snapshot
assistant browser --session myflow click --element-id e3
```

Omitting `--session` uses the `default` session.

## Machine-Readable Output

Use `--json` on the `assistant browser` parent command to get structured JSON output suitable for parsing in scripts:

```bash
assistant browser --json navigate --url https://example.com
# {"ok":true,"content":"Page title: Example Domain"}

assistant browser --json snapshot
# {"ok":true,"content":"...element list..."}

assistant browser --json screenshot
# {"ok":true,"content":"...","screenshots":[{"mediaType":"image/jpeg","data":"<base64>"}]}
```

Error responses use `{"ok":false,"error":"..."}`.

## Screenshots

To save a screenshot to disk, use `--output <path>`:

```bash
assistant browser screenshot --output page.jpg
assistant browser screenshot --full-page --output full.jpg
```

To receive base64 screenshot data in JSON output:

```bash
assistant browser --json screenshot
```

The response includes a `screenshots` array with `mediaType` and `data` (base64) fields.

## Typical Workflow

1. `assistant browser --json status` to check backend readiness — if the extension is not available, help the user install it
2. (Optional) `assistant browser attach` to establish the session
3. `assistant browser navigate --url <url>` to load a page
4. `assistant browser snapshot` to discover interactive elements
5. Use `click`, `type`, `press-key`, `scroll`, `select-option`, or `hover` to interact
6. `assistant browser extract` or `assistant browser screenshot --output <path>` to capture results
7. **Always** `assistant browser detach` when you are done — this releases the debugger so the user can browse freely

## Human verification

Treat every CAPTCHA and bot-detection challenge as a request for human help, including drag-to-verify sliders, press-and-hold checks, verification checkboxes and image puzzles. Stop before interacting with the challenge, even if its controls look easy to automate. Do not try it yourself, retry it, or script a solution.

In the virtual desktop, request the desktop-help card immediately, using one short sentence for the needed action. Wait for Done or Skip. After Done, take a fresh snapshot; if verification remains, ask for help again. On other browser backends, ask the user to complete verification in their browser and wait for confirmation.

For ordinary logins, use saved credentials or securely prompt for missing credentials, then fill the form yourself. A CAPTCHA on a login page still requires human help.

## Interaction Strategies

**Date pickers / calendars:** Click the date input to open the picker, re-snapshot to see calendar controls, click month navigation arrows to reach the target month, then click the target date. For `<input type="date">`, use `type` with `YYYY-MM-DD` format.

**Native `<select>` elements:** Use `select-option` with `--value`, `--label`, or `--index`. Do not try to click individual `<option>` elements.

**ARIA / custom dropdowns:** Click to open, take a new `snapshot`, then click the desired option by `--element-id`.

**Autocomplete inputs:** Type the search text, wait 500-1000ms (`wait-for --duration`), re-snapshot for suggestions, then click the suggestion or use `press-key --key ArrowDown` + `press-key --key Enter`.

**Multi-step forms:** Complete each step, wait for the next section to load, re-snapshot to discover new elements, then proceed.

**Dynamic content:** After interactions that change the page, use `wait-for` (with `--selector` or `--text`) or re-snapshot to see updated elements before continuing.

**Scrolling:** Use `scroll --direction down` to reveal below-the-fold content before snapshotting. Long pages may require multiple scrolls.

**Hover menus / tooltips:** Use `hover` to reveal hidden menus or tooltips, then re-snapshot to see newly revealed elements.

## Verification

After critical actions (form submission, booking confirmation, checkout), take a screenshot and then read the saved image to visually verify results before reporting success to the user:

```bash
assistant browser screenshot --output /tmp/verify.jpg
```

Then read the saved image to inspect it before reporting success. Use `file_read` if the screenshot was taken via `bash`, or `host_file_read` if it was taken via `host_bash`.