--- name: agent-browser description: Automates browser and Electron app interactions for user-flow validation. --- # Browser Automation with agent-browser ## One-time setup (fresh machines) If you see an error like "Executable doesn't exist ... chrome-headless-shell", install the bundled Chromium: ```bash agent-browser install ``` ## Quick start ```bash agent-browser open # Navigate to page agent-browser snapshot -i # Get interactive elements with refs agent-browser click @e1 # Click element by ref agent-browser fill @e2 "text" # Fill input by ref agent-browser close # Close browser ``` ## Core workflow 1. Navigate: `agent-browser open ` 2. Snapshot: `agent-browser snapshot -i` (returns elements with refs like `@e1`, `@e2`) 3. Interact using refs from the snapshot 4. Re-snapshot after navigation or significant DOM changes ## Command chaining Commands can be chained with `&&` in a single shell invocation. The browser persists between commands via a background daemon, so chaining is safe and more efficient than separate calls. ```bash # Chain open + wait + snapshot in one call agent-browser open https://example.com && agent-browser wait --load networkidle && agent-browser snapshot -i # Chain multiple interactions agent-browser fill @e1 "user@example.com" && agent-browser fill @e2 "password123" && agent-browser click @e3 # Navigate and capture agent-browser open https://example.com && agent-browser wait --load networkidle && agent-browser screenshot page.png ``` **When to chain:** Use `&&` when you don't need to read the output of an intermediate command before proceeding (e.g., open + wait + screenshot). Run commands separately when you need to parse the output first (e.g., snapshot to discover refs, then interact using those refs). ## Commands ### Navigation ```bash agent-browser open # Navigate to URL (aliases: goto, navigate) # Supports: https://, http://, file://, about:, data:// # Auto-prepends https:// if no protocol given agent-browser back # Go back agent-browser forward # Go forward agent-browser reload # Reload page agent-browser close # Close browser (aliases: quit, exit) agent-browser connect 9222 # Connect to browser via CDP port ``` ### Snapshot (page analysis) ```bash agent-browser snapshot # Full accessibility tree agent-browser snapshot -i # Interactive elements only (recommended) agent-browser snapshot -i -C # Include cursor-interactive elements (divs with onclick, cursor:pointer) agent-browser snapshot -c # Compact output agent-browser snapshot -d 3 # Limit depth to 3 agent-browser snapshot -s "#main" # Scope to CSS selector ``` ### Interactions (use @refs from snapshot) ```bash agent-browser click @e1 # Click agent-browser click @e1 --new-tab # Click and open in new tab agent-browser dblclick @e1 # Double-click agent-browser focus @e1 # Focus element agent-browser fill @e2 "text" # Clear and type agent-browser type @e2 "text" # Type without clearing agent-browser press Enter # Press key (alias: key) agent-browser press Control+a # Key combination agent-browser keydown Shift # Hold key down agent-browser keyup Shift # Release key agent-browser hover @e1 # Hover agent-browser check @e1 # Check checkbox agent-browser uncheck @e1 # Uncheck checkbox agent-browser select @e1 "value" # Select dropdown option agent-browser select @e1 "a" "b" # Select multiple options agent-browser scroll down 500 # Scroll page (default: down 300px) agent-browser scrollintoview @e1 # Scroll element into view (alias: scrollinto) agent-browser drag @e1 @e2 # Drag and drop agent-browser upload @e1 file.pdf # Upload files ``` ### Get information ```bash agent-browser get text @e1 # Get element text agent-browser get text body > page.txt # Get all page text to file agent-browser get html @e1 # Get innerHTML agent-browser get value @e1 # Get input value agent-browser get attr @e1 href # Get attribute agent-browser get title # Get page title agent-browser get url # Get current URL agent-browser get count ".item" # Count matching elements agent-browser get box @e1 # Get bounding box agent-browser get styles @e1 # Get computed styles (font, color, bg, etc.) ``` ### Check state ```bash agent-browser is visible @e1 # Check if visible agent-browser is enabled @e1 # Check if enabled agent-browser is checked @e1 # Check if checked ``` ### Screenshots & PDF ```bash agent-browser screenshot # Save to a temporary directory agent-browser screenshot path.png # Save to a specific path agent-browser screenshot --full # Full page screenshot agent-browser screenshot --annotate # Annotated screenshot with numbered element labels agent-browser pdf output.pdf # Save as PDF ``` ### Video recording ```bash agent-browser record start ./demo.webm # Start recording (uses current URL + state) agent-browser click @e1 # Perform actions agent-browser record stop # Stop and save video agent-browser record restart ./take2.webm # Stop current + start new recording ``` Recording creates a fresh context but preserves cookies/storage from your session. If no URL is provided, it automatically returns to your current page. For smooth demos, explore first, then start recording. ### Diff (compare page states) ```bash agent-browser diff snapshot # Compare current vs last snapshot agent-browser diff snapshot --baseline before.txt # Compare current vs saved file agent-browser diff screenshot --baseline before.png # Visual pixel diff agent-browser diff url # Compare two pages agent-browser diff url --wait-until networkidle # Custom wait strategy agent-browser diff url --selector "#main" # Scope to element ``` Use `diff snapshot` after performing an action to verify it had the intended effect. This compares the current accessibility tree against the last snapshot taken in the session. ```bash # Typical workflow: snapshot -> action -> diff agent-browser snapshot -i # Take baseline snapshot agent-browser click @e2 # Perform action agent-browser diff snapshot # See what changed (auto-compares to last snapshot) ``` `diff snapshot` output uses `+` for additions and `-` for removals, similar to git diff. `diff screenshot` produces a diff image with changed pixels highlighted in red, plus a mismatch percentage. ### Wait ```bash agent-browser wait @e1 # Wait for element agent-browser wait 2000 # Wait milliseconds agent-browser wait --text "Success" # Wait for text (or -t) agent-browser wait --url "**/dashboard" # Wait for URL pattern (or -u) agent-browser wait --load networkidle # Wait for network idle (or -l) agent-browser wait --fn "window.ready" # Wait for JS condition (or -f) ``` ### Mouse control ```bash agent-browser mouse move 100 200 # Move mouse agent-browser mouse down left # Press button agent-browser mouse up left # Release button agent-browser mouse wheel 100 # Scroll wheel ``` ### Semantic locators (alternative to refs) When refs are unavailable or unreliable, use semantic locators: ```bash agent-browser find role button click --name "Submit" agent-browser find text "Sign In" click agent-browser find text "Sign In" click --exact # Exact match only agent-browser find label "Email" fill "user@test.com" agent-browser find placeholder "Search" type "query" agent-browser find alt "Logo" click agent-browser find title "Close" click agent-browser find testid "submit-btn" click agent-browser find first ".item" click agent-browser find last ".item" click agent-browser find nth 2 "a" hover ``` ### Browser settings ```bash agent-browser set viewport 1920 1080 # Set viewport size agent-browser set device "iPhone 14" # Emulate device agent-browser set geo 37.7749 -122.4194 # Set geolocation (alias: geolocation) agent-browser set offline on # Toggle offline mode agent-browser set headers '{"X-Key":"v"}' # Extra HTTP headers agent-browser set credentials user pass # HTTP basic auth (alias: auth) agent-browser set media dark # Emulate color scheme agent-browser set media light reduced-motion # Light mode + reduced motion ``` ### Cookies & Storage ```bash agent-browser cookies # Get all cookies agent-browser cookies set name value # Set cookie agent-browser cookies clear # Clear cookies agent-browser storage local # Get all localStorage agent-browser storage local key # Get specific key agent-browser storage local set k v # Set value agent-browser storage local clear # Clear all ``` ### Network ```bash agent-browser network route # Intercept requests agent-browser network route --abort # Block requests agent-browser network route --body '{}' # Mock response agent-browser network unroute [url] # Remove routes agent-browser network requests # View tracked requests agent-browser network requests --filter api # Filter requests ``` ### Tabs & Windows ```bash agent-browser tab # List tabs agent-browser tab new [url] # New tab agent-browser tab 2 # Switch to tab by index agent-browser tab close # Close current tab agent-browser tab close 2 # Close tab by index agent-browser window new # New window ``` ### Frames ```bash agent-browser frame "#iframe" # Switch to iframe agent-browser frame main # Back to main frame ``` ### Dialogs ```bash agent-browser dialog accept [text] # Accept dialog agent-browser dialog dismiss # Dismiss dialog ``` ### JavaScript (eval) Use `eval` to run JavaScript in the browser context. **Shell quoting can corrupt complex expressions** -- use `--stdin` or `-b` to avoid issues. ```bash # Simple expressions work with regular quoting agent-browser eval 'document.title' agent-browser eval 'document.querySelectorAll("img").length' # Complex JS: use --stdin with heredoc (RECOMMENDED) agent-browser eval --stdin <<'EVALEOF' JSON.stringify( Array.from(document.querySelectorAll("img")) .filter(i => !i.alt) .map(i => ({ src: i.src.split("/").pop(), width: i.width })) ) EVALEOF # Alternative: base64 encoding (avoids all shell escaping issues) agent-browser eval -b "$(echo -n 'Array.from(document.querySelectorAll("a")).map(a => a.href)' | base64)" ``` **Rules of thumb:** - Single-line, no nested quotes -> regular `eval 'expression'` with single quotes is fine - Nested quotes, arrow functions, template literals, or multiline -> use `eval --stdin <<'EVALEOF'` - Programmatic/generated scripts -> use `eval -b` with base64 ### Profiling ```bash agent-browser profiler start # Start Chrome DevTools profiling agent-browser profiler stop trace.json # Stop and save profile (path optional) ``` ## Global options ```bash agent-browser --session ... # Isolated browser session agent-browser --session-name ... # Named session with auto-save/restore agent-browser --json ... # JSON output for parsing agent-browser --headed ... # Show browser window (not headless) agent-browser --full ... # Full page screenshot (-f) agent-browser --cdp ... # Connect via Chrome DevTools Protocol agent-browser --auto-connect ... # Auto-discover running Chrome with remote debugging agent-browser -p ... # Cloud browser provider (--provider) agent-browser --proxy ... # Use proxy server agent-browser --headers ... # HTTP headers scoped to URL's origin agent-browser --executable-path

# Custom browser executable agent-browser --extension ... # Load browser extension (repeatable) agent-browser --allow-file-access ... # Allow file:// URL access for local files agent-browser --ignore-https-errors ... # Ignore HTTPS certificate errors agent-browser --config ... # Custom config file path agent-browser --help # Show help (-h) agent-browser --version # Show version (-V) agent-browser --help # Show detailed help for a command ``` ### Proxy support ```bash agent-browser --proxy http://proxy.com:8080 open example.com agent-browser --proxy http://user:pass@proxy.com:8080 open example.com agent-browser --proxy socks5://proxy.com:1080 open example.com ``` ## Environment variables ```bash AGENT_BROWSER_SESSION="mysession" # Default session name AGENT_BROWSER_EXECUTABLE_PATH="/path/chrome" # Custom browser path AGENT_BROWSER_EXTENSIONS="/ext1,/ext2" # Comma-separated extension paths AGENT_BROWSER_PROVIDER="your-cloud-browser-provider" # Cloud browser provider (select browseruse or browserbase) AGENT_BROWSER_STREAM_PORT="9223" # WebSocket streaming port AGENT_BROWSER_HOME="/path/to/agent-browser" # Custom install location (for daemon.js) AGENT_BROWSER_ENCRYPTION_KEY="" # Encrypt session state at rest AGENT_BROWSER_CONFIG="/path/to/config.json" # Custom config file path ``` ## Configuration file Create `agent-browser.json` in the project root for persistent settings: ```json { "headed": true, "proxy": "http://localhost:8080", "profile": "./browser-data" } ``` Priority (lowest to highest): `~/.agent-browser/config.json` < `./agent-browser.json` < env vars < CLI flags. Use `--config ` or `AGENT_BROWSER_CONFIG` env var for a custom config file. All CLI options map to camelCase keys (e.g., `--executable-path` -> `"executablePath"`). Boolean flags accept `true`/`false` values. Extensions from user and project configs are merged, not replaced. ## Ref lifecycle (important) Refs (`@e1`, `@e2`, etc.) are invalidated when the page changes. Always re-snapshot after: - Clicking links or buttons that navigate - Form submissions - Dynamic content loading (dropdowns, modals) ```bash agent-browser click @e5 # Navigates to new page agent-browser snapshot -i # MUST re-snapshot agent-browser click @e1 # Use new refs ``` ## Annotated screenshots (vision mode) Use `--annotate` to take a screenshot with numbered labels overlaid on interactive elements. Each label `[N]` maps to ref `@eN`. This also caches refs, so you can interact with elements immediately without a separate snapshot. ```bash agent-browser screenshot --annotate # Output includes the image path and a legend: # [1] @e1 button "Submit" # [2] @e2 link "Home" # [3] @e3 textbox "Email" agent-browser click @e2 # Click using ref from annotated screenshot ``` Use annotated screenshots when: - The page has unlabeled icon buttons or visual-only elements - You need to verify visual layout or styling - Canvas or chart elements are present (invisible to text snapshots) - You need spatial reasoning about element positions ## Timeouts and slow pages The default Playwright timeout is 60 seconds for local browsers. For slow websites or large pages, use explicit waits instead of relying on the default timeout: ```bash agent-browser wait --load networkidle # Wait for network activity to settle (best for slow pages) agent-browser wait "#content" # Wait for a specific element to appear agent-browser wait @e1 # Wait for element by ref agent-browser wait --url "**/dashboard" # Wait for URL pattern (useful after redirects) agent-browser wait --fn "document.readyState === 'complete'" # Wait for JS condition agent-browser wait 5000 # Wait fixed duration (milliseconds) as last resort ``` When dealing with consistently slow websites, use `wait --load networkidle` after `open` to ensure the page is fully loaded before taking a snapshot. ## Example: Form submission ```bash agent-browser open https://example.com/form agent-browser snapshot -i # Output shows: textbox "Email" [ref=e1], textbox "Password" [ref=e2], button "Submit" [ref=e3] agent-browser fill @e1 "user@example.com" agent-browser fill @e2 "password123" agent-browser click @e3 agent-browser wait --load networkidle agent-browser snapshot -i # Check result ``` ## Example: Authentication with saved state ```bash # Login once agent-browser open https://app.example.com/login agent-browser snapshot -i agent-browser fill @e1 "username" agent-browser fill @e2 "password" agent-browser click @e3 agent-browser wait --url "**/dashboard" agent-browser state save auth.json # Later sessions: load saved state agent-browser state load auth.json agent-browser open https://app.example.com/dashboard ``` ## Session persistence ```bash # Auto-save/restore cookies and localStorage across browser restarts agent-browser --session-name myapp open https://app.example.com/login # ... login flow ... agent-browser close # State auto-saved to ~/.agent-browser/sessions/ # Next time, state is auto-loaded agent-browser --session-name myapp open https://app.example.com/dashboard # Encrypt state at rest export AGENT_BROWSER_ENCRYPTION_KEY=$(openssl rand -hex 32) agent-browser --session-name secure open https://app.example.com # Manage saved states agent-browser state list agent-browser state show myapp-default.json agent-browser state clear myapp agent-browser state clean --older-than 7 ``` ## Sessions (parallel browsers) ```bash agent-browser --session test1 open site-a.com agent-browser --session test2 open site-b.com agent-browser session list ``` Always close your browser session when done to avoid leaked processes: ```bash agent-browser close # Close default session agent-browser --session test1 close # Close specific session ``` If a previous session was not closed properly, the daemon may still be running. Use `agent-browser close` to clean it up before starting new work. ## Data extraction ```bash agent-browser open https://example.com/products agent-browser snapshot -i agent-browser get text @e5 # Get specific element text agent-browser get text body > page.txt # Get all page text # JSON output for parsing agent-browser snapshot -i --json agent-browser get text @e1 --json ``` ## Connect to existing Chrome ```bash # Auto-discover running Chrome with remote debugging enabled agent-browser --auto-connect open https://example.com agent-browser --auto-connect snapshot # Or with explicit CDP port agent-browser --cdp 9222 snapshot ``` ## Local files (PDFs, HTML) ```bash # Open local files with file:// URLs agent-browser --allow-file-access open file:///path/to/document.pdf agent-browser --allow-file-access open file:///path/to/page.html agent-browser screenshot output.png ``` ## iOS Simulator (Mobile Safari) ```bash # List available iOS simulators agent-browser device list # Launch Safari on a specific device agent-browser -p ios --device "iPhone 16 Pro" open https://example.com # Same workflow as desktop - snapshot, interact, re-snapshot agent-browser -p ios snapshot -i agent-browser -p ios tap @e1 # Tap (alias for click) agent-browser -p ios fill @e2 "text" agent-browser -p ios swipe up # Mobile-specific gesture # Take screenshot agent-browser -p ios screenshot mobile.png # Close session (shuts down simulator) agent-browser -p ios close ``` **Requirements:** macOS with Xcode, Appium (`npm install -g appium && appium driver install xcuitest`) **Real devices:** Works with physical iOS devices if pre-configured. Use `--device ""` where UDID is from `xcrun xctrace list devices`. ## Debugging ```bash agent-browser --headed open example.com # Show browser window agent-browser --cdp 9222 snapshot # Connect via CDP port agent-browser connect 9222 # Alternative: connect command agent-browser console # View console messages agent-browser console --clear # Clear console agent-browser errors # View page errors agent-browser errors --clear # Clear errors agent-browser highlight @e1 # Highlight element agent-browser trace start # Start recording trace agent-browser trace stop trace.zip # Stop and save trace agent-browser record start ./debug.webm # Record video from current page agent-browser record stop # Save recording agent-browser profiler start # Start Chrome DevTools profiling agent-browser profiler stop trace.json # Stop and save profile ``` ## HTTPS Certificate Errors For sites with self-signed or invalid certificates: ```bash agent-browser open https://localhost:8443 --ignore-https-errors ``` ## Electron App Automation Automate any Electron desktop app using agent-browser. Electron apps are built on Chromium and expose a Chrome DevTools Protocol (CDP) port that agent-browser can connect to, enabling the same snapshot-interact workflow used for web pages. ### Core Workflow 1. **Launch** the Electron app with remote debugging enabled 2. **Connect** agent-browser to the CDP port 3. **Snapshot** to discover interactive elements 4. **Interact** using element refs 5. **Re-snapshot** after navigation or state changes ```bash # Launch an Electron app with remote debugging open -a "Slack" --args --remote-debugging-port=9222 # Connect agent-browser to the app agent-browser connect 9222 # Standard workflow from here agent-browser snapshot -i agent-browser click @e5 agent-browser screenshot slack-desktop.png ``` ### Launching Electron Apps with CDP Every Electron app supports the `--remote-debugging-port` flag since it's built into Chromium. #### macOS ```bash # Slack open -a "Slack" --args --remote-debugging-port=9222 # VS Code open -a "Visual Studio Code" --args --remote-debugging-port=9223 # Discord open -a "Discord" --args --remote-debugging-port=9224 # Figma open -a "Figma" --args --remote-debugging-port=9225 # Notion open -a "Notion" --args --remote-debugging-port=9226 # Spotify open -a "Spotify" --args --remote-debugging-port=9227 ``` #### Linux ```bash slack --remote-debugging-port=9222 code --remote-debugging-port=9223 discord --remote-debugging-port=9224 ``` #### Windows ```bash "C:\Users\%USERNAME%\AppData\Local\slack\slack.exe" --remote-debugging-port=9222 "C:\Users\%USERNAME%\AppData\Local\Programs\Microsoft VS Code\Code.exe" --remote-debugging-port=9223 ``` **Important:** If the app is already running, quit it first, then relaunch with the flag. The `--remote-debugging-port` flag must be present at launch time. ### Connecting to Electron Apps ```bash # Connect to a specific port agent-browser connect 9222 # Or use --cdp on each command agent-browser --cdp 9222 snapshot -i # Auto-discover a running Chromium-based app agent-browser --auto-connect snapshot -i ``` After `connect`, all subsequent commands target the connected app without needing `--cdp`. ### Tab Management Electron apps often have multiple windows or webviews. Use tab commands to list and switch between them: ```bash # List all available targets (windows, webviews, etc.) agent-browser tab # Switch to a specific tab by index agent-browser tab 2 # Switch by URL pattern agent-browser tab --url "*settings*" ``` ### Common Electron Patterns #### Inspect and Navigate an App ```bash open -a "Slack" --args --remote-debugging-port=9222 sleep 3 # Wait for app to start agent-browser connect 9222 agent-browser snapshot -i # Read the snapshot output to identify UI elements agent-browser click @e10 # Navigate to a section agent-browser snapshot -i # Re-snapshot after navigation ``` #### Take Screenshots of Desktop Apps ```bash agent-browser connect 9222 agent-browser screenshot app-state.png agent-browser screenshot --full full-app.png agent-browser screenshot --annotate annotated-app.png ``` #### Extract Data from a Desktop App ```bash agent-browser connect 9222 agent-browser snapshot -i agent-browser get text @e5 agent-browser snapshot --json > app-state.json ``` #### Fill Forms in Desktop Apps ```bash agent-browser connect 9222 agent-browser snapshot -i agent-browser fill @e3 "search query" agent-browser press Enter agent-browser wait 1000 agent-browser snapshot -i ``` #### Run Multiple Apps Simultaneously Use named sessions **only** when you need to control multiple Electron apps at the same time. Each session spawns a separate Chromium process (~300MB RAM), so only create multiple sessions when explicitly required and close each one when done. ```bash # Connect to Slack agent-browser --session slack connect 9222 # Connect to VS Code agent-browser --session vscode connect 9223 # Interact with each independently agent-browser --session slack snapshot -i agent-browser --session vscode snapshot -i # Close sessions when done agent-browser --session slack close agent-browser --session vscode close ``` ### Color Scheme Playwright overrides the color scheme to `light` by default when connecting via CDP. To preserve dark mode: ```bash agent-browser connect 9222 agent-browser --color-scheme dark snapshot -i ``` Or set it globally: ```bash AGENT_BROWSER_COLOR_SCHEME=dark agent-browser connect 9222 ``` ### Electron Troubleshooting #### "Connection refused" or "Cannot connect" - Make sure the app was launched with `--remote-debugging-port=NNNN` - If the app was already running, quit and relaunch with the flag - Check that the port isn't in use by another process: `lsof -i :9222` #### App launches but connect fails - Wait a few seconds after launch before connecting (`sleep 3`) - Some apps take time to initialize their webview #### Elements not appearing in snapshot - The app may use multiple webviews. Use `agent-browser tab` to list targets and switch to the right one - Use `agent-browser snapshot -i -C` to include cursor-interactive elements (divs with onclick handlers) #### Cannot type in input fields - Try `agent-browser keyboard type "text"` to type at the current focus without a selector - Some Electron apps use custom input components; use `agent-browser keyboard inserttext "text"` to bypass key events ## CRITICAL: 1. Screenshots are required for UI testing. Text snapshots do not catch layout issues, styling problems, alignment, z-index issues, etc. Always use `agent-browser screenshot --annotate` to get numbered element labels overlaid on the screenshot, which enables both visual verification and immediate interaction via refs. 2. **Use a single browser session.** Do not create multiple `--session` names — each spawns a separate Chromium process (~300MB RAM). Use one session and call `agent-browser open ` to navigate between pages within it. If you must create a new session, close the previous one first with `agent-browser --session close`. **Exception:** When controlling multiple Electron apps simultaneously on different CDP ports, multiple named sessions are acceptable — but only when the user explicitly needs it, and each session must be closed when no longer needed. 3. When done, close your browser with `agent-browser close` (or `agent-browser --session close`) to free resources. 4. **Electron apps: quit first.** If the Electron app is already running, you must quit it completely before relaunching with `--remote-debugging-port`. The flag only takes effect at launch time.