AutoCua
Automatic Computer Use Agent.
Mac, Windows, Linux, real phone, simulator, any browser.
A multi-agent harness engine.
[](https://pypi.org/project/AutoCua/)
[](https://pypi.org/project/AutoCua/)
[](https://github.com/AutoCua/AutoCua/blob/main/README.md#requirements-and-setup)
[](https://github.com/AutoCua/AutoCua/blob/main/LICENSE)
[](https://github.com/AutoCua/AutoCua/blob/main/README.md#how-it-acts-on-a-screen)
[](https://github.com/AutoCua/AutoCua/blob/main/README.md#the-browser-agent)
[](https://github.com/AutoCua/AutoCua/blob/main/README.md#providers-and-models)
|
```bash
pip install AutoCua
```
|
| | Windows | Linux | macOS |
|:--|:--:|:--:|:--:|
| **Computer use** — any desktop app, with real mouse and keyboard input | ✓ | ✓ | ✓ |
| **Browser skills** — Chrome, Firefox, Edge, Brave and Opera | ✓ | ✓ | ✓ |
| **Shell use** — a coding agent in a real terminal | ✓ | ✓ | ✓ |
| **Web use** — your own Chrome, driven through the AutoCua extension | ✓ | ✓ | ✓ |
| **Mobile use** — your iPhone, iPad or an iOS Simulator | — | — | ✓ |
Windows 10 and 11 · Linux with GNOME · macOS on Apple Silicon or Intel
**Full capabilities:** **[agent_capabilities.md](https://github.com/AutoCua/AutoCua/blob/main/agent_capabilities.md)**
---
## Computer use
Any desktop app, driven with real mouse and keyboard input. Agent mode: computer use.
https://github.com/user-attachments/assets/e6c4b637-baeb-441f-8dfb-e550d88b4223
## Web agent: fast mode
6.1 seconds for a flight search. Agent mode: web.
https://github.com/user-attachments/assets/8cc797d5-5556-40a9-bbfc-5d6aa8880fb8
## Web agent: quality mode
Under a minute for three tasks in one run: an iPhone 18 Pro into the Apple bag, a case into the
Amazon basket, and a Google Sheet of Meta's five-year financials. Agent mode: web.
https://github.com/user-attachments/assets/82bf6163-55c9-4cc4-95e7-9c6466f2a25d
## iOS agent: iPhone use
A paired iPhone, driven over the USB cable. Agent mode: mobile use, iOS.
https://github.com/user-attachments/assets/3c22eb45-d939-4742-9053-22ecc75b416f
---
## Pick a surface
One string decides what the agent drives. Everything else stays the same.
```python
from AutoCua.agent_launcher import run_agent
run_agent(mode="web use", provider="openrouter", model="gemini-3.8-flash",
task="Open wikipedia.org and summarise today's featured article.")
```
| `mode` | What it drives | Where it runs |
|---|---|---|
| `"computer use"` | The desktop. Clicks, typing, hotkeys, apps, AppleScript | macOS, Windows or Linux, auto detected |
| `"shell use"` | A coding agent, straight into a real terminal | macOS, Windows or Linux |
| `"web use"` | Your own Chrome through the AutoCua extension; headless runs on a debugging port | Any host with Chrome, installed for you when it is missing |
| `"mobile use, ios"` | An iOS Simulator by default, or your paired iPhone or iPad | macOS host |
| `"mobile use, android"` | Not available yet | |
On Linux, computer use needs a GNOME desktop (tested on Wayland), and computer use and shell use
both need a few distro packages that pip cannot install. See [Linux, from pip](#linux-from-pip).
---
## Why it holds up on real software
**No pixel guessing.** The model never sees a coordinate. It picks a numbered element from a
view that fuses the operating system's accessibility tree with OCR, and a number it was never
shown is rejected before anything moves.
**No debugging port on your desktop.** Input goes in at the operating system's own input
layer. Mouse events posted at `kCGHIDEventTap` on macOS, pywinauto's real input path on
Windows, and a kernel driver when Windows UAC blocks everything else. An ordinary
application sees the same events it would get from your hand on the mouse.
**No debugging port on your browser either.** A normal `"web use"` run drives your own Chrome
through the AutoCua extension, which Chrome loads at launch with no store install. Chrome shows
its own "AutoCua started debugging this browser" bar while the agent works, and its Cancel
button hands the browser back to you. A debugging port is opened only for headless runs, on a
browser of the agent's own. The page scanner never calls `Runtime.enable`, and the
numbered boxes are painted into the screenshot, not into your DOM.
**No context wall.** A separate compression agent watches the transcript and splices a handoff
summary into it mid-run, so a long task finishes instead of dying at the context limit. In the
app you watch the memory ring's count fall.
**No telemetry.** AutoCua sends nothing anywhere except to the model provider you chose, and
to Telegram if you connect it yourself.
---
## Contents
- [Quick start](#quick-start)
- [How it sees a screen](#how-it-sees-a-screen)
- [How it acts on a screen](#how-it-acts-on-a-screen)
- [The agent team](#the-agent-team)
- [The browser agent](#the-browser-agent)
- [iPhone and iPad](#iphone-and-ipad)
- [Windows, UAC and the kernel driver](#windows-uac-and-the-kernel-driver)
- [Automation testing and workflows](#automation-testing-and-workflows)
- [Running tasks in parallel](#running-tasks-in-parallel)
- [The desktop app](#the-desktop-app)
- [Memory, vault, chats and skills](#memory-vault-chats-and-skills)
- [Remote control from Telegram](#remote-control-from-telegram)
- [Providers and models](#providers-and-models)
- [Requirements and setup](#requirements-and-setup)
- [Limits and safety](#limits-and-safety)
- [Project layout](#project-layout)
- [Roadmap](#roadmap)
- [Licence, credits and citation](#licence-credits-and-citation)
## Quick start
```bash
pip install AutoCua
```
On Linux, install a few distro packages first: see [Linux, from pip](#linux-from-pip).
Put the four lines from [Pick a surface](#pick-a-surface) in a `main.py`, then:
```bash
python main.py
```
That is the whole API.
### What the wheel brings, and what it cannot
Two of the native pieces arrive on their own, and one, the desktop app's Electron, when you
say yes to it. Two cannot: a Windows kernel driver, which `pip`
is structurally unable to install, and on Linux the desktop's accessibility and input bindings,
which come from your distro.
| | How it reaches you |
|---|---|
| **Browser agent** — Rust | Ships **prebuilt** in the Windows x64, Apple Silicon macOS and Linux x86_64 wheels. On any other platform pip builds it from the source distribution, which needs [Rust](https://rustup.rs) and a C compiler. |
| **WebDriverAgent** — iOS | **Cloned automatically** the first time you run a `"mobile use, ios"` task. Needs full Xcode and git. Nothing to do by hand. |
| **Electron** — the desktop app's Chromium window | **Downloaded only if you say yes.** The first `ui=True` launch from a terminal asks (120 to 160 MB, once) on macOS 13+, Windows 10+ and Linux, except stock Ubuntu 23.10+ ([below](#the-desktop-app)); without it the app opens in pywebview. `python -m AutoCua.utils.electron` fetches it any time. No Node.js needed. |
| **Interception** — Windows UAC | **Not installed by pip.** One manual step, once — below. |
| **AT-SPI, GTK and the desktop portal** — Linux | **Not installed by pip.** One `apt` line, once — [below](#linux-from-pip). |
### Answering UAC prompts on Windows
Windows' user-mode `SendInput` cannot deliver input to the UAC secure desktop, so an elevation
prompt can only be answered from kernel mode. That is the one thing the
[Interception](https://github.com/oblitum/Interception) driver does here.
`pip install` cannot set it up, and no packaging trick changes that: a wheel is unpacked and
never executed, pip does not run elevated, and the driver binds its input slots only when it
loads **at boot**. Without it **every mode still works** — you simply lose UAC handling, and a
task that hits an elevation prompt stops there.
To enable it, pick one:
```bat
setup_windows.bat
```
from a checkout — it asks first (answer N and nothing is downloaded or installed), then downloads
the pinned release, verifies its SHA-256, installs it, binds it to your **built-in keyboard only**
so external keyboards are never filtered, and asks before rebooting. Or install Interception
yourself from its own release page and reboot.
> It is never bundled because it is dual-licensed **LGPL-3.0 non-commercial / paid commercial**.
> Shipping it inside the wheel would make this project its distributor to every person who runs
> `pip install`, which that licence does not allow for free. You receive it from its author.
> See [THIRD_PARTY_NOTICES.md](https://github.com/AutoCua/AutoCua/blob/main/THIRD_PARTY_NOTICES.md).
### Linux, from pip
The Linux wheel is built for x86_64 and needs a GNOME desktop (tested on Ubuntu with Wayland;
X11 is not tested yet). The agent reads and drives the desktop through AT-SPI, GTK and the XDG
RemoteDesktop portal. Those bindings come from your distro, not from pip, so install them once:
```bash
sudo apt install python3-gi python3-cairo python3-gi-cairo python3-venv gir1.2-atspi-2.0 gir1.2-gtk-3.0 \
gir1.2-webkit2-4.1 at-spi2-core xdg-desktop-portal xdg-desktop-portal-gnome \
gir1.2-gstreamer-1.0 gstreamer1.0-pipewire gstreamer1.0-plugins-base \
libglib2.0-bin libgtk-3-bin xdg-utils
```
On KDE, swap `xdg-desktop-portal-gnome` for `xdg-desktop-portal-kde`. Fedora, Arch and openSUSE
ship the same packages under their own names; `setup_linux.sh` in the repository lists them.
Optional, for faster recordings: also install your distro's ffmpeg. Each run's recording is then
compressed on the GPU instead of the CPU, because the ffmpeg pip brings on Linux has no GPU encoders:
```bash
sudo apt install ffmpeg # Debian, Ubuntu
sudo pacman -S --needed ffmpeg # Arch
```
Skip it on Fedora and openSUSE: their own ffmpeg builds leave out codecs the app needs (RPM Fusion
and Packman have full ones). Without it, recordings are still compressed, on the CPU.
Ubuntu and Debian refuse `pip install` into the system Python, so create a virtualenv first,
inside the folder that holds your `main.py` and `.env` so the agent finds that `.env`. Build it on
your **distro's own `python3`** (3.10 to 3.14): AutoCua loads the system PyGObject from inside
the venv only for the same 3.x version, so a venv on another version stops with "cannot import gi".
```bash
cd my-agent # the folder with your main.py and .env
python3 -m venv .venv
source .venv/bin/activate
pip install AutoCua
python main.py # the main.py from Quick start
```
The browser agent is prebuilt in the wheel, so you need no Rust or C compiler. On the first
computer-use run GNOME asks for screen-share consent: turn **Allow Remote Interaction** on
before you click **Share**, because GNOME leaves it off. Your choice is saved in `AutoCua_data/`.
The first scan also downloads the PP-OCRv6 models that read text the desktop does not publish.
### A few things worth knowing on the first run
- Add `ui=True` to open the desktop app instead of running the task in the terminal. The first
time, it asks in the terminal whether to download Electron for its Chromium window (`y`, `n`
for no and don't ask again, or Enter for not now); without it the same app opens in
pywebview ([The desktop app](#the-desktop-app)).
- Put your provider key in a `.env` file, or paste it into Settings in the app.
- Model names come from **[model_list.txt](https://github.com/AutoCua/AutoCua/blob/main/model_list.txt)** and are case sensitive. A name
that is not on the list is forwarded verbatim and comes back as a 404.
- Everything the agent keeps, chats, keys, your own skills and browser profiles, lands in an
`AutoCua_data/` folder created where you run Python from.
### From a checkout
```bash
git clone https://github.com/AutoCua/AutoCua.git
cd AutoCua
bash setup_mac.sh # macOS
setup_windows.bat # Windows, self-elevates, asks before installing the kernel driver
cp .env.example .env # add your provider key
python main.py
```
`main.py` is the whole configuration surface: one `run_agent(...)` call. Edit it in place:
```python
run_agent(
mode="computer use",
provider="anthropic",
model="claude-sonnet-5", # names come from model_list.txt
task="check the version of macOS.",
)
```
Every optional flag goes straight into that call (`ui=True`, `speed="fast"`, …). All of them,
`ui`, `save_conversation`, `speed`, `external_terminal`, `headless`, `browser_port`,
`browser_profile`, `extra_tasks`, `device`, `ios_version`, `sim_device`, `app_build`, `api_key`
and `os`, are documented in
**[agent_capabilities.md](https://github.com/AutoCua/AutoCua/blob/main/agent_capabilities.md)**.
---
## How it sees a screen
Every step the agent gets an element tree and a matching annotated image, then reasons over
both.
### macOS
1. **Find the real front window.** Window z-order comes from `CGWindowListCopyWindowInfo`, and
the tree is walked per process from `AXUIElementCreateApplication`. The front app, any
dialog owner, the Dock, Finder's desktop and menu-bar status-item owners are each walked
separately, so a sheet on top of a window does not hide what owns it.
2. **Wake the app up.** `AXEnhancedUserInterface` is set when it is not already on, so apps
publish their full tree, then the scanner waits 0.3 s for it to settle.
3. **OCR whatever the tree missed.** Apple Vision runs at the accurate level with a confidence
floor of 0.4, on worker threads, while the accessibility walk is still running. Recognising
at 1x logical resolution instead of 2x took a 2940 px capture from 1.17 s to 0.78 s with no
loss in hit rate.
4. **Drop what cannot be clicked.** Every element is re-tested against the real window stack
with a 20x20 grid of sample points. Anything under 1 percent visible is thrown away, so the
model is never handed a number that sits behind another window.
5. **Draw the numbers.** The capture is downscaled first to fit inside full HD (1920 x 1080,
aspect kept, never upscaled), then 13 px magenta labels with a 2 px black rim are stamped on
from cached tiles, and the frame is encoded as a **4:4:4 JPEG at quality 85**. That chroma
choice is deliberate. Default subsampling smears thin magenta digits into mush, and this is a
tenth of PNG's bytes.
### Windows
The same contract, built out of what Windows offers.
One UI Automation cached round trip pulls 18 properties for the whole subtree, then four
threads run in parallel: WinRT OCR, a taskbar walk, a Win32 backend scan and a raw UIA scan of
the Start menu. The results are merged and deduped at 5 px, with OCR lines nested into the
deepest container that does not already hold a labelled element there. Chromium browsers are
launched with `--force-renderer-accessibility` so they publish their full tree, and page loads
are gated by polling the toolbar reload button until it stops saying "Stop".
Clicks aim at the pixel centroid of the element's own content rather than the middle of its
box, so a wide button with a label on the left still gets hit where the label is.
### Both platforms
A step that only digests a tool result, the turn after a `web` lookup or after the coder agent
reports back, deliberately skips the scan entirely. No screenshot, no tree, just the payload
and an instruction to pull the findings into the scratchpad. Scanning a screen that nobody
looked at costs tokens and buys nothing.
---
## How it acts on a screen
Nothing in the desktop input path opens a debugging port, sets an automation flag or injects
JavaScript. Input goes in at the operating system's own input layer, so an ordinary
application sees the same events it would get from your hand on the mouse.
### macOS
Mouse events are posted at `kCGHIDEventTap`, the same tap a physical mouse feeds, from a
private event source, after a genuine cursor warp with `CGWarpMouseCursorPosition`.
- A partially visible element tries the accessibility `AXPress` action first, then falls back
to a synthetic click. A fully hidden element is refused with "scroll first" instead of being
clicked blind.
- Multi-line text is put on the clipboard and pasted, then the previous clipboard is restored,
because code editors auto-indent typed newlines and corrupt the text.
- Drags are 20 interpolated move events at 15 ms, not a teleport.
- `applescript` is a first-class tool, not a bash escape hatch. The model supplies a complete
`tell application` block and the runtime handles launching, foregrounding and a 30 second cap.
Typing and hotkeys go through pyautogui and pynput rather than the private event source, so
they are ordinary synthetic input.
### Windows
Element clicks use `pywinauto.click_input()`, which is a real `SendInput`. When that fails
because User Interface Privilege Isolation blocked it, the click is retried through the
**Interception kernel-mode driver**. Typing switches to the driver when the front application
is Windows Security. See [Windows, UAC and the kernel driver](#windows-uac-and-the-kernel-driver).
### A watcher for permission dialogs
macOS asks for consent the first time an app does something new, and a modal dialog will
deadlock a run. AutoCua fingerprints those dialogs **structurally**, any window or sheet
carrying a "Don't Allow" or "Deny" button, and clicks the affirmative. It runs on a one second
loop around every AppleScript call and every shell command, and if a dialog is present but
cannot be clicked, the command fails after 8 seconds with a permission-specific error instead
of hanging forever.
### Browser windows on the desktop
When a browser is in front, the agent gets browser rules injected into its prompt and the
runtime changes behaviour:
- It **reuses the browser you already have open**, with your profile, cookies and sessions. The
rules are emphatic that launching a second instance lands in a signed-out profile.
- macOS waits for a real `AXWebArea` before scanning, with a 1.5 second grace period and a 15
second load budget. No web area means it is not a web window, so scan it as it is. Firefox
publishes no load property, so it falls through to a children-ready check.
- Known destinations are opened in one command with the query already in the URL, rather than
opening a site to click its search box.
### Tools the desktop agent has
| | macOS | Windows |
|---|---|---|
| Click | `left_click` (1, 2 or 3 clicks), `right_click` | same |
| Type | `input`, `typewrite` | same |
| Navigate | `scroll`, `hotkey`, `screenshot`, `drag_drop` | `scroll`, `hotkey`, `screenshot` |
| System | `open_app`, `shell` (zsh), `applescript` | `open_app`, `shell` (PowerShell) |
| Delegate | `sub_agent` (coder_agent, browser_agent), `agent_wait`, `web` | same |
| State | `todo_list`, `update_todo`, `scratchpad`, `wait`, `done` | same |
19 tools on macOS, 17 on Windows. The names are a frozen allow-list. Anything else the model
invents comes back as an error result and is never executed.
---
## The agent team
Behind one task sits a hierarchy of agents, each in its own process.
**The parent agent** owns the screen and decides what a task needs. When it needs real code
work it delegates and can carry on, or block until the result comes back.
**The coder agent** has 14 tools: `shell`, `view`, `grep`, `glob`, `write`, `replace`, `web`,
`plan`, `todo_list`, `update_todo`, `wait`, `scratchpad`, `minion` and `exit`. Its operating
procedure is EXPLORE, PLAN, EXECUTE, VERIFY, and it is written into the prompt with teeth:
- Exploration is delegated by rule. "Never read the codebase first-hand to build first-time
understanding, send a minion."
- The plan is a structured Markdown document with real `path:line` anchors, not a restatement
of the request.
- Every change with logic in it needs a throwaway test under `./.AutoCua_verify/` that is
actually run, whose output is recorded, and which is deleted before exit.
- `replace` verifies the old block before writing and re-verifies after, so a stale line number
fails with "mismatch at line X" instead of corrupting your file.
**Minions** are read-only scouts with six tools and no recursion. The restriction on tool
*names* is structural: the minion's six-name set is the allow-list at the call router, so a
hallucinated `write` comes back as an error result and never runs. Several minions fire from
one coder step, run in parallel, and the loop blocks until all of them report back with
findings anchored to exact `path:line`. Even a crashed minion returns its partial scratchpad,
so the parent never hangs. To be precise about the boundary: the tool *names* are enforced, and
the "read-only commands only" rule on its `shell` tool is held by the prompt.
Minions exist to keep the coder's context small. Every tool result is budgeted too: 200 lines
or 15,000 characters, with `shell` keeping a 50-line tail, `view` capped at 2,000 lines and
`grep` at 8 MB, so one wide search or noisy build log can never blow up the conversation.
Run either directly. They start in their own workspace, so point them at an absolute path when
the work is in an existing project:
```bash
python -m AutoCua.mac.agent.coder --task "refactor the auth module in /Users/me/projects/api" --provider anthropic --model claude-sonnet-5
python -m AutoCua.mac.agent.minions --task "where is _validate_token defined in /Users/me/projects/api and who calls it?" --provider anthropic --model claude-sonnet-5
```
---
## The browser agent
`"web use"` is a different program from the rest of the framework. The whole browser agent,
the agent loop, the page scanner and the controller, is **one Rust crate** compiled into a
single Python extension module and imported like any other. Its LLM calls go through the
same Python providers every other mode uses, in `AutoCua/llm_provider/`.
A checkout builds it on first import with `cargo build --release` and rebuilds only when a
`.rs` file actually changed. The pip wheel ships it already compiled. One binary covers every
CPython from 3.10 up.
**Two ways into Chrome.** Every run with a window goes through the **AutoCua extension**, a
parallel run included: Chrome is started with the flags that load it, or taken as it is when it
is already running with it, and the agents drive your own browser, logged in as you are, with
no debugging port open; each agent is on a line of its own to the extension. A headless run
uses the **DevTools Protocol** on a Chrome of the agent's own. Every tool works the same way in
both. When Chrome is missing it is downloaded from Google and installed on the first web run.
### Raw CDP, hand written
No Playwright, no Selenium, no Puppeteer, no chromedriver. None of them appear anywhere in the
repository. In the debugging-port mode there is **one WebSocket** to Chrome plus hand-written
HTTP for tab management, one session per tab and a child session per cross-origin iframe,
attached in flatten mode up to six levels deep. In extension mode the same scanner runs as a
content script of the extension, in an isolated world the page cannot see, and the tools send
their input through `chrome.debugger` as the same trusted CDP events.
The property that matters:
> **The debugging-port scanner runs no JavaScript in the page, and `Runtime.enable` is never
> called in either mode.**
In the debugging-port mode the page model is two CDP calls per frame, `DOMSnapshot.captureSnapshot`
plus `Accessibility.getFullAXTree`; in extension mode it is one injection into every frame of
the tab, cross-origin frames included, reading the same facts from the DOM. Clicks and keystrokes go through CDP's trusted input path with a
rising click count. Even the numbered boxes the model sees are painted **into the JPEG** in
Rust with a hand-rolled 3x5 bitmap font, never into the DOM.
Two honest exceptions, both deliberate:
- `run_script` and `scrape` are the two doors for JavaScript in the page, and they are split by
what they are for: `run_script` runs what the model wrote, to DO something the ordinary tools
cannot reach; `scrape` runs a reader we wrote, which takes no selectors from the model and only
reads. Both are raced against a 10 second in-page timer and serialised in page, `run_script` is
parse-probed with `new Function` and capped at 30,000 characters, and both are screened by the
same `guardrails_check.rs`: web pages only, theft and cross-site sends refused before anything
runs, credential-shaped values taken out of the result. Both descriptions tell the model to use
them sparingly, because automation blockers can notice them.
- A `scrape` reads the screen the page is on plus the next two below it: three viewports at
most, whole records only, in one chunk of about 8k tokens. The reader finds the page's records
itself, by shape and never by site: the block in the page's main content (a sidebar or a menu
only when nothing else repeats), a custom element that repeats and carries its data in
attributes, or the repeated block on plain HTML, with nesting or indentation read as thread
depth; it falls back to the text in reading order on a page without records, so an article or
a table reads too. A thread comes back as the post, then its comments with each reply nested
under the comment it answers; a list as its items with title, url, score, counts, price,
phone or address where the page shows them. After the call the agent is in **scraping mode**
until it calls `exit_scrape_mode`: no scan, one clean picture per screen read, and a schema of
six tools: `index` (what the agent keeps from the chunk: a one-line title, what is still pending,
and the value as JSON in a shape of its own), `more` (the next three screens, scrolled to from
where the last read ended; nothing is read twice), `exit_scrape_mode`, `scratchpad`,
`update_todo` and `skills`. Every index is stored as JSON under
`AutoCua_data/scrapes/