# computer-use-mcp ("realhands") An MCP server that lets **Claude operate your real computer** the way a human does — moving the **actual mouse**, clicking, typing, and reading the **actual screen**. Unlike OpenAI Operator, browser-use, or Playwright agents (which spin up a separate, isolated, logged-out Chrome), this drives the **physical OS cursor and keyboard**. So it works in **your own Chrome with your own logged-in sessions** — and in every other app — because it's just a human at the keyboard, as far as any website can tell. **Status: LIVE and battle-tested.** Registered with Claude Code as the user-scope MCP **`realhands`** (tool `mcp__realhands__computer`) and ✓Connected since 2026-06-02. On 2026-06-09 it drove the user's real, logged-in Chrome through a **complete Google Play Console deployment** (app upload, release notes, submission) end-to-end. > The server is registered as `realhands` rather than `computer-use` because the name > "computer-use" is reserved in Claude Code. ## How it works Claude (Desktop or Code) is the agent loop. You type a task; Claude calls the single `computer` tool in a see → think → act cycle: > **See** — `screenshot` returns the real screen (downscaled to ~1280px for grounding accuracy) > → **Think** — Claude picks the next action + pixel coordinates > → **Act** — the server moves the real mouse / types on the real keyboard > → a fresh screenshot comes back automatically after every action, and it repeats. Two Windows-specific details make clicks land accurately (`src/screen.py`): - **DPI awareness** — `SetProcessDpiAwareness(2)` is set at import time so screenshot pixels == pyautogui cursor coordinates even under display scaling (125% / 150% / …). - **Stateless coordinate scaling** — screenshots are downscaled (LANCZOS) to at most `COMPUTER_USE_MAX_DIM` on the longest side before sending; incoming click coordinates are scaled back up to real pixels. The scale factor is a pure function of monitor geometry + `MAX_DIM`, so mapping never depends on which screenshot ran last. Coordinates are clamped inside the target monitor so a stray click can't fly off-screen. Each axis is rounded down to a whole 28px vision patch, so the two axes can scale very slightly differently (<2.2%); `to_real()` maps each with its own factor and stays exact. **Multi-monitor:** every call takes an optional `monitor` index (1 = primary, 2.. = others, 0 = the whole virtual desktop). `action="monitors"` enumerates the setup. Origins may be negative for screens left/above the primary — `to_real()` handles the offset. Use the **same** monitor for a click as for the screenshot you're clicking on. ## Architecture ``` src/realhands/ server.py FastMCP server "computer-use"; the single `computer` tool (action enum modeled on Anthropic's reference computer_20250124 tool); runs one action or a batch of `steps`, then returns status text + at most one screenshot screen.py DPI awareness, mss capture, patch-aligned downscale, model-space -> real-pixel mapping, unchanged-screen detection input.py pyautogui mouse/keyboard execution; xdotool-style key-name translation (Return, Page_Down, ctrl+a, super, ...); clipboard-paste fast path for long/Unicode/multiline typing (preserves your existing clipboard); activate_window via win32 AttachThreadInput safety.py kill switches + lazy arm / stand-down lifecycle config.py .env-driven configuration (all defaults are sensible; .env is optional) install.py one-shot installer: venv, deps, .env, Claude Desktop registration ``` Stack: Python 3.10/3.11 · `mcp` (FastMCP, stdio) · `pyautogui` · `mss` · `pillow` · `pynput` · `keyboard` · `pyperclip` · `python-dotenv` — plus `pygetwindow` and `pywin32` for `activate_window`. ## The `computer` tool A single tool with an `action` parameter: | Action | What it does | |---|---| | `screenshot` | Capture the screen (always start a task with this) | | `cursor_position` | Report the real mouse position | | `monitors` | List detected monitors (for multi-screen setups) | | `mouse_move` | Glide the cursor to `coordinate` | | `left_click` / `right_click` / `middle_click` / `double_click` / `triple_click` | Click at `coordinate` (or current position) | | `left_click_drag` | Drag from `text="x1,y1"` to `coordinate=[x2,y2]` | | `left_mouse_down` / `left_mouse_up` | Press / release the left button | | `scroll` | Scroll at `coordinate` (`scroll_direction` + `scroll_amount` notches) | | `type` | Type `text` (clipboard-paste path for long/Unicode/multiline) | | `key` | Press a key or chord — `"Return"`, `"ctrl+s"`, `"alt+Tab"` | | `hold_key` | Hold keys for `duration` seconds | | `activate_window` | Bring an app to the front by title substring (beats Windows' foreground-lock; far more reliable than clicking the taskbar) | | `wait` | Sleep `duration` seconds, then screenshot | | `stop` | Stand down: close the STOP overlay + release the panic hotkey (call as the final action) | Coordinates are in the pixel space of the most recent screenshot; its size is reported with every capture. After every non-screenshot action the tool waits ~0.4s for the UI to settle and returns a fresh screenshot. ### Token economy Screenshots dominate the cost of driving a desktop, and not just once: every image stays in the conversation and is re-sent as history on each later turn. Claude bills vision in 28×28 patches — `tokens = ⌈w/28⌉ × ⌈h/28⌉` — so this server does four things to keep the bill down. **Batch steps.** Pass `steps` (a list of action dicts) instead of one call per action and the whole run shares **one** screenshot at the end: ```jsonc {"steps": [{"action": "left_click", "coordinate": [420, 300]}, {"action": "type", "text": "hello@example.com"}, {"action": "key", "text": "Tab"}, {"action": "type", "text": "secret"}, {"action": "key", "text": "Return"}]} ``` That is 1 125 visual tokens instead of 5 625, and one round trip instead of five — the larger saving, since each avoided turn also avoids re-sending the entire transcript. A failing step stops the run, reports which step failed, and still returns the screen. Add `"screenshot": false` to skip the trailing image too. **Patch-aligned downscaling.** A dimension that isn't a multiple of 28 pays for a partial patch row/column carrying almost no pixels. From a 1920×1080 primary the default 1260×700 is exactly 45×25 patches = 1 125 tokens, versus 1 196 for 1280×720 — 6% off for 1.5% fewer pixels. **Unchanged-screen suppression.** If under `CHANGE_THRESHOLD` of pixels moved since the last image sent, the reply is a line of text instead of a screenshot. A real desktop never produces two byte-identical frames (clock, caret, hover states), so this is a threshold, not an equality check. After `MAX_SKIPS` suppressions in a row it force-sends one, so the model can't fly blind if it lost the earlier image to context compaction. **Right-sized images.** `COMPUTER_USE_MAX_DIM` trades grounding accuracy against cost — from a 1920×1080 primary: 1792 → 2 304 tokens, 1260 → 1 125 (default), 1036 → 777, 896 → 576. Keep the long edge ≤ 2576 px: an image returned inside a `tool_result` is *rejected* rather than downscaled when it exceeds the model's limit. > `COMPUTER_USE_IMAGE_FORMAT` is **not** a token lever — Claude bills by pixel > dimensions, so a JPEG and a PNG of the same screenshot cost exactly the same. JPEG > only cuts payload bytes (~977 KB → ~141 KB here), which helps latency at some risk > to small-text legibility. ## Safety — it controls your REAL machine This is **fully autonomous**: it does not ask before each action. Three independent kill switches (`src/safety.py`): 1. **Fail-safe corner** — slam the mouse into the **top-left corner** → pyautogui raises `FailSafeException` and the action aborts instantly. 2. **Panic hotkey** — **Ctrl+Alt+Q** (configurable) → hard-kills the server process (`os._exit(1)`). 3. **STOP overlay** — an always-on-top window (top-right) showing the current action, with a big red **■ STOP AGENT** button that also hard-kills the process. **Lazy arm / stand-down:** the overlay and the global panic hotkey are armed lazily on the **first action** of a task, not at server startup — idle sessions show nothing and grab no hotkeys. They stand down when the agent calls `action="stop"` at the end of a task, and re-arm automatically on the next action. (The STOP overlay is a single persistent window that is *hidden* when dormant, never destroyed — recreating it was a crash hazard.) An optional idle auto-stand-down is available via `COMPUTER_USE_IDLE_STOP` but is **disabled by default**: an agent's thinking time between tool calls easily exceeds any short idle window, so a non-zero value would stand the agent down mid-task. Pacing also helps you stay in control: every action is followed by a configurable pause (`COMPUTER_USE_PAUSE`) and the cursor glides rather than teleports (`COMPUTER_USE_MOVE_DURATION`), so you can watch and interrupt. **Don't leave it unsupervised on anything that can spend money, send messages, or delete data.** ## Install Requires **Python 3.10 or 3.11** (3.13+ untested; avoid the 3.14 beta). ### Recommended: `uvx` (always the latest version) Nothing to install up front, and **you get every release automatically** — `uvx` resolves the newest published version each time the server starts, so a restart of your MCP client is the whole upgrade process. Requires [uv](https://docs.astral.sh/uv/). ```powershell uvx realhands@latest ``` Drop the `@latest` (`uvx realhands`) if you would rather let uv reuse whatever version it already has cached. ### From PyPI (pinned) ```powershell pip install realhands ``` This installs the `realhands` console script and the importable `realhands` package. Run the server with either `realhands` or `python -m realhands.server`. Pick this over `uvx` when you want a **pinned** version that never changes underneath you, or when the machine may be offline at startup — `uvx realhands@latest` needs to reach PyPI each time it launches. The trade-off is that upgrades become manual: ```powershell pip install --upgrade realhands ``` Then restart your MCP client — clients bind their servers at session start, so a running session keeps the old code until it restarts. ### From source (with Claude Desktop registration) ```powershell git clone https://github.com/kanishka089/computer-use-mcp cd computer-use-mcp py -3.10 install.py ``` This creates `.venv/`, installs the package + deps (editable), copies `.env.example` to `.env` if missing, and registers the server in Claude Desktop's config (backing up any existing config). **Restart Claude Desktop**, then look for the `computer-use` tool. ### Claude Code Register it as a **user-scope** stdio server named `realhands`. This form auto-upgrades — each new session resolves the latest release, so you never have to think about it: ```powershell claude mcp add realhands --scope user -- uvx realhands@latest ``` The tool then appears as `mcp__realhands__computer` in every project. If you installed with `pip` instead and want a pinned version, point it at the Python that has the package (and run `pip install --upgrade realhands` yourself to move up): ```powershell claude mcp add realhands --scope user -- python -m realhands.server ``` > **Already registered on an older version?** Re-point it at the auto-upgrading form: > ```powershell > claude mcp remove realhands --scope user > claude mcp add realhands --scope user -- uvx realhands@latest > ``` > Removing and re-adding an MCP mid-session drops this session's connection to it — the > new registration is picked up on the next Claude Code start. ## Upgrading > **On 0.1.x?** Upgrade. `mcp` 2.0 removed the API 0.1.x imports, so while an install > made back when it resolved `mcp` 1.x still runs, reinstalling or moving machines will > produce a server that cannot start. 0.2.1 and later handle both. See > [v0.2.1](https://github.com/kanishka089/computer-use-mcp/releases/tag/v0.2.1). ### Best: switch to the auto-upgrading form (once) Then you never do this again — every client restart picks up the newest release: ```powershell claude mcp remove realhands --scope user claude mcp add realhands --scope user -- uvx realhands@latest ``` Restart Claude Code. (Removing and re-adding an MCP mid-session drops this session's connection to it; the new registration takes effect on the next start.) ### Staying on pip **Upgrade the same interpreter the server actually runs** — not whatever `pip` happens to resolve. If you installed into a virtualenv, or have several Pythons, a bare `pip install --upgrade realhands` will cheerfully upgrade a different environment and leave the server on its old version. This is the most common reason an upgrade "doesn't take". First, find the interpreter: ```powershell claude mcp get realhands ``` Read the `Command:` line — that is your Python. Then upgrade with it explicitly: ```powershell & "C:\path\from\that\Command line\python.exe" -m pip install --upgrade realhands ``` If `Command:` is a bare `python`, `pip install --upgrade realhands` is fine. Confirm it took, then restart Claude Code: ```powershell & "C:\path\to\python.exe" -c "import realhands; print(realhands.__version__)" ``` ### Installed from source? `git pull` is enough — `install.py` installs the package as editable, so the checkout *is* the installed version. Restart your client. ## Use Just ask. For example: > *Take a screenshot, open Chrome, go to YouTube, and search for "lofi".* Watch your real cursor move and your logged-in Chrome respond. Real-world proof: it has autonomously completed a full Google Play Console release flow in the user's own signed-in Chrome session. ## Configuration (`.env`, optional — defaults are fine) | Var | Default | Meaning | |-----|---------|---------| | `COMPUTER_USE_MAX_DIM` | `1260` | Longest screenshot side sent to Claude; the main accuracy-vs-cost dial (see [Token economy](#token-economy)) | | `COMPUTER_USE_PATCH_ALIGN` | `1` | Round each axis down to a whole 28px patch so no tokens go on a partial patch (~6% off every shot) | | `COMPUTER_USE_CHANGE_THRESHOLD` | `0.002` | Skip the screenshot when less than this fraction of pixels moved (`0` = always send) | | `COMPUTER_USE_MAX_SKIPS` | `6` | Force a real screenshot after this many suppressed in a row | | `COMPUTER_USE_MONITOR` | `1` | Default monitor (1 = primary, 2.. = others, 0 = all screens); overridable per call | | `COMPUTER_USE_IMAGE_FORMAT` | `png` | `png` (crisp text) or `jpeg` (smaller payload). **Not** a token lever — cost is by pixel dimensions, not bytes | | `COMPUTER_USE_PAUSE` | `0.15` | Delay after each pyautogui action (interruptibility) | | `COMPUTER_USE_PANIC_HOTKEY` | `ctrl+alt+q` | Global hard-stop hotkey | | `COMPUTER_USE_OVERLAY` | `1` | Show the STOP overlay window | | `COMPUTER_USE_MOVE_DURATION` | `0.4` | Cursor glide time (human-like movement) | | `COMPUTER_USE_IDLE_STOP` | `0` | Auto stand-down after this many idle seconds (`0` = never; stand down only on `action="stop"`) | ## Known gotchas - **MCP connection drops when the agent idles between turns.** The stdio connection to `realhands` can silently die while Claude is thinking/waiting between turns. Fix: issue a `screenshot` action — it silently reconnects. Importantly, an action that "failed" with *Connection closed* **often still executed** on the real machine — take a screenshot and check the actual screen state before retrying, or you may double-click / double-submit. - **Click coordinates must match the screenshot's monitor.** If you screenshot `monitor=2` and then click without passing `monitor=2`, the click lands on the primary. - **`activate_window` beats the taskbar.** Windows' foreground-lock makes taskbar clicks unreliable (the icon just flashes). `activate_window` uses `AttachThreadInput` + z-order toggling + a minimize/restore fallback, so prefer it for app switching. - **Don't run with Python 3.13/3.14.** Tested on 3.10/3.11 only; the installer warns. - **Typing long/Unicode text uses the clipboard.** Your clipboard is saved and restored, but anything watching the clipboard will see the pasted text momentarily. ## Self-test ```powershell python -m realhands.screen ``` Captures the screen, prints real vs. sent dimensions and the scale factor, writes `test_capture.png`, and runs a coordinate round-trip check (center + both corners).