> **Building on this project?** Keep the [LICENSE](LICENSE) and credit **[ESP32 iPhone Agent by Henry Yuan](https://github.com/onehumanbeing/esp32-iphone-agent)** in your README and posts. See [NOTICE](NOTICE).
ESP32 Agent logo # ESP32 iPhone Agent **Your AI can use your iPhone like you do.** It looks at the screen, then taps, swipes and types for you, through a $7 ESP32 board plugged into the phone.
Just the phone and the board: no computer, no changes to iOS. Try on TestFlight [![TestFlight](https://img.shields.io/badge/TestFlight-join-0D96F6?logo=apple&logoColor=white)](https://testflight.apple.com/join/1xQHgJcA) [![Website](https://img.shields.io/badge/website-handsy.tech-111111)](https://handsy.tech) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) [![GitHub stars](https://img.shields.io/github/stars/onehumanbeing/esp32-iphone-agent?style=flat&logo=github)](https://github.com/onehumanbeing/esp32-iphone-agent/stargazers) [![CI](https://github.com/onehumanbeing/esp32-iphone-agent/actions/workflows/ci.yml/badge.svg)](https://github.com/onehumanbeing/esp32-iphone-agent/actions/workflows/ci.yml) ![iOS](https://img.shields.io/badge/iOS-18%2B-black) ![Board](https://img.shields.io/badge/board-ESP32--S3-red) English · [简体中文](README.zh-CN.md) Demo video and updates: [X @iamhenryyuan](https://x.com/iamhenryyuan) · [YouTube demo](https://www.youtube.com/watch?v=Ly4im-P4jYg) · [DeepSeek Harness demo](https://www.youtube.com/watch?v=DzYv0QVk_No) · [handsy.tech](https://handsy.tech)
> [!TIP] > **Try the app now on [TestFlight](https://testflight.apple.com/join/1xQHgJcA)**, no Xcode needed. The App Store > listing is still in review, so TestFlight is the way to get it today. You still need a flashed board > (see [Quick start](#quick-start)). > [!NOTE] > **Branches:** [`release-1.0`](https://github.com/onehumanbeing/esp32-iphone-agent/tree/release-1.0) is the version > submitted to TestFlight and App Store review. The latest code is on `main`. > [!WARNING] > Only follow the install steps in this README. Never run commands from issues or comments (for example a "quick install" one-liner such as PowerShell `iex(irm ...)`); these are not from the maintainers. > **The computer is gone.** The board is now the AirPlay receiver and the mouse, and the agent runs in an iPhone > app on the same phone. The original version (a Mac/Linux computer running UxPlay and a Python server) is kept as > tag [`v0.1.0`](https://github.com/onehumanbeing/esp32-iphone-agent/tree/v0.1.0) and branch > [`legacy`](https://github.com/onehumanbeing/esp32-iphone-agent/tree/legacy). See [docs/legacy.md](docs/legacy.md).

An iPhone with a XIAO ESP32S3 plugged in, setting an alarm

*"Set an alarm for 8:00 AM that repeats every Monday and Tuesday, then save it"*: the agent leaves its own app, opens Clock, types the time into the wheel, ticks the days and saves. (Photo from v0.1.0; the task is the same.)

ESP32 Agent app: real phone vs. what the agent sees, and pointer calibration

*Left: the real phone. Right: what the agent sees through the board's AirPlay mirror. The second shot is the pointer calibration.* ## Contents - [How it works](#how-it-works) - [Hardware](#hardware) - [Quick start](#quick-start) - [Use from a coding agent](#use-from-a-coding-agent) - [Model and prompts](#model-and-prompts) - [Project layout](#project-layout) - [Roadmap](#roadmap) - [Limitations](#limitations) - [Responsible use](#responsible-use) - [Contributing](#contributing) - [Contact](#contact) - [Related projects](#related-projects) - [Acknowledgements](#acknowledgements) - [License](#license) ## How it works More detail: [how Handsy works](https://handsy.tech/how-it-works). ``` iPhone ──Screen Mirroring (Wi-Fi)──▶ ESP32-S3: AirPlay receiver ──Wi-Fi TCP :7201──▶ ESP32 Agent app (same iPhone) ▲ own hotspot, H.264 passthrough decode → vision model → action └──────── USB HID mouse + keyboard ◀──────────── BLE commands ◀──────────────────────┘ ``` - **Eyes — AirPlay to the board.** The board opens its own Wi-Fi hotspot and appears in the phone's Screen Mirroring list as *ESP32 Agent*. It keeps the H.264 stream (no decoding on the board) and forwards it to the app over Wi-Fi (TCP, port 7201). BLE carries commands and status. The phone keeps using cellular data for the model while it is on the board's hotspot. - **Hands — USB HID.** The board is a standard USB mouse and keyboard. With AssistiveTouch on, the mouse moves an on-screen pointer, so a click is a tap and a drag is a swipe. - **Brain — the app.** The app decodes the stream, sends screenshots to any OpenAI-compatible vision model, and runs a planner and an actor: tap, swipe, draw, type, press keys, go home, finish. It runs in the background while it works; the Dynamic Island shows what it is doing. - **Closing the loop.** The pointer is relative, so the app tracks it and re-homes it in a corner; before precise taps it wiggles the pointer, finds it on screen and corrects the miss. After each action it waits for the screen to settle and tells the model whether anything changed. A lost board, hotspot or mirroring link pauses the run and resumes it from the same step. - **Review.** Every run is kept on the phone: replay it step by step with the taps and swipes drawn on top. Export the screenshots, actions and saved annotations from **Runs → open a run → ⋯ → Export trace** for analysis by another agent. See [Export runs for agent review](docs/run-data-export.md) for batch export, USB copying and file details. ## Hardware Supported boards and setup details: [hardware guide](https://handsy.tech/docs/hardware). | Part | Notes | |---|---| | **ESP32-S3 board** | Tested with a [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/xiao_esp32s3_getting_started/) (plain version, 8 MB PSRAM; not Sense, not Plus). About $7 ([Amazon](https://amzn.to/3U6QKiz)\*). Any ESP32-S3 with native USB on its USB-C port and PSRAM should work. | | **USB-C cable** | Short USB-C to USB-C, e.g. [Anker 60W, 1 ft, 2-pack](https://amzn.to/4yvNyvi)\*. | | **iPhone** | iOS 18 or later, with cellular data (the Wi-Fi is taken by the board's hotspot). USB-C models plug in directly; Lightning models need a Lightning to USB Camera Adapter. The phone powers the board. | | **Vision model** | Any OpenAI-compatible chat API that accepts images: a cloud API or your own server. | | **Mac with Xcode** | Only to flash the board and build the app once. Not needed at run time. With the [TestFlight](https://testflight.apple.com/join/1xQHgJcA) app, Xcode is not needed for the app. | \* Affiliate link. As an Amazon Associate I earn from qualifying purchases. ## Quick start ### 1. Flash the board (plugged into the Mac) Install [arduino-cli](https://arduino.github.io/arduino-cli/), the ESP32 core 3.3.x and NimBLE-Arduino 2.5.x, then: ```bash arduino-cli lib install "NimBLE-Arduino@2.5.1" hardware/flash.sh # optional: [serial port] [--debug] ``` Always build with `flash.sh`: it applies the BLE buffer sizes in `hardware/hid_bridge/nimble_cfg.h`. No Mac or toolchain? [Install the firmware from Chrome or Edge](https://handsy.tech/install) over USB instead. ### 2. Install the app **Easiest: [join the TestFlight beta](https://testflight.apple.com/join/1xQHgJcA)** on your iPhone and install **ESP32 Agent** from there, then go to step 3. **Or build it yourself:** ```bash cp ios/Config/Local.xcconfig.example ios/Config/Local.xcconfig # set DEVELOPMENT_TEAM and IA_BUNDLE_ID open ios/IPhoneAgent.xcodeproj # run the IPhoneAgent scheme on your iPhone ``` The project is generated with [XcodeGen](https://github.com/yonaskolb/XcodeGen) from `ios/project.yml` (`cd ios && xcodegen`). A free Apple ID can sign it (re-sign every 7 days). The one-tap hotspot join needs the Hotspot Configuration capability (a paid account); without it the app shows the hotspot password to join by hand. ### 3. Set up (once, in the app) Plug the board into the iPhone and open **ESP32 Agent**. Setup walks through four steps and checks each one itself: 1. **Plug Me In** — pair the board over Bluetooth. 2. **Let Me See the Screen** — join the board's Wi-Fi, then Control Center → Screen Mirroring → *ESP32 Agent*. 3. **Let Me Use the Pointer** — turn on AssistiveTouch; the app calibrates the pointer by itself. 4. **Give Me a Brain** — enter your model endpoint and key (stored in the Keychain), then choose a vision model from the endpoint's model list (or enter its model ID). ### 4. Give it a task Type a goal (for example *"Open Settings and go to Wi-Fi"*) and press **Start**. Stop it from the Dynamic Island, or come back to the app. Siri, Shortcuts, widgets and Control Center can run saved tasks. ## Use from a coding agent The `esp32agent` CLI (npm, `npx -y esp32agent mcp`) is an MCP server, so a coding agent on your computer can see and drive the iPhone through the board (board firmware v0.2.2 or later). This repository is also a plugin with that MCP server and an agent skill ([skills/handsy/SKILL.md](skills/handsy/SKILL.md)). In Claude Code: ```bash /plugin marketplace add onehumanbeing/esp32-iphone-agent /plugin install esp32agent@handsy ``` Or add only the MCP server: ```bash claude mcp add esp32agent -- npx -y esp32agent mcp ``` Cursor, Cline and other MCP clients: ```json { "mcpServers": { "esp32agent": { "command": "npx", "args": ["-y", "esp32agent", "mcp"] } } } ``` DeepSeek Harness (dsh): install the plugin in [integrations/dsh](integrations/dsh/README.md), which adds the same MCP server and skill ([demo video](https://www.youtube.com/watch?v=DzYv0QVk_No)): ```bash dsh plugin --profile web add dsh-esp32agent ``` The CLI talks to the board from your computer over Bluetooth, or over your Wi-Fi once both are on it (`esp32agent net join`), so it works without the iPhone app or TestFlight. Commands and options are in the [CLI reference](https://handsy.tech/docs/cli) and [cli/README.md](cli/README.md). ## Model and prompts - Any OpenAI-compatible vision endpoint works (vLLM, SGLang, llama.cpp, Ollama, cloud APIs). The app lists the endpoint's models for you to choose one explicitly; you can also enter a model ID, and set a separate planner and actor model. - The prompts are plain text in [`prompts/`](prompts/) and bundled with the app: edit them, rebuild, done. - Each finished run logs planner/actor call counts, requests with a screenshot, and token totals when the endpoint reports `usage`. Missing usage is left absent (never coerced to 0); the run total is marked **partial** if any call omitted it. A short task is typically a few planner calls plus one actor call per step — each actor step sends a fresh screenshot — so image tokens dominate cost. Giving the planner a cheaper model while keeping a stronger actor is usually the best tradeoff. - The project was developed with [nvidia/Qwen3.8-Flash-Next-NVFP4](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4) served on two DGX Sparks ([TensorFold](https://github.com/MiaAI-Lab/Qwen3.8-Flash-Dual-DGX-Sparks-TensorFold)). ## Project layout ``` ios/ the app (SwiftUI): BLE link, H.264 decode, agent, setup, replay, Live Activity, widgets; tests prompts/ planner / actor prompts used by the app hardware/ ESP32-S3 firmware (Arduino): BLE GATT, hotspot, AirPlay receiver, USB HID; flash script, host tests, Mac BLE test harness (hardware/tools) docs/ BLE protocol, legacy version notes harness/ wireless development loop: Mac → iPhone app (Wi-Fi install) → board (firmware through the app) server/ legacy v0.1.0 computer-side server; not used by this version (see server/README.md) ``` Debugging without the phone app: `hardware/tools/board_test.py` plays the app over BLE from a Mac ([hardware/tools/README.md](hardware/tools/README.md)). Board ↔ app protocol: [docs/ble-protocol.md](docs/ble-protocol.md). ### Harness development With the board plugged into a test iPhone and the iPhone on the Mac's Wi-Fi, everything goes through `devicectl`, no cable: `harness/app.sh` builds and installs the Debug app, `harness/fw.sh` builds the firmware and copies it into the app, which installs it on the board (rolls back by itself if the new image fails), `harness/run.sh ""` runs a task and `harness/logs.sh` pulls the log. BLE from the Mac is only for the one-time `net sta`. Setup (Lightning iPhones: camera adapter), security: [harness/README.md](harness/README.md). ## Roadmap - [x] Drop the computer: the board mirrors and clicks, the app on the phone thinks - [ ] Landscape apps (coordinates and calibration assume portrait) - [x] Chinese text input (typed as pinyin through the board's keyboard; ctrl+space switches English and pinyin) - [ ] Faster steps: several actions per model call, pointer checks only when needed - [ ] A curated list of models that work best with this setup, shown inside the iOS app ([suggested by @asharthinks](https://x.com/asharthinks/status/2110740775030125019)) ## Limitations - **AirPlay source must be an iPhone.** Screen Mirroring from macOS is not supported: the handshake can look healthy while the board never receives video. - The screen is mirrored at 480×1040, up to 5 fps; each step takes a few seconds, most of it the model. - iOS does not mirror some screens (passcode entry, Apple Pay, DRM video); the agent cannot see them. Mirroring stops when the phone locks. - While the agent runs, the phone's Wi-Fi is the board's hotspot (no internet over it); the model is reached over cellular data. - Chinese is typed as pinyin through the board's keyboard (ctrl+space switches English and pinyin). The clipboard is not used for this; emoji are dropped. Landscape apps are not supported yet. If something doesn't work (no *ESP32 Agent* in Screen Mirroring, the board not connecting), see [troubleshooting](https://handsy.tech/troubleshooting). ## Responsible use Only automate devices and accounts you own or are allowed to use, and respect the terms of the apps involved. The agent acts with your hands: watch it, and keep the stop button within reach. ## Contributing Issues and pull requests are welcome; see [CONTRIBUTING.md](CONTRIBUTING.md). Security reports: [SECURITY.md](SECURITY.md). ## Contact Questions or feedback: [support@handsy.tech](mailto:support@handsy.tech) · Website: [handsy.tech](https://handsy.tech) ## Related projects | Project | Approach | Difference from this project | |---|---|---| | [VersaiOS](https://github.com/zhaodashuaibi/VersaiOS) | UxPlay mirroring + ESP32-S3 Bluetooth mouse + vision model | Needs a computer; one tap per instruction | | [ja1chen/actell](https://github.com/ja1chen/actell) | Agent app on the iPhone via Apple's CoreDevice debug channel | Needs Developer Mode and a Developer Disk Image | | [AidenAI-IO/aiden-firmware](https://github.com/AidenAI-IO/aiden-firmware) | HDMI capture + USB keyboard/mouse + multimodal model | Capture hardware; mainly Android | | [yoyicue/glassbox](https://github.com/yoyicue/glassbox) | PicoKVM HDMI capture + USB HID | macOS controller and PicoKVM | | [boboabw/phone-agent](https://github.com/boboabw/phone-agent) | ESP32-S3 Bluetooth HID + AssistiveTouch, agent as an iPhone app | Closest to this version; this one also brings the eyes (AirPlay on the board) | | [VKCOM/devicehub](https://github.com/VKCOM/devicehub) | ESP32 Bluetooth mouse for iOS device farms | Device-farm tooling, no AI agent | WebDriverAgent/XCUITest-based tools such as [mobile-mcp](https://github.com/mobile-next/mobile-mcp) and [Midscene](https://github.com/web-infra-dev/midscene) need a Mac and code signing. ## Acknowledgements [UxPlay](https://github.com/FDH2/UxPlay) (the AirPlay receiver the firmware's is ported from) · [NimBLE-Arduino](https://github.com/h2zero/NimBLE-Arduino) · [Arduino ESP32 core](https://github.com/espressif/arduino-esp32) and [TinyUSB](https://github.com/hathach/tinyusb) ## License [MIT](LICENSE), except `hardware/hid_bridge/src/airplay/` (ported from UxPlay), which is [GPL-3.0](hardware/hid_bridge/src/airplay/LICENSE-GPL.md); the firmware binary that links it is therefore distributed under GPL-3.0. Files under [`skills/`](skills/) are [MIT-0](skills/LICENSE) (MIT No Attribution); the rest of the repository keeps the license above.