# Drengr Runner

**Drive an iOS simulator over HTTP. 479 lines of Swift. No WebDriverAgent.**

A single XCTest target that hosts a loopback HTTP server inside a UI Test bundle. Three endpoints:
screenshot the screen, read the accessibility tree, and act on it — tap, swipe, draw, type, press
Home, rotate. Any language that can make an HTTP request can drive a simulator.

It was built as the iOS control plane for [Drengr](https://drengr.dev), replacing Facebook's
WebDriverAgent. It works on its own, and this repository is the whole thing.

![Drengr tapping a button drawn on an HTML canvas in an iOS simulator](docs/canvas-demo.gif)

*An HTML `<canvas>` has no accessibility tree, no DOM children and no selectors — there is nothing
to query. Coordinates still work, because a touch is a touch. (Deciding **where** to tap is
Drengr's job, not the runner's.)*

## Quickstart

Everything below was run end to end on a clean checkout. Requires Xcode and a simulator.

```bash
# 1. Build the test bundle
xcodebuild build-for-testing \
  -project DrengrRunner.xcodeproj -scheme DrengrRunner \
  -destination "id=$UDID" -derivedDataPath /tmp/dr-dd
# ** TEST BUILD SUCCEEDED **

# 2. Run it — this blocks and serves until you stop it (~30s to come up)
xcodebuild test-without-building \
  -project DrengrRunner.xcodeproj -scheme DrengrRunner \
  -destination "id=$UDID" -derivedDataPath /tmp/dr-dd

# 3. Talk to it
curl -s http://127.0.0.1:8200/status
# {"ok":true,"product":"drengr-runner","version":"0.6.0","ios_major":26,
#  "screen":{"width":402,"height":874,"scale":3}}

curl -s "http://127.0.0.1:8200/observe?bundle=com.apple.mobilesafari" | jq 'keys'
# ["screenshot_b64","tree_hint"]

curl -s -X POST http://127.0.0.1:8200/act \
  -H 'Content-Type: application/json' \
  -d '{"kind":"tap","x":201,"y":437}'
# {"ok":true}
```

Get a UDID with `xcrun simctl list devices available`.

## API

### `GET /status`
Health plus screen geometry. `scale` is the retina factor — coordinates are in **points**, not pixels.

### `GET /observe?bundle=<id>`
```jsonc
{
  "screenshot_b64": "…",   // JPEG, base64. Downscaled to ≤2× and quality 0.6
  "tree_hint": { … }       // accessibility snapshot, or null
}
```

**Pass `bundle`.** Without it you get SpringBoard only, and your app's elements are invisible. With
it, the foreground app's snapshot is merged with SpringBoard so system alerts and the keyboard stay
in the tree.

`tree_hint` nodes carry `type`, `frame` (`[x, y, w, h]`, points), and — when non-empty — `id`,
`label`, `value`, `enabled`, `children`.

### `POST /act`
The discriminator field is **`kind`**.

| `kind` | Fields | Notes |
|---|---|---|
| `tap` | `x`, `y` | points |
| `swipe` | `x1`, `y1`, `x2`, `y2`, `duration_ms?` | default 300ms |
| `draw_path` | `points: [{x,y}, …]`, `duration_ms?` | ≥2 points, default 800ms |
| `type` | `text`, `bundle_id?` | requires a focused field — see below |
| `button` | `button: "home"` | only `home` is wired in 0.6.0 |
| `orientation` | `orientation` | `portrait`, `portrait_upside_down`, `landscape_left`, `landscape_right` |

## Known limitations — read these before adopting

- **`draw_path` lifts the finger between points.** Each segment is a discrete drag, so it is not one
  continuous stroke. True continuous strokes need the private `XCSynthesizedEventRecord` path.
- **`type` needs something already focused.** It searches your app, Spotlight, then SpringBoard for
  an element with keyboard focus and types into the first it finds. Nothing focused → clean error.
  Tap the field first.
- **`button` supports `home` only** in 0.6.0.
- **Simulators only.** `SUPPORTED_PLATFORMS: iphonesimulator`. No physical devices.
- **Caller errors return HTTP 200** with `{"ok":false,"error":"…"}`. Check the `ok` field, not the
  status code. HTTP 500 is reserved for genuine runner faults; 404 and 413 behave normally.

## Design notes

**Loopback only.** `NWParameters.requiredInterfaceType = .loopback` — it cannot be reached off the
machine, so a running simulator is not an open port on your network.

**The listener reopens itself.** iOS tears down the listening socket when the runner becomes
suspension-eligible — which SpringBoard triggers whenever an overlay takes the screen (Spotlight) or
the device rotates. Left alone, every later request gets connection-refused. The listener watches for
`.failed`/`.cancelled` and reopens with a short backoff. This is the single fix that makes a
long-running XCTest server survive real use.

**Failures don't kill the server.** `continueAfterFailure = true`. A failed interaction — typing with
no focused field, say — would normally end the test method and take the server with it. Instead the
client gets an error and the server keeps serving.

**Screenshots are JPEG, not PNG.** The 3× retina grab is downscaled to ≤2× and JPEG-compressed at
0.6: roughly 10× smaller with negligible loss, and far fewer image tokens when a vision model is
reading it. Falls back to PNG if JPEG encoding fails.

**Rotation resets through portrait.** Going landscape→landscape directly leaves `XCUIDevice` in a
stuck state, so the runner passes through portrait first.

## Why this exists

WebDriverAgent is a large dependency that needs its own build, its own host process, and breaks on
Xcode upgrades. For driving a *simulator* by coordinates, nearly all of it is unnecessary.

Drengr replaced it with this — one Swift file, one HTTP server, no host process. Drengr's bootstrap
still uninstalls `com.facebook.WebDriverAgentRunner.xctrunner` from simulators that ran the older
version.

This does not replace Appium or WebDriverAgent for **physical devices** or device clouds. Drengr
still speaks Appium/WebDriver for those (BrowserStack, Sauce Labs, LambdaTest, Perfecto, Kobiton, or
any Appium-compatible endpoint). The simulator path is the part that got simpler.

## What this deliberately does not do

The runner takes coordinates and returns pixels. It has no idea what is on the screen, what a button
means, or where to tap next. That is the entire hard part of mobile automation, and it is not in
this repository.

If you already know your coordinates — a fixed layout, a recorded script, a canvas game with known
hit boxes — this is all you need. Stop here.

If you *don't* know them, you need something to look at the screenshot and decide. That is
[Drengr](https://drengr.dev), which uses this runner as its iOS transport:

```bash
npm install -g drengr
drengr demo          # an agent drives a system app, no setup
drengr look          # numbered elements + screenshot
drengr do tap --element 5
```

Drengr reads the screen as text first (cheap) and escalates to vision only when elements are
unlabeled — which is exactly the canvas/Flutter/game case in the GIF above. Bring your own model key;
it never proxies a call on your behalf.

- **Run it in CI** → [`drengr-ci`](https://github.com/SharminSirajudeen/drengr-ci), a GitHub Action
- **Drive it from an AI client** → `drengr mcp` speaks MCP over stdio
- **Android too** → same interface, ADB transport

## Relationship to Drengr

This runner is one transport behind Drengr's device interface, alongside ADB for Android and Appium
for device clouds. It knows nothing about agents or models, which is why it is useful on its own to
anyone scripting a simulator from any language.

## License

Apache-2.0. See [LICENSE](LICENSE).