# Gateway, MCP and local API

Branch `claude/gateway-core-da93b5`. This covers the packages the main session owned in [WORKING.md](../WORKING.md): `internal/gateway` (except the shared `api.go`), `internal/policy`, `internal/approval`, `internal/history`, `internal/mcp` and `cmd/pinky`, plus `internal/api` (the local socket, listed in DESIGN's repository layout), `internal/demo` (scripted fake phones) and, from round three, `internal/recording`. The first round used fake phones only; the second ran the gateway on the Mi 10 Pro and the iPhone (see "Real phones") and added the Settings API; the third ([round3.md](round3.md)) added waking and unlocking, `phone_files`, one stream per device, recordings, retention and export, background displays and MCP over HTTP (see "Round three").

## What works

Everything below runs and is tested against fake phones; real drivers plug in later (see "Integration").

| Area | What it does |
| --- | --- |
| Gateway | `gateway.New(Options{Providers, DataDir, …})` discovers devices every 2 s through providers (no platform branches), opens them, and implements `gateway.API` for the client. Agents use `Session` methods instead, which are not in `API` |
| Pipeline | lease and scope → policy → approval → execute → settle (`driver.Settler` when the driver has it, else polling `Observe` until two reads match, 3 s limit) → read and diff → record. One `screen.Refs` per device; results are `screen.Diff` lines, or the whole screen on an app change |
| Leases | One MCP session can hold several phones. A lease ends on `phone_release`, session close, 10 idle minutes, Stop, or after 60 s offline. Another session gets `lease_held` with the holder's agent and task. A drop mid-action returns `uncertain`, then `device_offline`; a phone back within 60 s keeps its lease |
| Person controls | `TakeOver`, `HandBack`, `Stop`, `StopAll`, mirror input (`Touch`, `Key`, `Text`; taps and swipes are emulated when a driver lacks `Toucher`), lock state (from `Screen.Locked` and `driver.LockReporter`; an idle phone every `idleLockEvery`, 6 s, or `idleLockEveryWiFi`, 30 s, over Wi-Fi with no picture watched, GitHub issue #96), client events |
| Policy | Allow, Ask, Deny; commit detection (`commit`, `payment`, `send_message`, `install_app`, `open_url`, `call`, plus `change_setting`) with word tables in 11 languages; YAML rules, strictest wins; a bundled, editable app table; blocked apps hidden from agents; password and one-time-code fields refused (rule `credentials`); read-only mode; per-task app list; one-time codes masked on the screen and in events, and covered in screenshots |
| Approvals | 60 s from opening, then Deny; calls block 45 s, then `approval_pending` with a single-use id bound to session, device, lease, tool and arguments; decisions between calls are kept; on Allow the target and screen text are checked again (`stale`); the phone takes no other action meanwhile (reads still work); only the client's `Decide` answers |
| History | SQLite through `modernc.org/sqlite` with the tables from DESIGN (devices, agents, sessions, tasks, steps, approvals, media) plus `recordings` and `device_settings`; a JPEG before and after each step in `media/`, taken in the background (see "What the agent waits for"); a screen recording per task in `recordings/`; retention, export and `WipeHistory` |
| MCP | The 20 tools from DESIGN on the official Go SDK v1.8.0, with progress notifications when a request has a `progressToken`, over stdio (through the socket) and Streamable HTTP. Reference: [docs/mcp.md](../mcp.md); agent guide: [docs/SKILL.md](../SKILL.md) |
| Local API | A Unix socket only the user can reach (0700 folder, 0600 socket, peer uid checked), one gateway per data folder (flock on `pinky.lock` in the data folder, so two gateways never share one history and the same phones, even when their runtime folders differ: logind removes `/run/user/<uid>` at logout; a socket that still answers is left alone, so a gateway on other data in the same runtime folder, or an older Pinky that kept its lock there, keeps it); on Linux the socket is in `$XDG_RUNTIME_DIR/pinky`, and a Pinky started without that variable, by a service for one, uses the user's own `/run/user/<uid>/pinky` before `~/.local/state/pinky`, so it finds the desktop session's gateway; on Windows a named pipe `\\.\pipe\pinky-<hash of the runtime folder>` instead (MSIX-packaged apps such as the Claude and Codex desktop apps see a virtualized AppData and could not reach a socket file there, issue #69): owned by the user and open only to them, remote clients refused, the first instance exclusive so a second gateway cannot start beside it (Windows has no flock), clients check the pipe's owner before they send the agent token, and fall back to the old `pinky.sock` to reach an older gateway that is still running; a Unix socket path longer than the system takes (107 bytes on Linux, 103 on macOS) fails with an error that names `PINKY_HOME`; JSON-RPC control plus MCP sessions |
| Commands | `pinky` (the app: gateway and window in one process), `pinky daemon` (`--http addr` for remote agents), `pinky mcp` (stdio shim that starts the gateway when none runs), `pinky devices`, `pinky token`, `--fake`; hidden helpers `pinky needs` and `pinky decide` (answers only on a fake gateway) |

## Round three

Built on 2026-11-07 from [round3.md](round3.md). Each part has tests against fake phones (test doubles for the driver interfaces the drivers do not have yet), and the streams and recordings also ran on the emulators and both phones (see "Checked on real streams").

### Wake and unlock

- **Wake** (`wake.go`): before an agent's call reads or acts on the phone's own screen (connect, every action, `phone_observe`, `phone_wait_for`, an approved action, a hand-back), the gateway calls `driver.Waker.Wake`, at most once every 5 s (the shortest screen timeouts are 15 and 30 s). A woken phone that is locked shows its lock screen, and the agent gets `device_locked` and the NeedUnlock card as before. Tasks on a background display never wake the phone.
- **Unlock from the computer**: `Options.Unlock` is gone; `Options.PIN func(ctx, deviceID) (string, error)` hands out the saved PIN and the gateway enters it with `driver.Unlocker.Unlock`, only after asking the phone again whether it is locked (a PIN typed into an unlocked phone would land in the app in front), and once: a wrong PIN returns `ErrWrongPIN` and is never retried. It holds the device's action lock meanwhile, counts as input (no screenshot or recording of the PIN pad is kept), and adds a step to the running task: "Unlocked from the computer", or "Unlock from the computer failed: the phone did not accept the saved PIN" (status `wrong_pin`). The log says only whether it worked. `DeviceView.Features.Unlock` is the driver's `Unlocker`.
- **cmd/pinky** passes a PIN source that answers only while Settings › Unlock from this computer is on, checks `secret.Store.Has` first (so a phone without a saved PIN never brings up Touch ID), and calls `secret.System().Reveal(ctx, secret.PINAccount(id), "unlock "+name)`. On main, `secret.System()` is still the stub, so unlocking from the computer returns the keychain's `ErrUnsupported` until the platform session lands it.

### phone_files

`Session.Files` and the tool (`files.go`): `list`, `pull` and `push` between the phone's Pinky folder (`driver.Files`) and `<data>/files/<phone id>/` (the id made safe for a folder name). Results carry the local folder and the files, each with where it is (`both`, `phone`, `computer`, `differs`, by size and time to the second). Pull and push skip files that are the same on both sides and keep modification times. Push asks the person (rule `Files`, the usual 45 s block and `approval_id`; "Allow for this task" covers later pushes), read-only mode refuses it, and read-only and watch-only agents get `scope_denied`. Paths with `..` are refused before the driver sees them; symbolic links in the local folder are never pushed, a pull goes through a temporary file renamed into place, so a link planted where a file goes is replaced, not written through, and a folder on the way that is a link is refused before anything is made. Every call is a history step. iPhones (no `Files`) return `unsupported`. Docs: [mcp.md](../mcp.md), [SKILL.md](../SKILL.md).

`upload` (`upload.go`, GitHub DIYgod/pinky#8) is for agents that cannot write that folder, such as one on another computer over HTTP: it saves one file the agent sends into it, for a push after it, and never touches the phone, so it asks nothing; it follows push's scope and read-only rules. Names are checked more strictly than other paths (no hidden files, nothing Windows or Android's shared storage refuse, no Windows device names); a taken name needs `Replace` unless the bytes are the same; the file goes through a `.pinky-` temporary file renamed into place, so a link where it goes is replaced, and a folder on the way that is a link is refused. `FilesRequest.Body` carries the bytes (at most `MaxUpload`, 200 MB), and `Check` only checks. The MCP layer (`internal/mcp/upload.go`) takes the bytes in `content` (base64, up to 10 MB; HTTP requests may now be 16 MiB, as stdio messages are), or, over HTTP without `content`, checks the upload and hands out a link, `/upload/<token>` on the same listener, that takes one `PUT` of the file without the agent token within 10 minutes; links live on `mcp.Server` and end with their session. The address in the link is the request's host (or a proxy's `X-Forwarded-*`), which `HTTP.ServeHTTP` passes to the tools in the `X-Pinky-Link-Base` header over whatever the agent sent. The step reads "Received “photo.jpg” (1.2 MB) from the agent", which `internal/gwtext/gwtext.go` translates.

**Resuming sessions** (`resume.go`). Some clients over HTTP open an MCP session for every call, or for every turn, and never send an id twice (Grok Bot, reported on X); each call then found no phone, and the phone stayed leased to the session before until it idled out. `AgentInfo.Resume`, which `internal/mcp` sets for HTTP sessions, lets a newer session of the same agent name continue a lease its older sessions are not in a call on (`adoptLocked`, from `acquire`, `Connect` and `Release`): the lease, its task, grants and open approvals (`approval.Center.Rebind`) move to the newer session, which sessions are ordered by `Session.seq`. It never goes back, and other agents and local agents never adopt. A Resume session that closes keeps its leases for `Options.ResumeGrace` (2 min) for the next session (`keepLocked`); `Session.CloseNow` and `EndKeptLeases` (turning Remote agents off) and `Gateway.Close` end them at once. A Stop of a Resume agent's task holds for its newer sessions for `stopHold` (10 min, `resumeStops`). The new-agent approval of a Resume session is bound to the agent's name, so the next session can repeat it. Upload links follow `Session.Current`. `internal/mcp` also answers a `GET` without a session id with `405`, and serves clients that send no session id at all (`sessionless.go`, GitHub DIYgod/pinky#10): a `POST` without `Mcp-Session-Id` other than `initialize` goes into a session Pinky opens and keeps per `User-Agent`, initialized with the parameters of that client's newest `initialize` whose session no request ever named (a session that clients claim with an id is never used for others), or `clientInfo` "agent" without one; its id is injected as the client would have sent it, and an id-less `notifications/initialized` is answered without forwarding. Tests: `resume_test.go` in both packages, the MCP one with a hand-written client that opens a session per call (with and without `DELETE`), and one that keeps a `GET` stream open while it calls; `sessionless_test.go` and the HTTP end-to-end test send what #10 shows Grok Bot sending.

### One stream per device

A phone gives one stream at a time (the Android driver ends the first when a second opens), and the mirror's viewers and the recorder all want it. `streams.go` keeps one hub per device screen: the first member opens the driver's stream, every packet goes to each member, and the stream closes when the last member leaves. A member that joins late first gets the last configuration and the packets since the last key frame, so the mirror shows the current picture at once. Each viewer has a queue of its own and hands packets out through a channel of 4 as it reads them: a burst of the packets since the last key frame never looks like a backlog to the mirror's player (which skips to the next key frame past 30 queued packets). A viewer that falls more than 2048 packets or 64 MB behind skips to the next key frame. When the phone goes offline or the gateway stops, every member sees the stream end. `Gateway.Stream` (the mirror) and `AgentStream` (a background display) are hubs; the recorder taps them.

A stream that can ask the phone for a key frame now (an optional `RequestKeyFrame()` on the driver's stream; nothing in `driver.go` yet) is asked when a recording starts or resumes and when a viewer falls behind; otherwise members wait for the next one. The iOS driver could implement it with its `requestKeyFrame` (PLI), the Android driver with scrcpy's `ResetVideo`.

### Recordings

One fragmented MP4 per task, remuxed without re-encoding (`internal/recording`, on `github.com/Eyevinn/mp4ff`): an init segment with the first parameter sets (`avc1` or `hvc1`), then a fragment about every second, one starting early at a key frame, so a crash loses at most the fragment being written and leaves a file that players open. Parameter sets that change later (a rotation) go into the next key frame's sample. Files live in `<data>/recordings/<YYYY-MM>/<task id>.mp4` with a row in the `recordings` table; `Recording(ctx, taskID)` returns the path, start, length (read from the file, so a crash's file is right too) and a mark per step (`StepView.At` from the recording's start). `recording.Open(ctx, path, from)` plays one back as a `driver.Stream`: from the key frame at or before `from`, the frames up to `from` at once, then paced in real time, through a channel of 8 that waits for the reader. `SetRecording(on)` turns recording of new and running tasks on or off (off by default in the gateway; the app and the daemon apply `Settings.Record`, on by default); a task is recorded once, so turning it off and on again does not overwrite its file.

**Nothing of a blocked app, a locked screen or the person's own use is stored.** The gateway knows what is on the screen only from reads, so the recorder holds packets back until a read vouches for them: the packets that arrived before a read started are written when that read shows a recordable screen (unlocked, an app that is known and not blocked, the person not in control), and dropped, with everything held since the last read that did, when it does not. Until a read shows a recordable screen again, packets are dropped, and writing resumes at a key frame (a picture decoded against the wrong reference could show traces of what was dropped). Taking over, a lock the driver reports, an unlock from the computer, the phone going offline and a new stream drop what is held too. Every drop is a point in time: reads that started before it can no longer resume the recording, and packets that arrived before it and come later (a hub replays its last key frame to a member that joins again) are never written; the task's start is the first such point, and its end the last (later packets are not taken, so the person's own use after the task stays out, however late the last read comes). These points, the packets' arrivals and the reads' starts are times from `stamp` (`clock.go`), which never gives the same time twice: `time.Now` on Windows moves only with the system's clock tick (0.3 to 2 ms on the CI runners, up to 16 ms by default), and a picture of the person's own use stamped in the tick the task ended in was taken for the task's. When packets have waited 10 s and no action runs, the recorder reads the screen itself, so it holds little; past 48 MB it drops them. When the task ends it reads once more, except after Stop. Frames are placed by their PTS plus the shortest delay any frame of the stream took to arrive, which is smoother than arrival times and never places a frame after it arrived. Reads can overlap (`phone_observe` during an action, the recorder's own read): a recordable read writes nothing while a read that started before it still runs (in order, that one might have dropped it), a read that shows what must not be recorded drops what is held whenever it ends, and a recordable read older than one that already decided changes nothing. The recorder taps the screen's current device each time it opens the stream (a new one after the phone reconnects, a reopened background display), and its last read, when a task ends, reads that screen itself, waiting at most 15 s for actions in progress. A read that names no app (taken while one app hands over to the next; on the emulator, Settings opening) vouches for nothing, for history screenshots too, since a blocked app may be on its way in. A blocked app that comes and goes between two reads, neither of which sees it, is not caught; payment hand-offs stay in the bank's app until the person is done, so the next read sees them.

Agents' own screenshots (`phone_observe { screenshot: true }`) now follow the same rule: none while the phone is locked (its lock screen can show notifications from blocked apps) or when a read names no app.

Every screenshot the gateway takes (history, approval cards, agents' own) goes through `shotOf`, which asks a driver that prefers pictures from the video (`driver.FrameShooter`: an iPhone over Wi-Fi, whose screenshot is a PNG of up to 15 MB at about 1 MB/s) for the screen from a frame first (`frameShot`), and takes a screenshot when it has none. The rules above apply to either picture alike.

### Retention and export

- `SetRetention(days)` (`retention.go`) deletes tasks that ended more than `days` ago (or, for tasks that never ended, as after a crash, started), with their steps, approvals, screenshots and recordings, and sessions left without tasks; at once and every 24 hours after. Running tasks stay; a screenshot that another task's step still shows stays. 0 keeps everything (the gateway's default; the app and daemon apply `Settings.KeepDays`, 30). `Retention()` reads it. If reading a task's screenshots fails partway, that pass changes nothing: going on would delete the tasks and leave the screenshots the read never reached.
- `ExportHistory(ctx, taskIDs, dir)` (`export.go`) writes `pinky-history-<date>-<time>.html`, a report that opens in any browser on its own (screenshots inline as data URLs, each shown once, a click enlarges it; each recording in a `<video>` with its step marks as links), a `.json` file of the same tasks (steps with their arguments, intents, statuses, times, screenshot ids and the screen text after each step; approvals; recordings with marks), and each recording copied next to them. Empty `taskIDs` means every finished task. HEVC recordings are copied as they are (Safari plays them; other browsers may not), not transcoded.

### Background displays

`SetBackground(deviceID, on)` (saved in `device_settings`; `driver.ErrUnsupported` for a connected phone whose driver has no `Backgrounder`), `Background(deviceID)`, `AgentStream(ctx, deviceID, opt)` (`background.go`). Whether a task uses a background display is decided when it starts, so the setting applies from the next task, and watch-only agents never get one (they watch the phone's own screen). With background on, a lease opens a display (`OpenBackground`, once per lease, again after the phone dropped) when its first call needs the screen, and its reads, actions, screenshots and recording use it. The phone's own lock does not stop the agent (no `device_locked`, no unlock card), the phone is not woken, and the person's touches on the phone or the mirror do not pause it; Take over still does. The status line ends "· in the background". The display closes with the lease. When it cannot be opened, when an app refuses it (its `Launch` returns `driver.ErrUnsupported`: the app is then launched on the phone's screen, unless that is locked, which returns `device_locked`, or the person used that screen in the last minute, which puts them in control and returns `user_took_over`), or when it cannot be read (`Observe` returns `driver.ErrUnsupported`), the task moves to the phone's screen for the rest of the lease and the agent's next result says why. An agent can also ask for the phone's own screen (`ConnectRequest.OwnScreen`, `phone_connect own_screen`), for work on the home screen itself, which a background display does not have (`useOwnScreen`): a new lease never opens a display, and the same task connected again with it leaves its display. It is treated like an app refusing the display: no card (the person's rules still judge every action), but a person who used the phone's own screen in the last minute is put in control and the result is `paused`, and a locked screen returns `device_locked`. The connect step reads ", on the phone's own screen" (args `own_screen`), so the person sees it in the task. The connect result on a background display, and the read of an empty one, point agents at `own_screen` (`ownScreenHint`).

### MCP over HTTP

`mcp.Server.HTTP(token)` (`internal/mcp/http.go`) serves the tools over the SDK's Streamable HTTP handler at `/mcp`, on an SDK server of its own so that turning it off (`CloseSessions`) ends exactly its sessions. Agents over HTTP are named with " (remote)" after their client name ("Claude Code (remote)"): a name is only what the agent says, so a remote agent never inherits the approval and scope of the local agent it claims to be, and its first connection asks. Every request needs `Authorization: Bearer <agent token>`; cross-origin browser requests are refused (`http.CrossOriginProtection`) and the SDK refuses loopback requests that name another host, but for this computer's own tailnet name (`HTTP.OwnNames`, which cmd/pinky fills from `tailscale status --json`): `tailscale serve` forwards requests to 127.0.0.1 with the `Host` they came with and cannot change it, so before 2026-11-10 every request through it got `403`. `ownHost` serves such a request as one to `localhost` after `linkBase` has taken the name for links; a page can make a browser send only a name whose DNS its author controls, and a `ts.net` name is Tailscale's, so the rebinding protection stands (`TestHTTPThroughTailscaleServe`). One MCP session is one gateway session, closed when the MCP session ends (`DELETE`, an hour idle, or Remote agents off), for stdio sessions too now. cmd/pinky starts, moves and stops it from `Settings.RemoteAgents` and `Settings.RemoteAddr` (`ui.DefaultRemoteAddr` when empty), logging a warning for an address other computers can reach; `pinky daemon --http addr` starts it at startup, and `pinky token` prints the token. [mcp.md](../mcp.md) has the `claude mcp add --transport http` command.

### Demo phones

Every demo phone now wakes (`driver.Waker`; `SetScreen` turns its screen off) and takes a PIN (`driver.Unlocker`, `demo.PIN` = 1234). The demo's Android phones come out of `Provider.Open` wrapped in `demo.Android` (`demo.Of` unwraps), with a Pinky folder (`PutFile`, `File`; the fake Pixel has `Notes/groceries.txt`) and background displays (a demo phone of their own), so `--fake` and the client's runs on the real gateway show those controls; the fake iPhone has neither, as real iPhones.

### cmd/pinky

The app loads Settings before the gateway starts, so the PIN source knows the unlock setting; it applies Record (`SetRecording`), KeepDays (`SetRetention`) and Remote agents at start and on every change. The daemon applies Record and KeepDays from the app's `settings.json` when there is one (defaults otherwise), and `--http`.

### Checked on real streams

On 2026-11-07, holding each device's lock (`scripts/devlock.sh`) for every run:

| | What ran | Result |
| --- | --- | --- |
| emulator-5560 (Android 16) | `TestEmulatorRecording`: Clock with a mirror viewer open; Settings blocked for the run and brought up by adb, then scrolled for 3 s; five runs | A shared scrcpy stream (the mirror got 1100 to 1150 packets); recordings of 787 to 864 frames over about 40 s, none while Settings was in front, frames on both sides. Two runs first failed because Settings read as no app or as its search package; that led to the unknown-app rule |
| emulator-5560 | `TestEmulatorBackground`: background on, through a test adapter for the driver's `OpenBackground` | The display opened, but the companion cannot read it ("display 359 is not open to accessibility services"); the gateway moved the task to the phone's screen and said so |
| emulator-5562 (Android 10) | the same recording test | The driver's stream fails at once (`scrcpy: video stream: EOF`, also with `androidctl stream`), so no mirror and no recording; the recorder backs off (2 s to 30 s) |
| Mi 10 Pro | the device test with `PINKY_RECORD=1` (Calculator, Settings, take over, Stop), twice | All parts pass; last run's recording H.264 1080x2340, 1470 frames, 38.2 s, 21 marks, 21.2 MB (ffprobe). Action median 1.42 and 1.44 s, as without recording (1.44 s) |
| iPhone 13 mini | the same, three times | All parts pass (one earlier run failed the test's own switch check, fixed below); last recording HEVC 1136x2464, 3419 frames, 68.7 s, 21 marks, 26.3 MB. Action median 2.89 and 2.50 s (the driver's reads got faster on main) |

After the review fixes, the recording test ran once more on emulator-5560 (552 frames, none while Settings was in front); both phones had locked themselves by then, so their device tests skipped without input, as they should. Recordings and logs stayed in `/tmp/pinky-gw-*` and were deleted. One mistake on the way: the first emulator run did not limit the provider to the emulator, so its gateway also opened the Mi 10 Pro and emulator-5562 for 43 s (the driver reads their properties and listens to `getevent`; no input was sent); the emulator tests now list only their emulator. One read-only `dumpsys` on the Mi also ran outside its lock.

Also found: the device test's check that no tap reached a denied switch looked at every tap of the run, and on the iPhone a Calculator tap from the first subtest fell inside where Settings' Airplane Mode switch now sits (two new banners moved it down). It now counts only input since the approval opened.

### For the drivers

- `Backgrounder`: return `driver.ErrUnsupported` from the display's `Launch` when the app opened on the main display instead (Android: some apps refuse secondary displays), so the gateway can move the task; and from `Observe` when the display cannot be read. On the Android 16 emulator the companion cannot read scrcpy's virtual display; background work needs a display accessibility services see (or another read path). The display's `Screenshot` returns `ErrUnsupported`, so background tasks have no history screenshots or approval-panel picture.
- `RequestKeyFrame()` on streams (optional, above).
- `Waker`, `Unlocker` and `Files` are not in the drivers on main yet; the gateway's paths for them are tested with doubles (`extras_test.go`).
- The Android 10 emulator's stream (above).

### What is left

- HEVC recordings are not transcoded for the export.
- `recording.Open` plays what was written when it opened; it does not follow a recording that is still growing.
- The person cannot drive a background display from the client: `Touch`, `Key` and `Text` reach the phone's own screen.
- Remote API over WebSocket, history encryption.

## How to run

```sh
go build ./... && go vet ./... && go test ./...
```

`go test ./...` includes two end-to-end tests in `cmd/pinky` that build the binary and run `pinky daemon --fake` and `pinky mcp` as separate processes (`go test -short` skips them). They use `/tmp/pinky-e2e-*` because a Unix socket path must fit in 104 bytes on macOS; the folders are removed afterwards.

Try it by hand with fake phones (a "Pixel 9 (fake)" and an "iPhone 17 (fake)": home screen, Dinely with a $12.80 payment, Chatter, Settings, a blocked Google Wallet, and an account screen with a password field):

```sh
go build -o /tmp/pinky ./cmd/pinky
export PINKY_HOME=/tmp/pinky-try        # keeps everything out of your real Pinky folders
/tmp/pinky daemon --fake &
/tmp/pinky devices
claude mcp add pinky-fake --env PINKY_HOME=/tmp/pinky-try -- /tmp/pinky mcp --fake
```

Then ask the agent to "reorder last week's lunch on the fake Pixel". The Pay tap waits for approval; answer it from another terminal:

```sh
/tmp/pinky needs
/tmp/pinky decide <id> allow "enjoy"
```

Real phones (build tag `device`): `internal/gateway/device_test.go` runs the gateway in-process with the real provider for one phone and drives a session like an agent: Calculator (6 × 7 = 42, one tap at a time and as one `phone_batch`), Settings (scroll, open a sub-page, back, home), a Settings switch that must ask and is denied, ref stability, take over, hand back, Stop mid-action, and the stored screenshots. Every input passes `guardDevice` (`guard_test.go`), which refuses anything outside Calculator keys and Settings navigation and any tap on a switch, whatever the gateway decides, and stops without input when the phone shows anything unexpected or is locked. Build once, then hold the phone's lock only for the run:

```sh
go test -c -tags device -o /tmp/pinky-gw/gateway.test ./internal/gateway
scripts/devlock.sh 3e8be48e env PINKY_DEVICE=3e8be48e /tmp/pinky-gw/gateway.test -test.run 'TestDevice$' -test.v
scripts/devlock.sh 00008110-001E30C9262A801E env PINKY_DEVICE=00008110-001E30C9262A801E /tmp/pinky-gw/gateway.test -test.run 'TestDevice$' -test.v
```

Add `PINKY_RECORD=1` to record the run and check the recording with ffprobe (the guard then lets the stream through; it only reads the screen). The emulator tests block Settings for the run and drive it with adb, so they refuse anything but an emulator serial:

```sh
scripts/devlock.sh emulator-5560 env PINKY_EMULATOR=emulator-5560 /tmp/pinky-gw/gateway.test -test.run TestEmulator -test.v
```

Folders: `$PINKY_HOME/{config,data,run}` when set; otherwise `~/Library/Application Support/Pinky` on macOS, `%APPDATA%\Pinky` (config) and `%LOCALAPPDATA%\Pinky` (data, run) on Windows, XDG folders on Linux. Config holds `rules.yaml`, `apps.yaml` (both written with defaults on first start) and `agent-token` (0600). Data holds `pinky.db`, `media/` and `daemon.log`; a `--fake` gateway keeps its history in `data/fake`.

## Real phones

On 2026-11-07, the gateway ran in-process with the real drivers through the device test above, one phone at a time under its lock, using only the home screen, the Calculator and Settings. Last runs: the Mi at 04:06, the iPhone at 04:09; both passed every part.

| | Mi 10 Pro (Android 17, LineageOS) | iPhone 13 mini (iOS 27.0.1) |
| --- | --- | --- |
| Connect (first screen) | 2.2 s | 2.1 s (15 s once, while OCR loaded in a new process) |
| Calculator 6 × 7 = 42, tap by tap | Yes, by ref | Yes: digits by ref; ×, = by x and y (they have no position, see below) |
| The same five taps as one `phone_batch` | 21.6 s, 4.3 s a step | 11.9 s, 2.4 s a step |
| Settings: open, sub-page, back, scroll, home | Yes (Display) | Yes (Bluetooth) |
| A Settings switch asks, Deny, unchanged | Yes: Adaptive brightness asked under "Settings changes" after 2.7 s; still on afterwards (also `settings get system screen_brightness_mode` = 1 before and after) | Yes: Airplane Mode asked after 2.0 s; unchanged |
| Refs on two reads of one screen | 32 of 32 kept | 26 of 26 kept |
| Take over, then `user_took_over`, then `paused`; hand back | Yes | Yes |
| Stop while an action reads the screen | `stopped`, no input reached the phone | Same |
| History | 22 steps, 41 screenshot references, all readable JPEG | 22 steps, 39 |
| Lock state (`LockReporter`) | Unlocked, 55 to 70 ms | Unlocked, 17 to 32 ms |
| Read (`Observe`), median | 2.24 s (`uiautomator dump`) | 1.40 s (walk and OCR); 0.7 s in the Calculator |
| Settle (`Settler`), median | 1.25 s, never hit the 3 s limit | 1.19 s; hit the limit 4 times in 19 (launches, scrolls, the home screen) |
| Screenshot, median | 0.35 s | 0.34 s |
| Action from call to result, median | 6.1 s (read 2.2, settle 1.3, screenshot 0.35, read 2.2) | 6.0 s |

What the runs changed in the gateway:

- **Third-party launchers count as system apps.** The Mi's launcher is Lawnchair (`app.lawnchair.play`), which the app table did not know; agent scopes and task app lists would have treated its home screen as an app outside the list. Lawnchair, Niagara, Smart Launcher, Microsoft Launcher, POCO, Hyperion, Kvaesitso and some vendor launchers are added.
- **History screenshots no longer cost a screen read.** The check that no blocked app was captured used one more `Observe` per screenshot, 2.2 s a step on the Mi. History and approval-panel screenshots now come right before the step's own read, which checks them; only screenshots an agent receives keep the extra read. (Later the same night history screenshots left the agent's wait altogether; see "What the agent waits for".)
- **`phone_batch` steps start from the previous step's read** when it settled less than a second before, which saves one read a step (Mi: 4.3 s instead of 6.1 s a step). Separate tool calls always read again, since the agent's thinking time leaves room for the screen to change.
- **The read limit is 60 s** (was 20 s): the iPhone's first read took 15 s while OCR loaded.
- **A ref that is on the screen without a position gets one more read** before `not_found`, because OCR placement on the iPhone varies from read to read.
- Switch approvals read "Change “Adaptive brightness”?" instead of "Tap …".
- Defaults stay: `SettleMax` 3 s fits both phones' settle times; `PollInterval` only matters for drivers without a `Settler` and for `phone_wait_for`.

A note for whoever runs the device test: one Mi run reported "the switch changed after Deny". It was the test's own check comparing a different element with the same label; the switch had not changed (the setting read 1 before and after, and the guard saw no tap on it). The check now compares the same element.

## What the agent waits for

After the Android driver moved to the companion app (a read takes 70 ms instead of 2.2 s), the gateway's own share of each action became visible: an action waited for a read, a screenshot for the step's "before" picture whenever the screen had changed, the input, settle, a screenshot for the "after" picture, and a read. Now no screenshot is on that path:

- **The result returns once the read after the action is done.** The history screenshot is taken in the background after that read. The next action reuses it as its "before" picture when the screen still matches: same app, size, lock and keyboard, and the same tree elements by role, label and state outside the status bar (refs, positions and OCR lines do not count, since OCR placement varies between reads). Nothing takes a screenshot before an action any more. Other screens shown to the agent (`phone_connect`, `phone_observe`, `phone_wait_for`, a hand-back) get a background screenshot the same way, and a screenshot the agent asked for is reused too. Code: `internal/gateway/shots.go`.
- **`phone_launch` skips the read before it** when the policy allows the app (that depends on the app and the task, not on the screen), the driver reports locks (`LockReporter`, so `d.locked` is current), and no step's picture waits for a read (see below). A launch the policy refuses reads nothing either. A launch that asks still reads first, for the approval panel.
- So an action now waits for: a read (none for such a launch, or for a `phone_batch` step that starts from the previous step's read), the input, settle, and a read. The approval panel still takes its screenshot at once, since the person needs it; so does `phone_observe { screenshot: true }`, which also reads again to check it.

**The blocked-app guarantee.** A background screenshot is bracketed by two reads, like the screenshots agents get. It is kept only when the read just before it and a later read, started after the screenshot returned, show the same app, which is not blocked, on an unlocked phone, with no input from the agent or the person in between. Input means any agent action, input on the mirror, a touch on the phone, Take over, an unlock from the computer, or the phone going offline. The later read is the one the next call makes anyway (an action reads before it acts); when the task ends first, the gateway reads once more in the background, except after Stop, when it leaves the phone alone. Otherwise the screenshot is dropped, including after 10 minutes without a read. The app is checked against the rules again when the screenshot is kept, and a kept screenshot is written to history only when a step shows it.

Why two reads: the old after-screenshot came before the read that checked it, so an app that opened during the screenshot was caught. A screenshot taken after a read, with no read after it, would miss that, and payment flows do hand over to a banking app after a static "Redirecting…" screen that can pass for settled. `TestShotOfBlockedAppNeverStored` covers both cases; `shots_test.go` also covers the drops below, the reuse, the task-end read and the launch reads.

**What history gives up:**

- In `phone_batch`, the next step's input comes before any read, so the steps in between have no pictures; the first step's "before" and the last step's "after" remain.
- A call that comes before the previous screenshot returned (0.3 to 0.7 s) drops it: that step has no "after" and the next no "before". Agents think for longer; the device test, paced at 0.4 s, lost a few (table below).
- An action whose screen changed by itself since the agent's last result has no "before" picture, and neither does a launch at the start of a task.
- A locked screen gets no screenshot (notification previews on it may come from blocked apps).

**Measured** with the device test, the same harness for both builds (before: 33299a0; after: this change; drivers from main 1564080), one phone at a time under its lock (40 to 50 s a run on the Mi, about 2 minutes on the iPhone):

| | Mi 10 Pro before | after | iPhone 13 mini before | after |
| --- | --- | --- | --- | --- |
| Action, median of 14 | 1.80 s | **1.44 s** | 5.39 s | **4.97 s** |
| Tap, median of 6 | 1.81 s | **1.44 s** | 3.17 s | **2.86 s** |
| Key (home, back), median of 4 | 1.60 s | **0.94 s** | 7.54 s | **6.45 s** |
| Swipe (2) | 2.09, 2.37 s | **1.60, 1.61 s** | 7.23, 6.57 s | **6.41, 5.85 s** |
| Launch (Calculator, Settings) | 2.77, 1.64 s | **1.55, 1.49 s** | 4.95, 7.01 s | **3.07, 4.74 s** |
| `phone_batch` of 5 taps | 8.74 s, 1.75 s a step | **6.99 s, 1.40 s a step** | 12.03 s, 2.41 s a step | **10.83 s, 2.17 s a step** |
| Settle, median | 1.25 s | 1.25 s | 1.03 s (3 of 19 hit 3 s) | 1.43 s (4 of 19 hit 3 s) |
| Read, median | 0.07 s | 0.07 s | 1.38 s | 1.35 s |
| Screenshots taken (median) | 22 (0.36 s), all in the wait | 18 (0.36 s), none in the wait | 35 (0.34 s), all in the wait | 19 (0.36 s), none in the wait |
| Settled actions with a before / after picture | 19 / 19 of 19 | 13 / 14 of 19 | 19 / 19 of 19 | 13 / 14 of 19 |

The parent session's figures from the desktop app on the uiautomator driver (`phone_launch` 10.8 s, `phone_tap` 8.4 s) also included a second read per screenshot, which the previous round had already removed. On the Mi, settle is now most of an action (1.25 s of a 1.44 s tap); the rest is the two 70 ms reads and the input. On the iPhone, two reads of about 1.4 s and settle (1 to 1.4 s, and the full 3 s on the home screen and after scrolls) make up most of an action. Before, nearly every iPhone action also took two screenshots: the old reuse compared the whole screen text, and the read after an action (with OCR lines) never matched the read before the next one (without them). Settle varied between the two runs (median 1.03 s with 3 timeouts before, 1.43 s with 4 after), which hides part of the gain in the medians; the whole test took 123.6 s instead of 134.7 s. The first launch of a task skips its read: the Calculator launch went from 4.95 s to 3.07 s on the iPhone and from 2.77 s to 1.55 s on the Mi.

Of the 11 pictures missing in each "after" run, 8 belong to the middle steps of the `phone_batch`. The other 3 are most likely the test's 0.4 s pacing (the next call read the screen before the previous screenshot had returned) or a screen that changed between calls; the run does not log why a screenshot was dropped. `TestBeforeReusesTheAfterPicture` checks that steps with a pause between them keep both pictures.

## Driver issues seen from the gateway

Found while running the gateway on the phones; not changed here, since the driver sessions own that code.

iPhone 13 mini (iOS driver):

- **OCR placement changes between two reads of the same still screen.** The Settings "Battery" row had a position in one read and none in the next (twice in three runs); the Calculator's "7" key lost its position once "6×7" showed on the display (the text "7" was on screen twice). An agent then gets `not_found` for a ref it has just seen. The gateway now reads once more before answering `not_found`, which helps when the placement flips back; a steadier placement (keep a row's position while the screen does not scroll, or skip ambiguous matches more carefully) would fix it at the source.
- **Positions can be wrong, not only missing.** Airplane Mode was placed at y 627 in one read and at y 1511 in the next, with no input in between (Settings had just opened). A tap on a misplaced ref lands on whatever is really there; the gateway checks every element under the finger against the policy, but an element without a position cannot be checked. This matters most of all the iOS findings.
- **A swipe that did not scroll**: in one run a swipe up through the middle of Settings (562,1948 to 562,488 in 300 ms) settled at the 3 s limit and the next two reads still showed the top of the list; the same swipe on the Mi scrolls Settings to its end. In other runs the list did move. After a scroll the list also kept changing for seconds after the settle signal (43 elements, then 38, rows losing their positions).
- **Keys that show a symbol never get a position**: the Calculator's Multiply, Divide, Add, Subtract, Equals, Clear, Delete, Per cent and Point have labels that OCR cannot match to their glyphs (×, ÷, +, −, =, C, ⌫, %, .), and digits 1, 3 and 0 were missed in some reads. Agents have to tap by x and y from a screenshot. A small table of symbol labels in the OCR matcher would place most of them.
- The Calculator's display reads `other "StandardInputView;value:6×7"`: the identifier and value joined, instead of the value as the label.
- OCR turns the status bar into lines such as `ocr "3:42イ"` and `ocr "三の"` (clock and signal icons) on every screen; dropping OCR lines inside the status bar would save agents the noise.
- go-ios logs `ERROR no connection dispatcher registered for global channel module=go-ios/dtx_codec` on stderr when the device opens.
- The first read in a new process took 15 s once (OCR loading); later processes read in 0.7 to 2 s. The gateway's read limit is now 60 s.
- Settling hits the 3 s limit after most launches and scrolls and on the home screen (widgets), so those actions take 6 to 7 s.

Mi 10 Pro (Android driver):

- Each read is a `uiautomator dump` of 2.2 s, and an action needs two (before and after), so 4.4 s of the median 6.1 s per action is reading. The companion app's tree (about 30 ms) is not wired into the driver yet; that is the biggest speed-up available.
- No toasts or notifications reach `Screen.Events` without the companion (known).

## Settings API (for the client)

Everything Settings › Safety shows maps onto the policy engine (`g.Policy()`) and two gateway helpers. Each setter saves its file at once (`rules.yaml` or `apps.yaml` in the config directory, mode 0600, written atomically), on top of any hand edit of it (see 38 below); agents cannot reach any of it.

| Client setting (`ui.Settings`) | Read | Change |
| --- | --- | --- |
| `Preset int` (1, 2, 3) | `g.Policy().Preset()` returns `policy.Preset` | `g.Policy().SetPreset(policy.Preset(n)) error` |
| `HideCodes bool` | `g.Policy().HideCodes()` | `g.Policy().SetHideCodes(on) error` |
| `ReadOnly bool` | `g.Policy().ReadOnly("")` | `g.SetReadOnly("", on) error` (a device id instead of `""` sets one phone until restart) |
| `BlockedApps []BlockedApp{Name, Kind}` | `g.BlockedApps(ctx) []policy.BlockedApp` | `g.SetAppBlocked(screen.App{ID: id, Name: name}, blocked) error` |

- `policy.Preset`: `PresetCommits` (1: payments, messages, deletions and other commits ask, and so do calls, installs, links that leave the app and settings changes; the default until 2026-11-07), `PresetPayments` (2, the default: only payments ask), `PresetEvery` (3: every tap, type, swipe, key and launch asks), `PresetNone` (0: only the rules in the file). Blocked apps and the credentials rule apply under every preset. With read-only on and preset 3, looking around still asks (it is not denied), and commits are denied.
- `HideCodes` masks one-time codes on the screen and in events, and covers them in screenshots ([One-time codes](#one-time-codes)).
- `policy.BlockedApp{ID, Name, Category, Rule, ByPerson}`: `g.BlockedApps` returns the apps the person blocked, plus the apps on the connected phones that a rule blocks (banking, wallets, password managers), sorted by name. For the client's `Kind`, use `policy.CategoryLabel(b.Category)` ("Finance", "Passwords", "Blocked by you"); for its name, `b.Name`, or `b.ID` when the name is unknown (an app blocked by package name that no phone has).
- `SetAppBlocked(app, true)` adds the app to the `blocked` category in `apps.yaml` (rule "Blocked by you"); the client's free-text field can pass `screen.App{Name: text}`, or `{ID: text}` when the text looks like a package or bundle id. `SetAppBlocked(app, false)` removes it, and for a bundled banking or password app adds it to `exclude`. It returns `policy.ErrBlockedByRule` when a rule in `rules.yaml` names the app.
- `rules.yaml` gets `preset: 2` on first run. A file an earlier version wrote on its first run and nobody changed since (byte for byte one of `internal/policy/olddefaults/`, which set preset 1 or spelled out its rules) is replaced with today's defaults at start, so it asks before payments only too; any file Settings or a hand edit changed keeps its preset (`upgrade.go`). A rules file from before presets that spells out the six old default Ask rules, changed since, becomes preset 1 without them; any other old file keeps exactly its rules (preset 0).

## Integration (for the main session)

- **Real drivers.** Append factories to `providerFactories` in `cmd/pinky/providers.go`; `realProviders` skips one that fails to start. `--fake` uses only `demo.World()`.
- **Desktop app.** Run `gateway.New` in the app process and use it as `gateway.API`. Methods the client needs that are not in `API` (I did not change the interface, since the client session may have its own implementation of it): `SetReadOnly(deviceID, on)` (empty id = every phone), `SetTaskApps`, `MediaPath(ctx, mediaID)` for `StepView.Before/After`, `WipeHistory`, `Agents`, `SetAgentScope`, `Policy()` (rules, `Reload`), `PersonTouched(deviceID)`, `Flush`, `Refresh`.
- **Needs.** `NeedApproval` covers Ask actions, `phone_request_approval` and an agent's first connection (rule `New agent`). New: `NeedHandoff` (`phone_handoff`). `NeedUnlock` ids are `unlock:<device id>` and show only while an agent task waits on a locked phone. `Decide` on a handoff: allow = hand back with the note, Deny = decline, TakeOver = take over (the handoff stays open until `HandBack`). `Decide(…, TakeOver)` on an action approval calls `TakeOver`. `AllowTask` grants the rules that asked, in that app, or later file copies, for the rest of the task, where `NeedView.AllowTask` says so; `AllowAlways` keeps an exemption where `NeedView.AlwaysAllow` says so, and otherwise counts as `AllowTask`. On the agent's own questions both count as `AllowOnce` (a yes to that question only), so the history records `allow_once`.
- **Unlock.** `Options.PIN` hands out a phone's saved PIN (cmd/pinky: from the keychain, after Touch ID, only while Settings › Unlock from this computer is on); the gateway enters it through `driver.Unlocker`. Without `PIN`, or for a driver without `Unlocker`, `Unlock` returns `ErrNoUnlock`; a wrong PIN returns `ErrWrongPIN`. A successful `Unlock` also answers the handoffs the agent opened while the phone was locked, when unlocking is all their reason asks for (`unlockOnly` in `wake.go`, `NeedView.EndsOnUnlock`); a handoff that asks for more stays open until the person hands back.
- **Round three extras** (methods on `*Gateway`, not in `API`): `Recording(ctx, taskID) (RecordingView, error)`, `SetRecording(on)`, `SetRetention(days)`, `Retention()`, `ExportHistory(ctx, taskIDs, dir) (htmlPath, error)`, `SetBackground(deviceID, on) error`, `Background(deviceID) bool`, `AgentStream(ctx, deviceID, opt)`, `FilesDir(deviceID)`; `recording.Open(ctx, path, from)` for the player. `Stream` viewers share one stream per phone, so the mirror may open several.
- **In control without a task.** `DeviceInControl` can now show with no task ("You're in control") after the agent left; the device card needs a Hand back button in that state.
- **Touches on the phone itself.** `driver.go` has no way for a driver to report them, so I added `Gateway.PersonTouched(deviceID)`. Either wire the Android driver's getevent signal to it, or add an optional driver interface (for example `TouchWatcher`) that the gateway subscribes to.
- **Socket clients.** A socket client (such as a desktop app attached to a running daemon, which the app does not do: it takes the daemon over) cannot approve over the socket: `decide`, `hand_back`, `unlock` and `wipe_history` need user presence (Touch ID or Windows Hello), which is not built. Only a gateway with fake phones accepts them.
- **Tuning for real phones.** `Options.PollInterval` (250 ms) and `SettleMax` (3 s) drive settling when a driver has no `Settler`; each poll is a full `Observe`. History screenshots are taken in the background and never delay an agent (see "What the agent waits for"); `NoScreenshots` or `--no-screenshots` turns them off. `Flush` waits for screenshots in progress as well as history writes. A driver whose `Observe` takes a screenshot anyway (the iOS driver's OCR placement does) could hand that image to the gateway through an optional interface and save the second screenshot; none exists yet.

## Decisions I made, and why

Nobody was available to ask, so these are mine; each is easy to change.

1. **The MCP server runs inside the gateway, and `pinky mcp` only copies bytes** between stdio and the socket. One connection is then exactly one MCP session, so session close, cancellation and progress work without extra plumbing, and the tools exist once.
2. **Socket protocol.** The first line is a hello that picks the role (`mcp` with the agent token, or `control`); control is line-delimited JSON-RPC 2.0. MCP connections must present the token from `agent-token`.
3. **The socket cannot approve.** `decide`, `hand_back`, `unlock` and `wipe_history` are refused unless the gateway runs only fake phones. Any process of the same user can open the socket, and DESIGN's threat model says approval methods accept only the in-process UI or a client that proves user presence. `stop`, `stop_all` and `take_over` are allowed: they only take power away from agents. So are `virtual_phones`, `start_virtual` and `stop_virtual` (`VirtualParams`: platform and the id `virtual_phones` gave), which list, start and stop virtual phones as Add phone does, for scripts and e2e tests: any process of the person's can start a Simulator or an emulator on this computer anyway.
4. **First connection approval** is a regular approval card (rule `New agent`). Allowing grants `Options.DefaultScope` (everything), unless the person picked a scope on the card: the client calls `SetAgentScope` (which also approves the agent) right before `Decide`, and the gateway keeps that scope when the agent picks up the answer. Since `SetAgentScope` approves by itself, call it only on Allow. Approved agents are remembered by name in the `agents` table. A `--fake` gateway trusts agents without asking.
5. **Text-only tool results**, no `structuredContent` or output schema. Some clients show only structured content when it exists, which would hide the screen, and sending both doubles the tokens. The first line (`status · key: value …`) is easy to parse.
6. **Two extra error codes**: `invalid_argument` (bad arguments, a misused `approval_id`, no phone connected) and `internal` (cancelled calls, gateway failures). DESIGN's five did not cover these.
7. **`phone_connect` takes an optional `apps` list**, the task's allow-list. DESIGN has the list (M1) but not where it comes from. An agent can only narrow itself with it, which protects against prompt injection later in the task; the client can change it with `SetTaskApps`.
8. **`user_took_over` vs `paused`**: the first call after a takeover returns `user_took_over`, later ones `paused`, until the person hands back. Person control belongs to the phone, not the task: releasing, renaming the task, idling out or another session connecting does not end it, and `phone_connect` then returns `paused`. Only Hand back ends it. Input on the mirror takes over only while an agent task runs; the Take over button always does.
9. **Kept decisions.** A decision made while no call waits is kept for the timeout again (60 s) from the decision, not only until the original deadline, so an answer at second 59 still reaches the agent. A later call with an expired id opens a new approval instead of failing.
10. **`intent` is not part of the approval binding**: agents rephrase it on retries, and it does not change what the action does.
11. **Handoffs are approvals of kind `handoff`** (new `NeedHandoff` in `api.go`) with a 15-minute timeout. A handoff opened while the phone is locked ends by itself only when the person unlocks from the computer (`Unlock`) and unlocking is all its reason asks for (every clause of the reason is about the lock, a courtesy, or what the agent will do next); after an unlock on the phone, or when the reason asks for more, the person hands back explicitly, since they may be doing what the agent asked.
12. **Stop** ends the lease and bars that session from the phone, so a looping agent cannot reconnect; **Stop all** bars the sessions it stopped from every phone. A new session (the agent restarted) may connect. Take over and Stop cancel the driver call in progress, and every action checks the lease, person control and open approvals again right before the driver acts.
13. **Task states**: an agent that leaves or idles without `phone_release` shows as done, not failed; the exact outcome (`disconnected`, `expired`, `offline`, `replaced`, `stopped`, …) is in history.
14. **Blocked apps**: the agent sees `app: blocked app` with no elements and no screenshot (one the person allowed for the task shows as usual, with `ocr` lines read from a picture kept in memory only, and still without screenshots); their notifications are dropped, and toasts and dialogs read while one is in front are discarded; their steps carry no screenshots or screen text. A screenshot an agent receives is followed by a read, and dropped when the foreground app changed or is blocked; a history screenshot is kept only when a read before it and a read after it show the same app, with no input in between (see "What the agent waits for"). `home` and launching another app still work so the agent can leave.
15. **Read-only mode** also refuses typing outside search fields, as DESIGN's definition implies; it comes from `rules.yaml` (`read_only`, `read_only_devices`) and `SetReadOnly`.
16. **The policy checks every element under the finger** and the strictest verdict wins, so a price label inside a Pay button still counts as Pay.
17. **Allow for this task** grants every rule that asked, in that app; another rule that also asks (or the same rule in another app) still asks. It is not offered for the agent's own questions (`NeedView.AllowTask`): answering later questions by itself would let an agent claim consent the person never gave. **Always allow** does that and keeps an exemption in `rules.yaml` (`policy.Exemption`, `internal/gateway/always.go`; see DESIGN.md › Always allow).
18. **Typing without `ref`** is refused (rule `credentials`) when no field reports focus and the screen has a password or code field. On a virtual phone with `policy.File.VirtualCredentials` on (Settings › Safety › Allow passwords on virtual phones), this, the credentials rule and the hidden-keypad check let the action through; `prepare` reads `driver.Info.Virtual` into `policy.Input.Virtual`, and phones are refused whatever the switch says. Android tells an emulator by its adb serial, `emulator-` and a console port only (`emulatorSerial`), on no USB transport: a phone connected by an address is `host:port`, so one at a host named `emulator-…` is not virtual, and neither is a phone on a cable whose serial reads like an emulator's.
19. **Policy extensions**: a `change_setting` kind (switches and Remove/Reset/Forget… buttons in Settings) for DESIGN's "change account or security settings"; categories beyond finance, messaging and stores (passwords, social, browsers, phone, settings, system) because the defaults need them; the app table matches display names too (iOS backends may not know bundle ids) and `com.bank.*` prefixes; `apps.yaml` adds to and excludes from the bundled table, so table updates still arrive.
20. **Code masking** applies to notifications that mention a code (whole words such as "code", "PIN", "OTP", and CJK keywords) and masks runs of 4–8 digits.
21. **A line break typed into a chat or social app counts as sending** (`send_message`), since many keyboards send on Enter.
22. **Credential fields** are recognized by the password state, or by labels and resource ids (password words; one-time-code words, minus promo, postal and invite codes). A label counts with a code word anywhere in it; an identifier, which agents do not see, only by whole words (split at punctuation, digits and camelCase) or a code word joined to one that says what code it is (`smscode`, `verifycode`, `otpinput`), so `footprintTextView` or `barcodeNotes` is no code field. Password words count anywhere in both. Before 2026-11-10 identifiers matched code words anywhere, and Apple Notes' body on iOS 27.2 was refused (GitHub issue #47).
23. **Screenshots** are JPEG (quality 75, scaled down by a whole factor to at most 1280 px), as the task allowed; DESIGN says WebP, which needs a non-stdlib encoder.
24. **Wipe** deletes sessions, tasks, steps, approvals and media, and keeps agents and devices, which are settings.
25. **Swipe direction** is where the finger moves (`up` shows content below), stated in the tool description.
26. **`phone_type` with `ref`** must target a text field; it taps it first to focus. Typing into a button could otherwise tap a commit without the commit check.
27. **Several phones online and no `device`**: `phone_connect` returns `invalid_argument` with the list, rather than picking one.
28. **Presets are rules generated from a number** in `rules.yaml` (`preset: 2`, the default), added before the file's own rules; the file stays the one source of truth, as DESIGN asks, and keeps working when edited by hand. Preset 2 means what the client's text says: only payments ask, so sends, deletions, installs, calls and settings changes go through. Preset 3 asks before every tap, type, swipe, key and launch; with read-only on, looking around still asks rather than being denied.
29. **"Hide codes" is the notification code mask** (it does not blur screenshots); **blocked apps** the person adds go to a `blocked` category in `apps.yaml` with a built-in deny rule, and unblocking a bundled banking app excludes it by id and by name.
30. **Recordings hold packets until a read vouches for them** (see "Recordings"), rather than trusting the last read: reads are the only way the gateway knows what is on the screen, and a payment can hand over to a bank between two reads. The cost is a few seconds of video around each pause and, on Android without key frame requests, up to scrcpy's key frame interval after one.
31. **`phone_files` copies on request and needs a lease.** Copying to the phone asks, as DESIGN says ("writing to the phone is Ask"), under its own rule `Files`; a lease keeps every call in a task's history and under the agent's approval and scope.
32. **The gateway's own defaults are off**: no recording and no retention limit until the client (or the daemon, from the app's settings) says otherwise, so a gateway in tests or in another program never records or deletes by surprise.
33. **Background displays fall back to the phone's screen for the rest of the lease** whenever the display cannot be opened, read or used for an app, and say why in the next result, rather than failing the call: the person turned background on to be left alone, but the task matters more.
34. **One HTTP endpoint for all agents with the one agent token**, as the socket has; agents are told apart by their client name, as over stdio.
35. **Export copies the recordings next to the report** rather than inlining them (they are large) or leaving them out.
36. **Icon buttons** (`internal/policy/icons.go`): a tap on an element with no words of its own is a commit when what the app calls it is one: the resource id on Android (uiautomator's fallback label is its last part, `btn_send`), the identifier on iOS (`sendButton`), or an SF Symbol name (`paperplane.fill`, `trash`). A label with words still decides alone. Ids name nouns as often as actions, so fewer words count than for labels (`post`, `book`, `order` only next to `btn`, `button`, `fab`, `action` or `menu`), and ids with a state or place word (`status`, `time`, `methods`, `list`, `carousel`) never count. It only adds asks.
37. **Rules editor and Always allow** (`policy/editor.go`, `policy/exempt.go`, `gateway/rules.go`, `gateway/always.go`): Settings › Safety edits the file's rules through `SetFileRules` (checked as `ParseFile` does, plus names no other rule has); Always allow keeps exemptions under `always_allow`. A step that runs without asking because of one gets a second line, `Always allowed: “Payments” in Dinely` (`exemptedLine`, from `Verdict.Exempted` through `prep.exempted`, only on the paths that ran without asking and only when it settled or may have); recording marks and the export's report keep a step's first line apart from the rest. See DESIGN.md › Detecting a commit and › Always allow.
38. **Hand edits of the rules files** (`policy/disk.go`): the engine remembers the version of `rules.yaml` and `apps.yaml` it last read or wrote (the file, its size and time, and a hash). `Engine.Refresh`, which `gateway.rulesLoop` runs every `Options.RulesEvery` (2 s) and `Rules`, `Exemptions` and `FileProblems` run first, reads a file that changed once two looks 50 ms apart agree, and puts it in force when it parses; every setter does the same before it changes anything, so a change from Settings lands on top of a hand edit. A file that cannot be used is a `*policy.FileError`: the rules in force stay, setters for that file return the error and write nothing, `CanExempt` is false, and `Engine.Problems` (`Gateway.FileProblems`) lists it. `SetFileRules(base, rules)` returns `ErrRulesChanged` unless the file's rules are still `base`. A removed file is written again with the bundled one. Setters that change nothing no longer rewrite the file. `Engine.Version` counts the changes of the files (hand edits taken in, problems found and fixed, Pinky's own writes); `cmd/pinky`'s `policySettings` reads the client's settings again only when it moved. On Windows, a save that renames a new file over `rules.yaml` can be refused ("Access is denied") while Pinky looks at the file: Windows refuses a rename over a file that anything holds open, whatever it shares (on Windows Server 2025, 30 of 30 renames each over a handle open to read with every share mode, over one with no access, and over the open `os.SameFile` makes), and even `os.Stat` holds it open for a few microseconds (22 of 100 renames refused while one goroutine did nothing but look). So a share mode such as `FILE_SHARE_DELETE` cannot help; at one look every 2 s it comes to about one save in a few hundred thousand, and VS Code, Notepad and Notepad++ save in place on Windows, which a look does not hold up. `TestHandEditedRules` looks every 5 ms, and its saves were refused one in about 40 (5 of 200 runs on Windows CI, 2026-11-08); its `handEdit` tries again on Windows, as a person would. Pinky's own saves (Settings, and the defaults' upgrade at start) cannot be held up by its own looks, which run under the same lock, but a virus scanner, a backup or the person's editor may hold the file open for a moment, so `writeFile` tries the rename again on Windows while it is refused as busy (`replace`: access denied, a sharing or lock violation; 20 tries 25 ms apart), rather than have Settings say it could not save. `TestSaveWhileRulesAreOpen` holds rules.yaml open for 150 ms during a save from Settings; on Windows it fails without the retry.

## Differences from DESIGN

- Not built here: the remote API over WebSocket, encryption, Always allow beyond what is above (M2), transcoding iPhone recordings for the export.
- Retention is one limit for everything (Settings › Keep history, 30 days by default), not 30 days for video and 180 for steps.
- `phone_files` copies on request (`list`, `pull`, `push`) rather than keeping the folders in sync.
- One agent token for all agents, rather than one per agent; agents are told apart by their MCP client name, and the `agents.token_hash` column stays empty.
- Takeover by touches on the phone itself needs a driver signal (see Integration).
- Screenshots are JPEG, not WebP.

## Known issues

- Commit detection is word-based. It catches the common buttons in 11 languages, but a commit with an unusual label (an icon without a label, a slider in a language without a table) is missed, and some navigation buttons (Checkout, Confirm in a wizard) ask needlessly. `rules.yaml` can add label rules.
- The status-bar exclusion for the stale check is a heuristic (systemui ids, `statusbar` in the path, elements in the top 4.5 % of the screen). Screens with a ticking clock or countdown in the app will always come back `stale`.
- Settling by polling costs one `Observe` every 250 ms on drivers without `Settler`. Each `Observe` has a 60 s limit. A screenshot an agent asks for is followed by one more `Observe` to make sure no blocked app came to the front; history screenshots wait for the next read instead, and the approval panel's screenshot is checked only by the read just before it.
- History pictures are missing for the steps in the middle of a `phone_batch`, after a call that follows the previous result within about half a second, and for screens that changed by themselves between calls (see "What the agent waits for").
- Take over and Stop cancel the driver call in progress through its context; a driver that ignores the context finishes the call anyway (the result then says it may have happened).
- DESIGN wants a pending status for a `phone_type` that would block longer than 45 s (long text on iPhone). Typing runs to completion here; the iOS driver's typing speed decides whether this matters.
- Windows: builds, but untested; the single-gateway lock there only checks whether the socket answers.
- A `--fake` gateway accepts approvals from any local process; that is intended (fake phones only) but worth knowing.
- `pinky needs` and `pinky decide` are development aids and not documented in `pinky help`.
- Recordings: a blocked app that comes and goes between two reads, neither of which sees it, is not caught (see "Recordings"). After a pause, writing resumes at the next key frame: at once on streams that take `RequestKeyFrame`, within scrcpy's key frame interval on Android otherwise.
- Background tasks have no history screenshots and no approval-panel picture (the Android display's `Screenshot` is unsupported).

## Review

An independent review of the pipeline, approvals and socket (after the first commit) found these, all fixed in the second round with regression tests in `internal/gateway/regression_test.go`:

- After Take over, an agent could get control back by connecting with another task name, or after release, idling out or a new session. Person control now belongs to the phone until Hand back.
- An action that had passed the checks still ran if the person took over or pressed Stop while its screen was being read. Every action now checks again right before the driver acts, and Take over cancels the call in progress.
- Tapping a child element inside a Pay button (a price label) skipped the payment check. The policy now checks every element under the finger.
- A blocked app's toasts and dialogs could reach the agent after it left the app, or through `phone_wait_for`; a screenshot could catch a blocked app that came to the front between the read and the shot. Both are closed.
- Approvals opened at the moment of a Stop or Take over could stay open with no owner and block the phone. They are now withdrawn.
- "Allow for this task" on one rule skipped other rules that also asked. Grants now need every asking rule, per app.
- Smaller: history steps of one approval used by two concurrent calls, a data race on the agent approval flag and on the device's provider, approvals opened after `Close`, the daemon exiting on a transient `Accept` error, typing with unknown focus near a password field, stale checks ignoring element states.

### Round three review

An independent review of the round three code (read-only, by a subagent) found these, all fixed with tests:

- A mirror viewer that gave up waiting for a stream to open left its goroutine running (`TestStreamGivenUpLeavesNothing`).
- Recording: a drop from outside (a lock, a takeover) could be undone by a read that had started before it; a clean read that finished first could write what an earlier, still running read was about to drop; a hub's replay of its last key frame to a recorder joining again could bring back packets from before a drop; packets after the task's end were taken until the recorder stopped; and its last read read whatever screen the device showed next (`TestRecordingOverlappingReads`, `TestRecordingEndsWithTheTask`).
- The recorder kept the device it started with, so after a reconnect it never opened the stream again (`TestRecordingAfterReconnect`); a driver read that returned no screen and no error stopped the recording for good.
- Turning background on moved a running task to an empty display, and watch-only agents got one too (`TestBackgroundFromTheNextTask`); an app refusing the display took the phone's screen from a person using it (`TestBackgroundFallbackWaitsForThePerson`).
- A remote agent could take a local agent's approval and scope by its name (now " (remote)"); Unlock could type the PIN into an unlocked phone (`TestUnlockOfAnUnlockedPhone`); a folder link could make empty folders outside the local folder (`TestPhoneFilesFolderLink`); turning remote agents off waited for event streams before ending sessions.
- Not fixed here: no driver's stream has `RequestKeyFrame` yet, so after a drop a recording resumes at the next key frame (scrcpy: every few seconds; an iPhone with a still screen may send none), and a mirror viewer that falls behind on an iPhone no longer asks the phone for a key frame, which the iOS driver did for its own subscribers.

## Shared contracts

- `internal/gateway/api.go`: added `NeedHandoff NeedKind = "handoff"`. Nothing else changed.
- 2026-11-08: `api.go` gained `DeviceNeedsSetup DeviceStatus = "needs_setup"` (see "Phones that wait for a step"). Additive: a client that does not know it shows it as it shows an unknown status. The MCP results gained the status `device_needs_setup` and the `setup:` detail (`StatusNeedsSetup`, `Result.Setup`), both additive; no status or error code changed. `driver.go` gained `SetupPasscode` (iOS's passcode before automation).
- `internal/screen`, `internal/driver/driver.go`, `internal/driver/fake`: unchanged.
- 2026-11-08: `screen.Element` gained `Masked bool` (additive): the policy masked a one-time code in the element's label, and pictures of the screen cover its bounds. It is not part of `Line`, so agents never see it.
- `go.mod` gains `github.com/modelcontextprotocol/go-sdk v1.8.0`, `modernc.org/sqlite v1.61.1`, `go.yaml.in/yaml/v3 v3.0.6`, `golang.org/x/sys`, and in round three `github.com/Eyevinn/mp4ff v0.62.0`; other branches add their own requires, so expect a `go.mod` merge and a `go mod tidy`.
- 2026-11-09: `driver.App` gained `CloneOf *App` with `App.Original`, `driver.CloneID` and `driver.ParseCloneID` (additive), for Android's app clones ("App clones" below).
- 2026-11-09 (pinching, all additive): `driver.Pinch`, `driver.Pincher` and `driver.MultiToucher` (`internal/driver/pinch.go`); `gateway.Action.Scale` and `Rotation`; `NeedView.Point2`/`To2` and `approval.Request.Point2`/`To2` (the second finger); `Features.Pinch`; `Gateway.Touch2`; the MCP tool `phone_pinch` (15 tools) and the batch step `pinch`; `ui/fakeapi`'s `Input.Kind` "touch2" with `Point2`. The fake and demo phones pinch.
- 2026-11-09: `driver.go` gained the optional `LinkOpener` interface (`LinkApp(ctx, link)`, `OpenLink(ctx, link, app)`) and `*LinkChoiceError` (additive), for opening links ("Links" below). The gateway's own `linkOpener` (`LinkApp() driver.App`, iOS's fixed Safari) is gone, and with it `ios.Device.LinkApp()`'s old signature.
- 2026-11-09 (device switches, all additive): `driver.Switcher`, `driver.LocaleLauncher` and `driver.Locale` (`internal/driver/switches.go`); `gateway.Action.Language` and `Region`; `Result.Switches` with `SwitchView`; `Session.Device` with `DeviceRequest`; the rule `Phone settings`; the MCP tool `phone_device` (18 tools) and `language`/`region` on `phone_launch` and batch steps. The design is in [device-switches.md](device-switches.md).
- 2026-11-10 (dragging, all additive): `driver.Drag`, `driver.DragStop`, `driver.DragStep` and `driver.Dragger` (`internal/driver/drag.go`); `gateway.Action.HoldMS`, `Via` and `PauseMS`; `gateway.Waypoint`; `NeedView.Path` and `approval.Request.Path` (every point of a drag). `NeedView` holds a slice now, so it is no longer comparable with `==`; nothing in the tree compared it except `ui/gwtext_test.go`, which keyed a map by it. The MCP tool `phone_drag` (19 tools) and the batch step `drag`. The fake and demo phones drag and record an input of Kind "drag".
- 2026-11-10 (GitHub issue #71, additive): `screen.Screen` gained `KeyboardFrame Rect` and `Covered []Rect`, where touches go to the keyboard or a system bar instead of the app; empty when the driver cannot tell. Not part of `Header` or `Render`. See "Gestures clear of the keyboard" below.
- 2026-11-11 (GitHub issue #78, pausing a phone, all additive): `driver.Pauser` and `driver.CaptureReporter`; `DeviceView.Paused` and `Streaming`; the MCP error `device_paused`; history schema 5 (`paused_devices`). See "Pausing a phone".
- Round three: `Options.Unlock` became `Options.PIN` (nothing outside the gateway set it); `Result` gained `Folder` and `Files`; streams may implement an optional `RequestKeyFrame()` (gateway-side interface, not in `driver.go`).

## Tests

| Package | Covers |
| --- | --- |
| `internal/policy` | commit words in 11 languages and non-commits, default rules for every kind, blocked apps by id, name and prefix, read-only, task and agent scopes, rules file parsing and strictest-wins, files created 0600, `apps.yaml` overrides, code masking, notification filtering |
| `internal/approval` | decide while waiting, pending then a kept decision, timeout as Deny, expiry, binding mismatch, single use, withdraw, cancelled waits |
| `internal/history` | round trip of every table, search, paths with spaces, JPEG downscaling, reopen, wipe |
| `internal/gateway` | approval allow (with history and media), deny, timeout, pending then repeat (busy phone, mismatched and reused ids), stale (screen text and target), take over and handoff, take over during an approval, stop and stop all (cancelling a slow action), lease conflict, session close, MCP-style cancellation, read-only, blocked app, credential fields, send message, batch, offline and reconnect, idle expiry, new-agent approval, locked phone and unlock from the computer, wait_for and diffs, `Settler`, watch-only scope, task app list, unclaimed approvals, events, wipe. Regression tests from a review: takeover kept across tasks and sessions, take over and Stop while an action is in flight, no orphan approvals, blocked-app events and screenshots, a price label inside a Pay button, per-app task grants, one approval id used twice, agent approval race, typing with unknown focus. Background screenshots (`shots_test.go`): the result does not wait, "before" reuses "after", a blocked app is never stored (also when it opens during the screenshot), drops on the next input and on the person's input, the read at task end (not after Stop), and when a launch reads first |
| `internal/mcp` | over the SDK's in-memory transport: the 20 tools and their schemas, a full flow with approval and progress notifications, pending and errors, session close releasing the lease, batch, list reading and screenshots, handoff. Over HTTP (`http_test.go`): no token or a wrong one is 401, a session works like stdio and its close frees the phone, `CloseSessions`, the first-connection approval |
| `internal/recording` | H.264 and HEVC from ffmpeg test patterns (`recordingtest`, stored as base64 since the repository ignores raw streams) recorded and checked with `Probe` and ffprobe, no file before a key frame, a file cut short still opens, playback from the start and from the middle (paced, pre-roll at once), Close, changed parameter sets, a live ffmpeg stream with the Mi's YCgCo tag |
| `internal/api` | single gateway per user, socket and token modes, MCP over the socket with the token, control methods and the trust rule |
| `internal/gateway`, round three | `streams_test.go` (one stream for every viewer, late joiners from the key frame, the stream closes after the last viewer and ends for all, a slow viewer skips to a key frame), `video_test.go` (only what reads vouch for is recorded: a blocked app, a takeover and a Stop drop what is held; no recording when off; packets from before the task), `files_test.go`, `background_test.go` (reads and actions on the display, the phone's lock and the person's touches, refusal, unreadable display, open failure, the setting kept across restarts, waking), `retention_test.go` (retention with screenshots and recordings, a running task kept; export of HTML, JSON and the recording), unlock with the PIN and a wrong PIN in `gateway_test.go`; doubles in `extras_test.go` |
| `internal/gateway` (tag `device`) | the real Mi 10 Pro and iPhone, see "Real phones"; with `PINKY_RECORD=1` the run is recorded and checked with ffprobe; `emulator_test.go` (`PINKY_EMULATOR`): the shared stream and the recording around a blocked app, and background displays |
| `cmd/pinky` | end to end with real processes: `daemon --fake`, a refused second daemon, `devices`, two `pinky mcp` agents (approval answered through `needs`/`decide`, credentials, `lease_held`, iPhone `recents` unsupported, release, history), stdin close releasing leases, SIGTERM; `pinky mcp --fake` starting the daemon itself; `daemon --fake --http` with `pinky token`. Settings reaching the gateway (preset, codes, read-only, blocked apps, retention, remote agents on and off) and the PIN source (off, saved, nothing saved without a prompt, cancelled) |

## Field test follow-ups: blocked apps for a task, read-only tasks (2026-11-08)

- **Allow a blocked app for the task** (`unblock.go`, field test #18). A `phone_launch` the policy refuses only because a rule blocks the whole app becomes an approval (rule `Finance apps` or `Blocked by you`, `NeedView.Unblock`, `NeedView.App` the app to open) when the app is a finance app or one the person blocked; password managers stay refused. Any Allow adds the app to `lease.unblocked`; from then on `prepare` passes `policy.Input.AllowBlocked` for it (the engine skips app-level deny rules, never for password managers), `agentView` shows its screen and `observe` keeps its toasts and dialogs. Screenshots, recordings, history pictures (`recordableScreen`, `rawShot`) and history screen text still treat it as blocked. Deny adds it to `lease.kept`: later launches in the task are refused at once. A new task starts blocked again.
- **Read-only tasks** (`ConnectRequest.ReadOnly`, `phone_connect read_only`, #19): the lease and task are read-only whatever the agent's scope (`policy.Input.ReadOnly`), never widened by connecting again; `TaskView.ReadOnly`; an agent's first connection that asks for it opens its card with `NeedView.ReadOnly` and "read_only access" in the detail.


## Removing a phone

`Gateway.RemoveDevice(ctx, id, RemoveOptions{History})` (`remove.go`) is what the client's **Remove this phone…** calls (`phoneRemover` in `ui/removephone.go`). The person removes a phone from Pinky, not from the computer:

- **What goes.** The phone's task ends as with Stop (stopped, and the session may not use the phone again); approvals about the phone that no lease holds, such as an agent's first connection to it, are withdrawn; its streams stop and the driver's device closes (in the background, since an Android phone kept dark can take 20 s). The phone leaves `Devices` and agents can no longer find it. Its settings go: the background choice (`g.background` and `device_settings`), its read-only switch (`policy.Engine.ForgetDevice`: the device card's switch and its entry in `read_only_devices` in `rules.yaml`, which is saved), how Pinky connects to it (`device_connections`, and the provider's `driver.ConnectionChooser` set back to `ConnectionAuto`, which on an iPhone also stops a WebDriverAgent helper waiting for the passcode; not `Gateway.SetConnection`, which would take the phone offline and discover it again) and its row in `devices` (`history.Store.ForgetDevice`). The client deletes its PIN (`pin:<id>`) from the keychain, which only the client writes.
- **History.** Kept by default. With `History`, `history.Store.DeleteDeviceHistory` deletes the phone's tasks that started before the removal, with their steps, approvals and recordings, the screenshots no other step shows, and the ended sessions that used the phone and have no task left, after `Flush`; the recent tasks (`g.ended`) drop them too. The copy of the phone's Pinky folder (`FilesDir`) stays: those are files, not history. When the phone went but its history did not, the error wraps `ErrHistoryNotDeleted`.
- **Staying away.** The gateway remembers removed phones (`g.removed`, the `removed_devices` table, history schema 4, read before the first discovery). While a phone is removed, discovery skips it when a provider lists it with `Connection` "wifi": a phone the providers reach over the network by themselves, such as an Android phone the adb server reconnects. A phone listed any other way (a cable, an emulator) is added back at once, as a new phone: it is plugged in, so it is the person's doing. A pairing (`StartPairing`'s `Wait`, `PairWithCode`) or `ConnectWiFi` adds back the phone it returns. A phone removed while the gateway opens it does not come online, and the new session closes (`device.removed`).
- **Drivers** (`driver.Forgetter`, an optional provider interface): `Forget` at the removal and, at start, for each removed phone, so the provider stops connecting to it by itself; `Remember` when it is added back; `Disconnect` at the removal, which drops the phone's connections over the network that the provider can drop. Android (`forget.go`): the Wi-Fi reconnector skips the paired phones whose guid (`adb-<serial>-<suffix>`) names a removed phone, and `Disconnect` runs `adb disconnect` on the phone's connections by address. A connection the adb server's own discovery made (`adb-<serial>-<suffix>._adb-tls-connect._tcp`) cannot be dropped by name: platform-tools 37 reads the argument as an address, adds `:5555` and answers no such device. It stays, and the server would make it again anyway; the gateway keeps the phone off its list.
- **Why not edit `~/.android/adb_known_hosts.pb`.** The adb server, and every tool that uses it (Android Studio, scrcpy), share that file: the person removed the phone from Pinky, not from the computer. Writing adb's private protocol buffer would race the server's own writes during a pairing and break when its format changes, and it would neither drop the connection that is open nor change the phone's trust of this computer. So Pinky keeps its own list, and the phone stays paired; to stop the computer reaching it at all, forget the computer under Wireless debugging on the phone.
- **iPhones.** Pinky does not unpair an iPhone (its lockdown pairing record, the phone's trust): Finder, Xcode and Photos use that pairing, and the phone would ask to trust the computer again. Removing one is offered without that choice. iPhones show up over USB (`ios.Provider.Wireless` is off), so a removed iPhone stays away until it is plugged in again.

The per-iPhone connection choice landed first (history schema 3, `device_connections`); `removed_devices` is schema 4. Its parts of removing a phone were agreed with that session: the row goes with `ForgetDevice`, the provider's choice is reset as above, and the client drops the choice `ui/connection.go` caches for 15 seconds (`a.conn`), so a phone added back at once reads it again.

Tests: `internal/history/devices_test.go` (forgetting a phone keeps its history; the removed list across a restart; deleting one phone's history, with a screenshot another phone's step shows, a recording's file, the sessions, and a task started after the removal kept), `internal/policy/forget_test.go`, `internal/gateway/remove_test.go` (a phone on Wi-Fi stays away across discovery passes and a restart, and is back once paired by code or QR or connected by address; one on a cable comes back at once, and after being removed unplugged, when plugged in; with history; removed while opening; its connection choice dropped without reconnecting it), `internal/driver/android/forget_test.go` (which connections `Disconnect` drops, the reconnector leaving a removed phone alone), `ui/removephone_test.go`. On the Android 16 emulator `pinky-api36` (`emulator-5560`), connected by address to an adb server of the test's own: `go test -tags device -run TestRemoveEmulatorOverWiFi ./cmd/pinky` (the real adb server, driver and gateway: removed, disconnected, kept away when the connection comes back and after a restart, back after `ConnectWiFi`) and `go test -tags e2e -run TestRemovePhoneOnEmulator ./cmd/pinky` (the app, removing it in Settings › Phones), both with `PINKY_E2E_EMULATOR=emulator-5560` under `scripts/devlock.sh emulator-5560`; `TestRemovePhoneInTheApp` does the same with the fake phones. All passed on 2026-11-08.

## Pausing a phone

GitHub issue #78. `Gateway.PauseDevice(id)` and `ResumeDevice(id)` (`pause.go`) are what the client's **Pause connection** and **Resume** call (`phonePauser` in `ui/pausephone.go`; the phone's ⋯ menu, its right-click menus, and the paused phone's page). Pausing stops Pinky using the phone without removing it:

- **What stops.** The phone's task ends with the outcome `paused` (shown as stopped), approvals about it are withdrawn, its streams stop and the driver's device closes: on an iPhone the tunnel and the WebDriverAgent helper (a helper waiting for the passcode too), on Android scrcpy and the companion app. The phone shows as offline with the status "Paused" (`DeviceView.Paused`), and its record, pairing and settings stay.
- **Staying paused.** Drivers implement the optional `driver.Pauser`: Android's Wi-Fi reconnector skips the phone (`forgot`), and the iOS provider lists it from what it knew without connecting. The gateway opens no paused phone, and a paused phone opened meanwhile closes at once. The state is saved (`paused_devices`, history schema 5) and read before the first discovery, so it survives restarts.
- **Agents.** A call on a paused phone returns the error `device_paused` (503 for uploads); `phone_connect` lists the phone as paused by the person, and picks no paused phone by itself.
- **Resuming.** By hand from the client, by pairing or connecting it by address (`unremove`), or by plugging it in. Only a new plug-in counts: a cable listing after the phone was seen away from its cable (unlisted, or listed over Wi-Fi). A phone paused while on its cable stays paused until it is unplugged and plugged in again, so pausing a phone that charges from the computer does not undo itself. After a restart, the first listing only notes where the phone is. Removing a paused phone clears the state.
- **Streaming.** `DeviceView.Streaming` says whether the phone sends its screen to Pinky now (`pollCapture`, with the discovery loop): the device's `driver.CaptureReporter` when it has one (iOS stops capturing 30 s after the last view or input; Android while its video runs), else whether a stream hub is open. The phone's header shows "Streaming".

Tests: `internal/gateway/pause_test.go` (pause and resume, across a restart; paused on its cable; paused on Wi-Fi then plugged in; the only phone and phone_connect).

## Phones that wait for a step (2026-11-08)

A phone the gateway has not opened used to be `DeviceOffline` with "Offline · <Detail>", also when its driver listed it as waiting for a step. So the app showed iPhones waiting for Trust, Developer Mode or the developer disk image, and Android phones waiting for Allow USB debugging or a USB driver, as offline, with the driver's English and Reconnect; only the fake API (`ui/fakeapi`) showed them as needing setup.

- **Needs setup.** `statusLocked` gives a phone that is not open but listed as `unauthorized` or `needs_setup` the status `DeviceNeedsSetup`, line "Needs setup" (`waitsForStep`), also while a task waits for it to reconnect: the person has a step to do before the task can go on. `Info` keeps the state, `Setup` and `Detail` for the client. The fake API gives the same.
- **Unplugged.** `discover` only marked online phones as gone, so a phone unplugged while it waited kept claiming its step in `Devices` and in its checkup. Now a phone its provider stops listing that is not open, and not being opened, is listed offline without `Setup` or `Detail` (`unlisted`, with an `EventDevices` so the client reads it again); one listed offline with a reason loses the reason the same way. A phone that goes away while open is unchanged: `setOffline` closes it and `Devices` shows it offline. A provider whose `List` fails changes nothing, as before.
- **Agents** (second round, same day). `phone_connect` leases such a phone instead of returning `device_offline`, and returns the new status `device_needs_setup` without a screen: `Result.Setup` (`setup:` on the result's first line) names the step (`Info.Setup`, or `authorize` for an Android phone that waits for Allow USB debugging, as `ConnectionChecks` names it), the message has the driver's `Detail` and what to do (`setupResultLocked`: a handoff for most steps; wait and observe for `preparing`; end the task for `update`; connect again after `usb_driver`, as the phone then shows up under its adb serial). Without `device`, with no phone online, the one that waits is chosen; the phone list in errors says "connected, needs setup". `gate` answers every other call on the lease the same way ("Nothing was done."), and `readError` too when the phone needs a step again in the middle of a call. `phone_handoff` passes the gate for such a phone, and its handoff is kept in `lease.readyHandoffs`: `open` answers it once the phone is open (`readyNow`, note "The phone is ready."), and the agent gets the whole screen as after a hand-back; a hand-back before that returns `device_needs_setup` with the person's note. `open` also starts the task's recording, which `firstScreen` could not. `expireOffline` keeps a lease while its phone waits for a step; once the phone leaves that state (unplugged, or listed offline) the 60 s grace starts then.
- **Phones that never came online** leave the list: a phone the gateway never opened (`device.opened`) is forgotten once its provider has not listed it for `unopenedGrace` (2 minutes), unless a lease holds it (`discover`, log "device forgotten", an `EventDevices`). The grace lets an iPhone that restarts for Developer Mode come back as itself rather than as a new row. Examples: an iPhone unplugged before it trusted the computer, and the `usbdriver:` entry Windows gets for a phone without a USB driver, which goes once the driver is installed and the phone shows up under its adb serial. A phone that was open once stays, offline, as before: the person knows it. An Android emulator is the exception (`emulator`): it is listed by its adb serial, `emulator-<port>`, so its next run on another port is another phone, and a day of emulators started and stopped by scripts or tests left a hundred offline rows. Once its provider has not listed it for `emulatorGrace` (10 minutes) and no lease holds it, it is forgotten, unless there is something to keep for it (`keepEmulatorLocked`): Pinky started its AVD in this run (`StartVirtual`, `Gateway.started`), so it runs again on the same port as the same phone, or the person gave it a setting of its own (a background display choice, a connection, read-only). Its saved PIN is the client's and stays either way. It is forgotten like a removed phone without its history (`forgetSettings`: its record and settings, read-only switch, connection choice and the approvals no task holds), but it is not added to the removed phones, so the same serial comes back as a new phone. An iOS Simulator keeps its UDID from run to run and stays.
- **`pinky devices`** without a gateway gives such a phone `needs_setup` in `--json` (was `offline`); through the gateway, the STATUS column no longer repeats the detail the gateway's "Offline · <detail>" line already has (`listedView`, `statusColumn` in `cmd/pinky/devices.go`).

Tests: `internal/gateway/needsetup_test.go` (waiting to be allowed, another step and its checkup, unplugged with its event and checkup, listed offline with a reason and then unplugged, online and open, a step after it was open; an agent connecting, its calls, a lease past the grace, a handoff that ends once the phone is ready; a hand-back before the step; the driver hearing an agent works once the phone opens (`driver.WorkWatcher`); forgetting a phone never opened, one with a task on it, and keeping one that was open; forgetting an emulator that was open once it is gone, `TestEmulatorForgotten`, and keeping one with a setting or whose AVD Pinky started, `TestEmulatorKept`), `internal/mcp` `TestStatusLine`, `cmd/pinky` `TestDevicesStatus`, `ui/needsetup_test.go` (the client on the real gateway: the pages, sidebar, tiles and flyout of an Android phone and an iPhone in English and Chinese, See what's wrong, Show the steps, the next step, unplugged; an agent's task card and handoff card on a phone that waits; the iPhone's passcode step).

## Phones over Wi-Fi (2026-11-08)

iPhones and iPads over Wi-Fi are on by default (`SetWiFi`, from Settings › Phones and the daemon's settings.json); the iOS side and the measurements are in `docs/notes/ios.md`, "Wi-Fi".

- **Whose phone.** Providers that reach phones over Wi-Fi by themselves implement `driver.WiFiFinder` (additive). The gateway tells them which phones are the person's (`UseOverWiFi`): every id in the history store's `devices` table at start (`DeviceIDs`, without removed phones), every phone listed on a cable (`knowLocked`, which also saves it there), and every phone the person chose to use (`UseFoundPhone`). Any other phone such a provider reaches only over Wi-Fi it lists with `driver.SetupConfirmWiFi`: status `DeviceNeedsSetup`, line "Found on Wi-Fi", hidden from agents (`findDeviceLocked`, `deviceListLocked`). `IgnoreFoundPhone` is `RemoveDevice` without history. A found phone its provider stops listing leaves the list (`discover`), as it was never the person's; one they chose to use stays, offline.
- **Moves.** `device.over` is the connection a session was opened over. When the provider lists an open phone over another one, `updateDevice` closes the session (`setOffline`, "moving to Wi-Fi" or "moving to the cable") and opens the phone again in the same pass; `open` waits up to `closeWait` (6 s) for the last session's `Close`, so the new one does not start its stream while the old one stops. `device.moving` makes `gate` wait (20 s, `moveWait`) instead of answering `device_offline`; `SetConnection` sets it too. A session over a cable that drops (`ErrOffline` from the driver) of a phone the person uses over Wi-Fi waits `dropWait` (6 s) for the next discovery, which extends it once the phone is listed again; `phone_observe` reads again after such a drop. A phone that dropped and is back, listed while Pinky opens it (Connecting), makes calls wait for that open too (`connectWait`, 20 s from when it began; `backBy`), and a failed open wakes them; before, they got `device_offline` although the app showed the phone connecting (`TestOfflineAndReconnect` on Windows CI, 2026-11-08; `TestCallsWaitWhileConnecting`).
- **Locked on Wi-Fi.** The iOS provider lists a phone that is locked on Wi-Fi as waiting for a step, `needs_setup` with `driver.SetupUnlock`, and does not open it (opening a sleeping iPad lit its screen). Without a lease `statusLocked` shows it at rest, `DeviceOffline` with the line "Locked on Wi-Fi"; with one it needs setup like any phone that waits, and agents hand it to the person (`device_needs_setup`, `setup: unlock`). `discover`'s two-minute forgetting of phones never opened skips the person's phones (`known`), such as one that stayed locked on Wi-Fi all along; a found phone that goes leaves at once.
- **Passes take turns (2026-11-09).** Discovery passes start from the loop, `discoverSoon` (`SetWiFi`, `UseFoundPhone`), pairings, `SetConnection` and the tailnet, and used to overlap. One that listed a found iPad just before Use could apply its listing after the pass that Use started had opened the iPad: `updateDevice` took it offline ("Found on Wi-Fi.") and the next pass opened it again. `TestWiFiFoundPhone` failed that way once on Windows CI (opens `[wifi wifi]`), and 5 times in 200 on a Mac with a random 0–30 ms delay in the fake provider's List. A listing older than a move between the cable and Wi-Fi would undo the move the same way. Now `discover` holds `listMu` while it lists and applies, so passes apply listings in the order they took them; the opens run outside it. `TestWiFiOlderListing` holds a listing across Use (it failed every time without the lock; with it, 400 runs of `TestWiFiFoundPhone` with the delay passed).

Tests: `internal/gateway/wifi_test.go` (found, Use and remembered after a restart, Ignore and back on a cable, moves both ways with a call waiting, Wi-Fi turned off, a found phone that goes away, a pull in the middle of a read, a locked phone). Shared contracts: `driver.WiFiFinder`, `driver.SetupConfirmWiFi` (additive); `api.go` unchanged.

Later the same day:
- **Connecting.** `statusLocked` gives a phone being opened (`d.opening`, listed online) `DeviceConnecting` (additive in `api.go`) with "Connecting", or "Moving to Wi-Fi" / "Moving to the cable" while it moves (`connectingLine`); `Info.State` still reads offline.
- **Lost phones.** A phone listed `needs_setup` with `driver.SetupDeviceService` reads "Lost on Wi-Fi". One listed with `driver.SetupMediaInUse` (2026-11-11) reads "Camera or microphone in use", and `setupResultLocked` adds to its message that the person may be on a call, to ask with `phone_handoff`. `RestartDeviceService(ctx, deviceID, reason)` calls the phone's provider (`driver.DeviceServiceRestarter`), then discovers. A phone listed without a name gets the one the history store saved (`SeenDevices`, `names`), else the one it had.
- **Frames for OCR.** `observe` offers reads with OCR `driver.ObserveOptions.Frame`, a picture from the live video through `frameShot` (the screen's hub, shared with the mirror); drivers whose screenshots are slow use it.
- **OCR on Android (2026-11-08).** Reads ask for OCR in one of two ways (`readOCR`, `observeOCR`): `ocrAuto` (`observe(…, true)`: the reads that show the agent the screen), where the Android driver reads text only over web views and canvases its tree is sparse in, and `ocrAll` (`ObserveOptions.OCRAll`) for `phone_observe { ocr: true }`, `phone_wait_for`'s reads with OCR and `ocrForVerdict`. iPhones read the same either way. `ObserveOptions.MayCapture` (`mayCapture`) lets a driver capture a screen only when `recordableScreen` allows it or it shows a blocked app the person allowed for the task (the person's choice on 2026-11-08): never a locked phone or an app Pinky cannot name. It is set on every read, with OCR or without: an iPhone's inspector path takes a picture to place its elements either way, and the iPhone driver asks it before it takes any picture (docs/notes/ios.md). The picture stays in the driver's memory; `TestAllowedBlockedAppReadByOCR` checks that nothing of such an app reaches a result, the history's pictures or its screen text. `ocrForVerdict` now also reads again on phones whose driver says it reads text (`driver.TextReader`, `readsText`), not only on phones that returned OCR text before, so "Pay" drawn in an image button asks from the first tap. `phone_observe { ocr: true }` on a computer without OCR says so (`noOCR`).

## One-time codes (2026-11-08)

While Settings › Safety › Hide codes from agents is on (`mask_codes` in rules.yaml, on by default), one-time codes never reach an agent as digits.

- **Text.** `observeOCR` calls `policy.MaskScreen` on every read before anything else sees it, refs included, so the agent's screen and diffs, `phone_wait_for`, history's step texts and screen text, approval titles and the client all get masked labels; masked elements are marked `Masked`. `FilterEvents` masks events of every kind (it masked only notifications before). The rules are in `internal/policy/codes.go`:
  - A text carries a code when it talks about one (`codeWords`, `codePhrases`, `cjkCodeKeywords`; "promo code", "zip code", "error code" and the like, `notCodePhrases`, do not count) and holds a number shaped like one: 4 to 8 digits, or 3+3 or 4+4 joined by a space or hyphen. Every such number in the text is masked, so a sender's short number in the same message row is masked too.
  - Never a code: three digit groups or more (phones, cards, dates), a number after `+`, `(` or `)` (phones), after a currency sign or code or before a currency, unit or `元`/`年`/`%` (amounts, dates), with a decimal part, after a month or a day (`November 1, 2026`), or touching Latin, Greek or Cyrillic letters (`25A1234`). `MaskCodes` for notifications uses the same number shapes, so it no longer masks a phone number, a date or an amount in a code message.
  - Across elements: OCR lines that wrap one text (stacked closely, about as tall, overlapping sideways) count as one text, so `【某银行】您本次操作的验` / `证码为482913` is masked; and a number nearly on its own (`482913`, `G-482913`, not a year) is masked when an element right above, below or before it names a code (`codeNouns` or the CJK words, not only "Sign in" or "Security") and shows no number itself, unless a neighbour names a promo, postal or invite code (`otpNegatives`).
- **Pictures.** `codes.go` in the gateway covers masked elements' bounds (scaled to the picture, padded) with gray boxes:
  - `phone_observe { screenshot: true }`: the boxes of the read before and the read after the screenshot, plus what the picture itself shows: on a computer with OCR (Vision, Windows.Media.Ocr) it is read again (`readPicture`, 0.1 to 0.4 s) and code lines are covered by the same rules. That catches a code in an image or one that came between the reads. A masked element without bounds and no OCR here: no screenshot, and the message says "No screenshot: a one-time code is on the screen, and Pinky could not tell where to cover it." When boxes were drawn, the message says "One-time codes on the screen are covered in the screenshot."
  - History screenshots (`writeShotLocked`): the boxes of the reads the picture belongs to (`bgShot.reads`: the read it was taken for and the one that confirmed it), drawn in the recorder's goroutine. A picture with a masked element without bounds is not kept; the step has no picture. No OCR runs for history.
  - The approval card's picture is the person's and stays as it is; history keeps it covered (`keepImage(…, covered=false)`).
- **Recordings** are not covered: the recorder remuxes without re-encoding, and they stay on the computer; an agent gets one only when the person allows it on a card at an export ([below](#screenshots-and-reports-for-the-agent-2026-11-09)). Codes expire within minutes. The setting's line says so ("Recordings show them, so an agent gets one only when you allow it"), and so do both READMEs.
- **Not covered.** A code that comes and goes between two reads, neither of which sees it, can be in a history picture (agents' screenshots are read once more). An element whose label holds the code in a long text gets its whole bounds covered (a whole message row). ID and card numbers are not codes.

Tests: `internal/policy/codes_test.go` (codes in Chinese, English, Japanese, Korean, German, Spanish and Russian; prices, phones, dates, years, order numbers, promo and zip codes left alone; message rows, wrapped OCR banners, a code under its label or typed after it, containers and far numbers), `internal/gateway/codes_test.go` (the demo's Messages screen masked and covered, `phone_wait_for` does not find the code, a code only the picture shows, an unplaced code with and without OCR, history pictures, the approval card, and this Mac's Vision on drawn code messages). `demo.Device.SetOverlay` adds elements over any screen for such tests.

## Blocked apps on other screens (2026-11-09)

A user reported that the agent's screen capture still showed a bank app they had blocked (not reproduced; platform and version unknown). Checked against 2026.1108.22, a blocked app's own screen never reaches an agent, but these did:

- **Android's notification shade, lock screen and heads-up banners** list a blocked app's notification rows as elements, and the agent's screenshot shows them.
- **Recent apps** shows a blocked app's card (Android's task snapshot, an iPhone's app switcher): the element names the app, the screenshot shows its last screen, and OCR could read it.
- **Notification events** were matched by package only (`FilterEvents` passed the package as the name too), so an app blocked by its name in Settings, or an unknown bank found only by its name ("某某银行"), had its notifications listed.
- **Settings › Safety › Add app** took free text with no check, so a typo blocked nothing.

What changed:

- **Hidden parts** (`internal/gateway/hidden.go`). `observeOCR` calls `hideBlocked` right after `MaskScreen`, before refs, so everything downstream sees the result. It replaces each blocked app's part with one element, `other "Hidden by Pinky: a notification from a blocked app"` or `"… a blocked app's card"`, marked `Hidden` (new field of `screen.Element`, additive), with the part's bounds; every element mostly inside those bounds goes, OCR lines included.
  - *Notification rows*, on any screen (heads-up banners come over apps): the system UI's `expandableNotificationRow`, one merged element with the companion app and with uiautomator alike (`"Shell, •, now, Card statement ready, …"`); elements inside a row that did not merge are gathered by place (`gatherRows`). A row is a blocked app's when it starts with a blocked app's name (Android shows the name only while a row is expanded; uiautomator joins with " · "), when its `app_name_text` names one, or when it holds what a blocked app's notification says.
  - *What notifications say*: the phone's list (`driver.NotificationLister`, new and additive; Android reads `dumpsys notification --noredact` filtered on the phone, about 70 ms, at most every 2 s and only while rows show), and the texts of notification events from blocked apps the device saw in the last day (`hideState.seen`). Texts shorter than four letters (two for Chinese, Japanese, Korean) are not matched; longer ones by their first 40.
  - *Cards*, on system screens (system UI, launchers, SpringBoard, the lock screen): a labelled element of 2% of the screen or more whose label, or its first part, names a blocked app (a task snapshot, an app switcher card, a widget, an iPhone notification cell). An element of recent apps (`task_view`, `task_thumbnail`, `recent`, `overview`, `snapshot` or `switcher` in its id or path; a taskbar is not) that names a blocked app but is too small to be its card is hidden without a place. Cards inside a larger labelled element (an app switcher labelled as a whole) are looked at one by one.
  - *Names*: `blockedName` is `policy.Blocked` by name, plus the names and aliases of the phone's apps that are blocked by id (掌上生活 is blocked as `com.cmbchina.*`). The apps come from `driver.Device.Apps`, read once and then again in the background every 2 minutes; the blocked names are rebuilt when the policy's version changes.
- **Pictures**: the agent's screenshot is covered where the reads before and after it hid parts (`coverHidden`, before one-time codes); a hidden part without a place withholds it ("No screenshot: a blocked app's card is on the screen, and Pinky could not tell where to cover it."). The message says "Notifications and cards of blocked apps are covered in the screenshot." History pictures are covered the same way in `writeShotLocked`, and not kept when a part has no place. The approval card stays the person's. Recordings are not covered (they stay on the computer, as with codes); READMEs say so.
- **Events**: `policy.FilterEventsNamed` takes the device's id→name map from `hideBlocked`; `NotificationBlocked` matches by id and name, and an event without an app (an iPhone banner whose app the driver could not find) by the name its text starts with.
- **Add app** (`ui/blockapps.go`, `gateway.InstalledApps`): the field lists the connected phones' apps that match what is typed, without those blocked already; a click blocks the app by its id, as does typing its exact name or id. Text that matches no app at all gets a warning, and Block still blocks it by the name as typed.
- **Finance table**: Chinese banks' credit card apps whose names have no 银行 (买单吧, 发现精彩, 动卡空间, 浦大喜奔, 全民生活, 阳光惠生活, 兴业生活, 华彩生活, 建行生活, 工银e生活, 邮储信用卡; 掌上生活 was already in `com.cmbchina.*`), by Android package, iOS bundle id and name, and the iOS bundle ids of the big banks. Each Android package was checked in Xiaomi's store (`app.mi.com/details?id=`) and Tencent's (`sj.qq.com/appdetail/`), each bundle id with Apple's lookup (`itunes.apple.com/search?country=cn`). `com.spdb.mobilebank.per` was wrong: 浦发银行 is `cn.com.spdb.mobilebank.per`. "信用卡" in an app's name now marks a finance app, as 银行 does.

Checked on the Android 16 emulator (`pinky-api36`) through the real driver and gateway (`TestEmulatorHidesBlockedNotificationsAndCards`, tag `device`; the shell's notification and Settings stand in for a bank): the expanded row (named "Shell"), the same row collapsed (found by its text through `dumpsys`), and Settings' card in recent apps were hidden from the element list and covered in the screenshots; the other notification, the shade's buttons and Chrome's card stayed. The companion trees of those screens are in `internal/gateway/testdata/` for the unit tests.

Not done:

- **iPhone banners over an app** on iOS 17.4 to 26 (WebDriverAgent): that path reads only the app in front, not SpringBoard, so a blocked app's banner can reach the agent as OCR lines and in a screenshot taken while it shows. On iOS 27 they are hidden (below, 2026-11-09 follow-up).
- **iPhones' Notification Center and app switcher** are handled by the generic rules (SpringBoard elements whose label starts with a blocked app's name, large elements named after one), not checked on a phone: reading the owner's notifications is off limits, and the simulator's WebDriverAgent tree differs from iOS 27's accessibility path. If Notification Center comes as one SpringBoard overlay over the app, as banners do, its event's text could hold several apps' notifications and only the first is matched by name.
- **Recent apps on other launchers** (MIUI/HyperOS, One UI, ColorOS) were not seen. Where the card is not one large element named after the app, the name alone is hidden and screenshots are withheld only if the element's id or path says recents; a launcher that names its cards otherwise could show a blocked app's snapshot.
- **Widgets** of a blocked app on the home screen are hidden only when their element is labelled with the app's name.

### iPhone banners (2026-11-09, follow-up)

On iOS 27 (remoteAXService) the driver now lists each notification SpringBoard shows over the app as one element on every read while it shows, besides the event it gives once (`notificationElement` in `internal/driver/ios/springboard.go`): a button with the banner's whole text, the app's name first, at the overlay's place, with ID `screen.NotificationID(bundle id)` (`pinky:notification:<bundle id>`, or `pinky:notification` when the app is not found; new and additive in `internal/screen`). The bundle id is found once, when the banner comes (`postingApp`), and kept while it shows (`springBoardWatch.posted`). The gateway takes such an element for a notification row (`isRowID`): a blocked app's, by its bundle id (`hideHints.blockedApp`) or its name, is hidden and covered like an Android row, the OCR lines over it go with it, and one whose frames are another process's (no place) is hidden without one, which withholds the screenshot. Other apps' banners stay listed, with one-time codes masked in them and covered in screenshots like any element's. Tests: `TestOverlayEvents`, `TestNotificationElementUnknown` (iOS driver), `TestHideBannerByItsApp`, `TestIPhoneBannerHidden` (gateway, on the fake iPhone). Not run on an iPhone.

### Checking on a person's phone

`TestDeviceBlockedHidden` (`internal/gateway/device_hidden_test.go`, tag `device`) checks one screen of a real phone without reading anyone's content into its output. The person makes something only the app under test shows, with a marker text (`7731`), and sets the screen; the test blocks that app for the run only (in memory, nothing saved), connects a read-only task, reads the screen with a screenshot, and reports: the app in front (only when it is the system UI or a launcher), how many parts were hidden, whether the marker is in the element list, in events, or readable in the screenshot (this Mac's OCR, picture kept in memory only), and, for an element still showing the marker, its role, resource id, ancestry and place. PASS, FAIL or NOT CHECKED (nothing hidden and no marker: the app's notification or card was not on the screen). It sends no input. On the Android 16 emulator: PASS for the shade with the shell's notification and for recent apps with Settings' card; FAIL, with the row's id and place, when nothing is blocked.

```sh
go test -c -tags device -o /tmp/pinky-gw/gateway.test ./internal/gateway
scripts/devlock.sh <id> env PINKY_DEVICE=<id> PINKY_BLOCK=<app id> PINKY_MARKER=7731 [PINKY_WAIT=90] \
  /tmp/pinky-gw/gateway.test -test.run TestDeviceBlockedHidden -test.v
```

What still needs a phone, and the screen the person sets for each run:

| Phone | Screen | Marker made by | `PINKY_BLOCK` |
| --- | --- | --- | --- |
| iPhone, iOS 27 | A banner over Settings: Settings open when the reminder goes off (`PINKY_WAIT=90`, started before it) | A reminder "Pinky 7731" with an alert a minute or two ahead | `com.apple.reminders` |
| iPhone, iOS 27 | Notification Center pulled down, the reminder in it | The same reminder, after it went off | `com.apple.reminders` |
| iPhone, iOS 27 | The app switcher, the Calculator's card in view | `7731` typed into the Calculator | `com.apple.calculator` |
| Xiaomi, HyperOS or MIUI | The notification shade pulled down | `adb shell "cmd notification post -t 'Pinky 7731' pinky 'Pinky test 7731'"` (the shell posts it) | `com.android.shell` |
| Xiaomi, HyperOS or MIUI | The shade again, the shell's row collapsed (no app name) | the same | `com.android.shell` |
| Xiaomi, HyperOS or MIUI | Recent apps, the Calculator's card in view | `7731` typed into the Calculator | `com.miui.calculator` |

Afterwards: delete the reminder and the shell notification (swipe it away), clear the Calculator. A FAIL's output names the resource ids and places to make the rules fit that phone; the output has no screen text, but the session that runs it should not add any either, and should keep no screenshot.

## App clones (GitHub issue #18, 2026-11-09)

Android drivers list app clones (应用分身, Dual apps, App Twin) as apps of their own: `driver.App{ID: "com.tencent.mm@999", Name: "微信（分身）", CloneOf: &{com.tencent.mm 微信}}` (docs/notes/android.md, "App clones"). A clone reads like its original on the screen, so the gateway judges it as its original wherever the rules look (`driver.App.Original`): the launch's policy input, `leaves`, `phone_apps`' blocked and scope filters, the hidden-icon check, and the task's app list, which gets both the clone's and the original's id and name. `BlockedApps` and `InstalledApps` (Settings › Safety) skip clones. The policy engine reads a clone id as its package in `appIn`, `Categories`, exemptions and `SetAppBlocked` (`clonePackage`), in case one reaches it. The launch approval, the unblock card and the history name the clone by its name. Tests: `internal/gateway/clones_test.go`, `TestAppCloneIDs` in `internal/policy`.

## Screenshots and reports for the agent (2026-11-09)

`phone_observe { save: "<name>" }` and `phone_release { export: true }` write files into the phone's folder on this computer (`FilesDir`, the one `phone_files` uses) for an agent to put in a bug report or a pull request (`internal/gateway/save.go`). Saving asks nothing and is allowed to read-only and watch-only agents and in read-only mode: nothing on the phone changes, and the files show no more than the agent may see. The one exception is a task's screen recording, which shows more and so goes only with the person's yes (below). It works on iPhones too.

- **The screenshot** is `g.screenshot`'s picture, the one `screenshot: true` returns: refused while the phone is locked, a blocked app (allowed for the task or not) is in front or a code cannot be placed; codes and hidden blocked items covered. It is written at full resolution as PNG (JPEG for `.jpg`/`.jpeg`) through a temporary file and a rename, with the upload name rules (`uploadName`, `uploadTarget`, `localPath`); a file of the same name is replaced, since the screen moment it showed may not come back. The image block is returned only with `screenshot: true` as well, which keeps the agent's context small. The step "Saved a screenshot as “x”" goes to history.
- **The report** is written by `exportTo`, the code behind Activity's Export…, for the one task, after its screenshots and steps are written (`shots.wait`, `rec.flush`, not the recording's last read, which can take 15 s). Its screenshots are history's, which are covered like the agent's. The step "Saved the task's report for the agent" goes in before "Finished: …".
- **The recording only with a yes.** Recordings cover neither codes nor blocked apps' notifications and catch a blocked app only when a read sees it (video.go), so an MP4 shows what screenshots and screen text hide. When a released task has a recording (`taskVideo.view()`), `Release` asks first (`askRecordings`): an action card with rule `Recordings` (`recordingsRule`), "Give Claude Code the screen recording of “Reorder lunch”?" (or "… of 2 tasks?", one card for every phone released), and a detail that says what is at stake. Only Allow, Deny and Deny with a note: no Allow for this task (an export happens once a task, so asking each time costs little) and no Always allow, and the card is opened outside `policy`, so no preset, rule or Always allow answers it, as with uninstalling apps. Read-only and watch-only agents get the same card. It waits like any approval (45 s, then `approval_pending` with the id; the same release waits on the open card), the task keeps running, and `Busy` holds other actions on the phone. A takeover does not withdraw it (it is about files on this computer, not the phone), and the card offers no Take over: a `TakeOver` answer counts as Deny. A yes copies `base-<task id>.mp4` next to the report (`exportTo(…, videos)`, after `g.Flush()`, which waits for the recording's end) and the step reads "Saved the task's report and screen recording for the agent". A no or no answer is not an error: the task ends, the report is saved, and the message says the recording was not given and why, with the person's note in `note`. The lease ending meanwhile (Stop) returns `stopped` and saves nothing.
- **Why not B or C.** B was to write the MP4 without asking and change the promise to say recordings show codes: simplest for agents, but it hands codes and blocked apps' notifications to every agent that exports, and nothing in the moment tells the person. C was to keep the MP4 from agents altogether and leave it to the person's own Export… in Activity: safest, but an agent filing a bug report then cannot attach the video that shows the bug, and the person has to find the task and export it by hand. A card per export keeps the promise (recordings go to an agent only when the person allows it, that time) and still lets an agent attach the video in one step.
- **Not `phone_record`.** The task is already the unit of recording, its history and its report, and its files are final only once it ends; a start/stop tool would add a 15th tool and a second recorder state for what `phone_release` gives.
- **Download links** (`internal/mcp/download.go`): an agent over HTTP gets `curl -fsS -o '<name>' '<base>/download/<token>'` per file instead of paths. One use, 10 minutes, `rand.Text()` token, no agent token. Unlike upload links they are not tied to the MCP session, because clients that open a session per call, or close it right after `phone_release`, would lose them; `CloseSessions` (Remote agents off) drops them all. The file is opened only if it is still a regular file (no link swapped in). HTML is sent as `application/octet-stream`, attachment, `CSP: sandbox`, so a report never renders at Pinky's origin.
- Tests: `internal/gateway/save_test.go`, `internal/mcp/download_test.go`.

## Pinching (`phone_pinch`, 2026-11-09)

`pinch.go`. `phone_pinch` takes a centre (`ref`, x,y, or the middle of the screen), `scale` (0.25 to 4; 2 spreads the fingers to twice their distance) and/or `rotation` (-180 to 180 degrees, clockwise when positive), and `duration_ms` (200 to 2000, default 500).

- **Planning** (`pinchGesture`). The fingers keep inside the screen less margins of 7% of its width and 8% of its height (status bar, home indicator or navigation bar, the back gesture's edges) and, with a ref, inside the element. They lie on the line through the centre with the most room over the whole turn (tried every 5 degrees), use 90% of that room at their widest, and start or end at the widest by `scale`, so a pinch never asks for more room than there is. Under 160 px at the widest is `invalid_argument`, before anything happens.
- **Rules.** A pinch is `policy.OpSwipe`, judged with the elements under both fingers' touch-down points (`fingers()`), as a swipe is with its one. The approval's screenshot gets both fingers' arrows (`Point`, `To`, `Point2`, `To2`).
- **Phones that cannot pinch.** A device without `driver.Pincher` (an iPhone through Device Hub or without the helper) gets `unsupported` ("This phone cannot pinch. Nothing was done.") before the person is asked.
- **History and recordings.** "Zoomed in on “Map”", "Zoomed out", "Turned “Photo” with two fingers" (`describePinch`, localised in the client); the step's point is the pinch's centre and its target the element, so the recording's step marks and the mirror's overlay show it there. An approval asks the same in the imperative: "Zoom in on “Map”?" (see "Approval titles ask").
- **The person's pinch** (`Gateway.Touch2`) goes to a `driver.MultiToucher` and takes over like the person's `Touch` (`PersonTouched`).

Checked on emulator-5900 through a temporary device test (android.md, "Pinching"): `ref` with scale 2 and x,y with scale 0.5 and rotation 45 settled on the main display and on a background display, the app's content followed, and history read "Zoomed in on “Zoom area”"; a pinch at 20,20 gave `invalid_argument`.

## Dragging (`phone_drag`, 2026-11-10)

`drag.go`, for issue #44: iOS turns a home screen page only while the finger holding an app rests near the edge for about 0.6 s, which `phone_swipe` never does. `phone_drag` presses (`ref` or `from`), may hold there (`hold_ms`, 0 to 5000) to pick something up, moves through up to 8 `via` points, resting at each for its `pause_ms`, then to `to`, rests there (`pause_ms`, default 300) and lets go. Each move takes `duration_ms` (100 to 3000, default 500); the finger may stay down 30 s in all. All of it is checked before anything happens (`checkDrag`, and every point on the screen in `prepareDrag`).

- **Why a tool of its own** rather than options on `phone_swipe`: a swipe scrolls and flings, a drag moves something and stops before it lets go; a drag needs the hold and the rests, its own words in history and in the approval, and keeping them off `phone_swipe` keeps that tool simple. The issue's `edge_hold_ms` would turn a page but could not say "turn a page, then drop onto a folder"; `via` points can. A raw touch-down/move/up API was turned down because it leaves the finger down between calls. `hold_ms`, `via` and `pause_ms` on any other tool are `invalid_argument`.
- **The gesture** (`driver.Drag`): From, Hold, and Stops (point, Move, Pause). `Drag.Steps(step)` samples it for drivers that play touch events (ease-in-out moves, a step for each rest) and the waits add up to `Drag.Duration()`. A driver that can drag implements `driver.Dragger`; one that cannot gets `unsupported` ("This phone cannot drag. Nothing was done.") before the person is asked.
- **Rules.** A drag is `policy.OpSwipe`. Its fingers (`p.fingers()`) are the press point, every `via` point and the drop point, so the strictest verdict over the elements under all of them stands: a drag that rests on, or lets go over, Place order asks under Payments as a tap there would (`TestDragRulesEveryPoint`). The points are judged on the screen as it is before the drag: what appears only mid-drag (a folder opening, the next page) is judged by what was there before. The target is the element pressed.
- **Approval and history.** The card's title is "Drag “Calculator”?" ("Drag from 200,2000 to 540,1700?" when nothing labelled is under the finger) and its screenshot draws the whole route (`NeedView.Path`, `approval.Request.Path`: press, via points, drop) with a ring at each point. History reads "Dragged “Calculator”", "Dragged from 200,2000 to 540,1700" when nothing labelled is under the finger, or a bare "Dragged" when even the points are unknown; `internal/gwtext/gwtext.go` localises all three. The step's point is the press point.
- **Timings** measured on the Simulator (iOS 26.5) and the emulator (pinky-api36), which the tool's description and docs/SKILL.md give agents: on iOS `hold_ms` 600 to 1000 picks an app icon up and 300 does not; a 700 ms rest near the left or right edge turns one page and 1200 ms turned two. On Android's launcher `hold_ms` 800 picks an icon up and an 800 ms edge rest turns one page; a folder is made by letting go over the target icon where it is during the drag. Pixel Launcher shrinks the page (about 0.78 toward about 540,990 on the 1080x2400 emulator; the dock stays) once it picks an icon up and takes the page out of the element tree until the finger lifts, so the gateway follows the page (below); the rest at the drop does not matter (300 ms and 2.5 s both made a folder there). iOS stays in edit mode after a drag (Done, or home).

- **Following a home screen's page** (`dragfollow.go`, `dragmatch.go`). Agents give `to` from the screen they read, but Pixel Launcher's page is shrunk where they let go. When the drag starts on a launcher's screen (`policy.IsSystem`, not System UI) that `mayCapture` allows, with `hold_ms` 300 or more, on a driver that implements `driver.DragReplanner`, the gateway:
  1. adds a pickup stop before the first one: a tenth of the width toward it, kept off the edges, 120 ms and a 250 ms rest;
  2. takes a picture as the finger touches down, in a goroutine, while the hold runs;
  3. at the pickup stop (`DragReplan`'s `at` 0, the finger down and still), waits for that picture and takes another;
  4. runs `findDrop` and gives the driver the rest of the stops with the last one moved.
  Pictures come from screencap on a phone on a cable, and over Wi-Fi from a hub that is open already (`frameShotOpen`, `watchOpen`) or else screencap; a background display only from its open hub. Never from a stream it would open: opening one waits for the input the drag holds.
  `findDrop` matches on grayscale cells of W/360 screen pixels (and 4× coarser first). The marks are the tree's named elements from before the drag, without the status bar, the element pressed and anything over a sixth of the screen. Each mark is compared where it was; correlation 0.9 or more counts as still (the dock). When the mark nearest `to` stayed, or none moved, nothing changes. Otherwise the moved marks inside a window around `to` (0.3 W, growing to 0.9 W until it holds enough cells) are the template, and only their cells count, so the wallpaper does not. It is searched at scales 0.6 to 1 for a shrink toward a centre anywhere in 20–80 % across and 15–75 % down (masked normalized cross-correlation), then the best three places are refined at the fine level. It accepts a correlation of 0.6 or more that beats any place 1/12 of the width away by 0.05, and a scale of 0.97 or less that moves `to` 3 px or more.
  The result gets a sentence: where Pinky let go and why, or why it could not tell (it lets go at `to` then). History's drop point is where the finger let go. The log has the scale, the score and the timings (`drag: follow`). On the synthetic home screens in `dragmatch_test.go` the match takes 20 to 30 ms on an M-series Mac and lands within 2 px. A drag in an app, or one without a hold, is played as before.

Drivers: iPhone through Device Hub plays `Drag.Steps` at 10 ms as HID touch events (`gestureSteps`); with WebDriverAgent, and on Simulators, one W3C touch pointer with pauses for the hold and rests and moves in 40 ms steps (docs/notes/ios.md, "Dragging"); the accessibility-only `AXDevice` cannot. Android plays the steps through scrcpy's touch injection on the main or a background display (docs/notes/android.md, "Dragging"). The fake phone records a `drag` input with every point; the demo phones too.

Tests: `internal/driver/drag_test.go` (steps, timing, easing), `internal/driver/ios/drag_test.go` (WDA actions), `internal/gateway/drag_test.go` (arguments, history, approval path, rules at every point, batch, a phone that cannot), `internal/gateway/dragmatch_test.go` (`findDrop` on synthetic shrunk pages at 0.66 to 0.85 and at half size, the dock, still and unrelated pictures, the pickup stop, and `TestDragFollowsHomeScreen` through the gateway), `internal/mcp` `TestPhoneDrag`, `ui/approvalshot_test.go` `TestDragMarks`, `ui/gwtext_test.go`.

## Links (2026-11-09)

`phone_launch` takes a link as well as an app: `https://…`, an app's own link (`myapp://…`), `tel:`, `sms:`, `mailto:` and the like (`policy.IsLink`: a scheme with `//`, or one of the schemes the policy knows without it, so "Notes: today" stays an app name). The rule the work keeps: a link never gets past a rule a tap would meet.

**Before anything opens** (`prepare`, `resolveLink`):

1. A scheme Pinky never opens (`policy.LinkRefused`: `file`, `content`, `data`, `blob`, `intent`, `android-app`, `javascript`, `vbscript`, `itms-services`, Shortcuts' `shortcuts`, `shortcuts-production` and `workflow`) skips the driver and is denied by the engine under `Unsafe links` (`RuleUnsafeLinks`), whatever the preset, and also when the person allowed the app for the task.
2. Otherwise the driver names the app the phone would open the link in (`driver.LinkOpener.LinkApp`). Several apps with none the default (`*driver.LinkChoiceError`) is `unsupported`, naming them: Pinky does not choose, and the agent can launch the app it means and open the link from inside. No app is `not_found`. A driver without `LinkOpener` is `unsupported`.
3. On a phone with app clones, the link goes to the clone the agent opened last (`launch.clone`, set by a launch by name) while that clone's app is in front and it is the app the link opens in; otherwise to the app itself, as the phone would.

**Judging.** The link is judged as `OpLaunch` of that app (`Input.Launch` is the app's original, `LaunchURL` set, `Input.Link` the link), so blocked apps, finance apps, password managers, the agent's scope, the task's app list and the person's "allow for this task" all apply to it as to a launch. On top, `detect.go` adds the kinds of the tap the link stands for (`linkKinds`): `tel:`, `facetime:` and their kin are a call, `sms:`, `mailto:`, `imessage:` sending a message, payment schemes (`upi`, `alipay(s)`, `paypal`, `bitcoin` and other wallets, `weixin://` and `wechat://` with a pay path) a payment, under the rules a tap on Call, Send or Pay meets. `OpenURL` is always there, so each link asks under `Installs and links` where that rule asks. The Shortcuts app is refused as a link's opener (`refusedLinkApps`): it runs shortcuts under schemes a list cannot hold. Launching Shortcuts by name is unchanged.

**Approval and history.** The approval reads "Open “<link>” in <app>?" (`gwAskOpenLink`), the step "Opened “<link>” in <app>" (`gwDidOpenLink`), the link clipped to 120 characters (`maxLinkText`). The approval's snapshot keeps the app (`snapshot.opener`), so an approval for a link the phone would now open in another app is stale and asks again. `perform` calls `OpenLink(ctx, link, app)` with the judged app; the drivers open it there and nowhere else (Android starts that app's activity by name; iPhone gives the bundle id to Device Hub or WebDriverAgent).

**Drivers.** Android: docs/notes/android.md, "Links". iPhone: docs/notes/ios.md, "Links". The fake phone (`fake.Device.SetLinkApps(prefix, apps...)`: one app opens such links, several are a `LinkChoiceError`, none `ErrNotFound`; `OpenLink` records a `link` action) and the demo phones (Dinely opens `dinely://` and pages on dinely.app, Chatter `chatter://`; no app opens other links) implement `LinkOpener` too.

Tests: `internal/policy/links_test.go` (refused schemes, links as the taps they stand for in presets 1 and 2, scope and task apps), `internal/gateway/linkapp_test.go` (judged as the app that opens it, a blocked opener and the person allowing it for the task, a payment link asks, refused schemes, choice and not found, a phone without links, stale approvals, step text, clones), `internal/driver/fake/fake_test.go`; on the emulator `TestEmulatorLinks` (tag `device`, docs/notes/android.md has the command). `internal/gwtext/gwtext.go` translates the new texts and the rule.

## Waiting for one of several texts, or in part of the screen (`phone_wait_for`, 2026-11-10)

`phone_wait_for` took one text. An agent that cannot tell which of several things comes next (the order goes through or the payment fails; a round ends, a dialog asks, or the run fails) could not use it, and slept and observed instead. Three days of an agent playing a game through Pinky on an iPhone (12 sessions) show what that costs: 271 sleeps of 30 to 45 s, 131 minutes in all, and after 59% of them the next `phone_observe` found nothing to do; 20 scripts the agent wrote for itself that poll `phone_observe` and match several texts with a regular expression; and of its 46 `phone_wait_for` calls, 19 ran to the timeout. Twice a wait was answered within a second by the same word in a list that is always on the screen. Field test 24 on the iPhone was the Latin case of that, which whole words solved (`containsText`); Chinese has no spaces to tell by.

- **`any`** (`WaitForRequest.Any`) lists more texts beside `text`, up to `MaxWaitTexts` (10) in all, blanks and repeats dropped (`waitTexts`); `text` is no longer required when `any` is there. Every read looks for them in the order given and the wait ends at the first found (`screenShows`). With `gone`, it ends once none of them shows.
- **`area`** (`WaitForRequest.Area`, `{x, y, width, height}` in screen pixels) counts an element only when its centre, the `@x,y` its line shows the agent, lies inside. Events (toasts, notifications) have no place on the screen and do not count then, nor do elements without bounds. `x` and `y` below 0, an empty size, or an area that starts beyond the screen's size on the first read are `invalid_argument`. The screen in the result is the whole screen as before; the area only decides what answers the wait.
- **The message** names the text: "“Payment failed” appeared after 4 s.", "None of “a”, “b” or “c” has appeared after 45 s.", and for `gone` "“a”, “b” and “c” disappeared after 2 s." or, on a timeout, the one still there: "“b” has not disappeared after 45 s.". With an area it says "in the area". One text reads as it did.
- **History.** The step's text keeps one pair of quotes around all the texts ("Waited for “Order placed / Payment failed”"), so the client's pattern for it (`internal/gwtext/gwtext.go`, `reWaited`) and its translations read it unchanged; the step's arguments keep `any` and `area`.
- **Not changed:** the 45 s limit, how a text matches a label, and the OCR read at most once a second (`waitForOCREvery`).

Tests: `internal/gateway/waitfor_test.go` (`TestWaitForAny`, `TestWaitForArea`, `TestScreenShows`), `internal/mcp` `TestWaitForAnyAndArea` (the schema and a call of each).

## iPhone switches tapped by ref (GitHub issue #51, 2026-11-10)

Settings and SwiftUI give an iPhone switch the frame of its whole row, label included, so a tap by ref at its centre landed on the label and flipped nothing, while the result said settled. `switchPoint` (pipeline.go, iOS only) taps the switch itself: WebDriverAgent lists it as a narrower, unlabelled switch inside the row (Settings › General › Keyboard on the iOS 26.5 Simulator: row `{96 1239 978 87}`, switch `{891 1239 189 87}`), and its centre is used when there is one. Otherwise, for a row-shaped switch (four times as wide as tall or more), the tap lands 38 points in from the trailing end (a 51-point switch in a 16-point margin), or from the leading end when an element with the switch's label lies in the row's right half (right to left); points are 3 pixels on screens 1000 to 1400 pixels across and 2 elsewhere (`pointScale`). `phone_drag` by ref presses the same point. A tap by ref on a switch or checkbox whose checked state did not change adds "[N] is still on/off after the tap. If nothing else came up, tap the switch itself by x and y." to the result (`unflipped`).

On the iOS 26.5 Simulator through the gateway: by ref, Haptic Feedback and Sound flipped both ways with the inner switch and with the trailing-end estimate alone; with the old centre tap they stayed as they were and the result carried the new sentence. Android is unchanged: on the API 36 emulator the switch is its own element (Display › Dark theme `{901 1661 137 126}`, beside the row's button) and a tap by ref flipped it both ways.

## Gestures clear of the keyboard (GitHub issue #71, 2026-11-10)

An element's bounds may reach under the on-screen keyboard: a search page's results list spans the screen below the field, and its middle lies just above the keyboard. `phone_swipe` by ref went across 60% of the bounds around their middle, so a swipe up started on the keyboard and typed a character. The screen model now says where the keyboard lies (`Screen.KeyboardFrame`) and where system bars take touches (`Screen.Covered`), and `reach.go` keeps gestures to the part of an element (or of the screen) clear of them:

- `phone_swipe` by ref, or across the screen, swipes inside that part, and is refused with what to do (hide the keyboard first) when it is empty or less than a tenth of the screen along the swipe. A swipe between `from` and `to` is the agent's to place and is left alone.
- Taps and `phone_drag` by ref press inside that part (`tapPoint` on the cut bounds), and are refused when nothing of the element shows; `phone_pinch` keeps both fingers inside it, by ref and across the screen.
- A part that takes three quarters of the screen or more is a keyboard's full-screen sheet the agent acts on and does not count. An element wholly inside a system bar is the bar's own. When the keyboard's own nodes are listed (Android's second read with `DumpOptions.IME` over an app that shows nothing, or a panel the app walked unasked), the companion leaves `KeyboardFrame` empty, since the agent acts on the keyboard there; whatever else lies wholly under a frame is refused.
- `phone_pinch` blames the keyboard or a bar only when there would be room for the fingers without it; an element too small or at the screen's edge gets the usual message.

The drivers fill the frames where they cheaply can. Android's companion: the input method window's largest touchable part is the keyboard, its other parts (Gboard's stylus strips) and the system bars go in `Covered`; this covers background displays too. The uiautomator fallback knows only that the keyboard shows, so nothing is cut there. iOS: the box of the keyboard's keys on iOS 27 (rax), the keyboard element's frame through WebDriverAgent, or from OCR's top row of keys, 6% of the screen higher for the suggestions bar.

## A time limit for every call (2026-11-10)

A field test on an iPhone through WebDriverAgent over Wi-Fi (see docs/notes/ios.md, "Slow reads through WebDriverAgent over Wi-Fi") had calls of a minute and more: each read waited up to 60 s, an action read twice or more, and a `phone_batch` ran its steps whatever they took. Claude Desktop gives up on a call after about a minute and then starts the MCP server again, which ended the task with its session.

- **The budget** (`budget.go`, `Options.CallBudget`, 45 s): a value on the call's context, set when the gateway receives the call, not a context deadline, so it never cancels an input in progress. Reads take the shorter of their own 60 s and what is left; a read cut by the budget is `errOutOfTime`, which before the action is `internal` ("Nothing was done") and after it `uncertain`. The action lock, approval waits (leaving 10 s), `awaitLeave`, `awaitApp`, `phone_wait_for` and `phone_batch` (before each step, as long as its slowest step so far, at least 5 s) keep within it.
- **The internal launch of 0 ms** in the field logs was a call the client had already cancelled after about 60 s spent before the action: `preflight` saw the cancelled context. The step's clock starts just before `preflight`, hence the 0 ms.
- **Local sessions** (`AgentInfo.Reconnect`, set by the MCP server for agents on this computer): a closed session's leases are kept for `LocalResumeGrace` (60 s), and a new session of the same agent continues the task when its `phone_connect` names the same task, or when the old session has closed. Another agent's `phone_connect` ends such a kept lease at once, so a person who switches agents is not refused for a minute. Remote agents (`Resume`) keep their rule: any newer session continues.
- **Web pages before their tree** (`tree.go`): after an action, a read where OCR found 8 lines or more and at least twice as many as the tree's elements is read again, up to 4 s and leaving 5 s of the budget, until the tree fills. On the iPhone 17 Simulator a Wikipedia page opened from a link had no element of the page in the tree after 1 s and 51 after 3.5 s. An app whose tree stays sparse for the whole wait (a game) is remembered per phone and not waited for again until another app's is.


## One session, several tasks (2026-11-11)

Field report: Claude Cowork (`local-agent-mode-pinky`, a local agent) ran two conversations over one MCP session. Conversation A worked on the iPad; Cowork opened a new session and closed the old one, so A's next call continued the kept iPad lease there. Conversation B then called `phone_connect` without `device` for its own task: `Connect` defaulted to the session's one phone, the iPad, and ended A's task as `replaced`. B released it and connected the iPhone, and A's next `phone_tap` without `device` went to the session's one phone, B's iPhone.

Pinky cannot tell two conversations of one session apart, except by the task `phone_connect` names (`shared.go`):

- **phone_connect never replaces.** A phone that runs another task of the session, or one a closed session of its agent keeps for it, returns `lease_held` naming the task ("end it with phone_release first … or pass device"). `continues` takes the task's title into account, so a `phone_connect` for another task never adopts a lease either; for Resume agents that means a per-call client must release before it starts a new task on the same phone, where it used to replace. `OutcomeReplaced` is no longer produced; history keeps it.
- **Without device**, `phone_connect` goes to the phone that runs the same task (`taskPhoneLocked`: one the session holds or may adopt for that title); otherwise it chooses as for a session that holds none, with a note listing the phones that run the session's other tasks.
- **Shared sessions.** `Session.shared` is set once the session's own leases and those kept for it (`keptForLocked`) run tasks of different names, once a `phone_connect` of it is refused for another task, when it adopts from a shared session, and when an older session of its agent closes and keeps leases that make the mix (`noteNewerSharedLocked`). It is never cleared. Every call of a shared session without `device` (`acquire`, `Release`) returns `invalid_argument` listing the session's phones and their tasks; the result that marks it says "pass device on every call from now on".
- **What a single agent sees:** nothing, as long as it releases a task before it starts the next. Starting a second task on a phone it holds is refused, and marks the session shared.
- **Not caught:** a kept lease that ran out (60 s) before the other conversation connects leaves the new session with one task, so Pinky cannot know a second conversation shares it, and a call without `device` goes to its one phone as before.

Tests: `shared_test.go` (the field report's timeline with a fake iPad and iPhone; a refused `phone_connect` on the session's own phone; one before the kept lease is continued; a close after the new session connected, with the kept lease running out; a single agent; a per-call Resume client).

## Approval titles ask (2026-11-12)

An approval's title asks a question: "Pay: “Place order · $12.80”?", "Open Northbank?", "Tap “Delete”?". When nothing more particular fits (nothing labelled under the finger, a key, a pinch), it used to be history's words with a question mark, in the past tense: "Pressed back?", "Tapped at 540,1200?", "Swiped up?". It now asks for the action itself in the imperative (`question`, which says what `describe` says in another tense; `describePinch` and `describeDrag` take the same choice): "Press back?", "Tap at 540,1200?", "Swipe up?", "Type “…” and submit?", "Zoom in on “Map”?", "Drag from 200,2000 to 540,1700?". This is what the Needs you card shows in English, what Pinky's phone apps show, and what the waiting agent reads ("Waiting for the person: Press back? (…)"). History's steps keep the past tense, "Waiting for you: Pressed back" included. The client reads both tenses (`parseQuestion` in `internal/gwtext/gwtext.go`), since an earlier daemon, and the approvals it stored, still send the old titles. Tests: `TestApprovalTitlesAsk`, `TestPinch`'s title, and `TestApprovalTitlesInEitherTense` in `ui`.

## Reading a list in one call (`phone_read_list`, PR #45, 2026-11-12)

A community PR, reworked before merging so that every swipe goes through the action pipeline: `ReadList` loops `act` with a `swipe` action whose `list` field (`listSwipe`) gives the area, travel and lane, and `Call.noAsk` turns an Ask verdict into `blocked_by_policy` (`noAskMessage`) instead of an approval. Stop, a pause and a takeover end the call with nothing more, as in `phone_batch`. The continuation is an id for a cursor on the lease (`lease.readLists`), never encoded screen content. The viewport merging lives in `internal/gateway/viewport.go`, unexported. Details: [read-list.md](read-list.md).