EYRA — Eye + Ear + Action

EYRA

Eye + Ear + Action for AI agents on Windows.
One ~1.9 MB pure-Rust binary that is an MCP server, an agent skill and a CLI.
It sees any window as compact text, hears system audio and the microphone, and drives the mouse and keyboard.
Works with any MCP-capable agent: Gemini (Antigravity, Gemini CLI), Claude (Claude Code, Claude Desktop), Cursor, VS Code and others.

Downloads Latest release GitHub stars License: Apache-2.0 Platform: Windows 10/11 MCP 2025-06-18 Tests: 40 passing

https://github.com/user-attachments/assets/023b3238-3dee-4a1b-b20c-65e0447f14b8 --- ## Why EYRA? Most computer-use agents send a full screenshot on every step and then guess pixel coordinates. EYRA reads what Windows already knows about the screen. Example `see` output (abridged): ```text WIN hwnd=0x3C051E proc=chrome.exe title="New repository - Google Chrome" mode=uia ms=22 #15 inp "Address and search bar" val="github.com/new" [158,46,1060,34] #49 cmb "Owner" val="octocat" [325,333,150,32] #51 inp "Repository name *" (focused) [496,333,530,32] #68 chk "Add README" (unchecked) [978,682,42,24] ``` - **Exact, not guessed.** The text and coordinates come from the OS accessibility tree, so the agent doesn't have to estimate them from pixels. `click(id=51)` hits the right control. - **Fewer tokens.** A text map of the UI is cheaper than a screenshot. `see_diff` returns only what changed, typically **under 40 tokens per step**. - **Not just the web: the whole Windows desktop.** Any browser (Chrome, Edge, Firefox, Brave; including logged-in tabs), File Explorer, Office, Settings, Notepad, Electron apps (VS Code, Discord, Slack, Telegram), canvases, games and video (through OCR). No browser extension, per-site API or cloud service. - **Model-agnostic.** It is a standard MCP server, so the same binary serves Gemini, Claude, GPT-based hosts or your own agent. Models that accept audio input (for example Gemini) can listen to the WAV clips that `listen` / `watch` produce. Text-only models still get Eye + Action. - **Zero runtime dependencies.** No Python, FFmpeg, Whisper or API keys. Only Windows APIs: UI Automation, WinRT OCR, WASAPI and SendInput. - **Safe by default.** Password values are masked, password managers are blocked, there's a `--read-only` mode, a failsafe corner, a kill-switch hotkey and an audit log. ## Measured performance Measured on Windows 10 at 1366×768 with `eyra bench` and the MCP server. ### Latency of `see` (one window) | Mode | Chrome (GitHub page, ~100 elements) | Electron desktop app | |---|---|---| | MCP server, warm (`see`) | **22 ms** | ~25 ms | | MCP server, warm (`see_diff`) | **23 ms** | ~25 ms | | CLI, one-shot process | ~45 ms | ~65 ms | | OCR mode (`mode="ocr"`) | ~80 ms | ~80 ms | File Explorer showing a very large folder is the slowest case tested: ~195 ms. ### Tokens per observation Each window is compared with one screenshot of the same window (image tokens = w·h/750). Text tokens are estimated as chars/4. | Window (screenshot cost) | `see` | `interactive_only` | `interactive_only` + `coords="center"` | `see_diff` (no change) | |---|---|---|---|---| | Chrome, dense GitHub repo page, 1036×701 (~969) | ~1,137 | ~1,065 | **~901 (−7%)** | **~42 (−96%)** | | Electron desktop app, 1366×728 (~1,326) | ~675 | ~603 | **~528 (−60%)** | **~28 (−98%)** | | Windows desktop / Explorer, 1366×768 (~1,399) | ~294 | ~294 | **~250 (−82%)** | **~31 (−98%)** | **How to read this:** - The first look at a very dense web page costs about as much as a screenshot, but it gives exact `#id`s and coordinates instead of pixels to guess. - The big savings come from the steps after that. A 20-step task that observes with `see_diff` costs about 900 + 19×40 ≈ **1.7k tokens instead of about 19–26k** with screenshots. Run it on your own machine: ```powershell eyra bench --window Chrome ``` ## When to use EYRA (and when not) | Task | Best tool | |---|---| | Native Windows apps (Explorer, Office, Settings, Telegram, WinForms/WPF/UWP) | **EYRA** | | Electron apps (VS Code, Slack, Discord, Teams) | **EYRA** | | Repeatedly watching the same window for changes | **EYRA** `see_diff` / `wait_change` | | Canvas, games, remote desktop, video | **EYRA** `see(mode="ocr")` / `screenshot(diff_only)` | | Hearing system audio or the microphone (for audio-capable models) | **EYRA** `listen` / `watch_*` | | Deep web automation: hidden DOM, network, console, JS | A browser-extension tool (DOM access beats OS-level access here) | EYRA can only see what's on screen. It can't read DOM nodes that aren't rendered, network traffic or a page's JavaScript state. ## Install ### One-click install | Agent | How | |---|---| | **Claude Desktop** | Download `eyra-.mcpb` from the [latest release](https://github.com/layer8labs8/EYRA/releases/latest) and open it (double-click, or drag it into *Settings → Extensions*). A *Read-only mode* switch is available in the extension's settings. | | **Gemini CLI** | `gemini extensions install https://github.com/layer8labs8/EYRA` | | **Claude Code** | `/plugin install eyra --marketplace layer8labs8/EYRA` | All packages contain the same standalone `eyra.exe`; nothing else is installed. ### Download 1. Download `eyra.exe` from the [latest release](https://github.com/layer8labs8/EYRA/releases/latest). Or take the `.zip`, which also includes `SKILL.md`, the example MCP config and the license. 2. Put it anywhere, for example `C:\Tools\eyra\eyra.exe`. Nothing else needs to be installed: it runs on a stock Windows 10/11. 3. Connect it to your agent (see below). The binary isn't code-signed yet, so Windows SmartScreen may show a warning on first run: choose *More info* → *Run anyway*. You can check the download against the SHA-256 published with the release: ```powershell Get-FileHash .\eyra.exe -Algorithm SHA256 ``` ### Build from source Requirements: Windows 10/11 and a stable Rust toolchain. The repo pins `stable-x86_64-pc-windows-gnu`, which needs MinGW-w64. To use MSVC instead, delete `rust-toolchain.toml`. ```powershell cargo build --release # -> target\release\eyra.exe (~1.9 MB, no DLLs needed) cargo test --release ``` To build every release asset (exe, zip, Claude `.mcpb`, Gemini extension archive, checksums) into `dist\`: ```powershell powershell -ExecutionPolicy Bypass -File scripts\package.ps1 ``` ### Connect it to an agent EYRA speaks standard MCP over stdio, so the setup is the same everywhere: point the host at `eyra.exe mcp`. | Host | Where to add it | |---|---| | **Gemini – Antigravity** | MCP servers → *Manage* → *View raw config* (`mcp_config.json`), add the JSON below | | **Gemini CLI** | `gemini extensions install https://github.com/layer8labs8/EYRA`, or `%USERPROFILE%\.gemini\settings.json` with the JSON below | | **Claude Code** | `claude mcp add eyra -- C:\path\to\eyra.exe mcp` | | **Claude Desktop** | open the `.mcpb` from the release, or `claude_desktop_config.json` with the JSON below | | **Cursor / VS Code / others** | the host's MCP config file, add the JSON below | See [`mcp-config.example.json`](mcp-config.example.json): ```json { "mcpServers": { "eyra": { "command": "C:\\path\\to\\eyra.exe", "args": ["mcp"] } } } ``` Use `"args": ["mcp", "--read-only"]` to expose only the Eye, Ear and Watch tools, with no mouse or keyboard. The server negotiates MCP revisions `2025-06-18`, `2025-03-26` and `2024-11-05`. It sends usage `instructions` on `initialize` and marks every tool with `readOnlyHint`/`destructiveHint` annotations, so hosts can auto-approve observation tools and ask before input actions. **Agent skill.** [`SKILL.md`](SKILL.md) teaches an agent the lowest-token workflow. ## MCP tools (16) ### Eye | Tool | What it does | Key arguments | |---|---|---| | `list_windows` | Visible windows: hwnd, process, title, rect, flags | — | | `see` | Numbered UI element map (UIA, OCR fallback); includes open popup menus | `window`, `mode` = auto/uia/ocr/both, `interactive_only`, `coords` = rect/center, `max_elements`, `ocr_lang` | | `see_diff` | Only `+` added, `~` changed and `-` removed elements since the last `see` of **that window** | same as `see` | | `screenshot` | Compact JPEG/PNG, including occluded windows | `window`, `crop`, `max_width`, `quality`, `diff_only`, `som` | | `wait_change` | Blocks until the pixels change (page load, animation end) | `window`, `timeout_ms`, `threshold_pct` | ### Ear and Watch | Tool | What it does | Key arguments | |---|---|---| | `list_audio_devices` | Output (loopback) and input endpoints | — | | `listen` | Records system audio, the mic, or both, trims silence, saves a 16 kHz WAV | `seconds`, `source`, `trim_silence` | | `watch_start` / `watch_poll` / `watch_stop` | Rolling timeline of scene-change keyframes and audio clips | `window`, `audio`, `chunk_sec`, `scene_threshold_pct` | ### Action (disabled in `--read-only`) | Tool | What it does | Key arguments | |---|---|---| | `click` | Clicks by `#id` (background UIA Invoke/Toggle/Select first) or by `x,y` | `id`, `x`, `y`, `button`, `clicks`, `background` | | `type_text` | Types Unicode text (background UIA `SetValue` first) | `text`, `id`, `clear`, `press_enter` | | `hotkey` | Sends a keyboard shortcut such as `ctrl+l` or `alt+tab` | `keys`, `window` | | `scroll` / `drag` | Scrolls the mouse wheel or drags with the left button | `delta_y`, `delta_x` / `x1,y1,x2,y2` | | `focus_window` | Restores, focuses and optionally locks a default target window | `window`, `lock` | ### Reading the element map ```text #12 btn "Save" (disabled) [840,610,90,32] │ │ │ │ └ x,y,width,height in screen pixels │ │ │ └ state: focused, disabled, checked, unchecked, mixed, selected │ │ └ accessible name (val="…" follows when an input has a value) │ └ role: btn inp lnk chk rad cmb tab mnu itm sld hdr txt img doc pwd ocr el └ id: pass it to click / type_text ``` With `coords="center"` the position is printed as the click point only (`#12 btn "Save" (disabled) @885,626`), which is the cheapest format. `#id`s come from the **most recent `see`/`see_diff`** of that window, so call `see_diff` again after the UI changes. Open popup menus (File/Edit menus, context menus) are part of the window's map. ## CLI ```powershell eyra windows eyra see --window Chrome --interactive-only --coords center eyra diff --window Chrome eyra screenshot --window Chrome --som --out som.jpg eyra click --id 5 eyra type "hello world" --id 2 --enter eyra hotkey ctrl+l eyra wait-change --window Chrome eyra audio-devices eyra listen --seconds 8 --source both eyra watch --window Chrome --duration 60 eyra ocr-langs eyra logs --tail 25 eyra bench --window Chrome eyra serve --port 47821 # optional 127.0.0.1 HTTP bridge (token-authenticated) ``` `--window` accepts an hwnd (`0x3C051E`), a title substring, a process name or `screen`. Add `--json` for machine-readable output. ## Safety | Guard | Behaviour | |---|---| | Password fields | Values are always shown as `[REDACTED]` and blacked out in screenshots and before OCR | | Sensitive windows | 1Password, KeePass, Bitwarden and Windows Security can't be read or controlled unless you pass `--allow-sensitive` | | `--read-only` | Removes all input tools from the MCP tool list and rejects them in the CLI and bridge | | Corner failsafe | Moving the mouse to the top-left corner `(0,0)` halts input actions | | Kill switch | `Ctrl+Alt+Esc` latches an emergency stop until eyra restarts | | Dangerous hotkeys | `ctrl+alt+delete` and `win+l` are blocked, and so is `alt+f4` on the Windows shell | | Audit log | JSONL at `%LOCALAPPDATA%\eyra\audit.jsonl`: every call with its timing and token estimate. Text typed into password fields is redacted. Rotates at 10 MB | ## Troubleshooting | Symptom | Cause and fix | |---|---| | `see` says the window is minimized | Restore it first with `focus_window`. | | `see` returns nearly nothing for a game, canvas or remote desktop | There is no UIA tree there. Use `mode="ocr"`; `auto` already falls back to OCR below 5 elements. | | OCR misses your language | Run `eyra ocr-langs`, then install the language's OCR pack in Windows Settings → Time & Language → Language, and pass `ocr_lang`. | | Clicks or typing do nothing in an admin window | Windows UIPI blocks input to elevated windows from a non-elevated process. Run eyra elevated, or the target app non-elevated. | | Coordinates are off on a multi-monitor or 150% DPI setup | eyra is per-monitor DPI aware and coordinates are physical pixels. Use `#id` clicks rather than hand-computed points. | ### Environment variables | Variable | Effect | |---|---| | `EYRA_LOG_FILE` / `EYRA_NO_LOG` | Audit log path / disable logging | | `EYRA_READ_ONLY` | Forces read-only mode | | `EYRA_OCR_SCALE=2` | Upscales frames before OCR (slower; can help with very small text) | ## Architecture ```text src/ ├─ eye/ uia.rs (cached UIA subtree scan) · ocr.rs (WinRT OCR) · capture.rs (PrintWindow/BitBlt) │ diff.rs (element and pixel diff) · som.rs (Set-of-Mark badges) · windows.rs ├─ ear/ wasapi.rs (loopback + mic) · vad.rs (resample + energy VAD) · wav.rs ├─ hand.rs UIA patterns → SendInput fallback ├─ watch.rs ring-buffered video and audio timeline ├─ mcp.rs stdio JSON-RPC server ├─ bridge.rs local HTTP bridge ├─ safety.rs · audit.rs · state.rs └─ cli.rs ``` ## License Licensed under the [Apache License 2.0](LICENSE).