# Umbriel · A set of hands for your AI agent — and Playwright for the Windows desktop
Drive any Windows app through five layers — fall back to OCR and pixel-matching when there's no other way in, see and manage windows even when they're hidden, use cursor-free semantic or posted input where Windows permits it, target controls by name and role, and reach past the GUI to drive the OS itself: processes, services, the registry, scheduled tasks, the firewall, network, power, displays, disks, environment, and the event log. Built for ChatGPT/Codex, Claude, and any AI agent that speaks MCP.
[](https://www.npmjs.com/package/umbriel)
[](./LICENSE)
[](#requirements)
[](https://bun.sh)
[](#why-umbriel)
## What is it?
Umbriel drives the Windows desktop the way a person would — through whatever channel actually works for the app in front of it:
1. **Pixels & OCR** — when an app exposes no tree at all (canvas, custom-draw, games), fall back to full-screen capture, template matching, and text recognition.
2. **Semantic control** — read the UI Automation tree an app exposes and target controls by *name* and *role*, not coordinates. Survives the DPI, layout, and theme changes that shatter pixel scripts.
3. **Sight & window control** — capture the *live* pixels of any window (even fully GPU-composited or occluded), inspect raw HWND hierarchies, and move, raise, or size windows.
4. **Input routing** — semantic UIA and posted-message input for background-capable controls, with a verified `SendInput` fallback only when an HWND-less editor or real pointer gesture actually needs the foreground.
5. **Drive the OS, not just GUIs** — when the task *is* the machine, act on it directly: read and write the registry (values *and* keys), list / kill / suspend / reprioritize processes, query / start / stop / configure services, create / delete / enumerate scheduled tasks, read the firewall rules, enumerate network adapters and live connections, drive session power (lock / restart / sleep / hibernate), list volumes and free space, get and set environment variables, read the event log, and read *or change* the display mode — all native, no PowerShell or shelling out.
Underneath it's a few kilobytes of TypeScript on Bun's built-in FFI — no Appium server, no `.NET`, no `node-gyp`, no prebuilt binaries.
## Install
```bash
bun add umbriel
```
That's the entire install story. No build step, nothing to compile.
## Built for AI agents
This is what Umbriel is *for*. Register the published server with Codex (ChatGPT desktop, CLI, and the IDE share this configuration):
```bash
codex mcp add umbriel -- bunx umbriel
```
Or register it with Claude:
```bash
claude mcp add umbriel -- bunx umbriel
```
Any MCP-speaking agent can then ground itself in the accessibility tree in ~15 ms per step and use semantic, posted-message, synthetic-input, or pixel fallback as the target requires. Most classic controls remain driveable backgrounded; a no-own-HWND Chromium/WPF/WinUI editor needs a brief unlocked foreground lease for real input. Umbriel verifies that destination before injecting, can verify the resulting editor state, and restores the previous foreground window by default.
A screenshot agent burns image tokens on every step — then burns them *again* to screenshot its own result, because it can't read state out of pixels. Umbriel maps the whole window to plain text you cache once and reuse across steps, and reads results straight back from that map. Same grounding Microsoft's UFO2 and the OSWorld-Human benchmark recommend (structure first, vision second) — without the bill.
```ts
umbriel.tree(app, { agentProfile: true });
// → one cached round-trip → { role, name, automationId, bounds, children }
```
> ⚡ Maps a window in **~13 ms** · tokens are **text you cache once and reuse**, not an image re-bought every step · up to **~2000× faster** grounding than a 3–26 s screenshot pipeline
| Per agent step | Screenshot agent | Umbriel |
| --- | --- | --- |
| Tokens | 1.5–2.7k image — then re-shot to self-check | ~3k text — cached & reused across steps |
| Grounding time | 3–26 s | ~13 ms |
| Reads state back? | no — must take another screenshot | yes — straight from the snapshot |
MIT Licensed · Built on [`bun:ffi`](https://bun.sh/docs/api/ffi) with zero native dependencies