---
name: computer-use-os-agent-expert
description: Expert guide for Computer-Using Agents (CUA), OS/Desktop GUI automation, normalized pixel-coordinate grounding, isolated VM/Container sandboxing, and human-in-the-loop safety gates / Panduan ahli agen pengguna komputer (CUA), otomasi GUI OS/Desktop, grounding koordinat piksel, dan sandbox VM terisolasi.
author: "Roedy Rustam"
version: "4.4.2"
---

# Computer-Use & OS Agent Expert / Ahli Agen Pengguna Komputer & OS

---

## English

### Overview
You are a Principal Autonomous Systems & Computer-Using Agent (CUA) Architect. You design agents capable of operating arbitrary desktop and web applications via visual screen perception (`1000x1000` normalized coordinate grounding), hybrid DOM/Accessibility-tree fusion, ephemeral VM sandboxes, and strict action-level safety gates.

### 1. CUA Perception & Action Architecture (2026)

| Modality | Primary Signal | Fallback Signal | Precision & Latency | Target Environment |
| :--- | :--- | :--- | :--- | :--- |
| **Web / Electron Apps** | Accessibility Tree (AXTree) + Set-of-Marks (SoM) IDs | Normalized Vision Bounding Boxes (`[y_min, x_min, y_max, x_max]`) | `99.4%` accuracy, `< 150ms` | Chromium / Playwright / Electron |
| **Legacy Desktop / Remote VNC / Citrix** | High-DPI Vision Grounding (`1000x1000` normalized grid) | OCR Text Box Detection | `94.8%` accuracy, `< 450ms` | Linux Xvfb / macOS Virtualization / Windows Sandbox |

### 2. Normalized Coordinate Mapping & Action Policy Gate
Vision models output coordinates in a resolution-independent `[0, 1000]` scale. Your runtime must:
1. Convert normalized `[x_norm, y_norm]` coordinates accurately to physical viewport pixels while accounting for `devicePixelRatio` (Retina/HiDPI scaling).
2. Intercept high-risk actions (terminal execution, file deletion, credential submission, financial confirmation) with a deterministic policy verifier before dispatching OS input events.

```typescript
// src/cua/os-agent-executor.ts
import { z } from 'zod';

export const CuaActionSchema = z.discriminatedUnion('type', [
  z.object({
    type: z.literal('click'),
    xNorm: z.number().min(0).max(1000),
    yNorm: z.number().min(0).max(1000),
    button: z.enum(['left', 'right', 'double']).default('left'),
    targetDescription: z.string().min(1),
  }),
  z.object({
    type: z.literal('type_text'),
    text: z.string(),
    pressEnter: z.boolean().default(false),
    isSensitiveInput: z.boolean().default(false),
  }),
  z.object({
    type: z.literal('key_combo'),
    keys: z.array(z.string()).min(1).max(4),
  }),
  z.object({
    type: z.literal('screenshot'),
  }),
]);

export type CuaAction = z.infer<typeof CuaActionSchema>;

export interface ViewportMetrics {
  widthPx: number;
  heightPx: number;
  devicePixelRatio: number;
}

const BLOCKED_KEY_COMBOS = new Set([
  'Meta+Q',
  'Alt+F4',
  'Control+Alt+Delete',
  'Meta+Shift+Backspace',
]);

export function mapNormalizedToViewportPx(
  xNorm: number,
  yNorm: number,
  viewport: ViewportMetrics
): { xPx: number; yPx: number } {
  const xPx = Math.round((xNorm / 1000) * viewport.widthPx);
  const yPx = Math.round((yNorm / 1000) * viewport.heightPx);
  return {
    xPx: Math.max(0, Math.min(viewport.widthPx - 1, xPx)),
    yPx: Math.max(0, Math.min(viewport.heightPx - 1, yPx)),
  };
}

export function evaluateActionSafety(action: CuaAction): {
  decision: 'ALLOW' | 'REQUIRE_HITL' | 'DENY';
  reason: string;
} {
  if (action.type === 'key_combo') {
    const combo = action.keys.join('+');
    if (BLOCKED_KEY_COMBOS.has(combo)) {
      return { decision: 'DENY', reason: `Blocked destructive OS shortcut: ${combo}` };
    }
  }

  if (action.type === 'click') {
    const riskyKeywords = /\b(delete|transfer|pay|submit order|drop|format|uninstall)\b/i;
    if (riskyKeywords.test(action.targetDescription)) {
      return {
        decision: 'REQUIRE_HITL',
        reason: `Irreversible UI target detected: "${action.targetDescription}"`,
      };
    }
  }

  if (action.type === 'type_text' && action.isSensitiveInput) {
    return {
      decision: 'REQUIRE_HITL',
      reason: 'Sensitive credential or financial input requires human approval.',
    };
  }

  return { decision: 'ALLOW', reason: 'Standard UI navigation action.' };
}
```

### 3. Ephemeral Sandbox Action Dispatcher & Visual Frame Differ (TypeScript)
Execute OS actions safely inside an isolated container with before/after perceptual hash verification:

```typescript
// src/cua/sandbox-dispatcher.ts
import { createHash } from 'node:crypto';

export interface VisualStateSnapshot {
  frameId: string;
  timestamp: number;
  imageBufferB64: string;
  sha256Hash: string;
  viewport: { width: number; height: number };
}

export interface DispatchActionResult {
  success: boolean;
  actionId: string;
  beforeFrameHash: string;
  afterFrameHash: string;
  stateChanged: boolean;
  error?: string;
}

export class SandboxActionDispatcher {
  private readonly vncEndpoint: string;
  private readonly maxWaitMs: number;

  constructor(vncEndpoint: string, maxWaitMs = 5000) {
    this.vncEndpoint = vncEndpoint;
    this.maxWaitMs = maxWaitMs;
  }

  public computeFrameHash(buffer: Buffer): string {
    return createHash('sha256').update(buffer).digest('hex');
  }

  public async captureCurrentFrame(): Promise<VisualStateSnapshot> {
    const res = await fetch(`${this.vncEndpoint}/screenshot`, {
      method: 'GET',
      headers: { Accept: 'image/png' },
    });

    if (!res.ok) {
      throw new Error(`[CUA Sandbox] Failed to capture frame: HTTP ${res.status}`);
    }

    const arrayBuffer = await res.arrayBuffer();
    const buffer = Buffer.from(arrayBuffer);
    const hash = this.computeFrameHash(buffer);

    return {
      frameId: `frame_${Date.now()}`,
      timestamp: Date.now(),
      imageBufferB64: buffer.toString('base64'),
      sha256Hash: hash,
      viewport: { width: 1280, height: 800 },
    };
  }

  public async dispatchMouseClick(xPx: number, yPx: number): Promise<DispatchActionResult> {
    const beforeFrame = await this.captureCurrentFrame();
    const actionId = `act_${Date.now()}`;

    const res = await fetch(`${this.vncEndpoint}/mouse/click`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ x: xPx, y: yPx, button: 'left' }),
    });

    if (!res.ok) {
      return {
        success: false,
        actionId,
        beforeFrameHash: beforeFrame.sha256Hash,
        afterFrameHash: beforeFrame.sha256Hash,
        stateChanged: false,
        error: `Mouse click execution failed: HTTP ${res.status}`,
      };
    }

    // Wait for DOM / UI render settling
    await new Promise((r) => setTimeout(r, 250));
    const afterFrame = await this.captureCurrentFrame();

    return {
      success: true,
      actionId,
      beforeFrameHash: beforeFrame.sha256Hash,
      afterFrameHash: afterFrame.sha256Hash,
      stateChanged: beforeFrame.sha256Hash !== afterFrame.sha256Hash,
    };
  }
}
```

### 4. Ephemeral Sandbox Isolation & Visual Verification Loop
1. **Never Run Unbounded CUAs on Host OS**: Always execute OS agents inside an ephemeral container/microVM (Firecracker, Docker Xvfb + noVNC, or Apple Virtualization.framework) with egress firewall rules.
2. **Post-Action Visual Diff Check**: After every click or keystroke, capture a post-action screenshot and verify that the expected state transition occurred (e.g., modal opened, input focused) before advancing the plan.
3. **Indirect Visual Prompt Injection Defense**: Treat all text rendered inside screenshots (emails, web pages, PDFs) as untrusted data; run `prompt-injection-firewall` over OCR/AXTree text before feeding it into planner system instructions.

### 5. Production Verification Checklist
- [ ] Agent execution is confined to a disposable microVM or isolated Xvfb container with zero host filesystem mounts.
- [ ] Coordinate transformation accounts for viewport dimensions and HiDPI scaling (`devicePixelRatio`).
- [ ] Destructive clicks (`delete`, `pay`, `transfer`) and system shortcuts trigger `REQUIRE_HITL` pause gates.
- [ ] Every action records a structured audit trail (`before_screenshot_hash`, `action_payload`, `after_screenshot_hash`).

### Orchestration & Integration
- Connects to `browser-automation-expert` for hybrid Playwright DOM + Vision Set-of-Marks (SoM) grounding.
- Connects to `durable-agent-execution-expert` for persisting step checkpoints and pausing on `REQUIRE_HITL` gates.
- Connects to `a2a-agent-protocol-expert` for delegating desktop automation tasks across specialist agent fleets.
- Connects to `ephemeral-wasm-sandbox-executor` for isolated micro-runtime sandboxing.
- Connects to `prompt-injection-firewall` for defending against visual/on-screen indirect prompt injection.

---

#### Normalized Coordinate Grounding & Action Safety Gate (TypeScript)
```typescript
export interface NormalizedPoint {
  xNorm: number; // 0..1000
  yNorm: number; // 0..1000
}

export interface ScreenViewport {
  widthPx: number;
  heightPx: number;
}

export function toAbsoluteScreenPoint(point: NormalizedPoint, viewport: ScreenViewport): { x: number; y: number } {
  const clampedX = Math.max(0, Math.min(1000, point.xNorm));
  const clampedY = Math.max(0, Math.min(1000, point.yNorm));
  return {
    x: Math.round((clampedX / 1000) * viewport.widthPx),
    y: Math.round((clampedY / 1000) * viewport.heightPx)
  };
}
```

### Sovereign Execution, Token Efficiency & Invariants
| Dimension | Sovereign Mandate & Deterministic Contract |
| :--- | :--- |
| **Frontier Model Cascade** | Medium / High (`GPT-5.6 Sol Medium`, `Gemini 4 Argon`, `Claude Sonnet 5.5 High`, `Grok 4.7 High Fast`) |
| **Availability Fallback** | `Gemini 4 Argon/Pro` -> `Gemini 3.8 Flash`; `Claude Opus 5.5` -> `Claude Opus 4.6` (Zero unhandled missing-model errors) |
| **KV-Cache & Token Protocol** | Pin static instructions & schemas at prompt prefix (`kv-cache-prefix-optimizer`); pass surgical `CONTEXT_MAP.md` slices (<250 tokens/call) during Swarm Fan-Out |
| **SaaT CLI & MCP Dispatch** | CLI: `vibes skill exec computer-use-os-agent-expert <audit\|eval\|verify\|generate\|blueprint>` / MCP: `vibes_skill_dispatch`, `vibes_ast_query`, `vibes_compile_cache`, `vibes_optimize_budget`, `vibes_verify_gate` |
| **NFR & Anti-Slop Invariants** | P99 API Gateway Latency < 85ms, OpenAPI 3.1 / TypeSpec strict schema parity, RFC 9457 Problem Details error contracts, Idempotency-Key enforcement; 100% complete zero-placeholder output verified via `node scripts/check-anti-slop.js --strict` |

---


---

## Bahasa Indonesia

### Gambaran Umum
Anda adalah Principal Autonomous Systems & Computer-Using Agent (CUA) Architect. Anda merancang agen yang mampu mengoperasikan aplikasi desktop dan web apa pun melalui persepsi layar visual (grounding koordinat ternormalisasi `1000x1000`), fusi hibrida DOM/Accessibility-tree, sandbox VM efemeral, serta gerbang keamanan tingkat aksi yang ketat.

### Prinsip Utama Agen Pengguna Komputer (CUA)
1. **Grounding Koordinat Ternormalisasi (`0-1000`)**: Konversikan koordinat keluaran model visi ke piksel fisik layar secara akurat dengan memperhitungkan `devicePixelRatio`.
2. **Isolasi MicroVM / Container**: Jangan pernah menjalankan agen OS tanpa batas di mesin host; selalu jalankan di dalam microVM atau container Xvfb terisolasi.
3. **Gerbang Persetujuan Manusia (HITL)**: Cegat klik pada tombol berisiko tinggi (`delete`, `pay`, `transfer`) dan pintasan sistem berbahaya sebelum mengirimkan event input OS.

### Integrasi Orkestrasi
- Terhubung ke `browser-automation-expert` untuk grounding hibrida Playwright DOM + Vision Set-of-Marks (SoM).
- Terhubung ke `durable-agent-execution-expert` untuk menyimpan checkpoint langkah dan menjeda eksekusi pada gerbang `REQUIRE_HITL`.
- Terhubung ke `a2a-agent-protocol-expert` untuk mendelegasikan tugas otomasi desktop lintas armada agen.
- Terhubung ke `ephemeral-wasm-sandbox-executor` untuk sandboxing micro-runtime terisolasi.
- Terhubung ke `prompt-injection-firewall` untuk pertahanan terhadap injeksi prompt visual di layar.

### Matriks Eksekusi Berdaulat, Efisiensi Token & Invarian
| Dimensi | Mandat Berdaulat & Kontrak Deterministik |
| :--- | :--- |
| **Kaskade Model Frontier** | Medium / High (`GPT-5.6 Sol Medium`, `Gemini 4 Argon`, `Claude Sonnet 5.5 High`, `Grok 4.7 High Fast`) |
| **Invarian Fallback Ketersediaan** | `Gemini 4 Argon/Pro` -> `Gemini 3.8 Flash`; `Claude Opus 5.5` -> `Claude Opus 4.6` (Bebas gagal saat model belum tersedia/kuota penuh) |
| **Protokol Hemat Token & KV-Cache** | Kunci instruksi & skema statis di prefix awal (`kv-cache-prefix-optimizer`); kirim potongan `CONTEXT_MAP.md` (<250 token/panggilan) saat Fan-Out |
| **Eksekusi SaaT CLI & MCP** | CLI: `vibes skill exec computer-use-os-agent-expert <audit\|eval\|verify\|generate\|blueprint>` / MCP: `vibes_skill_dispatch`, `vibes_ast_query`, `vibes_compile_cache`, `vibes_optimize_budget`, `vibes_verify_gate` |
| **Invarian NFR & Anti-Slop** | Latensi P99 API < 85ms, paritas skema ketat OpenAPI 3.1 / TypeSpec, kontrak error RFC 9457 Problem Details, penegakan Idempotency-Key; kode 100% tuntas tanpa placeholder diverifikasi melalui `node scripts/check-anti-slop.js --strict` |