---
name: computer-use
description: Control local desktop applications through Computer Use for tasks that require reading or operating app UI. Prefer purpose-built connectors, APIs, or CLIs when available.
---

## node_repl + @qwen-code/cua-sdk (Computer Use)

- Use `node_repl` (JavaScript) for all Computer Use actions.
- If both servers are available, use `node_repl` from the
  `desktop-node-repl` MCP server; the regular server runs on the Qwen Code host.
- Do not use other technologies besides `node_repl` for computer interactions, unless specifically requested by the user (e.g. AppleScript, `osascript`, JXA, System Events, synthesized input).
- Prefer a dedicated plugin or skill when it can complete the task; use Computer Use for app interactions that are not exposed through a more specific interface.
- `node_repl` state is persistent across calls.
- For text output, use `nodeRepl.write(...)`. It takes a string; use `JSON.stringify(...)` only for textual metadata. For an observation, write its `.text` and emit its screenshots as images; do not stringify an observation or driver result containing image bytes.
- Omit `yield_time_ms` for ordinary UI calls to use the default 10-second wait. A shorter yield does not speed up the action and can add a `node_repl_wait` round. Use a shorter yield only when you need control back before completion; if a cell is still running, collect its result with `node_repl_wait` before issuing dependent actions.

## Forwarding results in Codex code mode

When calling `node_repl` through `tools.*` inside Codex's outer `functions.exec`,
forward each returned `content` block by its type. `nodeRepl.emitImage(...)`
produces an MCP image block; the outer script must pass that block to `image()`
for the model to receive an image. Prefer the desktop relay tool when it is
present; otherwise use the regular `node-repl` server:

```js
// A code-mode `tools` object throws on an unknown key, so probe with `in`:
// reading an unbound tool would abort the script before any fallback ran.
const DESKTOP_NODE_REPL = 'mcp__desktop_node_repl__node_repl';
const nodeReplTool =
  DESKTOP_NODE_REPL in tools
    ? tools[DESKTOP_NODE_REPL]
    : tools.mcp__node_repl__node_repl;
const result = await nodeReplTool({ code });
for (const block of result.content ?? []) {
  if (block.type === 'text') {
    text(block.text);
  } else if (block.type === 'image') {
    image(block);
  }
}
```

Here `code` is the JavaScript to run in the persistent Node REPL. Keep the
forwarding loop in the outer code-mode script, outside that `code` string.
Apply the same loop to `node_repl_wait` results: images may arrive only when a
running cell completes. Forward text blocks too, including running-cell IDs and
errors. Direct MCP tool calls do not need this outer forwarding loop.

Do not use `text(result)`, `text(block)`, or `JSON.stringify(result)` to forward
an MCP result containing images: this turns image base64 into text, consuming
context without showing the image. `image()` accepts one image block, not the
whole result, so do not use `image(result)` either.

## Bootstrap

If `desktop-node-repl` is connected, skip the installation commands below: it
already provides `node_repl` and the SDK on the connected computer. Continue
with the `computer` initialization below. Otherwise, if `node_repl` is
unavailable, run:

```bash
qwen mcp add --scope user node-repl npx -y @qwen-code/node-repl-mcp@0.1.7
npm install --no-save --package-lock=false @qwen-code/cua-sdk@0.20.11
```

Tell the user to restart Qwen Code, then stop. If only the SDK import is missing,
run the second command and retry.

Reuse an existing `computer` connected to the intended desktop. Otherwise import
once per fresh `node_repl` session. The same App workflow below applies to macOS,
Linux and Windows; no additional platform resource is needed:

```js
globalThis.computer = await (
  await import('@qwen-code/cua-sdk/computer-use')
).ComputerUse.create();
var platform = await computer.getPlatform();
nodeRepl.write(`Connected platform: ${platform}`);
```

Use the connected platform for shortcuts, not the CLI or Node host operating system.
When the task identifies an app, combine initialization with `computer.getApp()`
and its first `getState()` in the same call. Read that state before editing or input.

## API surface

```ts
type Point = number | { x: number; y: number };
type ComputerUse = {
  getPlatform: () => Promise<'macos' | 'linux' | 'windows'>;
  getApp: (nameOrIdentifierOrPath: string) => Promise<App>;
  listApps: () => Promise<
    Array<{ id: string; displayName: string; isRunning: boolean }>
  >;
  close: () => Promise<void>;
};
type App = {
  getState: (options?: {
    disableDiff?: boolean;
    includeScreenshot?: boolean;
    maxTextChars?: number;
  }) => Promise<State>;
  click: (
    point: Point,
    options?: { button?: 'left' | 'right' | 'middle'; count?: number },
  ) => Promise<object>;
  doubleClick: (point: Point) => Promise<object>;
  rightClick: (
    point: Point,
    options?: { modifier?: string[] },
  ) => Promise<object>;
  setValue: (element: number, value: string) => Promise<object>;
  performSecondaryAction: (element: number, action: string) => Promise<object>;
  typeText: (text: string) => Promise<object>;
  paste: (
    text: string,
    options?: { format?: 'text' | 'md' | 'html'; signal?: AbortSignal },
  ) => Promise<object>;
  selectText: (
    element: number,
    text: string,
    options?: {
      prefix?: string;
      suffix?: string;
      selection?: 'text' | 'cursor_before' | 'cursor_after';
      signal?: AbortSignal;
    },
  ) => Promise<object>;
  pressKey: (
    key: string,
    options?: { modifiers?: string[] },
  ) => Promise<object>;
  hotkey: (keys: string[]) => Promise<object>;
  scroll: (
    point: Point,
    options: { direction: 'up' | 'down' | 'left' | 'right'; amount?: number },
  ) => Promise<object>;
  drag: (options: {
    fromX: number;
    fromY: number;
    toX: number;
    toY: number;
  }) => Promise<object>;
};
type State = {
  app: string;
  window: string;
  mode: 'full' | 'diff' | 'no_change';
  text: string;
  screenshot?: { images: Array<{ mimeType: string; dataBase64: string }> };
};
```

## Workflow

`paste` and `selectText` currently require macOS. On Linux/Windows, use the shared
`typeText`, `setValue`, `pressKey` and observed actions. Unsupported text methods
fail with `unsupported_platform`; do not retry them as window failures.

### 1. Initialize

If initialization already bound the task's app and returned its state, reuse
that `app` and observation. Otherwise, bind the app named by the task, then read its state.
`getApp()` binds identity; `getState()` can open a discovered stopped app. Combine these
steps in one Node REPL call:

```js
var app = await computer.getApp('Microsoft Excel');
nodeRepl.write((await app.getState()).text);
```

The app handle tracks its current window and dialog. Read the returned window
title to confirm the intended document. If the app is unknown or its name is
ambiguous, discover applications with `computer.listApps()` and use a matching
application `id` when it distinguishes the app. Two running instances may have
the same ID; retrying that ID cannot resolve the ambiguity. Ask the user to keep
only the intended instance open rather than guessing a target or retrying it.

AX text uses short numeric IDs, such as `[37] TextField "Name"`. Use IDs from
the current observation for element actions. IDs can change when the app's
window or session changes. Disabled and static-text rows are observation-only.

For token efficiency, the accessibility tree will be returned
as a diff when appropriate. Prefer this default diff output. A full state
replaces the previous state; a diff updates it; no-change preserves it. If you
need a full replacement, use `disableDiff: true` only when the previous state
is unavailable or no longer useful. Do not disregard the text and then assume
that a subsequent diff will reproduce the information you skipped.

Returned text defaults to at most 12,000 characters. Set `maxTextChars` (minimum 512) to adjust the limit. A truncation notice means some captured rows were
omitted; request `app.getState({ disableDiff: true, maxTextChars: 24000 })` when
you need more full text. An omitted row does not prove an element is absent.
Traversal-limited captures cover only the captured nodes: identical captures
can return no-change, while changes return full captured state. Use current
captured IDs; after a read failure, use only IDs from the latest observation.

### 2. Actions using app

After performing one or more UI actions, call `app.getState()` before deciding
what to do next. Batch actions whose target remains the same, then print only
the state needed for the next decision:

```js
await app.click(37);
await app.hotkey([platform === 'macos' ? 'super' : 'ctrl', 'a']);
await app.typeText('hello');
await app.pressKey('Return');
nodeRepl.write((await app.getState()).text);
```

Use the actual ID from your observation; `37` is only an example.

An observation is a decision boundary. When the current state already identifies
the controls and the next actions are known, combine those actions and saving
in the same call. Read state after the batch. End the batch at a new dialog,
menu, changed target or uncertain result; use that state before choosing the
next action. Do not split a known sequence merely to put each action in its own
call.

- Prefer element IDs to coordinates. `setValue(id, value)` changes a writable control, and `performSecondaryAction(id, action)` invokes a secondary action listed for that element. Use an observed action name rather than guessing.
- When an action opens or closes a dialog, sheet or menu, end the batch and call `app.getState()` to read the new window and IDs before continuing.
- `No open application window.` means the app is still running without a document window. If closing it completed the task, finish instead of retrying actions; otherwise open the intended file or window first.
- An action error can occur after the UI already changed. Read state before deciding whether to retry. Partial, unconfirmed or cancelled actions must not be blindly repeated.
- Coordinate actions use pixels in this app's current screenshot, with `(0, 0)` at its top-left. Every App observation refreshes that frame internally. Request `includeScreenshot: true` when you need to inspect the image, especially after a window change. Do not infer coordinates from another window or desktop screenshot.
- `pressKey` sends one key, optionally with modifiers. `hotkey` sends a combination such as `['super', 's']`. Use the connected platform's appropriate shortcut: macOS generally uses `super`; Linux/Windows generally use `ctrl`.
- App input manages any required activation internally and restores the previous focus, unless the user has moved it elsewhere. If the platform cannot confirm the target or restore focus, the action reports an error. There is no delivery-mode choice and no automatic replay after an uncertain result.
- On Linux, some compositor sessions cannot confirm an exact App target. An `app_window_unavailable` result can mean the session does not support this workflow; ask the user to use a supported desktop session instead of retrying input.
- Literal `\n` or `\r` in `typeText` sends Return. In a composer or form this may submit rather than insert a newline.
- If AX is incomplete or does not explain the interface, request a screenshot and inspect it. Only currently captured actionable IDs can be used for element actions.

## Paste and select text

These App methods are available on macOS only. Read current state first and use
an observed element ID for selection.

```js
await app.selectText(37, 'draft', { prefix: 'Status: ' });
nodeRepl.write((await app.getState()).text);
```

Use a real ID and text from your observation; the example assumes that the
selected element contains exactly one matching `draft` immediately after
`Status: `.

After confirming the selection, replace it with plain text:

```js
await app.paste('ready');
nodeRepl.write((await app.getState()).text);
```

- `selectText(element, text)` selects one exact, case-sensitive match. Optional
  `prefix` and `suffix` must be immediately adjacent to that match. Missing or
  ambiguous matches fail; provide enough observed context to identify one.
- `selection` defaults to `'text'`. Use `'cursor_before'` or `'cursor_after'` to
  place the insertion point at a match boundary instead of selecting the text.
  The element must support a writable text selection; static labels may not.
- `paste(text)` defaults to plain text. Use `{ format: 'md' }` for Markdown or
  `{ format: 'html' }` for HTML; the receiving app determines which supplied
  format it accepts. Use plain text when markup should remain literal.
- Paste uses the clipboard temporarily. It restores the previous clipboard only
  while it still owns that transaction; a newer external clipboard change is
  preserved.
- Both methods accept an optional `signal`. A completed dispatch, error or
  cancellation does not by itself prove what changed. Observe state before
  deciding whether to retry; do not blindly repeat unconfirmed text operations.

## Reading screenshots

Every App observation captures the current screenshot internally, independent
of whether AX returns full state, a diff or no-change. The default return omits
the image. `includeScreenshot: true` exposes it when visual inspection is needed.
Prefer the text-only default when it identifies the controls and confirms the
requested change. Request an image to resolve missing or ambiguous information,
choose coordinates or verify an appearance that AX does not describe.

```js
var state = await app.getState({ includeScreenshot: true });
nodeRepl.write(state.text);
for (const image of state.screenshot?.images ?? []) {
  await nodeRepl.emitImage(`data:${image.mimeType};base64,${image.dataBase64}`);
}
```

Include connection cleanup at the end of the call that emits the final
verification. Inspect that result before reporting success; reconnect and
continue if it reveals unfinished work. A separate cleanup-only model turn is
unnecessary:

```js
await computer.close();
globalThis.computer = undefined;
```

Reset the Node REPL only when no other persistent state is needed.