vercel-labs/agent-desktop

agent-desktop

Computer use for AI agents: read and operate desktop apps through the OS accessibility tree.

A CLI and C library for macOS, Windows and Linux. A snapshot returns JSON with a ref for every element an agent can act on. The agent acts on a ref and snapshots again to check, with no pixel matching; the loop stays in your agent.

Every CLI call runs in its own process. Headless calls still prohibit foreground activation and synthesized input.

Snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content. Keep them out of public logs and uploads. The optional JEV helper sends screen descriptions to a configured model provider; it is separate from the native CLI. See privacy and data handling.

Installation

Install on macOS, Windows or Linux with npm:

npm install -g agent-desktop
agent-desktop version

Bun, Yarn and pnpm can install the same package. It includes native binaries and platform helpers for x64 and ARM64. For direct execution without a JavaScript launcher, download the matching native archive from GitHub Releases and keep its helper beside the executable.

Official Homebrew and Scoop packages are being submitted; they are not yet available in the default catalogs. The repository no longer maintains a custom Homebrew tap or Scoop bucket.

• macOS needs Accessibility permission: agent-desktop permissions --guide.
• Windows needs no grant for apps at your terminal's integrity level.
• Linux needs the user's graphical session with at-spi2-core.

See installation and permissions for package-manager behavior, release archives, and building from source.

Quick start

Observe, act on a ref, observe again. Any app name from agent-desktop list-apps works; Finder is the macOS example. Each command prints one JSON envelope; a trimmed snapshot follows the commands.

agent-desktop snapshot --app Finder -i      # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8            # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i      # observe again to verify

{
  "version": "2.6",
  "ok": true,
  "command": "snapshot",
  "data": {
    "app": "Finder",
    "window": { "id": "w-204749", "title": "Downloads" },
    "snapshot_id": "s8f3k2p9",
    "ref_count": 59,
    "complete": true,
    "tree": {
      "role": "window",
      "name": "Downloads",
      "children": [
        { "role": "treeitem", "name": "Documents", "ref_id": "@s8f3k2p9:e8", "available_actions": ["Click"] }
      ]
    }
  }
}

• Refs belong to their snapshot. A bare @e8 is rejected with INVALID_ARGS. When the UI changes, an action returns STALE_REF or AMBIGUOUS_TARGET: take a new snapshot and use its refs.
• Several windows give AMBIGUOUS_TARGET; snapshot lists the candidate window ids in error.details. Pass one with --window-id.
• Dense apps: map first with snapshot --skeleton --app Slack -i --compact, then drill into one region with snapshot --root @s8f3k2p9:e3 -i --compact.
• Headless by default. Ref actions go through the accessibility API without moving the pointer or taking focus. The global --headed flag allows physical input.
• PowerShell. Quote every ref ('@s8f3k2p9:e8'), because a bare @token is the splatting operator.

For AI agents

The binary carries its own docs, matched to its version:

agent-desktop skills get desktop      # the loop, refs, verification and retry rules
agent-desktop skills get platform     # what differs on the OS you run on
agent-desktop <command> --help        # flags for one command

Install the skill into an agent harness from skills.sh:

npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop

Or add this to your AGENTS.md:

Use `agent-desktop` to see and operate desktop apps. Before the first use, run
`agent-desktop skills get desktop` and `agent-desktop skills get platform`, and follow them.

Every command prints one envelope with ok and either data or error. An error carries a code, a disposition and usually a suggestion; retry only when error.disposition.retry is safe. The contract, error codes and exit codes are in docs/json-output.md.

Commands

It ships 60 command names, 56 operational commands. The other four (key-down, key-up, mouse-down, mouse-up) refuse to run until a daemon can release a held key or button.

Platform support

The Windows surface and launch notes and the Linux X11 and Wayland notes are in docs/install.md.

agent-desktop skills get platform prints the full capability table for the OS you run on.

Language bindings

Each release also ships libagent_desktop_ffi for macOS, Linux and Windows, so a program can call the tool in process instead of starting the CLI for each command. agent-desktop skills get ffi covers the entry points, memory ownership and threading rules.

import ctypes
lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib")
lib.ad_adapter_create.restype = ctypes.c_void_p
lib.ad_adapter_destroy.argtypes = [ctypes.c_void_p]
assert lib.ad_init(7) == 0  # AD_RESULT_OK when the header's AD_ABI_VERSION_MAJOR matches the library
adapter = lib.ad_adapter_create()
lib.ad_adapter_destroy(adapter)

Development

Each platform adapter builds only on its own OS, so scope cargo to the host's packages. On macOS:

HOST_PKGS=(-p agent-desktop-core -p agent-desktop-macos -p agent-desktop -p agent-desktop-ffi)
cargo test "${HOST_PKGS[@]}"
cargo clippy "${HOST_PKGS[@]}" --all-targets -- -D warnings

On Windows, replace agent-desktop-macos in that list with agent-desktop-windows; on Linux, with agent-desktop-linux. CONTRIBUTING.md lists every gate. Build from source: docs/install.md.

Documentation

License

Apache-2.0