Vercel Labs has open-sourced Agent Desktop, a Rust command-line tool and C library that lets AI agents inspect and operate native applications through operating-system accessibility trees. It supports macOS, Windows and Linux under the Apache-2.0 license, and its core workflow requires no screenshot matching or bundled vision model. The repository documents npm installation and native release archives.
The October 30, 2026 release is version 1.0.2. The official changelog dates the integrated 1.0.0 release, which added Linux support, to October 27. Version 1.0.2 improves macOS permission guidance and warns npm users when a JavaScript launcher is adding overhead to each call.
The practical distinction is how an agent identifies its target. Instead of interpreting an image and choosing screen coordinates, it receives structured JSON describing accessible UI elements, then acts through an element reference. That changes the targeting problem, but application coverage, state verification and agent decision-making remain important engineering work.
Accessibility Trees Give Agents Named Targets
Operating systems expose accessibility interfaces so software can inspect controls such as buttons, text fields, menus and checkboxes. Agent Desktop uses those interfaces to build a machine-readable view of an application.
Its platform adapters use Apple’s accessibility API on macOS, UI Automation on Windows and AT-SPI on Linux, as described in MarkTechPost’s technical overview. The official documentation describes the resulting snapshot as JSON with references for elements the agent can operate.
For builders, the difference looks like this:
| Practical question | Screenshot-coordinate workflow | Agent Desktop’s structured workflow |
|---|---|---|
| What does the agent inspect? | A rendered image of the interface | An accessibility-tree snapshot |
| How is a target selected? | Visual interpretation and coordinates | A snapshot-qualified element reference |
| What information is available? | Visible pixels and inferred meaning | Exposed roles, labels, values and structure |
| What happens when layout changes? | Coordinate targeting may need to be recalculated | Target resolution uses the reference, subject to validity checks |
| What limits coverage? | Visual interpretation and input delivery | The app’s accessibility implementation and supported actions |
This comparison covers mechanisms, not a measured ranking of products.
An accessibility tree can make a named button easier to address even after a window moves. Conversely, a custom canvas with little accessible structure may offer very few usable targets. A screenshot can still reveal content that the accessibility interface fails to expose.
My view is that accessibility coverage should be the first integration check. A structurally rich app is a promising target; a visually elaborate interface with sparse accessibility data may still require another automation method. Agent Desktop also offers screenshots separately, but its documented core workflow does not require them. The project FAQ explains that separation.
Stable References Still Have a Lifetime
Agent Desktop assigns qualified references such as @s8f3k2p9:e1 to interactive elements. The first portion identifies the snapshot; the final portion identifies an element within it. A bare reference such as @e1 is rejected with INVALID_ARGS, according to the FAQ.
That qualification matters. An agent cannot safely assume that “element 1” from one observation means the same thing in another.
The documented control loop is:
- Take a snapshot of the application.
- Select an element from its structured output.
- Perform an action using its qualified reference.
- Take another snapshot to inspect the result.
- If the reference is stale, obtain a fresh snapshot before retrying.
References are also scoped. A reference created within a session still requires the corresponding --session or AGENT_DESKTOP_SESSION context. Including a snapshot ID in the reference does not bypass that namespace boundary.
Stable does not mean permanent. Elements can disappear, and changes can make resolution ambiguous. The changelog documents STALE_REF and AMBIGUOUS_TARGET outcomes, including changes that preserve those specific errors instead of reducing them to a generic timeout.
The current README describes a further macOS safeguard: CLI calls reuse a private command host that retains native accessibility objects. This is intended to prevent a deleted row from silently resolving to a surviving row with the same identifier. The host exits after 15 idle minutes.
That detail carries more weight than the reference syntax. A failed lookup gives an agent a chance to recover; silently selecting the wrong surviving row could produce a valid-looking action on the wrong item. Builders should make stale-reference recovery an explicit branch in their agent loop.
These are documented behaviors, not results from hands-on testing here. The supplied sources contain no independent benchmark establishing reliability across diverse desktop applications.
Install Through npm, Then Check Permissions
The documented installation works through npm:
npm install -g agent-desktop
agent-desktop version
The first command installs the CLI globally. The second reports the installed version, which is useful when matching behavior against the changelog.
Bun, Yarn and pnpm can install the same package. It includes native binaries and platform helpers for x64 and ARM64. Developers who want direct execution without a JavaScript launcher can download the matching native archive from GitHub Releases; the documentation says to keep its helper beside the executable.
Permissions and session requirements differ by platform:
| Platform | Documented requirement | Deployment consequence |
|---|---|---|
| macOS | Accessibility permission | The launching application needs the appropriate grant |
| Windows | No separate grant for apps at the terminal’s integrity level | Access is bounded by the terminal’s privilege context |
| Linux | The user’s graphical session and at-spi2-core | A generic server shell alone is insufficient |
On macOS, the permission helper is:
agent-desktop permissions --guide
Version 1.0.2 improves that guide so it offers the application that launched Agent Desktop, including terminals LaunchServices does not know about. It also restarts the command host after a missing-permission response, allowing subsequent commands to work after access is granted. These fixes are recorded in the changelog.
For Linux deployment, the 1.0.0 entry specifies x86_64 and aarch64 support with glibc 2.35 or later. That is worth checking before choosing a base image.
Homebrew and Scoop Are Still Pending
The current README says official Homebrew and Scoop packages are being submitted and are not yet in the default catalogs. The project no longer maintains its former custom Homebrew tap or Scoop bucket.
For now, deployment instructions should point to npm or native archives, not assume an official package-manager route exists.
Frequently Asked Questions
4 questions
1Does Agent Desktop Need a Vision Model?
No. Its core workflow uses accessibility-tree snapshots and element references. The native CLI does not require a remote model service; the calling agent supplies its own decision-making.
