The desktop
Screenshots, pointer, keyboard, clipboard, and the display geometry every coordinate is in.
The desktop is an ordinary Linux desktop, driven the way a person would drive it. The image
ships an agent user that owns the session, and a root user.
Coordinates
Every coordinate is in the pixel space of the machine’s current display. A coordinate outside it is refused, not clamped: a click 200 pixels off the edge is a bug worth surfacing, not one to land somewhere plausible.
Nothing is scaled for you. Take a screenshot, read the coordinates off it, click them.
const shot = await machine.screen.screenshot({ format: "png" });
shot.display.width; // the geometry this image was captured atResizing changes that space, so a screenshot taken before a resize is stale for clicking.
Screen
| Call | What it does |
|---|---|
GET /machines/{id}/screenshot |
capture the screen; png, jpeg or webp |
GET /machines/{id}/display |
width, height, and whether the desktop is up |
POST /machines/{id}/display/resize |
change the display geometry |
A screenshot comes back base64 in image, with the display it was taken at alongside it, so a
result is self-describing and a coordinate can be checked against the frame that produced it.
quality applies to the lossy formats.
For continuous frames rather than one at a time, use the screen channel.
Pointer
| Call | Notes |
|---|---|
POST /machines/{id}/mouse/move |
absolute x, y |
POST /machines/{id}/mouse/click |
at optional — omit it to click where the pointer is |
POST /machines/{id}/mouse/drag |
from optional, to required, steps for the path |
POST /machines/{id}/mouse/scroll |
direction, amount (default 3), optional at |
Buttons are left, middle and right. count: 2 is a double click; the SDKs spell that
doubleClick. down and up let you hold a button across several calls.
There is no read-only pointer query. Moving the pointer to find out where it is defeats the
question, so the Anthropic adapter withholds cursor_position rather than declaring a tool that
cannot work.
Keyboard
| Call | Notes |
|---|---|
POST /machines/{id}/keyboard/type |
a string, with an optional per-character delay |
POST /machines/{id}/keyboard/key |
one key by name, e.g. Return, Tab, F6 |
POST /machines/{id}/keyboard/hotkey |
a combination, e.g. ctrl + l |
type is for text and key is for keys. Typing "Return" types seven characters.
Clipboard
await machine.clipboard.write("hello");
const { text, selection } = await machine.clipboard.read();X11 has two selections and both are addressable: clipboard (the default, what Ctrl-V pastes)
and primary (what a middle click pastes). Reading is a read, so it needs no input lease.
The input lease
Anything that types or clicks needs the machine’s single input lease. Reads — screenshots, display geometry, clipboard reads — never do. Both SDKs acquire and renew the lease for you on the first call that needs it, and never re-acquire one they lost.
That last part is the point: a lost lease means a person took over, and silently taking it back would put an agent and a person on the same keyboard. See sessions and input control.