Input

Mouse, keyboard, clipboard, and the single lease that governs all three.

Coordinates

Every coordinate is in the pixel space of the machine’s current display, and one outside it is rejected, not clamped: a click 200 pixels off the edge is a bug worth surfacing, not one to land somewhere plausible. Nothing is scaled for you.

Screenshot first, read the coordinates off the frame, then click.

Mouse

await machine.mouse.move(100, 200);

await machine.mouse.click({ at: { x: 640, y: 400 } });
await machine.mouse.click({ at: { x: 640, y: 400 }, button: "right" });
await machine.mouse.click(); // wherever the pointer already is
await machine.mouse.click({ at: p, down: true }); // hold across several calls

await machine.mouse.doubleClick({ x: 640, y: 400 }); // `at` is positional here
await machine.mouse.rightClick({ x: 640, y: 400 });

await machine.mouse.drag({ from: { x: 10, y: 10 }, to: { x: 200, y: 200 } });
await machine.mouse.scroll({ direction: "down", amount: 3, at: { x: 640, y: 400 } });

machine.mouse.move(100, 200)

machine.mouse.click(at={"x": 640, "y": 400})
machine.mouse.click(at={"x": 640, "y": 400}, button="right")
machine.mouse.click()                                   # wherever the pointer already is
machine.mouse.click(at=p, down=True)                    # hold across several calls

machine.mouse.double_click({"x": 640, "y": 400})        # `at` is positional here
machine.mouse.right_click({"x": 640, "y": 400})

machine.mouse.drag(to={"x": 200, "y": 200}, from_={"x": 10, "y": 10})
machine.mouse.scroll(direction="down", amount=3, at={"x": 640, "y": 400})

Buttons are left, middle and right. count: 2 is what doubleClick sends. drag takes an optional steps to shape the path; scroll defaults to amount: 3 and moves the pointer first when you give it an at.

There is no read-only pointer query. Moving the pointer to find out where it is defeats the question, which is why the Anthropic adapter withholds cursor_position rather than declaring a tool that cannot work.

Keyboard

await machine.keyboard.type("hello");
await machine.keyboard.type("slowly", { delayMs: 40 });
await machine.keyboard.press("Return");
await machine.keyboard.hotkey("ctrl", "l");

machine.keyboard.type("hello")
machine.keyboard.type("slowly", delay=0.04)
machine.keyboard.press("Return")
machine.keyboard.hotkey("ctrl", "l")

type is for text and press is for keys. Typing "Return" types six characters; pressing it sends the key. Key names are the guest’s own — Return, Tab, Escape, F6, BackSpace.

Clipboard

await machine.clipboard.write("hello");
await machine.clipboard.write("hello", { selection: "primary" });

const { text, selection } = await machine.clipboard.read();

machine.clipboard.write("hello")
machine.clipboard.write("hello", selection="primary")

clip = machine.clipboard.read()
clip["text"], clip["selection"]

X11 has two selections and both are addressable: clipboard (the default, what Ctrl-V pastes) and primary (what a middle click pastes). Reading is a read, so it needs no lease.

The input lease

At most one session may send input to a machine at a time. Anything that types, clicks, drags, scrolls, resizes the display or opens a URL acquires the lease first. Reads never do.

Both SDKs handle it for you:

await machine.keyboard.type("hello"); // acquires the lease if it does not hold one
machine.holdsInput; // true

machine.keyboard.type("hello")          # acquires the lease if it does not hold one
machine.holds_input                     # True

The lease is opened on first use with a 60-second TTL and renewed at half of it. machine.close() releases it.

A lost lease is never taken back

If a renewal fails, the SDK stops holding the lease and does not re-acquire it:

machine.holdsInput; // false after a takeover
await machine.keyboard.type("more");
// RasterError: conflict — someone else holds the input lease

machine.holds_input   # False after a takeover
machine.keyboard.type("more")
# RasterError: conflict - someone else holds the input lease

That is the important half. A lost lease means a person took over, and silently taking it back would put an agent and a person on the same keyboard, which is the one thing single-holder input exists to prevent. Catch the conflict and decide deliberately.

The SDKs never send force. Taking a lease from its holder is human takeover — a deliberate act by a person, not something a library does on a click. To do it on purpose, see take input control.

Not everything is leased

Terminals are outside the lease entirely, and so are screen streams. A viewer that cannot type can still see.