The realtime gateway

Live screen, input, terminal and event channels over one websocket protocol.

Frames, input and pty bytes do not go through the API — they go through a websocket gateway the API mints a ticket for. The dashboard’s desktop, terminal and file views run on it, and so do machine.screen.stream() and machine.terminal.connect().

Getting a ticket

POST /v1/machines/{machine_id}/connect
  ->  { token, gateway_url, expires_at, capabilities }

The ticket is short-lived and carries no credential of yours. The gateway exchanges it server-side for the machine’s actual connection details, so the machine’s address and its guest bearer token never cross into a browser.

capabilities is the subset of screen:read, input:write and terminal:write this ticket was granted. A ticket is scoped to one machine and one session.

Connecting

wss://gateway.example/machines/{machine_id}/screen
wss://gateway.example/machines/{machine_id}/input
wss://gateway.example/machines/{machine_id}/terminal/{terminal_id}
wss://gateway.example/machines/{machine_id}/events

The token is presented as the websocket subprotocol:

Sec-WebSocket-Protocol: computer.v1, ticket.<token>

Not a query parameter: query strings land in access logs, proxy logs and browser history, and this token is a credential. Browsers cannot set headers on a websocket handshake, so the subprotocol field is the only place left.

Close codes

The gateway accepts the handshake and immediately closes it, rather than refusing at the HTTP layer — a browser cannot read the status of a rejected websocket handshake, so refusing there would leave a client with “it failed” and nothing else.

4503 is listed separately because it is not a client error. A client that treats an unreachable control plane as a bad ticket will mint a new one and fail the same way forever.

Framing

One rule across every channel, so a client needs one dispatcher:

• Text frames are JSON control messages, always an object with a type.
• Binary frames are payload — pixels or pty bytes, never JSON.

Unknown type values are ignored rather than fatal, so a newer gateway can add a message without breaking an older viewer.

The channels

screen

Server to client, capability screen:read. The gateway consumes the guest’s multipart MJPEG stream and republishes each part as one binary frame, so the browser never parses multipart. Each frame is a 20-byte little-endian header — magic CSF1, sequence, geometry, timestamp — followed by the image.

input

Client to server, capability input:write. All control messages, no binary frames: mouse.move, mouse.click, mouse.drag, mouse.scroll, key.type, key.press, key.hotkey, screen.resize.

Every message carries the lease fence. Coordinates are guest pixels, not CSS pixels — the viewer owns the mapping across scaling and letterboxing, because only the viewer knows how it is drawing, and a gateway that guessed would be wrong for every client that draws differently.

Renewal is not on this socket. Renew through the API and, if the fence changed, reconnect the input socket with a fresh ticket. See input control.

terminal

Bidirectional, capability terminal:write. Bytes go both ways unmodified. A terminal that already has a client refuses a second at the handshake with 4409.

events

Server to client. The machine’s event log as it happens, rather than polled.

Through the SDKs

const stream = await machine.screen.stream();
for await (const frame of stream) render(frame.image);
stream.close();

with machine.screen.stream() as stream:
    for frame in stream:
        render(frame.image)

Both mint the ticket, open the socket, decode the header and hand you frames. Screen streams open a session but never take the input lease, so watching a desktop cannot evict whoever is driving it.

In Python this needs the realtime extra: pip install "raster[realtime]". Most callers are agent loops that screenshot and click over ordinary HTTP and never open a socket, so the websocket dependency is opt-in rather than paid for by everybody.