OpenAI

Two surfaces over one translation — the Responses API's computer tool, and the Agents SDK's Computer.

@raster/openai ships two adapters over the same nine-action translation: computerUse for the Responses API’s computer_use_preview tool, and agentComputer for the Agents SDK’s computerTool. Neither owns a conversation, and neither is imported by the canonical SDK.

npm install @raster/openai openai

The Responses API

import OpenAI from "openai";
import { Client } from "@raster/sdk";
import { computerUse } from "@raster/openai";

const machine = await new Client().machines.create({ size: "standard" });
const display = await machine.waitForDesktop();

// the display the adapter declares is the display it just measured, so the
// coordinate space the model reasons in cannot drift from the real one.
const adapter = await computerUse(machine, { display });

const model = new OpenAI();
let input = [{ role: "user", content: task }];
let previousResponseId;

for (let turn = 0; turn < 24; turn++) {
  const response = await model.responses.create({
    model: "computer-use-preview",
    tools: [adapter.tool],
    input,
    truncation: "auto", // computer use needs this: a screenshot per turn
    ...(previousResponseId ? { previous_response_id: previousResponseId } : {}),
  });

  previousResponseId = response.id;
  const next = await adapter.handleAll(response);
  if (next.length === 0) break;
  input = next;
}

adapter.tool is sized to the machine’s real display. handle returns the input items to send back — the computer_call_output carrying a fresh screenshot, plus, when something went wrong, one user message saying what. The wire format has no error field on a computer output, so a failure that is not said out loud is a model staring at an unchanged screen with no idea why.

Safety checks

const adapter = await computerUse(machine, {
  display,
  onSafetyCheck: (call) => askAHuman(call),
});

There is no default yes. A pending safety check is the API asking a human-owned question, and a library that answers it for everyone has removed the only point of asking. Calls whose checks are not acknowledged are answered without being performed.

Other options: environment (what the model is told it is driving; the image is ubuntu), waitSeconds (the protocol carries no duration for a wait action), scrollPixelsPerClick, and sleep.

The Agents SDK

import { Agent, computerTool, run } from "@openai/agents";
import { agentComputer } from "@raster/openai";

const agent = new Agent({
  name: "operator",
  model: "computer-use-preview",
  instructions: "You drive a real Linux desktop. Take a screenshot whenever you need to look.",
  tools: [
    computerTool({
      computer: await agentComputer(machine, { dimensions: [display.width, display.height] }),
    }),
  ],
  modelSettings: { truncation: "auto" },
});

const result = await run(agent, "Open Chromium, load https://example.com, and read the heading.");

agentComputer returns a machine as the Agents SDK’s Computer interface. The SDK runs the loop itself, so unlike the Responses adapter there is nothing to call per turn: it asks for actions and this performs them. Same nine actions, same translation, different door.

The package depends only on @openai/agents-core types — install @openai/agents yourself to run it.

Options are the same minus the safety-check hook, which the Agents SDK handles itself: dimensions (skips the display lookup when you already know the geometry), environment, waitSeconds, scrollPixelsPerClick, sleep.

Adding a shell

The computer tool covers the GUI. One canonical tool from the SDK gives the model a shell:

const generic = machine.tools.generic();
const exec = generic.definitions.find((d) => d.name === "terminal_exec")!;

const tools = [
  adapter.tool,
  {
    type: "function",
    name: exec.name,
    description: exec.description,
    parameters: exec.input_schema,
    strict: false,
  },
];

The adapter does not touch your own function calls. Handle them as usual and push the outputs alongside what handleAll returned:

const next = await adapter.handleAll(response);

for (const item of response.output) {
  if (item.type !== "function_call") continue;
  const result = await generic.execute(item.name, JSON.parse(item.arguments));
  next.push({
    type: "function_call_output",
    call_id: item.call_id,
    output: result.content.map((p) => (p.type === "text" ? p.text : "[image]")).join("\n"),
  });
}

Also exported

perform, describe and DEFAULT_RUNTIME are the shared translation the two adapters sit on, and keyName / keyNames map OpenAI’s key vocabulary onto the guest’s. Reach for them if you are building a third surface rather than using one of these two.

Working examples live at packages/openai/examples/computer-use.ts and packages/openai/examples/agents-sdk.ts.