open-compute

<img src="assets/banner.png" width="100%" alt="open-compute banner"/>

EN | DE

A model-agnostic computer-use core: one agent loop, any reasoning model behind a single interface.

open-compute is a small, dependency-light Python core for building computer-use agents (LLM-driven GUI / desktop / browser automation). It implements the perception → model-tool-call → action → feedback loop and keeps the reasoning model swappable behind a single ComputerBackend interface. No provider is privileged: Anthropic Claude and OpenAI CUA are two equally-ranked API backends, and the offline mock backend is the default. A keyless path also exists today via Mode A, where the host model itself reasons — and it can run that loop either inline or in a self-spawned subagent for context economy (see usage pattern). The core has zero runtime dependencies; vendor SDKs (anthropic, openai) are optional, lazily imported extras — import open_compute works with none of them installed, and the default mock wiring runs fully offline.

[!NOTE] AI / LLM Integration Notice: open-compute includes a machine-readable llms.txt file designed for AI agents, RAG crawlers, and LLM-assisted workflows.

---

Why

Every computer-use model — Anthropic's Claude computer tool and OpenAI's computer-use tool — shares the same agent-loop shape but differs in transport, coordinate frame, and action names. open-compute factors out the common parts so you write the loop once and swap the reasoning model freely behind one ComputerBackend interface:

• A canonical action schema with one mapper per backend.
• Normalized (0..1) coordinates internally, denormalized per backend / resolution / DPI in one tested utility — the DPI problem solved centrally.
• A central safety gate ("confirm before risky actions") evaluated before every action.
• A hybrid perception interface (screenshot + Set-of-Marks / accessibility / DOM), so you can move from pure pixel-vision to semantic targeting later.

---

Architecture

                        +-----------------------------------------+
                        |        AGENT LOOP / ORCHESTRATOR        |
                        |  goal -> perceive -> backend -> safety  |
                        |        -> execute -> re-perceive        |
                        +-------------------+---------------------+
                                            |
        +-----------------------------------+-----------------------------------+
        |                                   |                                   |
+-------v---------+              +----------v-----------+            +----------v----------+
| PERCEPTION      |              | CANONICAL ACTIONS    |            | SAFETY / POLICY     |
| - screenshot    |              | click/type/key/      |            | - confirm-at-action |
| - set-of-marks  |              | scroll/drag/wait/    |            | - allow / deny list |
|   (OmniParser)* |              | screenshot + OS ext  |            | - read-only mode    |
| - accessibility*|              | (launch/activate)    |            | - audit log         |
+-------+---------+              +----------+-----------+            +----------+----------+
        |                                   |                                   |
        +-----------------+-----------------+----------------------------------+
                          |
              +-----------v------------+   COORDINATE / DPI NORMALIZATION
              | BACKEND ABSTRACTION    |   - internal: normalized (0..1)
              | (ComputerBackend)      |   - denormalize per backend:
              +-----+--------+---------+     * Claude: global px (display_w x display_h)
                    |        |    |          * OpenAI: px (computer_call)
        +-----------+        |    +-----------+   * Mock: synthetic
        |                    |                |
+-------v-------+   +--------v-------+  +-----v---------+
| Claude        |   | OpenAI CUA     |  | Mock backend  |
| computer_2025 |   | computer-use-  |  | (no SDK,      |
| 1124 + beta   |   | preview [?]    |  |  offline)     |
| (host runs)   |   | (host runs)    |  |               |
+---------------+   +----------------+  +---------------+

  * = stub / interface in this release (see Status)

---

Install

[!IMPORTANT] Not on PyPI — install from Git. This project has no PyPI release yet. The name open-compute on PyPI is taken by an unrelated project ("multi-agent systems for healthtech"), so a plain pip install open-compute installs someone else's package. Always install from this repository:

pip install "git+https://github.com/ellmos-ai/open-compute.git"                        # core only, zero runtime deps
pip install "open-compute[claude] @ git+https://github.com/ellmos-ai/open-compute.git" # + anthropic SDK

The same extra @ git+… form works for every extra below:

| Extra | Adds | |---|---| | claude | anthropic SDK | | openai | openai SDK | | local | mss — real Windows screenshots + input | | wgc | WGC fallback for DirectX surfaces (pulls numpy/OpenCV) | | compose | Pillow — Before|After composite + annotated shots | | watch | watchdog — native FS events for the directory-watch feed | | clirec | external clirec package for oc rec workflows | | record | clirec[record] capture backend compatibility | | mcp | mcp SDK — MCP server (console script: open-compute-mcp) | | dev | pytest | | all | anthropic, openai, playwright, mss, WGC, Pillow, watchdog, clirec, mcp |

Extras combine as usual, e.g. open-compute[local,wgc,claude]. Working from a clone instead? pip install -e ".[local,claude]" from the repository root.

Until clirec has a package release, install it directly when using oc rec:

pip install git+https://github.com/ellmos-ai/clirec.git

Python 3.10+.

---

Quick start

Mode A — No API key: session-agent as reasoner (chat skill)

Run oc capture / oc do manually from a Claude Code session. The session model sees the PNG via the Read tool and decides the next action:

# 1. Install the local extra (Windows only; provides real screenshots + input)
pip install "open-compute[local] @ git+https://github.com/ellmos-ai/open-compute.git"

# 2. Capture a screenshot — saved automatically to _session/ (never loose on Desktop)
oc capture
# -> {"path": ".../_session/0001_20260620_143200.png", "width": 1920, "height": 1080}
# Then: read the PNG with your Read tool to see the screen.

# 3a. Execute one canonical action (single, backwards-compatible)
oc do '{"type":"mouse_move","x":0.5,"y":0.5}' --mode allow_all
oc do '{"type":"left_click","x":0.25,"y":0.1}' --yes   # --yes = agent pre-approved

# 3b. Execute with Before|After composite (Pillow optional)
oc do '{"type":"left_click","x":0.5,"y":0.3}' --label "click_ok" --yes
# -> {"result":"executed","action":"left_click","composite":"_session/0002_click_ok.png"}

# 3c. Execute a batch/macro (JSON array, one call = multiple actions)
oc do '[{"type":"mouse_move","x":0.5,"y":0.5},{"type":"left_click","x":0.5,"y":0.3}]' --yes
# -> {"result":"batch","count":2,"width":1920,"height":1080}

# 3d. Ensure the target window is in the foreground before acting
oc do '{"type":"left_click","x":0.5,"y":0.3}' --ensure-foreground "Word" --yes

# 3e. Save a full-res after-shot + annotated click marker (v0.5, Pillow optional)
oc do '{"type":"left_click","x":0.5,"y":0.3}' --yes --fullres
# -> {"result":"executed",...,"fullres_annotated":"_session/...fullres.png"}

# 3f. Capture only the active window's bounding rect (v0.5, Windows)
oc capture --window "Word"
# -> {"path":"...","width":800,"height":600,"window":"Word","region":{...}}

# 3g. Watch a directory for changes (v0.5)
oc watch-dir ~/Downloads --for 5       # collect 5 s, print JSON events
oc watch-dir ~/Downloads --once        # one-time snapshot diff

# 3h. Explicit companion handoff (mutations need a granted, scoped lease)
oc session companion --owner local-user
oc session request-control --owner agent-a --scope window:42 --ttl 60
oc session grant --lease-id <lease_id-from-previous-output>
oc window minimize --hwnd 42 --yes

# 3i. Bounded, deduplicated window capture (full screen needs explicit opt-in)
oc capture-series --window "Word" --max-frames 8 --stable-frames 2

# 4. Recapture and repeat until done (or read the "composite" After-shot directly).

See SKILL.md for the full loop protocol, action schema, coordinate guide, and environment variable reference.

Mode B — Autonomous loop with an API backend

The backend is selected by name; claude and openai are equally supported (each needs its own key + extra). For a keyless path, use Mode A above — the host model reasons itself, optionally in a self-spawned subagent (see usage pattern).

# Claude (needs ANTHROPIC_API_KEY + open-compute[local,claude]):
oc run "Find the latest invoice in the Downloads folder" --backend claude --max-steps 15

# OpenAI (needs OPENAI_API_KEY + open-compute[local,openai]):
oc run "Find the latest invoice in the Downloads folder" --backend openai --max-steps 15

Or in Python — get_backend(name, ...) builds whichever you name; inject your own executor or use LocalExecutor:

from open_compute import AgentLoop, Config, get_backend
from open_compute.drivers.local import LocalExecutor   # Windows; needs mss
from open_compute.safety import SafetyPolicy

executor = LocalExecutor()   # real display + input
config = Config(backend="claude", scope="os",
                display_width=executor.width, display_height=executor.height)
backend = get_backend("claude", executor.width, executor.height, model="claude-opus-4-8")

loop = AgentLoop(
    config,
    backend=backend,
    executor=executor,
    policy=SafetyPolicy(mode="confirm",
                        confirm_callback=lambda a: input(f"run {a.type.value}? [y/N] ") == "y"),
)
loop.run("Find the latest invoice in the Downloads folder")

Offline dry-run (no API key, no display, mock only)

from open_compute import AgentLoop, Config

loop = AgentLoop(Config(backend="mock", safety_mode="allow_all"))
result = loop.run("Open the settings page and enable dark mode")
print(result.done, result.steps)
for trace in result.traces:
    print(trace.index, trace.backend_message, [a.type.value for a in trace.executed])

---

MCP server (native tool-calls, keyless)

Expose the keyless Mode A loop to any MCP client as native tools — the client is the reasoner (no API key, model-agnostic). Versus driving oc by hand, a long-lived server keeps one warm LocalExecutor resident (no Python restart per action) and returns screenshots as MCP image blocks. Windows-only for real capture/input.

pip install "open-compute[mcp,local,uia,wgc] @ git+https://github.com/ellmos-ai/open-compute.git"
open-compute-mcp          # stdio server (console script)

Tools: capture · do (single or