CopperOS

<img width="1230" height="454" alt="CopperOS-Banner" src="https://github.com/user-attachments/assets/188a612f-5722-4d58-9184-9ea006a7e903" />

CopperOS is an open-source AI agent for Chrome. You type a task in a side panel, and it reads, clicks and types in your own tabs, signed in as you. Every step shows in the panel, and it asks before it submits anything.

Add to Chrome · Website · Setup guide · Pricing

Run it on CopperOS's hosted model (sign in with Google, free weekly trial, no key needed), or on your own computer with the model you choose: Ollama, OpenRouter or OpenAI. See Known limits before assuming something is a bug, and Contributing if you'd like to help.

Perception is accessibility-tree first (cheap, exact), with badged screenshots escalated automatically only when the tree goes blind. Input goes through CDP so events are genuinely trusted rather than synthesized in page JS.

┌─────────────────────┐   WebSocket    ┌──────────────────────┐
│  broker/  (Node/TS) │◄──────────────►│  extension/  (MV3)   │
│  • Ollama /v1 API   │  {id,op,params}│  • chrome.debugger   │
│  • tool definitions │                │  • ref → coordinates │
│  • agent loop       │  {id,ok,data}  │  • humanized input   │
│  • holds API key    │                │  • AX snapshots      │
└─────────────────────┘                └──────────────────────┘

The extension is hands only — it holds no config and makes no decisions. The broker is brains only — it never sees a pixel coordinate.

The model is reached through Ollama's OpenAI-compatible endpoint (/v1) rather than the native /api/chat. The compat layer gives real tool_call ids, so parallel tool calls pair up unambiguously, and it accepts image blocks inside tool-result messages — which the vision escalation depends on.

Setup

Step-by-step guide: copper.jassydev.com/setup.

You need Node.js 20 or newer and a model that supports tool calling — an OpenRouter or OpenAI API key, or a model in Ollama. Vision is optional: it lets the agent use screenshots when the accessibility tree goes blind.

git clone https://github.com/Jesse0502/CopperOS.git
cd CopperOS/broker
npm install
npm start

Then load the extension: chrome://extensions → enable Developer mode → Load unpacked → select extension/. Keep only one copy of CopperOS enabled — the broker serves one client at a time (see Troubleshooting).

Click the CopperOS toolbar icon to open the side panel. It opens on the chat, whoever you are. On the hosted service (see The hosted service) nobody has to sign up before they have seen it: the first message they send brings up a popup asking them to Continue with Google, and the message goes as soon as they are in. If a broker is running on this computer, the extension notices (a plain HTTP request to port 7331, every few seconds while the panel is on the cloud; a WebSocket would knock out the broker's live client) and asks, signed in or not: a popup says CopperOS is running on this computer and offers Switch to this computer or Stay in the cloud. Switching lasts as long as that broker runs: if it stays gone for 15 seconds, the panel goes back to the cloud, where a sign-in is kept. Staying, or choosing Cloud in Settings, is not asked again until the broker stops and starts again. While a broker is running (or the panel is on it), Settings shows Cloud / This computer at the top; a choice made there is remembered. The dot in the panel's header is green when connected, grey when nobody is signed in, red when the broker or the cloud cannot be reached; a broker chosen in Settings that is not running shows how to start it. Choose a provider, key and model under Settings, then type a task.

Using Ollama

Pick a model with tools capability (ollama show <model> lists them):

ollama serve                  # or just run the desktop app
ollama list                   # what you already have
cp .env.example .env          # optional: set OLLAMA_MODEL to a model from `ollama list`

With Ollama as the provider, the broker checks it at startup and warns if it is unreachable or if the model is not installed, rather than failing mid-task.

CLI alternative:

npm run task -- "find the pricing page and tell me the cheapest paid tier"

How perception works

snapshot returns the accessibility tree as ref-tagged lines:

[e11] link "Products"
[e12] textbox "Search" value=""
[e28] button "Sign in"

The model acts with click(ref: "e28"). The extension resolves the ref to a backendNodeId, gets a live box model, and drives the pointer there. The model never handles coordinates, so it cannot invent one.

A ref names one element for as long as it is on the page, across snapshots, and refs are kept per tab. A navigation clears the table, but numbering carries on, so a ref from the page before raises an unknown ref error instead of pointing at some other element. The error goes back to the model as a normal tool result, and it re-snapshots. That is the intended recovery path, not a failure.

Every action reports what it changed. Since refs are stable, click, paste, press_key and the rest re-read the page afterwards and return the difference: + for what appeared, ~ for an element whose line changed (a value, checked, expanded), - for what is gone. After a navigation, or a change too big to read as a list, the result carries the whole page instead. The model no longer spends a turn on a snapshot after every action, and it can fill several fields in one response.

Actions wait for what they set off before reporting: the page going quiet (300ms with no requests that matter), or a new tab opening. A click that changes nothing is back in about half a second. Filling a field without submitting sets nothing off, so it does not wait at all.

Vision escalates automatically. snapshot scores the tree and attaches a badged screenshot when it looks unreliable: >30% of interactive elements unlabeled, a <canvas> present, or nothing interactive at all. Badge N on the image is ref eN — one vocabulary across both channels. Badges are drawn on the captured bitmap in an OffscreenCanvas, never injected into the page, so nothing shifts under the click you are about to make.

A text-only model gets no screenshots at all: OpenRouter refuses a request with an image in it for such a model ("No endpoints found that support image input"). The broker checks OpenRouter's model catalog, or Ollama's /api/show, before each task. Without a screenshot, an unreliable snapshot comes with a note to wait and snapshot again. When a provider can't say (OpenAI), the first refusal marks the model as text-only, and the request is resent without its images.

This is the main cost lever: an accessibility snapshot runs 2–5KB against 100KB+ for a screenshot of the same page.

Google Sheets

Sheets draws its grid on a canvas, so the accessibility tree has the toolbar, the menus and the sheet tabs, and not one cell. On a sheet, three tools reach the cells another way:

• sheet_read fetches the sheet's own CSV export with your existing Google login and renders it as a table with column letters and row numbers. A snapshot of a sheet starts with a preview of A1:J20 rendered the same way, in place of a screenshot.
• sheet_write enters a whole block of values in one call. It goes to the first cell through the Name box, types each value (the first character as a real key press, which opens the cell for editing), and presses Tab and Enter between cells. Then it reads the block back and lists any cell that does not show what was written, such as a value Sheets autocompleted from the column.
• sheet_select selects a cell or range through the Name box, so toolbar buttons, menus and press_key shortcuts (Delete, Mod+b, Mod+z) apply to it.

press_key takes modifiers anywhere: Shift+Tab, Mod+z (⌘ on a Mac, Ctrl elsewhere), Alt+Enter, with an optional repeat. Copy, cut and paste shortcuts are refused, because they would use your clipboard. type and paste can leave out the ref to write into whatever has focus.

Configuration

broker/.env:

| Variable | Default | Meaning | |---|---|---| | OLLAMA_MODEL | minimax-m3:cloud | Must support tools; vision too, or escalated screenshots are wasted. | | OLLAMA_HOST | http://127.0.0.1:11434 | Ollama server. /v1 is appended. | | OLLAMA_NUM_CTX | 32768 | Context window. Ollama's own default is far too small here — one snapshot can be several thousand tokens. | | OPENROUTER_API_KEY | — | Enables OpenRouter as a provider. Choose it in the extension's Settings page. | | OPENROUTER_MODEL | deepseek/deepseek-v4.1-flash | Any OpenRouter model id that supports tools. | | APPROVAL_MODE | submits | all gates every click, submits gates form submissions and clicks the model flags destructive, none gates nothing. | | PORT | 7331 | Broker WebSocket port. Must match BROKER_URL in extension/background.js. | | JEV_AI_API_KEY | — | Enables Jev: task rounds, check-ins, memory ranking and the other judgments. Nothing else depends on it; loops run without it. | | TASK_MAX_ROUNDS | 15 | Most fresh-context rounds one message can start while Jev says there is more to do. | | SUPERVISOR | off | on starts new chats with the supervisor on. Each chat's own setting is the Supervisor button in the panel. | | CHECK_IN_EVERY | 10 | Steps between check-ins on a tracked task. 0 turns them off in every chat, whatever the Supervisor button says. | | FRESH_CONTEXT_TOKENS | 25000 | Context size at which a check-in hands the task over to a fresh round. |

Answers and tasks

With Jev enabled, each message is sorted first: work to do, a question or comment the agent can answer from the chat ("how many did you apply to?", "why did you skip that one?"), or small talk. The last two get a reply in words with the agent's tools switched off. The task on record is left as it was, so a follow-up question never starts browsing or replaces the task. A message that may carry the task on, including a complaint about how it's going ("I asked you to colour the rows"), counts as work. If an answer really needs a page, the agent says so, and "go ahead" sends it to look.

Loops

"Keep applying until I tell you to stop", "do this nonstop" or just "loop" makes a task a loop: it repeats until you press Stop. Rounds end and fresh ones start from a brief, as with any long task, but nothing judges whether it is finished. A question for you (or a detail it doesn't know) is skipped and recorded instead of waited on, so a loop never sits idle on one. An action that needs your approval is still put to you, and the loop carries on once you answer. "Continue" picks a stopped loop back up. A message that says when to finish ("50 more, then stop", "for the next 30 minutes") turns it back into a task that ends. A loop also stops itself after 5 rounds in a row that record nothing (LOOP_STALLED_ROUNDS), because by then it is stuck.

With Jev, classifyIntent decides whether a message asks for a loop, sets an end, or carries on the task on record. Without Jev, or when Jev doesn't answer, the message's plain words decide (loopFromWords in jev.ts): "loop", "don't stop", "nonstop" and "until I tell you to stop" start one; "stop the loop", "don't continue" and questions like "what does the loop do?" never do. Loops are the one kind of task tracked without Jev, since only you end one. A loop started that way carries your last few messages into every round's b