computer-use-automation
Decides when a GUI-driving (computer-use) agent beats an API and lints planned screen actions for missing confirmation, verification and grounding
Install / Use
npx skills add borghei/Claude-Skills --skill computer-use-automationInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AutomationSupported Platforms
Our assessment of computer-use-automation
computer-use-automation scores 87/100 on our quality scale, 1705th of 2,769 Automation skills we index.
Its SKILL.md is 5.9 KB long, well organised into 16 sections with 1 code example: a thorough specification that gives an agent plenty to work with.
It has 817 GitHub stars, a meaningful sign that others use it.
Maintenance, license and trust
- The repository was last updated about 2 months ago, so computer-use-automation is actively maintained.
- Our last check on 2026-10-11 found the source still online.
- No license is declared. By default that means all rights are reserved: you can read it, but reusing or redistributing it is not clearly permitted. Ask the author before building on it commercially.
- Its trust signals score 88/100, with 1 caution from licensing, adoption, age or documentation. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.
AI review by kimi-k2.7-code on 2026-10-07. Automated pattern scan on 2026-10-07. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
computer-use-automation compared with similar skills
All 4 of these similar skills score higher than computer-use-automation; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| computer-use-automation (this skill)by borghei | 87 | 817 | 57d ago | SKILL.md |
| taste-skillby Leonxlnx | 100 | 117.6k | 9d ago | CLAUDE.md |
| Agent-Reachby Panniantong | 100 | 112.7k | 9d ago | CLAUDE.md |
| rufloby ruvnet | 100 | 82.0k | today | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 79.9k | today | CLAUDE.md |
Frequently asked questions
- How do I install computer-use-automation?
- Run
npx skills add borghei/Claude-Skills --skill computer-use-automation. The install tabs above show the steps for each supported agent. - Which AI agents does computer-use-automation work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is computer-use-automation safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It declares no license and scores 88/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is computer-use-automation still maintained?
- The repository was last updated about 2 months ago, so computer-use-automation is actively maintained.
Skill content
View source on GitHubname: computer-use-automation description: > This skill should be used when the user asks to "build a computer-use agent", "automate a GUI with an AI agent", "when to use computer use vs an API", "make browser automation reliable", or "design screenshot-driven agent actions". license: MIT + Commons Clause metadata: version: 1.0.0 author: borghei category: engineering domain: ai-agents updated: 2026-06-29 tags: [computer-use, browser-automation, agents, gui, reliability]
Computer Use Automation
Category: Engineering Domain: AI Agents
Overview
The Computer Use Automation skill helps you design AI agents that operate a graphical interface the way a person does — take a screenshot, reason about what is on screen, then click, type, scroll, or navigate, and repeat. It covers the core perception→reason→action loop, the decision of when computer-use is the right tool versus a structured API/MCP tool (prefer a real API whenever one exists; reach for computer-use only for GUIs with no programmatic surface), reliability patterns (grounding every action in the current screenshot, verifying after each step, recovering from misclicks), safety guardrails (confirmation gates for destructive actions, sandboxing, avoiding blocking dialogs), and how to evaluate a computer-use agent. It is model-agnostic — the patterns apply to any computer-use-capable model and any GUI tool surface.
Clarify First
Before designing or auditing a computer-use agent, confirm these inputs. If any is unknown or vague, ASK — do not assume:
- [ ] Does a real API/MCP tool exist? — whether the target exposes an API, SDK, CLI, or MCP server, or is GUI-only (the single biggest factor; if a real API exists, prefer it and skip computer-use)
- [ ] Task & risk — what the agent must accomplish and whether any step is destructive or irreversible (delete, send, pay, submit), which sets the confirmation gates and sandboxing
- [ ] Which tool — advise on tool choice for a target, or lint a planned action sequence for safety (selects
tool_choice_advisor.pyvsaction_safety_linter.py)
Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact.
Quick Start
# Decide computer-use vs API/MCP for a target
python scripts/tool_choice_advisor.py --api-exists no --gui-stability high --volume low --json
# Lint a planned action sequence for safety/reliability gaps
python scripts/action_safety_linter.py --file planned_actions.json
# Read actions from stdin and emit a markdown risk report
echo '[{"type":"click","target":"Delete"},{"type":"submit","target":"Confirm"}]' \
| python scripts/action_safety_linter.py --format markdown
Tools Overview
| Tool | Purpose | Key Flags |
|------|---------|-----------|
| tool_choice_advisor.py | Recommend computer-use vs structured API/MCP for a target, with rationale | --api-exists, --gui-stability, --volume, --reversible, --json |
| action_safety_linter.py | Scan a planned action list for destructive verbs, missing verification, missing confirmation gates, and dialog-triggering patterns | --file, --format, --json |
All scripts: Python 3 standard library only, argparse CLI, --json and human-readable output. Run --help for full usage.
Workflows
Decide and Design a Computer-Use Agent
- Run
tool_choice_advisor.pywith the target's API/MCP availability, GUI stability, and volume — if it says "use API/MCP," stop and build against the real interface instead. - If computer-use is justified, draft the action plan as the screenshot→reason→action loop: each step re-grounds on a fresh screenshot before acting.
- Add a verification observation after every state-changing action (read back the resulting screen, not the intent).
- Insert confirmation gates before any destructive/irreversible step and choose a sandbox (throwaway profile, test account, isolated VM/container).
Audit a Planned Action Sequence
- Express the plan as a JSON/text list of actions (
type,target, optionalverified/confirmed). - Run
action_safety_linter.py --file plan.jsonto flag risky verbs, unverified state changes, ungated destructive actions, and dialog-triggering patterns. - Resolve each finding — add verification steps, add confirmation gates, replace blocking-dialog flows.
- Re-run until clean, then dry-run in the sandbox before any real target.
Reference Documentation
- Computer Use Patterns - The action loop; computer-use vs structured-tool decision matrix; reliability patterns (grounding, verification, recovery); safety guardrails (confirmation gates, sandboxing, blocking dialogs); evaluation approach; and common failure modes.
Common Patterns
Ground Every Action in the Current Screenshot
- Never act on a stale screenshot or a remembered layout — re-capture before each action.
- Reference elements by what is visible now (label, position) rather than a cached coordinate from a prior turn.
- After acting, take a fresh screenshot and confirm the expected change actually happened before continuing.
Gate Destructive Actions and Sandbox by Default
- Require an explicit confirmation step before delete, send, pay, submit, or any irreversible action.
- Run in a sandbox first: throwaway browser profile, test account, or isolated VM/container.
- Avoid flows that spawn blocking modal/native dialogs (file pickers, OS print dialogs) that the agent cannot see or dismiss; prefer paths that keep state on the page.
Prefer the Real Interface When It Exists
- A documented API, SDK, CLI, or MCP tool is more reliable, cheaper, and more verifiable than pixels — use it.
- Reserve computer-use for genuinely GUI-only targets, one-off tasks, or bridging gaps an API does not cover.
- For high-volume or business-critical flows, the cost of computer-use flakiness usually justifies building or requesting an API.
Related Skills
taste-skill
117.6kAgent skills that give coding agents taste: art direction, reference images and a browser check instead of generic AI pages
Agent-Reach
112.7kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
ruflo
82.0k🌊 The original agent harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, federation, vector RAG integration, and native Claude Code / Codex / Hermes and many more Integrated
headroom
79.9kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
