Touchpoint

<p align="center"> <h1 align="center">Touchpoint</h1> <p align="center"> <strong>Give your AI agent eyes and hands on any desktop.</strong> </p> <p align="center"> <a href="https://pypi.org/project/touchpoint-py/"><img src="https://img.shields.io/pypi/v/touchpoint-py?color=blue" alt="PyPI"></a> <a href="https://pypi.org/project/touchpoint-py/"><img src="https://img.shields.io/pypi/pyversions/touchpoint-py" alt="Python"></a> <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="MIT License"></a> <a href="#status"><img src="https://img.shields.io/badge/status-alpha-orange" alt="Alpha"></a> <br> <img src="https://img.shields.io/badge/Linux-FCC624?logo=linux&logoColor=black" alt="Linux"> <img src="https://img.shields.io/badge/macOS-000000?logo=apple&logoColor=white" alt="macOS"> <img src="https://img.shields.io/badge/Windows-0078D6?logo=windows&logoColor=white" alt="Windows"> </p> <p align="center"> <code>pip install touchpoint-py</code> </p> </p> <p align="center"><img src="https://raw.githubusercontent.com/Touchpoint-Labs/touchpoint/main/docs/demo.gif" width="720" alt="Touchpoint demo — AI agent creates a formatted Excel table using Touchpoint"></p> <p align="center"><em>AI agent researches data in Chrome, then creates a formatted Excel table — full task completed in ~12 minutes</em></p>

---

Touchpoint is a cross-platform Python library for reading and interacting with desktop UI through native accessibility APIs. One import, one API — works on Linux, macOS, and Windows, with built-in support for Chromium and Electron apps via CDP (Chrome DevTools Protocol).

Instead of scraping pixels or running vision models, Touchpoint reads the real accessibility tree — structured names, roles, states, and positions for every element on screen. Fast and reliable, with no vision model required. Ships with an MCP server so LLM agents like Claude, Cursor, or any local model can control any desktop app out of the box.

import touchpoint as tp

elements = tp.find("Send", role=tp.Role.BUTTON, app="Slack")
tp.click(elements[0])

Why Touchpoint?

| | Screenshot / vision | Browser automation | Touchpoint | |---|---|---|---| | Native desktop apps | ⚠️ inaccurate or slow | ❌ | ✅ | | Browsers | ⚠️ inaccurate or slow | ✅ | ✅ via CDP | | Electron apps (Slack, VS Code, ...) | ⚠️ inaccurate or slow | ⚠️ web content only | ✅ native + web | | Structured element data | ❌ needs OCR/vision model | ✅ web only | ✅ names, roles, states, positions | | Works with local / non-vision models | ❌ | ✅ web only | ✅ all apps | | Works across Linux, macOS, Windows | ✅ | ✅ | ✅ |

---

Table of Contents

• Table of Contents
• Install
  • Platform requirements
• Quick Start
  • Element IDs
  • Output formats
• MCP Server
  • Tools
  • Client setup
  • Environment variables
• Browser & Electron Apps (CDP)
  • Setup
• API Reference
  • Discovery
  • Search & Wait
  • Actions
  • Input
  • Screenshot & Config
• Architecture
• Configuration
• Development
• Status
  • Known limitations
• License

---

Install

Requires Python 3.10+.

pip install touchpoint-py

Everything is included: your platform's native backend, CDP support for browsers and Electron apps, the MCP server, and screenshot capabilities. Platform-specific dependencies are installed automatically via pip environment markers.

Platform requirements

| Platform | Backend | Requirement | |----------|---------|-------------| | Linux | AT-SPI2 | Install xdotool (required for input + minimize_window) and wmctrl (required for all window management — used for AT-SPI → X11 id mapping). Most desktops include python3-gi and gir1.2-atspi-2.0 — install them if missing. | | Windows | UI Automation | None — uses built-in COM APIs | | macOS | Accessibility (AX) | Grant permission: System Settings → Privacy & Security → Accessibility |

---

Quick Start

import touchpoint as tp

# Discover
apps = tp.apps()                            # ["Firefox", "Slack", "Terminal", ...]
windows = tp.windows()                      # Window objects with title, position, size
all_els = tp.elements(app="Firefox", named_only=True)  # only elements with text labels

# Find
results = tp.find("Search", role=tp.Role.TEXT_FIELD, app="Firefox")

# Act
tp.set_value(results[0], "touchpoint python", replace=True)
tp.press_key("enter")
tp.hotkey("ctrl", "s")                      # keyboard shortcuts

# Wait for UI changes
tp.wait_for("results", app="Firefox", timeout=10)

# Screenshot
img = tp.screenshot()                       # full desktop → PIL.Image
img = tp.screenshot(app="Firefox")           # cropped to app window

Element IDs

Every element has a unique ID like atspi:1234:1:2.0 or cdp:9222:TID:4. Action functions accept either an Element object or a bare ID string — useful for storing references across steps:

results = tp.find("Send", max_results=1)
element_id = results[0].id                  # "atspi:1234:1:5.2"

# later...
tp.click(element_id)                        # works with just the string

Output formats

Control how results are returned:

tp.elements(app="Slack", format="flat")     # one compact line per element (best for LLMs)
tp.elements(app="Slack", format="tree")     # indented parent/child hierarchy
tp.elements(app="Slack", format="json")     # full JSON with all fields

---

MCP Server

Touchpoint ships an MCP (Model Context Protocol) server ready for any MCP-compatible client. Use it to let LLM agents like Claude, Cursor, local models, or any tool that supports MCP control your desktop.

Two modes — vision and no-vision

Set TOUCHPOINT_MODE=no-vision (default: vision) to switch modes:

• Vision mode — agents use screenshot() to see the screen and interact by element ID or coordinates. Best for frontier models with strong vision capabilities.
• No-vision mode — agents use snapshot() to get a compact structured text tree of the active window, then act on element IDs directly. Works with any model including local ones that have no vision capability. Most action tools append auto-verify flags ((new window: ...), (focus moved), (no change detected)) so the agent can detect state changes without taking a screenshot.

Tools

| Category | Vision mode | No-vision mode | |----------|------------|----------------| | Orient | screenshot, snapshot, apps, windows | snapshot, diff_snapshot, apps, windows | | Find | find, get_element | find | | Read | read_text | read_text | | Actions | click (element or coordinates), set_value, set_numeric_value, select_text, focus, action | click (element only), set_value, set_numeric_value, select_text, focus, action | | Keyboard | type_text, press_key | type_text, press_key | | Mouse | mouse_move, scroll | scroll | | Window | activate_window, minimize_window, fullscreen_window, close_window, move_window, resize_window | activate_window, minimize_window, fullscreen_window, close_window | | Waiting | wait_for, wait_for_app, wait_for_window | wait_for, wait_for_app, wait_for_window | | Health | diagnostics | diagnostics |

The MCP server includes built-in instructions that teach agents the correct workflow for each mode — including the orient → act → verify loop, when to use read_text vs find, and how to recover from errors.

         ┌──────────┐
    ┌───▶│  ORIENT  │  screenshot · apps · windows
    │    └────┬─────┘
    │         ▼
    │    ┌──────────┐
    │    │  LOCATE  │  find · snapshot · get_element
    │    └────┬─────┘
    │         ▼
    │    ┌──────────┐
    │    │   ACT    │  click · set_value · type_text · press_key
    │    └────┬─────┘
    │         ▼
    │    ┌──────────┐
    │    │  VERIFY  │───▶ Done ✅
    │    └────┬─────┘
    │         │ not yet
    └─────────┘

Client setup

<details> <summary><strong>Claude Desktop</strong></summary>

Config file location:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "touchpoint": {
      "command": "touchpoint-mcp"
    }
  }
}

If using a virtualenv, use the full path: "/path/to/venv/bin/touchpoint-mcp"

</details> <details> <summary><strong>VS Code / GitHub Copilot</strong></summary>

Add to .vscode/mcp.json in your workspace:

{
  "servers": {
    "touchpoint": {
      "command": "touchpoint-mcp"
    }
  }
}

</details> <details> <summary><strong>Cursor</strong></summary>

Create or edit ~/.cursor/mcp.json:

{
  "mcpServers": {
    "touchpoint": {
      "command": "touchpoint-mcp"
    }
  }
}

</details> <details> <summary><strong>Windsurf</strong></summary>

Edit ~/.codeium/windsurf/mcp_config.json:

{
  "mcpServers": {
    "touchpoint": {
      "command": "touchpoint-mcp"
    }
  }
}

</details> <details> <summary><strong>Claude Code (CLI)</strong></summary>

claude mcp add touchpoint -- touchpoint-mcp

</details> <details> <summary><strong>OpenClaw</strong></summary>

Add to mcpServers in ~/.openclaw/openclaw.json:

{
  "mcpServers": {
    "touchpoint": {
      "command": "touchpoint-mcp"
    }
  }
}

</details>

Environment variables

<details> <summary>All optional — click to see available settings</summary> <br>

| Variable | Example | Description | |----------|---------|-------------| | TOUCHPOINT_CDP_DISCOVER | true | Auto-discover CDP ports from running processes | | TOUCHPOINT_CDP_PORTS | {"Chrome": 9222} | Explicit app-to-port mapping (JSON) | | TOUCHPOINT_CDP_APP | Google Chrome | Single app name (pair with _PORT) | | TOUCHPOINT_CDP_PORT | 9222 | Single port (pair with _APP) | | TOUCHPOINT_CDP_REFRESH_INTERVAL | 5.0 | Seconds between CDP port scans | | TOUCHPOINT_SCALE_FACTOR | 1.25 | Display scale override | | TOUCHPOINT_FUZZY_THRESHOLD | 0.6 | Minimum match score for find() (0.0–1.0) | | TOUCHPOINT_FALLBACK_INPUT | true | Use coordinate fallback when native actions fail | | TOUCHPOINT_MAX_ELEMENTS | 5000 | Maximum elements per query | | TOUCHPOINT_MAX_DEPTH | 20 | Default tree depth limit | | TOUCHPOINT_AX_MESSAGING_TIMEOUT | 1.0 | Max seconds to wait for a macOS AX app reply |

</details>

---

Browser & Electron Apps (CDP)

Native accessibility APIs return limited data for Electron and Chromium apps (Slack, Discord, VS Code, etc.). Touchpoint's CDP backend connects via Chrome DevTools Protocol to get the full web content.

Auto-discovery is enabled by default — Touchpoint automatically finds running browsers and Electron apps that were launched with a debug port. No manual configuration needed beyond launching the app with the flag.

Setup

1. Launch the app with a debug port:

# Linux
google-chrome --remote-debugging-port=9222 --user-data-dir=/tmp/tp-chrome

# macOS
open -na "Google Chrome" --args --remote-debugging-port=9222 --user-data-dir=/tmp/tp-chrome

# Windows
start chrome --remote-debugging-port=9222 --user-data-dir=%TEMP%\tp-chrome

2. Configure Touchpoint:

import touchpoint as tp

tp.configure(cdp_discover=True)             # auto-discover from running processes
# or
tp.configure(cdp_ports={"Google Chrome": 9222})  # explicit mapping