System Automation — Control Any Desktop Application

Control every aspect of your PC via MCP server, optimized for AI access

Windows, Mac, Linux. Click buttons. Read text. Move windows. Run commands. Your AI can finally interact with desktop applications like a human would.

---

Benefits

1. 🖱️ Click Anything, Read Everything

Not just automation — actual understanding. Scan UI elements, extract text from any window, click specific buttons by name. Your AI doesn't just run commands — it sees what's on screen and interacts intelligently.

2. 🌐 Cross-Platform Power

One API, three operating systems. Windows with full UI Automation, macOS with accessibility APIs, Linux with X11/Wayland support. Write once, automate everywhere.

3. 🚀 Background Execution Built-In

Start commands and move on. No waiting for long-running processes. Execute commands in background, check output later, terminate if needed. Perfect for builds, tests, or any long-running task.

---

Why This Tool Changes Desktop Automation

Most automation tools require exact coordinates. Click at pixel (500, 300). Hope the window hasn't moved. Hope the resolution hasn't changed. Fragile and frustrating.

RPA tools cost thousands. UiPath, Blue Prism, Automation Anywhere — enterprise pricing for enterprise features. This tool? Free, included, just works.

Platform-specific code is a nightmare. AutoHotkey for Windows, AppleScript for Mac, xdotool for Linux. Three completely different approaches, three codebases to maintain.

This tool solves all of that.

Click buttons by name, not coordinates. Extract text from any UI element. Move windows programmatically. Execute commands with background support. All cross-platform. All included.

---

Real-World Story: The Testing Nightmare

The Problem:

A software company needed to test their desktop application across Windows, Mac, and Linux. Manual testing took 3 days per release. Automated testing with platform-specific tools required three separate test suites.

"Can AI help?" they asked.

Standard solution: Selenium (web only), Appium (mobile focus), or expensive RPA platforms ($15K/year minimum).

With This Tool:

# Cross-platform test script - works on all OSes
# 1. Launch application
execute_command("./myapp", background=True)
time.sleep(2)

# 2. Find the application window
windows = list_windows()
app_window = [w for w in windows if "MyApp" in w['title']][0]

# 3. Scan UI elements
elements = scan_ui_elements(hwnd=app_window['hwnd'])

# 4. Click the "New Document" button by name
click_ui_element(hwnd=app_window['hwnd'], element_name="New Document")

# 5. Type into the text field
send_text(hwnd=app_window['hwnd'], text="Test document content")

# 6. Take screenshot for verification
screenshot = take_screenshot(hwnd=app_window['hwnd'])

# 7. Click "Save" button
click_ui_element(hwnd=app_window['hwnd'], element_name="Save")

Result: One test script, three platforms. Testing time reduced from 3 days to 3 hours. Zero RPA licensing costs. Complete automation with intelligent UI interaction.

The kicker: Same script now tests their web app (via browser automation) and their mobile app (via emulators). One tool, all platforms, all application types.

---

The Complete Feature Set

Window Management

List All Windows:

# Get all visible windows
windows = list_windows()

# Include minimized and popup windows
all_windows = list_windows(include_all=True)

# Returns: hwnd, title, class, position, size, style flags

Activate Windows:

# Bring window to foreground
activate_window(hwnd="0x00020828")

# Bring to foreground AND give keyboard focus
activate_window(hwnd="0x00020828", request_focus=True)

Move and Resize:

# Move single window
move_window(hwnd="0x00020828", x=100, y=100, width=800, height=600)

# Batch move multiple windows (layout management)
move_window(moves=[
    {"hwnd": "0x00020C4A", "x": 0, "y": 0, "width": 960, "height": 580},
    {"hwnd": "0x00030B12", "x": 960, "y": 0, "width": 960, "height": 580},
    {"hwnd": "0x00040C2E", "x": 0, "y": 580, "width": 1920, "height": 500}
])

Why batch moves matter: Arrange your entire workspace in one atomic operation. Perfect for multi-monitor setups or workflow-specific layouts.

UI Element Interaction

Scan UI Elements:

# Scan by window title
elements = scan_ui_elements(window_title="Notepad")

# Scan by window handle
elements = scan_ui_elements(hwnd="0x00020828")

# Returns: element names, types, positions, states, automation IDs

Get Clickable Elements:

# Find all clickable elements in foreground window
clickable = get_clickable_elements()

# Returns: buttons, links, menu items with names and positions

Click UI Elements:

# Click button by name (smart - finds it automatically)
click_ui_element(hwnd="0x00020828", element_name="OK")

# Click at specific coordinates (relative to window)
click_at_coordinates(hwnd="0x00020828", x=50, y=100, button="left")

# Click at screen coordinates (absolute position)
click_at_screen_coordinates(x=500, y=300, button="left")

# Supported buttons: left, right, middle

Send Text & Keystrokes (AutoHotkey-Style Syntax):

# Type text into active window
send_text(hwnd="0x00020828", text="Hello, World!")

# Press Enter after typing
send_text(hwnd="0x00020828", text="Hello{Enter}")

# Keyboard shortcuts
send_text(hwnd="0x00020828", text="^a")      # Ctrl+A (select all)
send_text(hwnd="0x00020828", text="^c")      # Ctrl+C (copy)
send_text(hwnd="0x00020828", text="^v")      # Ctrl+V (paste)
send_text(hwnd="0x00020828", text="^s")      # Ctrl+S (save)
send_text(hwnd="0x00020828", text="!{F4}")   # Alt+F4 (close window)
send_text(hwnd="0x00020828", text="#r")      # Win+R (Run dialog)

# Tab navigation
send_text(hwnd="0x00020828", text="{Tab 3}") # Press Tab 3 times

# Works with any text input field that has focus

AutoHotkey-Style Keystroke Syntax

The send_text operation supports full AutoHotkey-style syntax for sending keystrokes, making it easy to automate keyboard interactions.

Special Keys (in braces):

# Navigation keys
send_text(hwnd, text="{Enter}")      # Enter/Return key
send_text(hwnd, text="{Tab}")        # Tab key
send_text(hwnd, text="{Escape}")     # Escape key (also {Esc})
send_text(hwnd, text="{Space}")      # Space bar
send_text(hwnd, text="{Backspace}")  # Backspace (also {BS})
send_text(hwnd, text="{Delete}")     # Delete key (also {Del})
send_text(hwnd, text="{Insert}")     # Insert key (also {Ins})
send_text(hwnd, text="{Home}")       # Home key
send_text(hwnd, text="{End}")        # End key
send_text(hwnd, text="{PageUp}")     # Page Up (also {PgUp})
send_text(hwnd, text="{PageDown}")   # Page Down (also {PgDn})

# Arrow keys
send_text(hwnd, text="{Up}")         # Up arrow
send_text(hwnd, text="{Down}")       # Down arrow
send_text(hwnd, text="{Left}")       # Left arrow
send_text(hwnd, text="{Right}")      # Right arrow

# Function keys
send_text(hwnd, text="{F1}")         # F1 through {F24}
send_text(hwnd, text="{F5}")         # Refresh in many apps
send_text(hwnd, text="{F11}")        # Fullscreen toggle

# Lock keys
send_text(hwnd, text="{CapsLock}")   # Caps Lock
send_text(hwnd, text="{NumLock}")    # Num Lock
send_text(hwnd, text="{ScrollLock}") # Scroll Lock

# Numpad keys
send_text(hwnd, text="{Numpad0}")    # Numpad 0-9
send_text(hwnd, text="{NumpadEnter}")# Numpad Enter
send_text(hwnd, text="{NumpadAdd}")  # Numpad +
send_text(hwnd, text="{NumpadSub}")  # Numpad -
send_text(hwnd, text="{NumpadMult}") # Numpad *
send_text(hwnd, text="{NumpadDiv}")  # Numpad /
send_text(hwnd, text="{NumpadDot}")  # Numpad .

# Media/Browser keys
send_text(hwnd, text="{Volume_Mute}")      # Mute
send_text(hwnd, text="{Volume_Up}")        # Volume up
send_text(hwnd, text="{Volume_Down}")      # Volume down
send_text(hwnd, text="{Media_Play_Pause}") # Play/Pause
send_text(hwnd, text="{Browser_Back}")     # Browser back

Modifier Prefixes:

# ^ = Ctrl
send_text(hwnd, text="^c")           # Ctrl+C (copy)
send_text(hwnd, text="^v")           # Ctrl+V (paste)
send_text(hwnd, text="^a")           # Ctrl+A (select all)
send_text(hwnd, text="^s")           # Ctrl+S (save)
send_text(hwnd, text="^z")           # Ctrl+Z (undo)
send_text(hwnd, text="^f")           # Ctrl+F (find)

# + = Shift
send_text(hwnd, text="+{Tab}")       # Shift+Tab (reverse tab)
send_text(hwnd, text="+{Home}")      # Shift+Home (select to start)
send_text(hwnd, text="+{End}")       # Shift+End (select to end)

# ! = Alt
send_text(hwnd, text="!{F4}")        # Alt+F4 (close window)
send_text(hwnd, text="!{Tab}")       # Alt+Tab (switch windows)
send_text(hwnd, text="!f")           # Alt+F (File menu)

# # = Win (Windows key)
send_text(hwnd, text="#r")           # Win+R (Run dialog)
send_text(hwnd, text="#e")           # Win+E (Explorer)
send_text(hwnd, text="#d")           # Win+D (Show desktop)
send_text(hwnd, text="#l")           # Win+L (Lock screen)

# Combine modifiers
send_text(hwnd, text="^+s")          # Ctrl+Shift+S (Save As)
send_text(hwnd, text="^+{Escape}")   # Ctrl+Shift+Escape (Task Manager)
send_text(hwnd, text="^!{Delete}")   # Ctrl+Alt+Delete
send_text(hwnd, text="^+n")          # Ctrl+Shift+N (new incognito window)

Key Repetition:

# Press a key multiple times
send_text(hwnd, text="{Tab 5}")      # Press Tab 5 times
send_text(hwnd, text="{Enter 3}")    # Press Enter 3 times
send_text(hwnd, text="{Down 10}")    # Press Down arrow 10 times
send_text(hwnd, text="{Backspace 20}")# Delete 20 characters

Key Hold/Release:

# Hold a key down, do something, then release
send_text(hwnd, text="{Shift down}hello{Shift up}")  # Types "HELLO"
send_text(hwnd, text="{Ctrl down}ac{Ctrl up}")       # Ctrl+A, Ctrl+C

# Useful for drag operations or games
send_text(hwnd, text="{Ctrl down}")  # Hold Ctrl
# ... do something ...
send_text(hwnd, text="{Ctrl up}")    # Release Ctrl

Literal Characters (Escaping):

# Send literal brace characters
send_text(hwnd, text="{{}Hello{}}")  # Sends "{Hello}"

# Send literal modifier symbols
send_text(hwnd, text="Price: $100{!}")  # Sends "Price: $100!"
send_text(hwnd, text="2{+}2=4")         # Sends "2+2=4"
send_text(hwnd, text="C{#} code")       # Sends "C# code"
send_text(hwnd, text="100{^}2")         # Sends "100^2"

Raw Mode (No Parsing):

# Send text literally without any parsing
send_text(hwnd, text="{Raw}^c is Ctrl+C")  # Sends "^c is Ctrl+C" literally
send_text(hwnd, text="{Text}!@#$%^&*()")  # Sends special chars literally

Complete Examples:

# Fill a form and submit
send_text(hwnd, text="John Doe{Tab}john@example.com{Tab}password123{Enter}")

# Open Run dialog, type command, execute
send_text(hwnd, text="#r")           # Win+R
time.sleep(0.5)
send_text(hwnd, text="notepad{Enter}")

# Select all, copy, paste
send_text(hwnd, text="^a^c")         # Select all, copy
send_text(hwnd, text="{Tab}^v")      # Tab to next field, paste

# Navigate a document
send_text(hwnd, text="^{Home}")      # Go to beginning
send_text(hwnd, text="^+{End}")      # Select to end
send_text(hwnd, text="^c")           # Copy selection

# Close application gracefully
send_text(hwnd, text="^s")           # Save first
send_text(hwnd, text="!{F4}")        # Then close

Screenshot Capabilities

Capture Windows:

# Full window screenshot
screenshot = take_screenshot(hwnd="0x00020828")

# Specific region (x, y, width, height relative to w