<!-- mcp-name: io.github.IlyasKhallouki/hypruse -->

hypruse

Computer use for Hyprland. An MCP server that gives AI agents native hands on your Wayland desktop: workspaces, windows, mouse, keyboard, screenshots.

No ydotool daemon. No root. No portals. No X11.

Why

Computer use exists on macOS and Windows, and the vendors' Linux apps still ship without it: Claude Desktop's Linux docs say computer use is not available there, and ChatGPT's computer use covers macOS and Windows only. Anthropic's reference implementation is an X11 container. Other Wayland tools lean on ydotool and uinput, on GNOME-only portals, or on a compositor plugin built for one exact Hyprland version.

Meanwhile Hyprland already exposes everything an agent needs, better than any accessibility bridge: a complete IPC surface for state and window management, and first-class Wayland protocols for input. hypruse just wires them to MCP:

• Semantic first. desktop returns the real window/workspace tree (addresses, classes, titles, geometry) in one call. The agent switches workspaces and focuses windows the way you do (instantly, over IPC), not by squinting at pixels.
• Vision when it matters. Screenshots of a monitor, one window (with grim 1.5+ on Hyprland 0.54+, captured as itself even on a hidden workspace or under another window, though a hidden app may show how it last looked), or a zoomed region, with the geometry/scale metadata to map any pixel back to a clickable coordinate: a coarse-to-fine loop grounded in the GUI-agents research.
• Native input. Clicks and scrolls are spoken directly over the Wayland wire (zwlr_virtual_pointer_v1); typing goes through wtype's virtual keyboard with a proper XKB keymap, unicode-safe on any layout.
• Works beside you. The agent can work in apps on its own workspaces while you keep working on yours. It presses controls through the accessibility tree and types through it or with Hyprland's sendshortcut, so your cursor, keyboard focus and workspace stay where you left them. Stock Hyprland, no compositor plugin: see Background mode.

How it works

agent (Claude Code, or any MCP client)
   │ stdio
   ▼
hypruse
   ├── hyprctl -j ········▶ desktop state: monitors, workspaces, windows, layers
   ├── hyprctl dispatch ··▶ focus / move / close / launch / movecursor,
   │                        and sendshortcut for keys to a hidden window
   ├── grim ··············▶ screenshots: monitor, window, region
   ├── D-Bus (AT-SPI) ····▶ accessibility tree: named controls, current
   │                        values, exact coords (ui / marks / click_ui),
   │                        and presses and text in the background
   ├── wtype ·············▶ keyboard (zwp_virtual_keyboard_v1, real XKB keymap)
   └── raw Wayland wire ··▶ click & scroll (zwlr_virtual_pointer_v1)

Optional binaries gate two more tools: imagemagick draws the numbered overlay for marks, and wl-clipboard backs the opt-in clipboard tool.

Design decisions:

• No ydotool / uinput. That path needs a daemon, udev rules or root, and types US scancodes that break on other layouts. hypruse is just another Wayland client of your compositor, same standing as wlrctl.
• No portals. xdg-desktop-portal-hyprland does not implement the RemoteDesktop portal (InputCapture is capture, not injection), so anything built on libei/portals silently degrades on Hyprland. hypruse doesn't try.
• Cursor positioning through the compositor's own cursor dispatcher (global logical coordinates, exact on any monitor layout), with only button/axis events on the virtual pointer, sidestepping the known multi-monitor bugs of absolute virtual-pointer motion (hyprwm/Hyprland#6749).

Tools

| tool | what it does | |---|---| | desktop | One-call semantic snapshot: monitors, workspaces, windows (address/class/title/geometry), active window, cursor, and layer surfaces (launchers, bars, notification popups) with a best-effort kind and geometry. A listed layer is one the compositor tracks, not one you can see: transparent and dormant surfaces are reported too, so screenshot when visibility matters | | screenshot | Focused monitor, one window by address (with grim 1.5+ on Hyprland 0.54+, even hidden or covered; metadata visible says whether the compositor draws it, source whether the pixels are the window's own or a screen crop, and a hidden capture may be stale), or x,y,WxH region; returns image + coordinate-mapping metadata; fast JPEG by default (lossless=true for PNG); stable=true waits for the frame to settle | | zoom | Native-resolution re-capture around an estimated point (optionally clamped to a window): the precision step before clicking small controls, same metadata contract | | ui | Read a window's accessibility tree (AT-SPI, GTK/Qt apps that expose one) and return clickable elements by name with exact global coordinates, no screenshot; reports current values too (typed text, slider position, checkbox state); falls back to vision when an app exposes nothing | | marks | Set-of-Marks capture: the window screenshot with every accessible control drawn as a numbered mark, plus a JSON legend (role, name, current value, exact click point per number); needs ImageMagick for the drawing, degrades to the legend alone without it | | click_ui | Click a control by accessible NAME or by a marks number in one call: the coordinate comes from the tree. In the window you are in, the click goes through the real pointer (visible, same safety guarantees); in any other window, the control's accessibility action presses it in the background. An ambiguous name returns the candidates instead of guessing | | pointer | move / click / drag / scroll (discrete wheel notches) in global coordinates. Your real cursor, so by default it is refused where the input would land on a window other than yours | | keyboard | Type literal text (unicode-safe) or press app-level combos (ctrl+shift+t, esc, F5); an optional window address says where the keys go: the window you are in is focused and typed into, any other window gets them in the background. Compositor binds (super+...) go through use_bind, not here | | hypr | Switch workspace, focus/move/close windows, fullscreen, floating (pure IPC, milliseconds; close_window also waits up to 1s for the destroy event, so its result and then= observation reflect the close instead of racing it) | | launch | Start an app, silently on the next of the agent's own workspaces unless you name one (or pass background=false for yours), block on its actual openwindow event, return its address; detects single-instance apps (browsers) whose window ignores exec rules and moves it to the requested workspace | | binds | The user's own keybinds, decoded (SUPER+Q, action, description); the agent runs one with use_bind | | use_bind | Execute a keybind by combo (SUPER+F), running its bound action, so the agent drives the owner's own launchers and shortcuts. Not available for binds in a Lua hyprland.lua: those are anonymous closures Hyprland exposes no way to call, so binds shows them as action lua and use_bind says so instead of failing cryptically | | sequence | Run an ordered list of actions (pointer/keyboard/click_ui/hypr/wait_for) in one call; stops the moment the desktop changes in a way the current step did not expect, so a click/type/enter micro-sequence costs one round-trip instead of several | | wait_for | Block on real compositor events (window open/close, workspace change, title change, layer surfaces appearing/closing, urgency, screen sharing on/off) with a match filter and timeout; a filtered wait whose condition already holds (window already gone, workspace already active, layer already mapped) answers instantly with already: true instead of missing an event that fired before it subscribed. Replaces sleep-and-hope in multi-step automations | | clipboard | Read or write the text clipboard via wl-clipboard; opt-in, exists only with HYPRUSE_CLIPBOARD=1 in the server env |

The acting tools (pointer, keyboard, click_ui, hypr, use_bind, sequence) take an optional then argument that appends the result to the same call, so the agent sees the effect without a second round-trip: then='desktop' adds a fresh semantic snapshot (~20 ms, cheap, best for window/focus changes), then='screenshot' a stable capture (best for visual changes), then='ui' the acted-on window's controls with their current values (a few hundred exact tokens, best after typing or toggling; click_ui reads the window it clicked even if the click handed focus to a dialog, the others read the focused window), then='none' nothing (the default everywhere except sequence, which defaults to 'desktop').

Every tool is also a shell verb (hypruse desktop, hypruse click_ui Save --window 0x...) for agents that run commands instead of MCP, with a skill that teaches them: see Shell verbs and the agent skill.

Features

The tools group into five capabilities, ordered most-reliable-and-cheapest first. An agent that reaches for them in this order is both faster and more accurate, and many tasks never need a screenshot at all.

1. Semantic desktop control (start here)

desktop returns the entire window and workspace tree in one call: every window's address, class, title, and geometry, the active window, the cursor, and any layer surfaces the compositor is tracking (launchers, bars, notification popups). hypr and launch then act on it over IPC in milliseconds: switch workspace, focus/move/close/fullscreen/float a window by address, or start an app.

Use it well: never take a screenshot to find or arrange windows. Read desktop, act on the address you want. launch blocks on the real openwindow event and hands back the new window's address, so there is nothing to poll or guess; it also relocates single-instance apps (browsers) that ignore workspace rules.

2. Click controls by name, no pixels

When an app exposes an accessibility tree (most GTK and Qt apps), you can target controls by name instead of by pixel. ui lists every control with its exact global coordinate, and reports the current value of the controls that carry one: the text in a field, a slider's percentage, a checkbox's state. click_ui resolves a name and clicks it in one call, through the real cursor (so the beacon and every safety guarantee still apply), or, in a window you are not in, through the control's own accessibility action without touching your cursor. marks draws numbered marks over a screenshot with a legend, for when you want to see the options first and then click_ui(mark=N).

Use it well: reach for click_ui name="Save" before estimating any pixel, since it is exact and spends no image. Read a form's state with ui (did the box actually tick?) instead of screenshotting it. An ambiguous name returns the candidates rather than guessing. When an app exposes no tree (terminals, canvas apps, Electron/Chrome without --force-renderer-accessibility) the tool says so, and you fall back to vision.

3. Vision when it matters: the zoom loop

For everything the accessibility tree cannot name, screenshot (monitor, window, or region) and zoom (a native-resolution re-capture around a point) carry a strict coordinate contract, global = geometry + pixel / scale, that stays exact on every monitor and fractional scale.

Use it well: don't guess a small control from a full-screen image. Work coarse-to-fine: screenshot the window, estimate the target, zoom there, re-estimate on