uMouse MCP — Hardware Mouse Control for AI Agents

English | 简体中文

Give AI agents a real hardware mouse. uMouse MCP is a Model Context Protocol (MCP) server for USB HID mouse control and voice dictation on Windows. Mouse movement, clicks, scrolling and dragging use a compatible uMouse device; voice input supports pluggable speech recognition providers.

Requires compatible uMouse hardware and firmware, Windows x64, and a separately installed Node.js runtime (use Node.js 22 x64). An ordinary USB mouse is not sufficient. The MIT license covers the server code; native SDK binaries have separate terms. See License.

Download · Quick start · Reference demos · Documentation · Get hardware / support · uMouse SDK

Download

| Package | Download | Requirements | |---|---|---| | MCPB · Windows x64 · v0.1.1 | Download MCPB · Release notes and SHA-256 | MCPB-compatible client; compatible uMouse hardware; x64 Node.js 22+ | | Windows x64 · v0.1.0 | Download ZIP · Release notes and SHA-256 | Compatible uMouse device and firmware; Node.js installed separately |

All releases · Gitee release mirror

MCPB installation: import the .mcpb into a client that supports MCPB Node extensions. Version 0.1.1 wraps the legacy 0.1.0 runtime, includes no cloud credentials, and defaults speech recognition to disabled; windows-speech is the optional local provider. Data defaults to ~/.umouse-mcp-mcpb. See the bilingual MCPB guide. Published in the official MCP Registry as io.github.SonicloudTech/mouse-mcp; metadata in server.json, publication and validation.

The ZIP includes production Node dependencies, native SDK DLLs, the voice preview UI and VC runtime libraries. No npm install or compilation is required to use the release; Node.js itself is not bundled.

Release and source are different installation paths: the v0.1.0 installer registers the legacy stdio entry, mcp/dist/index.js. The shared-v2 tools, native host and acceptance lab described below belong to the source implementation. This ZIP is not a complete shared-v2 bundle: it does not contain that native host or the acceptance lab. Use the installed server's tool list and the README shipped inside the ZIP for release-specific behavior.

Quick start

The following steps use the original ZIP. For MCPB, use the import instructions above, then perform the status/device checks in step 5.

1. Install Node.js 22 x64 and confirm node --version works. Connect a compatible uMouse USB receiver/device and power on the mouse. For a device or firmware compatibility check, see Hardware & support.
2. Download the ZIP above and extract it. Open the umouse-mcp-os-win-x64 folder; keep it in a permanent location because client configuration points to this folder.
3. Double-click install-agents.cmd to register ZCode at user scope, or run one command for your chosen client from the extracted folder:

| Client | PowerShell command | |---|---| | ZCode | node mcp/scripts/install-agents.mjs zcode | | Claude Desktop | node mcp/scripts/install-agents.mjs claude | | Cursor | node mcp/scripts/install-agents.mjs cursor |

The installer adds or updates the umouse entry in the selected client's configuration. It also copies the bundled trial ASR configuration into the local data directory if no configuration already exists. Replace trial credentials with your own for production use.

4. Open a new ZCode session, fully quit and reopen Claude Desktop, or reload the Cursor window, as appropriate.
5. Ask your agent to call umouse_status, umouse_get_capabilities, and umouse_list_devices. Check SDK health, the device list, and the reported capabilities. A returned sdkVersion only confirms SDK access; it does not prove that hardware control works. Confirm the visible result of any requested mouse action before relying on it.

For a client that accepts the mcpServers JSON format, add this entry manually, replacing the path with your extracted folder. Other clients may require a different configuration format.

{
  "mcpServers": {
    "umouse": {
      "command": "node",
      "args": ["C:\\path\\to\\umouse-mcp-os-win-x64\\mcp\\dist\\index.js"]
    }
  }
}

Manual configuration does not run the installer's ASR setup. Configure a speech provider separately if you need voice dictation; see Voice dictation.

For source development, npm ci, npm run build, and npm test install dependencies, build TypeScript, and run the default tests. They do not produce a complete shared-v2 native bundle. A complete public shared-v2 packaging workflow is not yet provided. The shared-v2 development and maintenance notes (Chinese) and native host build notes (Chinese) are references for that separate development path.

Reference demos

The mouse lab recording demonstrates the shared-v2 source implementation, not the legacy ZIP installation.

Hardware mouse driving a real browser lab page (gold arrow = system cursor overlaid on the recording) — counters, wheel and hit feedback all come from real browser events:

Voice typing: speak, and it lands in any window

Hold the device voice key, speak, release — the recognized text is typed into whatever window currently has focus. Both clips recorded live at 1080p:

Agent client (terminal) — spoken command typed in and executed:

<video src="mcp/acceptance/demo/voice-typing-agent.mp4" controls muted width="720">mcp/acceptance/demo/voice-typing-agent.mp4</video>

Open / download video

WeChat — spoken message typed into the input box and sent:

<video src="mcp/acceptance/demo/voice-typing-wechat.mp4" controls muted width="720">mcp/acceptance/demo/voice-typing-wechat.mp4</video>

Open / download video

Shared-v2 capabilities (source implementation)

Model → MCP host → shared daemon (:47192) → C++ native host → USB HID 0xF0/0xF1 → uMouse device → Windows input pipeline

• Hardware actions: umouse_mouse_action — move / click / scroll / drag in desktop physical-pixel coordinates, multi-monitor with negative coords, optional precision: standard | high.
• Task exclusivity: umouse_control begin → actions → end; only one task drives the mouse and keyboard at a time; dead connections are cleaned up automatically.
• Read-only sensors: umouse_input_status — cursor position, live key state, foreground window identity; read-only, never intercepts.
• Voice dictation: mic capture → VAD segmentation → pluggable ASR provider → preview / final transcript; an independent lane that never preempts mouse control (see below).
• Shared daemon: one instance per machine (port 47192), shared across ZCode / Codex / Claude hosts, reclaimed when idle.

Voice dictation: pluggable ASR providers

ASR here is not a black box tied to one vendor — it is a tiny interface contract: implement begin / writePcm / finishInput / cancel and emit preview / segmentFinal / warning / terminal (exactly once), and you have a valid provider. Swapping recognition services means swapping one factory function; mcp/src/sinicloud-asr.ts is a complete reference implementation.

| Provider (UMSDK_ASR_PROVIDER) | Notes | |---|---| | sinicloud (default for the legacy ZIP/source on Windows) | Sonicloud open-platform streaming ASR: pure TypeScript, no DLL dependency; streams 16k/16bit/mono PCM, interim previews, low-balance warnings without dropping the stream | | windows-speech | Built-in Windows recognizer; offline, audio never leaves the machine | | whisper-cpp | Self-hosted whisper-cli + ggml model (paths configurable), offline CPU transcription | | http | Custom transcription endpoint: multipart WAV in, {"text"} out | | disabled | Dictation off; a missing provider degrades gracefully to asr_unavailable |

Credentials travel only through configuration and are hot-read per session — swap keys without a restart: by default a JSON file at <data-dir>/sinicloud_asr.json (appId / apiKey / apiSecret / userId / lanid / streamUrl / tokenUrl); environment variables UMSDK_ASR_APP_ID / UMSDK_ASR_API_KEY / UMSDK_ASR_API_SECRET / … override the file. Never hardcoded, never printed, never logged, never committed. The legacy v0.1.0 ZIP ships a trial credential for the open platform; replace it with your own key for production use. The v0.1.1 MCPB contains no cloud credentials and accepts only disabled (default) or windows-speech. See mcp/SHARED-V2.md for the full source provider matrix.

Shared-v2 adaptive positioning

Windows pointer ballistics ("Enhance pointer precision") make the protocol-dots → pixels gain vary with amplitude. umouse-mcp ships an 11-step ballistic gain curve: read-only seeded from local pointer settings on first use of a device, learned online frame-by-frame within [0.125, 10], decaying back to the seed after 60 s idle — no tuning when you change machines or devices; a stale seed costs 1–2 extra frames. Design and measurements: mcp/sdk-host/POSITIONING.md.

| Metric | standard | high | |---|---|---| | Stop tolerance | residual ≤ 4 px | residual ≤ 1 px | | Measured (reference rig) | median 356 ms / 2 frames | max 1 px, mean 0.67 px | | Failure mode | identical; high at worst degrades to ≤ 4 px best-effort, residuals reported as-is | |

Shared-v2 reference acceptance

After preparing a complete shared-v2 build, follow the acceptance setup guide. The lab is in this repository, not the v0.1.0 ZIP.

32 cases verify effects in closed loop on a real browser page: left / right / middle click counts, double-click interval, wheel scrollTop, drag linearity, 1 px bullseye hits.

cd mcp/acceptance
$env:UMCFG = "C:\path\to\client.json"
$env:UMBUNDLE = "C:\path\to\shared-v2-bundle"
node run-matrix.mjs
# Writes acceptance-report.json; the documented reference run passed 32/32 cases.

Shared-v2 security and verification

• Locked screen / UAC / secure desktop: key actions are rejected with button_feedback_input_desktop_unavailable — that is a safety property, not a bug.
• effectConfirmed is always false: transport success ≠ application effect; the acceptance lab's page evidence is the ground truth.
• ASR credentials enter only via local config file and environment variables — never committed, printed or logged; `recove