dsh-agent-toolchain

The engineering runtime that makes coding agents verifiably accountable for their own changes.

A suite of plugins and an MCP server that let a coding agent do more than write code — it can see the running desktop application, drive it, measure it, capture what it sends over the network, and verify the change it just made. Each plugin is a small, composable building block; together they close the loop between "the agent wrote a change" and "the change actually works" — and produce the evidence to prove it.

One command runs the whole loop against a throwaway WPF window that ships in this repo — build it, drive it, read the result back out, then have verify_report adjudicate what the run claimed:

npm run demo          # build → launch → drive → read → verdict
npm run demo:perf     # + catch a deliberately blocked UI thread

Real output (npm run demo, paths shortened):

1. environment self-check
  ok   toolchain_status reports what is configured        (toolchain_status, 542 ms)
2. build the sample app (and bind the result to a runId)
  ok   build_run → 0 errors                               (build_run, 3353 ms)
3. launch and look at the real window
  ok   ui_launch brings the window up                     (ui_launch, 11929 ms)
  ok   ui_observe(state) sees the controls                (ui_observe, 62 ms)
4. drive it — this is the part a code-reading agent cannot do
  ok   type into the Input box                            (ui_drive, 458 ms)
  ok   click Greet                                        (ui_drive, 314 ms)
  ok   read the result back out of the UI                 (ui_observe, 79 ms)
  ok   capture a screenshot as evidence                   (ui_observe, 35 ms)
5. adjudicate: claims vs evidence
  ok   verify_report returns a verdict                    (verify_report, 57 ms)

verdict: pass
steps: 9/9 ok

npm run demo:perf additionally reports stutterCount=1, maxMs=1474 — the sample app blocks its UI thread for exactly 1500 ms on purpose, and the perf probe has to catch it.

The demo is not demo-only code. mcp/demo/run-demo.mjs is an ordinary MCP client: it spawns mcp/server.mjs over stdio and calls the same tools Claude Code, Cursor, Cline or Codex would call. If it passes, the MCP face works.

Recording a GIF: npm run demo:perf while a screen recorder is running is the intended capture — the window does something visible at every step. docs/media/ is where the recording goes.

The problem

Coding agents are excellent at producing patches, but they are blind to what happens after the patch: Did the build break? Did the UI actually render the new page? Which API calls did the application fire, and did one of them hang for 20 seconds? Did the change introduce a memory leak or a UI stall?

Traditional agents verify by reading code. This toolchain lets them verify by observing the running application — the same way a human QA engineer would — and then makes them show the evidence.

The loop

  change code → build (dsh-build) → drive client (dsh-ui-drive)
                                      → capture APIs (dsh-api-visualizer, dsh-postman)
                                      → inspect perf (dsh-perf) / hang (dsh-hang-inspector)
                                      → remember lessons (dsh-memory)
  claim "done" → verify_report: claims vs evidence → one verdict
  fail / handoff → failure_record → failure corpus (the data flywheel)

Quick start

Path A — the demo path (no Rust, about five minutes)

Exactly the tools npm run demo exercises — build_run, the ui_* family, verify_report, perf_probe, http_request — need neither Rust nor MSVC. Verified on a checkout whose plugins/*/bin/ directories are empty.

1. Node.js 20+ and a .NET SDK (the sample targets net10.0-windows).
2. npm install --prefix mcp — the MCP server is the only thing with a dependency.
3. npm run demo.

Nothing is written into the repo tree: a demo run keeps its screenshots, build log, verify report and any failure records under .dsh-agent-toolchain/demo/ (gitignored).

Path B — full desktop mode

The API-capture store, the memory index, flame folding and the terminal inspector load Rust Node-API modules. Build the ones you need (Rust + Visual C++ build tools), or take a deployment that already ships them:

npm run build:capture-store        # dsh-api-visualizer (+ shared by dsh-verify)
npm run build:memory-store         # dsh-memory
npm run build:trace-fold           # dsh-perf flame folding
npm run build:terminal-inspector   # dsh-win-terminal-inspector

Then pwsh -File install.ps1 (dry run) or pwsh -File install.ps1 -Apply to copy the plugins into your DSH profile, register them in cordis.patch.yml, and restart the host.

What this does that a code-reading agent cannot

• Tell the difference between "0 errors" and "the file was compiled". Legacy .csproj projects do not include new .cs files automatically, so a build can report success while your file was never compiled. build_compile_check answers that question specifically, with three states — in the compilation set, provably not, or cannot be read (never silently "not").
• Operate and observe the real application. ui_observe / ui_drive / ui_flow find controls by name or AutomationId, click, type, wait for conditions, read values back, and capture window-scoped screenshots. Read-only actions never need permission; anything that clicks or types must pass allowSideEffects=true, and an optional snapshot-freshness gate rejects actions aimed at a stale UI.
• Jev-assisted UI decisions. ui_jev sends a sanitized UIA control list to Jev, lets it choose one bounded candidate action, then reuses the same deterministic UI executor and safety gates. Jev never receives pixels, generates arbitrary input, or bypasses authorization; see docs/jev-ui.md.
• Refuse to take the agent's word for it. A closing summary becomes a claims list, and verify_report adjudicates each claim against machine evidence — a build record, the API capture store, a file on disk, a real command, git state — producing pass / incomplete / fail. Claims that the evidence contradicts are recorded in the failure corpus (class agent-misjudge) automatically. When the tools cannot see something, they say "unverified", not "passed".

Tools — 55, of which 23 are read-only

The single source of truth is lib/tool-registry.mjs; both faces (DSH plugins and MCP) are generated from it, so they cannot drift apart. Every tool, one line each, sorted by what you are trying to do → docs/tools.md.

| Plugin | What it does for the agent | | --- | --- | | dsh-build | Run MSBuild / dotnet build as a tool: incremental build, structured error list (file/line/col/code), error re-parse, compile-membership check. Build defaults auto-resolve by repo layout (legacy client layouts keep their defaults byte-for-byte; stock repos get .sln/.slnx + platform auto-detection). | | dsh-ui-drive | Drive a running Windows desktop client via UIA: find/click/type/read/screenshot, visual-tree dumps, multi-step flows with assertions, screenshot + vision description. | | dsh-verify | The closing-adjudication surface: claims vs evidence verdicts, plus the failure corpus tools. Thin shell over lib/verify — the same engine behind the MCP verify_report. | | dsh-api-visualizer | Capture the client's HTTP traffic: live panel, JSONL store, caller attribution, auto-responder rules, baseline/contract regression, optional Fiddler-style local proxy. | | dsh-postman | Postman-style HTTP client inside the harness: compose/send requests from the host (no browser CORS), history store, WebSocket client, http_request agent tool. | | dsh-perf | UI stutter measurement (window-message latency, P50/P95/P99, stutter events), full-dump capture + ClrMD analysis, managed-heap stats, GC-root retention paths, ETW trace / hotstacks / flame and allocation folding, UI-freeze (wall-clock) analysis. | | dsh-hang-inspector | Hang diagnosis: monitor main-window responsiveness, auto-collect evidence packs (frozen screenshot, timeline, process info, net-trace tail, dump), analyze the managed thread stack and map the hang to project source. | | dsh-memory | Long-term memory for the harness: semantic search over indexed workspace docs, cross-session key-value conventions, mtime-incremental indexing, stale-chunk eviction, token/secret filter on save. | | dsh-win-terminal-inspector | Windows terminal (ConPTY) inspection for persistent shells. | | dsh-jev | Optional TypeSafe Jev decision layer: batched typed routing/evidence judgments with explicit remote-data opt-in; advisory only, never executes a selected action. |

Highlights

• One toolchain, every agent: the same tools power the DeepSeek Harness plugins and any MCP client — see mcp/. Claude Code, Cursor, Cline and Codex can drive the client, run builds, capture APIs and search memory with the exact same lib/ code.
• Safety-first UI automation: click/setvalue/key require an explicit allowSideEffects=true; read-only operations are always safe. An operational kill switch and an optional deny-first policy sit above every action, and recovery is deliberately not an agent tool.
• Honest measurement: perf numbers come from real window-message round trips and each report prints what the method cannot see; cost tables say "unknown" instead of inventing numbers; an empty enumeration is reported as "not read", never as "nothing there".
• The failure corpus is the moat: failure paths record themselves — build errors, ui_flow assertion failures, ui_drive/http failures, and verify_report claim-vs-evidence mismatches all append automatically under a fixed 7-class taxonomy. See docs/failure-corpus.md.
• Loopback-only control APIs, and every environment-specific value (client exe, window title, evidence dirs, tool paths, source roots) is an environment variable with a sane default: no hard-coded machines, no embedded credentials.

Works with any MCP client

claude mcp add --scope user dsh-agent-toolchain -- cmd /c node <repo>\mcp\server.mjs

Cursor, Cline and other stdio clients take the same command. Configuration details, progress notifications, output budgets and the honest platform matrix live in mcp/README.md.

Install (details)

Each plugin is a drop-in host plugin. Copy the plugin directory into your DSH profile's plugins/ (or node_modules/@dsh-agent-toolchain/ for the panels) and register it in cordis.patch.yml:

- insert:
    - id: ui-drive
      name: './plugins/dsh-ui-drive/index.js'

scripts/deploy-plugins.mjs (wrapped by install.ps1) is the supported way to do that copy; `--ch