AgentDeck

Stop Chatting. Start Steering.

AgentDeck is a physical control surface for AI coding agents. It started with an Elgato Stream Deck+ and now runs on 12 display surfaces simultaneously — tablets, e-ink readers, phones, ESP32 modules, LED matrices, and terminals.

One bridge. 12 surfaces. Steer your AI — without leaving your keyboard flow.

<img src="assets/AgentDeck_SNS_Collage.png" width="720" alt="AgentDeck collage — Stream Deck+, Android, e-ink, Apple devices, ESP32 displays, Pixoo64, and TUI dashboard"> <a href="https://youtu.be/s-f8ICBcC4o">Watch Demo on YouTube</a> <img src="docs/media/desk-setup-2.jpg" width="720" alt="AgentDeck — TUI dashboard, Pixoo64, Stream Deck+, and all surfaces running simultaneously">

| | Requirement | |---|---| | Platform | macOS 14+ (Sonoma) — Windows/Linux not supported | | Hardware | Elgato Stream Deck+ (8 keys, 4 encoders, LCD touch strip) | | Terminal | iTerm2 (required for session management and voice paste) | | Android | (Optional) Android 10+ tablet or e-ink reader for remote dashboard | | Apple | (Optional) iOS 17+ / iPadOS 17+ / macOS 14+ for SwiftUI dashboard | | TUI | (Optional) Any terminal with truecolor support for agentdeck dashboard |

What is AgentDeck?
Prerequisites
Quick Start
Manual Build & Install
Usage
- CLI Reference
Stream Deck+ Layout (v3)
Android Dashboard
Apple Dashboard
TUI Dashboard
ESP32 Display
Pixoo64 LED Matrix
Configuration
Troubleshooting
Uninstall
Development
Roadmap

Documentation

Voice Setup — sox, whisper.cpp install, model selection
Wake Word Detection — Porcupine (Mac) + microWakeWord (ESP32) setup
Stream Deck+ Layout Reference — per-state layouts, encoder details, button colors
Android Reference — device support, build/signing, creature behavior
Device Reference — 7 dashboard device types, transport protocols, broadcast architecture
Protocol & Architecture — state machine, WebSocket messages, project structure
Testing Guide — test structure, coverage, CI pipeline, writing tests

What is AgentDeck?

A control surface — like an audio mixing console, but for AI coding agents. It reads your agent's state in real-time and dynamically reconfigures buttons and encoders to match what's happening right now.

Respond instantly — YES / NO / ALWAYS buttons appear with semantic colors for permission prompts
Interrupt — STOP button sends Ctrl+C to a runaway agent
Switch modes — cycle Plan / Accept Edits / Default
Navigate options — encoder scrolls and selects multi-choice prompts on a wide-canvas LCD
Voice input — push-to-talk → whisper.cpp → auto-send (offline, <2s on M-series)
Voice assistant — wake word detection → whisper.cpp STT → LLM → TTS response (fully offline)
Display sync — macOS sleep dims all connected surfaces; wake restores instantly
Terminal postit — agent state shown in iTerm2 tab titles and badges
Monitor usage — animated water-gauge dashboard with rate limit countdowns
Quick actions — GO ON / REVIEW / COMMIT / CLEAR; encoder cycles custom prompts
System utilities — volume, mic, media, timer from the Utility encoder
Terminal sessions — iTerm dial switches sessions, auto-attaches tmux
Multiple coding agents — Claude Code, Codex CLI, and OpenClaw in one multi-agent daemon view
Works from anywhere — all 12 surfaces can monitor the agent; interactive surfaces (Stream Deck, Android, Apple) can also control it

The bridge is transparent: if it's off, Claude Code works exactly as before.

Supported Agents

| Agent | Status | |-------|--------| | Claude Code | Supported (primary) | | Codex CLI | Supported — PTY parser, model detection, and dashboard integration | | OpenClaw | Experimental — Gateway WebSocket, timeline panel, log stream |

Supported Surfaces — 12 Types

| # | Surface | Description | |---|---------|-------------| | 1 | Stream Deck+ | Primary — 8 keys, 4 encoders, LCD touch strip | | 2 | Android Tablet | Color terrarium + HUD overlay (60fps) | | 3 | E-ink Reader | B&W 16-level grayscale + Color E-ink (Kaleido 3, 4096 colors) + partial refresh | | 4 | iPhone | SwiftUI app — mobile agent monitoring | | 5 | iPad | SwiftUI app — terrarium second screen | | 6 | macOS | SwiftUI app — desktop monitoring window | | 7 | ESP32 Round AMOLED | 1.8" circular 360×360 — compact WiFi display | | 8 | ESP32 IPS LCD | 3.5" rectangular 480×320 | | 9 | ESP32 B86 Box | 4" wall-mount touch panel 480×480 | | 10 | Ulanzi TC001 | 8×32 RGB LED matrix — compact HUD pages and creature sprites | | 11 | Pixoo64 LED | 64×64 RGB LED pixel art terrarium | | 12 | TUI Terminal | Unicode braille terrarium + ANSI dashboard — SSH/remote |

<img src="docs/media/ipad-iphone-closeup.jpg" width="360" alt="iPad and iPhone showing terrarium with pixel art creatures">    <img src="docs/media/esp32-pixoo-closeup.jpg" width="360" alt="ESP32 Round AMOLED, IPS LCD, B86 Box, and Pixoo64 LED matrix"> Left: iPad + iPhone (SwiftUI)  |  Right: ESP32 3 types + Pixoo64 LED <img src="docs/media/eink-dual-display.jpg" width="360" alt="Dual E-ink display — Crema S (B&W) and Pantone 6 (Color Kaleido 3) showing multi-agent aquarium with ESP32 Round AMOLED">    <img src="docs/media/tui-dashboard.png" width="360" alt="TUI Dashboard in terminal — braille octopus, rate limit gauges, timeline"> Left: Dual E-ink — B&W (Crema S) + Color (Pantone 6) + ESP32 AMOLED  |  Right: TUI terminal dashboard

Architecture

                              ┌── Daemon (port 9120, sole hub) ──┐
Stream Deck Plugin ◄── WS ──►│                                   │
Android Dashboard  ◄── WS ──►│  WS Server + mDNS + Device Mods  │
Apple Dashboard    ◄── WS ──►│  Gateway Proxy + Usage Relay      │
TUI Dashboard      ◄── WS ──►│  Pixoo + ESP32 Serial + SSE      │
ESP32 Display      ◄ Serial ►│                                   │
Pixoo64 LED        ◄ HTTP ──►└───────────────┬───────────────────┘
                                             │ aggregates
                              ┌── Session Bridge (port 9121+) ──┐
User's Terminal ◄─ stdio ───►│  PTY Manager → claude CLI         │
Claude Code Hooks ─ HTTP ───►│  Output Parser → State Machine    │
                              │  Hook Server + Voice (whisper)    │
                              └──────────────────────────────────┘

The daemon is the sole hub for all dashboard clients. Session bridges handle PTY + hooks only. The daemon aggregates state from all sessions and broadcasts to all 12 surfaces. Local clients are auto-trusted; LAN clients authenticate with a token from ~/.agentdeck/auth-token. Interactive surfaces (Stream Deck, Android, Apple) can control the agent; monitoring surfaces (Pixoo, TUI, ESP32) display state.

Prerequisites

| Item | Required | Install | |------|----------|---------| | macOS 14+ (Sonoma) | Yes | Windows/Linux not supported | | Xcode Command Line Tools | Yes | xcode-select --install (node-pty native build) | | Node.js >= 22 | Yes | brew install node | | pnpm >= 9 | Yes | npm install -g pnpm | | Python 3 | Yes | brew install python (display sleep detection) | | Elgato Stream Deck app >= 6.7 | Yes | Elgato Downloads | | Stream Deck+ hardware | Yes | 8 keys + 4 encoders + LCD touch strip | | iTerm2 | Yes | Terminal management, voice paste, session switching | | Claude Code CLI | Yes | `npm in

AgentDeck

Install / Use

README

AgentDeck

Table of Contents

Documentation

What is AgentDeck?

Supported Agents

Supported Surfaces — 12 Types

Architecture

Prerequisites