ClawPuter 🦞

中文

A pixel-art desktop companion running on M5Stack Cardputer (ESP32-S3). Features an OpenClaw-themed lobster character with animations, real-time weather, AI chat with voice input/TTS, and a synced macOS desktop pet.

<p align="center"> <img src="assets/hardware.png" alt="ClawPuter on M5Stack Cardputer" width="500"> <img src="assets/desktop-pet.png" alt="ClawPuter Desktop Pet" width="500"> </p>

Why "ClawPuter"?

The idea started one afternoon at a friend's office. We were chatting about OpenClaw and I mentioned I'd been getting into hardware lately. He happened to have an M5Stack Cardputer lying around and lent it to me. That same afternoon, the concept clicked: what if we gave OpenClaw — a digital life — a more embodied, physical form? The Cardputer was the perfect vessel. So I mashed the names together: Claw (from OpenClaw) + Cardputer = ClawPuter.

Features

• Companion Mode — Pixel lobster with idle, happy, sleep, talk, stretch, and look-around animations. Movable with keyboard (hold to walk). Day/night background with time-travel sky (walk left = past, right = future). NTP clock display.
• Moisture System — Virtual hydration mechanic driven by real weather. Humidity affects decay speed, rain pauses decay, walking drains water faster. Press H to spray water and restore moisture. When dehydrated: movement slows, chat is locked, pet shows "so thirsty..." status. Water drop indicators in the top bar and on the desktop scene.
• Real-Time Weather — Fetches weather from Open-Meteo API every 15 minutes. Background effects for rain, drizzle, snow, fog, and thunderstorms. Pet wears weather-appropriate accessories (sunglasses, umbrella, snow hat, mask). Temperature displayed next to the clock.
• Weather Simulation — Fn+W toggles simulation mode, number keys 1-8 to preview all 8 weather types (Clear, Cloudy, Overcast, Fog, Drizzle, Rain, Snow, Thunder).
• Chat Mode — Full keyboard input, AI conversation with SSE streaming (token-by-token display), scrollable message history with word-wrap.
• Pixel Art Generation — /draw a cat generates 8x8 AI pixel art rendered as a 96x96 color grid in chat. /draw16 for 16x16. 16-color palette, mixed with text messages, auto-syncs to Mac desktop.
• Voice Input — Push-to-talk via Fn key, speech-to-text via Groq Whisper API (through a local proxy server).
• TTS Voice Replies — AI responses are spoken through the built-in speaker. Press any key to interrupt playback. Mic and speaker share GPIO 43 — the system handles switching automatically.
• Desktop Pet Sync — macOS companion app receives lobster state, position, and weather over UDP, rendering a synced desktop pet on your Mac. Supports Follow Mode (cursor tracking) and Scene Mode (weather panel).
• Desktop Bidirectional Control — Mac ↔ ESP32 two-way communication. Remote trigger animations, send text/messages to Cardputer, forward notifications as toast overlays, view synced chat history, and display pixel art in a high-res popover. Pet perches on the Chat Viewer window when open.
• Thinking Model Compatibility — Works with reasoning models (GPT-5.4, Claude, etc.) that output extended thinking. Thinking content is automatically filtered, with a "thinking..." indicator shown during the thinking phase. Gateway fallback injection is detected and handled gracefully.
• OpenClaw Integration — Connects to your local OpenClaw Gateway over LAN. Multi-model fallback (Kimi/Claude/GPT/Gemini), persistent memory, 5400+ community skills.
• Dual WiFi + Offline Mode — Auto-fallback to secondary WiFi (e.g. phone hotspot). Gateway host switches automatically. Offline mode if all WiFi fails (companion works, chat shows offline).
• Runtime Config — Setup wizard to configure WiFi, Gateway, STT host at runtime. Build-time values as defaults, Fn+R to reset.
• Boot Animation — Lobster pixel-art line-by-line reveal with pixel wipe transitions between modes.
• Sound Effects — Key clicks, happy melody, typing chirps during AI streaming, notification tones.

Quick Start

1. Set Environment Variables

# WiFi
export WIFI_SSID="<your-wifi-ssid>"
export WIFI_PASS="<your-wifi-password>"

# AI Backend (OpenClaw Gateway)
export OPENCLAW_HOST="<your-host-ip>"       # OpenClaw Gateway LAN IP
export OPENCLAW_PORT="<your-port>"           # Gateway port
export OPENCLAW_TOKEN="<your-gateway-token>"

# Voice Input (optional, for stt_proxy.py)
export GROQ_API_KEY="<your-groq-api-key>"    # Used by tools/stt_proxy.py, not the firmware
export STT_PROXY_HOST="<your-host-ip>"       # Machine running stt_proxy.py
export STT_PROXY_PORT="8090"                 # STT proxy port (default 8090)

# Weather (optional)
export DEFAULT_CITY="Beijing"                # City name for weather lookup

# Secondary WiFi (optional, for phone hotspot fallback)
export WIFI_SSID2="<your-hotspot-ssid>"
export WIFI_PASS2="<your-hotspot-password>"
export OPENCLAW_HOST2="<your-hotspot-host-ip>"  # Mac's IP on the hotspot network

2. Build & Flash

pio run -t upload

First-time flashing may require download mode: hold G0 + press Reset, then release G0. See Setup Guide for details.

3. Start STT Proxy (for voice input)

python3 tools/stt_proxy.py

The proxy runs on your Mac/PC, relays audio from the Cardputer to Groq Whisper API, and returns transcriptions. Requires GROQ_API_KEY in your .env or environment.

4. Serial Debug

pio device monitor

Controls

| Key | Companion Mode | Chat Mode | |-----|---------------|-----------| | TAB | Switch to chat | Switch to companion | | , (hold) | Move left | — | | / (hold) | Move right | — | | ; (hold) | Move up | — | | . (hold) | Move down | — | | Space / Enter | Happy animation | Send message | | Backspace | — | Delete character | | Fn (hold) | — | Push-to-talk voice input | | Fn + ; | — | Scroll up | | Fn + / | — | Scroll down | | H | Spray water (when moisture ≤ 1) | Spray water (when moisture ≤ 1), otherwise type 'h' | | Fn + W | Toggle weather simulation | — | | 1-8 (in weather sim) | Select weather type | — | | Fn + R | Reset config + setup wizard | — | | Fn + 0 | Debug: set moisture to 0 | — | | Any key (while sleeping) | Wake up lobster | — | | Any key (during TTS) | — | Stop voice playback | | TAB (in setup) | Exit setup, return to companion | — |

What Can It Do?

Companion Mode — Your Pixel Pet

The lobster lives on a 240x135 pixel screen with a dynamic background:

• Day/Night cycle — Sky color changes based on real time (NTP). Blue sky with sun and clouds during daytime, orange sunset at 17-19h, dark night with twinkling stars and crescent moon after 19h.
• Time-travel sky — Moving the pet left/right shifts the displayed sky ±12 hours. Walk to the left edge to see last night's starry sky, or to the right to see tomorrow's sunrise. The clock at the bottom always shows the real time.
• Spontaneous actions — The pet randomly stretches or looks around every 8-15 seconds when idle. After 30 seconds of no interaction, it falls asleep with animated "Zzz". Press any key to wake it up.
• Keyboard movement — Hold , / ; . for continuous walking. The sprite flips to face the direction of movement.
• Interaction — Press Space or Enter to make the pet jump happily with a cheerful melody.

Moisture System — Keep Your Lobster Hydrated

A virtual hydration mechanic tied to real-world weather:

• 4-level moisture indicator shown in the top bar (water drops). Desktop scene mode mirrors the same indicator.
• Weather-driven decay: low humidity (< 40%) = fast drain, high humidity (> 70%) = slow drain, rain/snow pauses decay entirely.
• Movement drain: walking around costs moisture — every 20 steps drains 1 level. Stay still to conserve water.
• Spray water: press H to restore 1 moisture level (3-second cooldown). Triggers a water spray particle animation and happy jump.
• Dehydration effects: at moisture 1, movement slows to 50% and the pet shows "need water...". At moisture 0, the pet cannot move, cannot jump, chat mode is locked ("thirsty... press H to spray!"), and shows "so thirsty...".
• Water bottle accessory: when moisture ≤ 1, the pet holds a tiny water bottle instead of weather accessories.
• Rain recovery: during rain/thunderstorms, moisture auto-recovers +1 every 15 minutes.
• Initial moisture: set based on first weather data — dry climate starts thirsty, humid climate starts comfortable.

Weather — Real-Time & Simulated

• Auto-updates every 15 minutes from Open-Meteo (free, no API key needed). City configured via DEFAULT_CITY env var.
• Background effects: rain drops fall, snow drifts with horizontal drift, fog dots shimmer, thunder flashes the screen white. Sky tints darker under heavy weather. Sun/moon/stars hidden during overcast conditions.
• Accessories: the pet automatically wears sunglasses (clear/cloudy), umbrella (rain/drizzle/thunder), snow hat (snow), or mask (fog/overcast). Accessories flip with the sprite direction.
• Temperature shown next to the clock with a ° symbol, separated by a vertical divider.
• Simulation mode: Fn+W enters weather simulation, keys 1-8 preview all 8 weather types (1=Clear, 2=Cloudy, 3=Overcast, 4=Fog, 5=Drizzle, 6=Rain, 7=Snow, 8=Thunder). A [SIM] status bar appears at the bottom. Press Fn+W again to return to real weather.

Chat Mode — AI Conversation

• Type messages and press Enter to send. AI replies stream token-by-token with typing sound effects.
• Pixel art: Type /draw a cat to generate an 8x8 pixel art, or /draw16 a heart for 16x16. AI returns hex-encoded pixel data, rendered as a 96x96 color grid inline with chat messages. 16-color fixed palette. Parse failures gracefully fall back to plain text.
• Voice input: Hold Fn to record (up