Bolt-Pet/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this is

A desktop pet (PySide6 window) that is a voice/click UI on top of an external Bolt server's desk API — same brain, memory, tools, and persona as that server's Discord bot and Linux desk client. This repo has no import dependency on the server repo; it's a standalone HTTP client configured via its own .env.

Pipeline: mic → openWakeWord ("thunderbolt", on-device) / push-to-talk / click → record utterance → Deepgram STT → + active-window context → POST /desk/converse → [server may relay a shell command to run on this machine, or a petctl pseudo-command that moves/emotes the pet instead] → reply → ElevenLabs streaming TTS (or offline pyttsx3 fallback) → speakers, with the pet sprite/speech bubble reflecting state throughout, and playback interruptible by talking over it (barge-in).

Side channels that let the pet act between turns: the heartbeat (proactive announcements), the desktop notification bridge, and autonomous wandering — all suppressed while it's napping (quiet hours / fullscreen DND).

Commands

# Setup + run (creates .venv and installs requirements.txt on first run)
./run.sh          # macOS/Linux
run.bat           # Windows

# Run tests (no pytest config file — tests self-insert repo root via sys.path).
# QT_QPA_PLATFORM=offscreen avoids a QApplication segfault on headless/no-display hosts.
QT_QPA_PLATFORM=offscreen .venv/bin/pytest tests/
.venv/bin/pytest tests/test_state.py::test_happy_path_transitions  # single test

# Convert a grid sprite sheet into the per-frame-PNG convention sprite.py expects
python scripts/slice_spritesheet.py path/to/sheet.png assets/sprites/idle --cols 6 --rows 1

There is no lint/build step configured beyond pytest. cp .env.example .env and fill in BOLT_SERVER_URL / DESK_API_KEY (+ DEEPGRAM_API_KEY, ELEVENLABS_API_KEY) before running — without server config the controller logs a missing-config message and exits its thread instead of starting.

Architecture

• config.py — loads .env from the project root (not via python-dotenv; a small hand-rolled parser matching the server repo's desk_client/bolt_desk.py convention) into module-level constants. Everything else reads config from here, never os.environ directly.
• state.py — PetStateMachine, pure logic with no Qt/audio imports (kept that way deliberately for cheap unit testing). Enforces a transition table; notably IDLE -> TALKING is legal directly (no LISTENING/THINKING leg) because the heartbeat can make the pet speak proactively/unprompted.
• controller.py — PetController(QObject), the pipeline orchestrator. Runs on a background QThread (wired in ui/app.py) so audio I/O/network never blocks the Qt event loop; communicates with the UI only through Qt signals (state_changed, said, log, action, napping), never touches a QWidget directly. Also drives the periodic heartbeat (_maybe_heartbeat, gated by HEARTBEAT_INTERVAL_SECONDS) which lets the server push proactive spoken announcements between user turns, and on the same tick re-evaluates nap state and drains queued desktop notifications. It owns the live wake-word threshold (wake_threshold() is passed to listen_for_wake_word as a callable so the tray slider takes effect mid-listen) and the conversation history.
• server_client.py — HTTP client for the desk API, dependency-free beyond requests so it's easy to mock in tests. converse() loops relaying server-issued shell commands (run_local_command, executed via subprocess.run(shell=True) as the desktop user, 30s default timeout) via /desk/tool_result until the server sends a final reply (capped at _MAX_RELAY_HOPS). This is the same "full desktop control" trust model as the server repo's other desk clients — commands only ever originate from the user's own voice/click requests in their own session.
• audio/ — mic.py (energy-based VAD utterance capture, ported from the server repo's bolt_desk.py), wake_word.py (openWakeWord thunderbolt.onnx detection + NearMissLog for threshold tuning — see below), stt.py (Deepgram), tts.py (ElevenLabs, streaming by default — stream_pcm() + play_stream() start playback on the first chunk; chunks_to_int16() carries odd bytes across HTTP chunk boundaries, without which everything after the first split sample plays as static — falling back to whole-clip PCM then offline pyttsx3), barge_in.py (BargeInDetector: N consecutive loud mic frames while the pet is talking cuts playback and starts the next turn; threshold is deliberately ~4x the VAD one because the mic hears the pet's own voice). Each accepts an injectable stream/model/protocol so tests don't need real audio hardware or a display.
• pet_actions.py — petctl pseudo-commands (petctl move top-left, petctl emote wave, say/wander/nap). The desk API has no "move the pet" payload type and this repo can't change the server, so these ride the existing shell-command relay: controller._handle_command parses them and they never reach subprocess; anything else is a real shell command exactly as before. Pure parsing; the UI half is PetWindow.apply_action.
• screen_context.py — active-window title (xprop/xdotool, Win32, osascript) appended to each utterance via context_for(), plus is_fullscreen_active() for do-not-disturb. Text only — the desk API takes no images. Every probe is best-effort and returns None/False rather than raising; the parsing is split into pure functions that are tested without a display server.
• quiet.py — quiet-hours spec parsing (23:00-08:00, wraps midnight, comma-separated). Napping suppresses proactive noise and wandering only; wake word / click / push-to-talk still work.
• notifications.py — Linux/D-Bus notification bridge: tails dbus-monitor, parses Notify calls (pure iter_notifications()), filters and rate-limits them (NotificationGate), and the controller forwards survivors through converse(). Off by default — each one is a round trip.
• history.py — rolling transcript (HISTORY_LIMIT turns) behind the tray's History window and click-to-copy on the bubble.
• hotkey.py — global push-to-talk via pynput; soft-fails with a logged reason (Wayland, missing package, macOS permissions) since the wake word is the primary trigger.
• speech_text.py — sanitizes server replies before they're heard/shown. for_speech() (called inside tts.speak(), so every path to the speakers is covered) strips markdown, emoji, URLs and stray symbols the voice would read literally ("asterisk asterisk"), turns bullet lists into full sentences, and words a few symbols (& → "and"). for_display() is the looser version for the speech bubble — markdown syntax gone, emoji kept. Pure string logic, no Qt/audio imports.
• ui/ — app.py wires QApplication + PetWindow + PetTray + the history/tuner windows + the push-to-talk hotkey + the controller thread together; pet_window.py is the frameless/translucent/always-on-top sprite window + speech bubble (non-square frames are centered in the square PET_SIZE window, see paintEvent), and also owns:
  • wandering — a ~30fps timer walks the window toward a random on-screen target every PET_WANDER_INTERVAL_SECONDS (randomized), suppressed whenever the pet is non-IDLE, napping, dragged, or has a bubble up. A commanded petctl move overrides all of that except the drag.
  • emotes — emote_transform() is pure maths (dx, dy, rotation, scale from a 0..1 progress) kept out of paintEvent so the curves are unit tested; every emote must return to the identity transform at progress 1.0 or the pet ends up permanently askew.
  • shaped input / click-through — PET_SHAPED_INPUT masks the window to the sprite's opaque pixels so the square window's transparent corners stop eating clicks (mask rebuilt only when the frame changes, and pinned to the resting position so a bob/spin doesn't thrash it); PET_CLICK_THROUGH makes the pet ignore the mouse entirely.
  • edge snapping (PET_EDGE_SNAP) after a drag or a stroll, and nap dimming (set_napping). sprite.py loads assets/sprites/<state>/*.png (filename-sorted, looping — currently Kenney's CC0 robot pack, see assets/sprites/README.md) and falls back to a procedurally-drawn placeholder blob per state if a folder has no frames; tray.py is the system tray menu (talk now / mute / nap / wander / click-through / history / wake-word tuning / quit) — the pet window has no title bar or taskbar entry; history_window.py and wake_tuner.py are the two dialogs it opens.

Wake-word detection

audio/wake_word.py uses a custom-trained openWakeWord model, thunderbolt.onnx (ships in the project root), the same way the server repo's desk_client/bolt_desk.py uses bolt.onnx for "hey bolt" — same runtime (openWakeWord, ONNX inference framework), same per-frame predict()/reset() loop. Every mic frame is scored; any class score at or above WAKE_WORD_THRESHOLD (default 0.5, in .env) counts as a detection. Swap WAKE_MODEL_FILE to point at a differently-trained .onnx model to change the wake phrase — everything downstream (STT, server call, TTS) is unaffected.

The threshold is tunable at runtime: the tray's Wake word tuning… window (ui/wake_tuner.py) shows the peak score seen and a rolling list of near misses (frames within WAKE_NEAR_MISS_MARGIN below the threshold — i.e. the times it nearly heard you), and its slider is read per frame because listen_for_wake_word accepts a callable threshold. Set the threshold just under the peak you can hit reliably, then persist it in .env.

Testing conventions

tests/ covers pure logic only (state machine, wake-word scoring loop, mic VAD, HTTP client against mocks) — nothing there needs real audio hardware or a display. Modules under test are written to accept fake streams/models/ on_command callables specifically to keep tests hardware-free; follow that pattern (inject a Protocol-typed collaborator) rather than mocking at the sounddevice/openwakeword import boundary when adding new testable logic. test_controller.py, test_controller_features.py, test_wander.py and test_pet_window_features.py need a QApplication, which segfaults without a display unless run with QT_QPA_PLATFORM=offscreen.

Newer subsystems follow the same rule — the testable part is separated from the part that needs hardware: dbus-monitor output is parsed by a pure iter_notifications(lines), xprop output by pure parse_xprop_* functions, HTTP chunk reassembly by chunks_to_int16, emote motion by emote_transform. Tests that touch the controller monkeypatch screen_context.context_for / is_fullscreen_active, otherwise they shell out to xprop on a headless box.

Security notes

The server can relay a shell command back to this machine to execute as the desktop user (see server_client.run_local_command). This is intentional ("full desktop control" for things like "open firefox" or disk checks) and matches the trust model of the server repo's other desk clients. Keep DESK_API_KEY private and don't expose the desk API port to the open internet.

Two newer features widen what leaves this machine, both switchable in .env: SCREEN_CONTEXT appends the focused window's title to each utterance (titles often contain file paths, document names, or subject lines), and NOTIFICATION_BRIDGE (off by default) forwards matching desktop notifications to the server. Neither sends screenshots or notification contents you haven't matched with NOTIFICATION_FILTER.