Upload

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

CLAUDE.md

@@ -0,0 +1,199 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What this is
+
+A desktop pet (PySide6 window) that is a voice/click UI on top of an external
+Bolt server's desk API — same brain, memory, tools, and persona as that
+server's Discord bot and Linux desk client. This repo has **no import
+dependency** on the server repo; it's a standalone HTTP client configured via
+its own `.env`.
+
+Pipeline: `mic → openWakeWord ("thunderbolt", on-device) / push-to-talk /
+click → record utterance → Deepgram STT → + active-window context → POST
+/desk/converse → [server may relay a shell command to run on this machine, or
+a `petctl` pseudo-command that moves/emotes the pet instead] → reply →
+ElevenLabs streaming TTS (or offline pyttsx3 fallback) → speakers`, with the
+pet sprite/speech bubble reflecting state throughout, and playback
+interruptible by talking over it (barge-in).
+
+Side channels that let the pet act between turns: the heartbeat (proactive
+announcements), the desktop notification bridge, and autonomous wandering —
+all suppressed while it's napping (quiet hours / fullscreen DND).
+
+## Commands
+
+```bash
+# Setup + run (creates .venv and installs requirements.txt on first run)
+./run.sh          # macOS/Linux
+run.bat           # Windows
+
+# Run tests (no pytest config file — tests self-insert repo root via sys.path).
+# QT_QPA_PLATFORM=offscreen avoids a QApplication segfault on headless/no-display hosts.
+QT_QPA_PLATFORM=offscreen .venv/bin/pytest tests/
+.venv/bin/pytest tests/test_state.py::test_happy_path_transitions  # single test
+
+# Convert a grid sprite sheet into the per-frame-PNG convention sprite.py expects
+python scripts/slice_spritesheet.py path/to/sheet.png assets/sprites/idle --cols 6 --rows 1
+```
+
+There is no lint/build step configured beyond pytest. `cp .env.example .env`
+and fill in `BOLT_SERVER_URL` / `DESK_API_KEY` (+ `DEEPGRAM_API_KEY`,
+`ELEVENLABS_API_KEY`) before running — without server config the controller
+logs a missing-config message and exits its thread instead of starting.
+
+## Architecture
+
+- **`config.py`** — loads `.env` from the project root (not via python-dotenv;
+  a small hand-rolled parser matching the server repo's `desk_client/bolt_desk.py`
+  convention) into module-level constants. Everything else reads config from
+  here, never `os.environ` directly.
+- **`state.py`** — `PetStateMachine`, pure logic with no Qt/audio imports (kept
+  that way deliberately for cheap unit testing). Enforces a transition table;
+  notably `IDLE -> TALKING` is legal directly (no LISTENING/THINKING leg)
+  because the heartbeat can make the pet speak proactively/unprompted.
+- **`controller.py`** — `PetController(QObject)`, the pipeline orchestrator.
+  Runs on a background `QThread` (wired in `ui/app.py`) so audio I/O/network
+  never blocks the Qt event loop; communicates with the UI only through Qt
+  signals (`state_changed`, `said`, `log`, `action`, `napping`), never touches
+  a `QWidget` directly. Also drives the periodic heartbeat
+  (`_maybe_heartbeat`, gated by `HEARTBEAT_INTERVAL_SECONDS`) which lets the
+  server push proactive spoken announcements between user turns, and on the
+  same tick re-evaluates nap state and drains queued desktop notifications.
+  It owns the live wake-word threshold (`wake_threshold()` is passed to
+  `listen_for_wake_word` as a *callable* so the tray slider takes effect
+  mid-listen) and the conversation `history`.
+- **`server_client.py`** — HTTP client for the desk API, dependency-free
+  beyond `requests` so it's easy to mock in tests. `converse()` loops relaying
+  server-issued shell commands (`run_local_command`, executed via
+  `subprocess.run(shell=True)` as the desktop user, 30s default timeout) via
+  `/desk/tool_result` until the server sends a final `reply` (capped at
+  `_MAX_RELAY_HOPS`). This is the same "full desktop control" trust model as
+  the server repo's other desk clients — commands only ever originate from
+  the user's own voice/click requests in their own session.
+- **`audio/`** — `mic.py` (energy-based VAD utterance capture, ported from the
+  server repo's `bolt_desk.py`), `wake_word.py` (openWakeWord `thunderbolt.onnx`
+  detection + `NearMissLog` for threshold tuning — see below), `stt.py`
+  (Deepgram), `tts.py` (ElevenLabs, streaming by default — `stream_pcm()` +
+  `play_stream()` start playback on the first chunk; `chunks_to_int16()`
+  carries odd bytes across HTTP chunk boundaries, without which everything
+  after the first split sample plays as static — falling back to whole-clip
+  PCM then offline `pyttsx3`), `barge_in.py` (`BargeInDetector`: N consecutive
+  loud mic frames while the pet is talking cuts playback and starts the next
+  turn; threshold is deliberately ~4x the VAD one because the mic hears the
+  pet's own voice). Each accepts an injectable stream/model/protocol so tests
+  don't need real audio hardware or a display.
+- **`pet_actions.py`** — `petctl` pseudo-commands (`petctl move top-left`,
+  `petctl emote wave`, `say`/`wander`/`nap`). The desk API has no "move the
+  pet" payload type and this repo can't change the server, so these ride the
+  existing shell-command relay: `controller._handle_command` parses them and
+  they never reach `subprocess`; anything else is a real shell command exactly
+  as before. Pure parsing; the UI half is `PetWindow.apply_action`.
+- **`screen_context.py`** — active-window title (xprop/xdotool, Win32,
+  osascript) appended to each utterance via `context_for()`, plus
+  `is_fullscreen_active()` for do-not-disturb. Text only — the desk API takes
+  no images. Every probe is best-effort and returns None/False rather than
+  raising; the parsing is split into pure functions that are tested without a
+  display server.
+- **`quiet.py`** — quiet-hours spec parsing (`23:00-08:00`, wraps midnight,
+  comma-separated). Napping suppresses *proactive* noise and wandering only;
+  wake word / click / push-to-talk still work.
+- **`notifications.py`** — Linux/D-Bus notification bridge: tails
+  `dbus-monitor`, parses Notify calls (pure `iter_notifications()`), filters
+  and rate-limits them (`NotificationGate`), and the controller forwards
+  survivors through `converse()`. Off by default — each one is a round trip.
+- **`history.py`** — rolling transcript (`HISTORY_LIMIT` turns) behind the
+  tray's History window and click-to-copy on the bubble.
+- **`hotkey.py`** — global push-to-talk via `pynput`; soft-fails with a logged
+  reason (Wayland, missing package, macOS permissions) since the wake word is
+  the primary trigger.
+- **`speech_text.py`** — sanitizes server replies before they're heard/shown.
+  `for_speech()` (called inside `tts.speak()`, so every path to the speakers is
+  covered) strips markdown, emoji, URLs and stray symbols the voice would read
+  literally ("asterisk asterisk"), turns bullet lists into full sentences, and
+  words a few symbols (`&` → "and"). `for_display()` is the looser version for
+  the speech bubble — markdown syntax gone, emoji kept. Pure string logic, no
+  Qt/audio imports.
+- **`ui/`** — `app.py` wires `QApplication` + `PetWindow` + `PetTray` + the
+  history/tuner windows + the push-to-talk hotkey + the controller thread
+  together; `pet_window.py` is the frameless/translucent/always-on-top sprite
+  window + speech bubble (non-square frames are centered in the square
+  `PET_SIZE` window, see `paintEvent`), and also owns:
+  - **wandering** — a ~30fps timer walks the window toward a random on-screen
+    target every `PET_WANDER_INTERVAL_SECONDS` (randomized), suppressed
+    whenever the pet is non-IDLE, napping, dragged, or has a bubble up. A
+    commanded `petctl move` overrides all of that except the drag.
+  - **emotes** — `emote_transform()` is pure maths (dx, dy, rotation, scale
+    from a 0..1 progress) kept out of `paintEvent` so the curves are unit
+    tested; every emote must return to the identity transform at progress 1.0
+    or the pet ends up permanently askew.
+  - **shaped input / click-through** — `PET_SHAPED_INPUT` masks the window to
+    the sprite's opaque pixels so the square window's transparent corners stop
+    eating clicks (mask rebuilt only when the frame changes, and pinned to the
+    resting position so a bob/spin doesn't thrash it); `PET_CLICK_THROUGH`
+    makes the pet ignore the mouse entirely.
+  - **edge snapping** (`PET_EDGE_SNAP`) after a drag or a stroll, and **nap
+    dimming** (`set_napping`).
+  `sprite.py` loads `assets/sprites/<state>/*.png` (filename-sorted, looping —
+  currently Kenney's CC0 robot pack, see `assets/sprites/README.md`) and falls
+  back to a procedurally-drawn placeholder blob per state if a folder has no
+  frames; `tray.py` is the system tray menu (talk now / mute / nap / wander /
+  click-through / history / wake-word tuning / quit) — the pet window has no
+  title bar or taskbar entry; `history_window.py` and `wake_tuner.py` are the
+  two dialogs it opens.
+
+### Wake-word detection
+
+`audio/wake_word.py` uses a custom-trained openWakeWord model,
+`thunderbolt.onnx` (ships in the project root), the same way the server
+repo's `desk_client/bolt_desk.py` uses `bolt.onnx` for "hey bolt" — same
+runtime (openWakeWord, ONNX inference framework), same per-frame
+`predict()`/`reset()` loop. Every mic frame is scored; any class score at or
+above `WAKE_WORD_THRESHOLD` (default `0.5`, in `.env`) counts as a
+detection. Swap `WAKE_MODEL_FILE` to point at a differently-trained `.onnx`
+model to change the wake phrase — everything downstream (STT, server call,
+TTS) is unaffected.
+
+The threshold is tunable at runtime: the tray's **Wake word tuning…** window
+(`ui/wake_tuner.py`) shows the peak score seen and a rolling list of near
+misses (frames within `WAKE_NEAR_MISS_MARGIN` *below* the threshold — i.e.
+the times it nearly heard you), and its slider is read per frame because
+`listen_for_wake_word` accepts a callable threshold. Set the threshold just
+under the peak you can hit reliably, then persist it in `.env`.
+
+### Testing conventions
+
+`tests/` covers pure logic only (state machine, wake-word scoring loop, mic
+VAD, HTTP client against mocks) — nothing there needs real audio hardware or
+a display. Modules under test are written to accept fake streams/models/
+`on_command` callables specifically to keep tests hardware-free; follow that
+pattern (inject a `Protocol`-typed collaborator) rather than mocking at the
+`sounddevice`/`openwakeword` import boundary when adding new testable logic.
+`test_controller.py`, `test_controller_features.py`, `test_wander.py` and
+`test_pet_window_features.py` need a `QApplication`, which segfaults without
+a display unless run with `QT_QPA_PLATFORM=offscreen`.
+
+Newer subsystems follow the same rule — the testable part is separated from
+the part that needs hardware: dbus-monitor output is parsed by a pure
+`iter_notifications(lines)`, xprop output by pure `parse_xprop_*` functions,
+HTTP chunk reassembly by `chunks_to_int16`, emote motion by
+`emote_transform`. Tests that touch the controller monkeypatch
+`screen_context.context_for` / `is_fullscreen_active`, otherwise they shell
+out to xprop on a headless box.
+
+## Security notes
+
+The server can relay a shell command back to this machine to execute as the
+desktop user (see `server_client.run_local_command`). This is intentional
+("full desktop control" for things like "open firefox" or disk checks) and
+matches the trust model of the server repo's other desk clients. Keep
+`DESK_API_KEY` private and don't expose the desk API port to the open
+internet.
+
+Two newer features widen what leaves this machine, both switchable in `.env`:
+`SCREEN_CONTEXT` appends the focused window's *title* to each utterance
+(titles often contain file paths, document names, or subject lines), and
+`NOTIFICATION_BRIDGE` (off by default) forwards matching desktop
+notifications to the server. Neither sends screenshots or notification
+contents you haven't matched with `NOTIFICATION_FILTER`.