Bolt-Pet/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this is

A desktop pet (PySide6 window) that is a voice/click UI on top of an external
Bolt server's desk API — same brain, memory, tools, and persona as that
server's Discord bot and Linux desk client. This repo has **no import
dependency** on the server repo; it's a standalone HTTP client configured via
its own `.env`.

Pipeline: `mic → openWakeWord ("thunderbolt", on-device) / push-to-talk /
click → record utterance → Deepgram STT → + active-window context → POST
/desk/converse → [server may relay a shell command to run on this machine, or
a `petctl` pseudo-command that moves/emotes the pet instead] → reply →
ElevenLabs streaming TTS (or offline pyttsx3 fallback) → speakers`, with the
pet sprite/speech bubble reflecting state throughout, and playback
interruptible by talking over it (barge-in).

Side channels that let the pet act between turns: the heartbeat (proactive
announcements), the desktop notification bridge, and autonomous wandering —
all suppressed while it's napping (quiet hours / fullscreen DND).

## Commands

```bash
# Setup + run (creates .venv and installs requirements.txt on first run)
./run.sh          # macOS/Linux
run.bat           # Windows

# Run tests (no pytest config file — tests self-insert repo root via sys.path).
# QT_QPA_PLATFORM=offscreen avoids a QApplication segfault on headless/no-display hosts.
QT_QPA_PLATFORM=offscreen .venv/bin/pytest tests/
.venv/bin/pytest tests/test_state.py::test_happy_path_transitions  # single test

# Convert a grid sprite sheet into the per-frame-PNG convention sprite.py expects
python scripts/slice_spritesheet.py path/to/sheet.png assets/sprites/idle --cols 6 --rows 1
```

There is no lint/build step configured beyond pytest. `cp .env.example .env`
and fill in `BOLT_SERVER_URL` / `DESK_API_KEY` (+ `DEEPGRAM_API_KEY`,
`ELEVENLABS_API_KEY`) before running — without server config the controller
logs a missing-config message and exits its thread instead of starting.

## Architecture

- **`config.py`** — loads `.env` from the project root (not via python-dotenv;
  a small hand-rolled parser matching the server repo's `desk_client/bolt_desk.py`
  convention) into module-level constants. Everything else reads config from
  here, never `os.environ` directly.
- **`state.py`** — `PetStateMachine`, pure logic with no Qt/audio imports (kept
  that way deliberately for cheap unit testing). Enforces a transition table;
  notably `IDLE -> TALKING` is legal directly (no LISTENING/THINKING leg)
  because the heartbeat can make the pet speak proactively/unprompted.
- **`controller.py`** — `PetController(QObject)`, the pipeline orchestrator.
  Runs on a background `QThread` (wired in `ui/app.py`) so audio I/O/network
  never blocks the Qt event loop; communicates with the UI only through Qt
  signals (`state_changed`, `said`, `log`, `action`, `napping`), never touches
  a `QWidget` directly. Also drives the periodic heartbeat
  (`_maybe_heartbeat`, gated by `HEARTBEAT_INTERVAL_SECONDS`) which lets the
  server push proactive spoken announcements between user turns, and on the
  same tick re-evaluates nap state and drains queued desktop notifications.
  It owns the live wake-word threshold (`wake_threshold()` is passed to
  `listen_for_wake_word` as a *callable* so the tray slider takes effect
  mid-listen) and the conversation `history`.
- **`server_client.py`** — HTTP client for the desk API, dependency-free
  beyond `requests` so it's easy to mock in tests. `converse()` loops relaying
  server-issued shell commands (`run_local_command`, executed via
  `subprocess.run(shell=True)` as the desktop user, 30s default timeout) via
  `/desk/tool_result` until the server sends a final `reply` (capped at
  `_MAX_RELAY_HOPS`). This is the same "full desktop control" trust model as
  the server repo's other desk clients — commands only ever originate from
  the user's own voice/click requests in their own session.
- **`audio/`** — `mic.py` (energy-based VAD utterance capture, ported from the
  server repo's `bolt_desk.py`), `wake_word.py` (openWakeWord `thunderbolt.onnx`
  detection + `NearMissLog` for threshold tuning — see below), `stt.py`
  (Deepgram), `tts.py` (ElevenLabs, streaming by default — `stream_pcm()` +
  `play_stream()` start playback on the first chunk; `chunks_to_int16()`
  carries odd bytes across HTTP chunk boundaries, without which everything
  after the first split sample plays as static — falling back to whole-clip
  PCM then offline `pyttsx3`), `barge_in.py` (two detectors behind one
  `reset()`/`check()` shape, chosen by `BARGE_IN_MODE` via `make_detector`:
  **wake** (default) scores every frame with the same openWakeWord model the
  idle listener uses, so only the wake phrase cuts playback; **energy** is the
  original N-consecutive-loud-frames rule, threshold ~4x the VAD one because
  the mic hears the pet's own voice. Wake mode shares `_default_model` with
  the idle listener — the two never run concurrently — and `reset()`s it on
  detection so the tail of one reply can't count toward the next). Each
  accepts an injectable stream/model/protocol so tests don't need real audio
  hardware or a display.
- **`pet_actions.py`** — `petctl` pseudo-commands (`petctl move top-left`,
  `petctl emote wave`, `say`/`wander`/`nap`). The desk API has no "move the
  pet" payload type and this repo can't change the server, so these ride the
  existing shell-command relay: `controller._handle_command` parses them and
  they never reach `subprocess`; anything else is a real shell command exactly
  as before. Pure parsing; the UI half is `PetWindow.apply_action`.
- **`screen_context.py`** — active-window title (xprop/xdotool, Win32,
  osascript) appended to each utterance via `context_for()`, plus
  `is_fullscreen_active()` for do-not-disturb. Text only — the desk API takes
  no images. Every probe is best-effort and returns None/False rather than
  raising; the parsing is split into pure functions that are tested without a
  display server.
- **`quiet.py`** — quiet-hours spec parsing (`23:00-08:00`, wraps midnight,
  comma-separated). Napping suppresses *proactive* noise and wandering only;
  wake word / click / push-to-talk still work.
- **`notifications.py`** — Linux/D-Bus notification bridge: tails
  `dbus-monitor`, parses Notify calls (pure `iter_notifications()`), filters
  and rate-limits them (`NotificationGate`), and the controller forwards
  survivors through `converse()`. Off by default — each one is a round trip.
- **`sudo_askpass.py`** — makes server-relayed `sudo` usable from a process
  with no terminal, by pointing sudo's `SUDO_ASKPASS` at a GUI helper and
  rewriting bare `sudo` to `sudo -A` (`add_askpass_flag`, a conservative regex
  that skips anything already carrying a flag and anything inside quotes).
  Prefers a real askpass binary and falls back to generating a
  zenity/kdialog wrapper in `~/.cache/bolt-pet/askpass.sh`. Resolution order
  is injectable (`is_executable`/`which`) so it's testable on a machine with a
  different set installed. See the security notes — the dialog is the boundary.
- **`updater.py`** — self-update from the Gitea releases API. Polls
  `<UPDATE_REPO_API>/releases/latest` for a tag newer than
  `bolt_pet.__version__` and moves the checkout to it with
  `git fetch --tags` + `git checkout tags/<tag>`, so "downloading an update"
  is just git and rolling back is one command. Three safety rules: a **dirty
  working tree is skipped, never stashed** (silently discarding your
  work-in-progress beats running an old version); everything after the
  checkout — dependency install, then an **import smoke test in a
  subprocess** (this process still has the old modules loaded, so importing
  in-process would prove nothing) — is guarded, and any failure rolls back to
  the exact ref that was live before, branch name or SHA; and the restart only
  happens once the new code imports, so a broken release costs a log line
  rather than a pet that won't start. Git goes through an injectable
  `run(args) -> (code, output)` callable so apply/rollback is unit-tested
  against a fake git; version comparison and release parsing are pure.
  `controller._maybe_update` drives it from the wake-listener tick (so the pet
  is IDLE and between turns by construction) and the actual `os.execv` happens
  in `ui/app.py` *after* `app.exec()` returns — that ordering is what
  guarantees the mic is released before the new process opens it.
- **`history.py`** — rolling transcript (`HISTORY_LIMIT` turns) behind the
  tray's History window and click-to-copy on the bubble.
- **`hotkey.py`** — global push-to-talk via `pynput`; soft-fails with a logged
  reason (Wayland, missing package, macOS permissions) since the wake word is
  the primary trigger.
- **`speech_text.py`** — sanitizes server replies before they're heard/shown.
  `for_speech()` (called inside `tts.speak()`, so every path to the speakers is
  covered) strips markdown, emoji, URLs and stray symbols the voice would read
  literally ("asterisk asterisk"), turns bullet lists into full sentences, and
  words a few symbols (`&` → "and"). `for_display()` is the looser version for
  the speech bubble — markdown syntax gone, emoji kept. `is_question()` decides
  whether a reply leaves the pet waiting on an answer: it tests the *spoken*
  form (so a '?' inside a stripped code block or URL doesn't count) and only a
  trailing one counts, since a question asked in passing isn't awaiting a
  reply. `controller._should_follow_up` uses it to keep listening without the
  wake word, capped by `FOLLOW_UP_MAX_TURNS` so a server that ends every reply
  with a question can't loop forever off mic noise. Pure string logic, no
  Qt/audio imports.
- **`ui/`** — `app.py` wires `QApplication` + `PetWindow` + `PetTray` + the
  history/tuner windows + the push-to-talk hotkey + the controller thread
  together; `pet_window.py` is the frameless/translucent/always-on-top sprite
  window + speech bubble (non-square frames are centered in the square
  `PET_SIZE` window, see `paintEvent`), and also owns:
  - **wandering** — a ~30fps timer walks the window toward a random on-screen
    target every `PET_WANDER_INTERVAL_SECONDS` (randomized), suppressed
    whenever the pet is non-IDLE, napping, dragged, or has a bubble up. A
    commanded `petctl move` overrides all of that except the drag.
  - **emotes** — `emote_transform()` is pure maths (dx, dy, rotation, scale
    from a 0..1 progress) kept out of `paintEvent` so the curves are unit
    tested; every emote must return to the identity transform at progress 1.0
    or the pet ends up permanently askew.
  - **shaped input / click-through** — `PET_SHAPED_INPUT` masks the window to
    the sprite's opaque pixels so the square window's transparent corners stop
    eating clicks (mask rebuilt only when the frame changes, and pinned to the
    resting position so a bob/spin doesn't thrash it); `PET_CLICK_THROUGH`
    makes the pet ignore the mouse entirely.
  - **edge snapping** (`PET_EDGE_SNAP`) after a drag or a stroll, and **nap
    dimming** (`set_napping`).
  `sprite.py` loads `assets/sprites/<state>/*.png` (filename-sorted, looping —
  currently Kenney's CC0 robot pack, see `assets/sprites/README.md`) and falls
  back to a procedurally-drawn placeholder blob per state if a folder has no
  frames; `tray.py` is the system tray menu (talk now / mute / nap / wander /
  click-through / history / wake-word tuning / quit) — the pet window has no
  title bar or taskbar entry; `history_window.py` and `wake_tuner.py` are the
  two dialogs it opens.

### Wake-word detection

`audio/wake_word.py` uses a custom-trained openWakeWord model,
`thunderbolt.onnx` (ships in the project root), the same way the server
repo's `desk_client/bolt_desk.py` uses `bolt.onnx` for "hey bolt" — same
runtime (openWakeWord, ONNX inference framework), same per-frame
`predict()`/`reset()` loop. Every mic frame is scored; any class score at or
above `WAKE_WORD_THRESHOLD` (default `0.5`, in `.env`) counts as a
detection. Swap `WAKE_MODEL_FILE` to point at a differently-trained `.onnx`
model to change the wake phrase — everything downstream (STT, server call,
TTS) is unaffected.

**openwakeword's `Model.reset()` is not enough to forget a detection.** It
clears the *prediction* buffer only; the rolling audio window the classifier
actually scores lives in `model.preprocessor` (`raw_data_buffer` — 10s of raw
audio — plus `melspectrogram_buffer` and a ~120-frame `feature_buffer`) and
`AudioFeatures` has no reset method at all. So after a detection the wake
phrase is still in the window, and the next frame fed to the model re-fires on
it. Symptom when this bites: the pet cuts itself off a word into every reply,
because wake-mode barge-in resumes feeding the model and instantly matches the
"thunderbolt" that *started* the turn. `wake_word.hard_reset(model)` restores
the preprocessor to its as-constructed (silence) state and is what both
`listen_for_wake_word` and `WakeWordBargeIn.reset()` call — use it, not
`reset()`, anywhere a detection needs to be genuinely forgotten. The blank
state is cached on the preprocessor object (not in an `id()`-keyed dict —
CPython reuses ids after GC), since rebuilding it costs an ONNX pass over 10s
of silence.

The threshold is tunable at runtime: the tray's **Wake word tuning…** window
(`ui/wake_tuner.py`) shows the peak score seen and a rolling list of near
misses (frames within `WAKE_NEAR_MISS_MARGIN` *below* the threshold — i.e.
the times it nearly heard you), and its slider is read per frame because
`listen_for_wake_word` accepts a callable threshold. Set the threshold just
under the peak you can hit reliably, then persist it in `.env`.

### Testing conventions

`tests/` covers pure logic only (state machine, wake-word scoring loop, mic
VAD, HTTP client against mocks) — nothing there needs real audio hardware or
a display. Modules under test are written to accept fake streams/models/
`on_command` callables specifically to keep tests hardware-free; follow that
pattern (inject a `Protocol`-typed collaborator) rather than mocking at the
`sounddevice`/`openwakeword` import boundary when adding new testable logic.
`test_controller.py`, `test_controller_features.py`, `test_wander.py` and
`test_pet_window_features.py` need a `QApplication`, which segfaults without
a display unless run with `QT_QPA_PLATFORM=offscreen`.

Newer subsystems follow the same rule — the testable part is separated from
the part that needs hardware: dbus-monitor output is parsed by a pure
`iter_notifications(lines)`, xprop output by pure `parse_xprop_*` functions,
HTTP chunk reassembly by `chunks_to_int16`, emote motion by
`emote_transform`. Tests that touch the controller monkeypatch
`screen_context.context_for` / `is_fullscreen_active`, otherwise they shell
out to xprop on a headless box.

## Security notes

The server can relay a shell command back to this machine to execute as the
desktop user (see `server_client.run_local_command`). This is intentional
("full desktop control" for things like "open firefox" or disk checks) and
matches the trust model of the server repo's other desk clients. Keep
`DESK_API_KEY` private and don't expose the desk API port to the open
internet.

`sudo_askpass.py` widens that further, by design: with `SUDO_ASKPASS_PROMPT`
on (the default), a relayed bare `sudo` is rewritten to `sudo -A` and the
password is collected in a desktop dialog, so commands can escalate to root
instead of hanging on a tty the pet doesn't have. The dialog is the security
boundary — it's the only thing between the server deciding to run `sudo` and
it running, so the prompt is deliberately not suppressible per-command and
those commands get their own longer timeout (`SUDO_COMMAND_TIMEOUT_SECONDS`)
rather than being made non-interactive. Set `SUDO_ASKPASS_PROMPT=false` to
take the capability away entirely; sudo commands then fail. Note that
`sudo -n` / `sudo -A` / `sudo -u …` in a relayed command are never rewritten,
so an explicit non-interactive sudo stays non-interactive.

**Don't run the pet as root.** It needs no privileges of its own, PortAudio
can't reach the user's PipeWire socket from a root session (raw ALSA devices
reject the 16 kHz capture rate — `paInvalidSampleRate`), and every relayed
command would run unconstrained.

Two newer features widen what leaves this machine, both switchable in `.env`:
`SCREEN_CONTEXT` appends the focused window's *title* to each utterance
(titles often contain file paths, document names, or subject lines), and
`NOTIFICATION_BRIDGE` (off by default) forwards matching desktop
notifications to the server. Neither sends screenshots or notification
contents you haven't matched with `NOTIFICATION_FILTER`.