Bolt-Pet/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this is

A desktop pet (PySide6 window) that is a voice/click UI on top of an external Bolt server's desk API — same brain, memory, tools, and persona as that server's Discord bot and Linux desk client. This repo has no import dependency on the server repo; it's a standalone HTTP client configured via its own .env.

Pipeline: mic → openWakeWord ("thunderbolt", on-device) / push-to-talk / click → record utterance → Deepgram STT → + active-window + screen-layout context → POST /desk/converse → [server may relay a shell command to run on this machine, or a petctl pseudo-command that moves/emotes the pet, jumps it to another monitor, reads a screen's text back, or plays a multi-voice scene instead] → reply (optionally tagged with a voice the server picked for it) → ElevenLabs streaming TTS (or offline pyttsx3 fallback) → speakers, with the pet sprite/speech bubble reflecting state throughout, and playback interruptible by talking over it (barge-in).

Because a relayed command's output goes back up the tool-result relay before the final reply, a petctl read mid-turn means Bolt can look at a monitor and then talk about what's on it in the same answer.

Side channels that let the pet act between turns: the heartbeat (proactive announcements), the desktop notification bridge, and autonomous wandering — all suppressed while it's napping (quiet hours / fullscreen DND).

Commands

# Setup + run (creates .venv and installs requirements.txt on first run)
./run.sh          # macOS/Linux
run.bat           # Windows

# Run tests (no pytest config file — tests self-insert repo root via sys.path).
# QT_QPA_PLATFORM=offscreen avoids a QApplication segfault on headless/no-display hosts.
QT_QPA_PLATFORM=offscreen .venv/bin/pytest tests/
.venv/bin/pytest tests/test_state.py::test_happy_path_transitions  # single test

# Redraw the pet's sprite frames (the committed PNGs are this script's output)
python scripts/generate_bolt_sprites.py             # --out /tmp/x to preview first

# Convert a grid sprite sheet into the per-frame-PNG convention sprite.py expects
python scripts/slice_spritesheet.py path/to/sheet.png assets/sprites/idle --cols 6 --rows 1

Cutting a release: bump __version__ in bolt_pet/__init__.py in the same commit you tag, because that string — not the git history — is what every already-installed pet compares against the newest Gitea tag (updater.py). A tag without the bump means nobody updates; a bump without the tag means the next tag looks older than what's running.

There is no lint/build step configured beyond pytest. cp .env.example .env and fill in BOLT_SERVER_URL / DESK_API_KEY (+ DEEPGRAM_API_KEY, ELEVENLABS_API_KEY) before running — without server config the controller logs a missing-config message and exits its thread instead of starting.

Architecture

• config.py — loads .env from the project root (not via python-dotenv; a small hand-rolled parser matching the server repo's desk_client/bolt_desk.py convention) into module-level constants. Everything else reads config from here, never os.environ directly.

• state.py — PetStateMachine, pure logic with no Qt/audio imports (kept that way deliberately for cheap unit testing). Enforces a transition table; notably IDLE -> TALKING is legal directly (no LISTENING/THINKING leg) because the heartbeat can make the pet speak proactively/unprompted.

• controller.py — PetController(QObject), the pipeline orchestrator. Runs on a background QThread (wired in ui/app.py) so audio I/O/network never blocks the Qt event loop; communicates with the UI only through Qt signals (state_changed, said, log, action, napping), never touches a QWidget directly. Also drives the periodic heartbeat (_maybe_heartbeat, gated by HEARTBEAT_INTERVAL_SECONDS) which lets the server push proactive spoken announcements between user turns, and on the same tick re-evaluates nap state, checks for delivered files, and drains queued desktop notifications. It owns the live wake-word threshold (wake_threshold() is passed to listen_for_wake_word as a callable so the tray slider takes effect mid-listen) and the conversation history.

  One thread drives all of it, so failure containment is structural. Every entry point that can raise runs inside _guarded(work, label), which logs and forces the machine back to IDLE (the only state it's always safe to resume from): the conversation turn, the heartbeat tick — which matters most, since on_tick is the one place control returns to us during a listen that blocks for minutes, and everything it drives touches the network or shells out — and the post-restart report. run() wraps the lot in try/finally because finished is what ui/app.py waits on to quit the thread and to run a pending os.execv; an exception escaping _loop used to skip it, so the failure mode of any bug below was "the pet goes deaf with the mic still open and the tray won't quit" rather than "one turn failed". _handle_command has the same shape for a different reason: it must always return a string, because the server is blocked on /desk/tool_result while it runs and an exception there means the relay never posts and the server sits out its own timeout on a turn that can't finish — silent on both ends. Handed back as command output instead, Bolt can read what broke and say so in the same turn.

• server_client.py — HTTP client for the desk API, dependency-free beyond requests so it's easy to mock in tests. converse() loops relaying server-issued shell commands (run_local_command, executed via a shell=True Popen as the desktop user, 30s default timeout) via /desk/tool_result until the server sends a final reply (capped at _MAX_RELAY_HOPS — exhausting which is reported as its own error, because "unknown server response" sent everyone looking at the payload shape when what happened is a model that kept calling tools and never answered). run_local_command is Popen rather than subprocess.run for the timeout path: the command is a shell, and run()'s timeout would kill only that shell, leaving whatever it spawned (a build, a tail -f, an ffmpeg) alive for the rest of the session with no parent watching — so the child gets its own process group (start_new_session, POSIX) and a timeout SIGTERMs the group, SIGKILLs it two seconds later, then drains the pipes with its own timeout so a grandchild holding stdout can't turn a timeout into a hang. Whatever the command printed before it hung is returned alongside the timeout notice, since the last line usually says exactly what it was stuck waiting for. This is the same "full desktop control" trust model as the server repo's other desk clients — commands only ever originate from the user's own voice/click requests in their own session. A final reply is returned as a Reply(text, voice_id, voice_name) rather than a bare string, because the server can tag it with a voice — see "Voices" below. list_outbox_files / download_outbox_file hit the same /desk/files and /desk/files/<id> endpoints the server's deliver_files tool queues onto — see file_delivery.py.

• file_delivery.py — the filesystem half of receiving files the server queues via its deliver_files tool (ai/desk_api.py in the main tmn-api repo — "send me that report" during a conversation spools the matched workspace files, zipping multiple into one, onto the session's outbox). controller._check_deliveries lists /desk/files and downloads anything queued — right after a conversation/notification turn (the common case) and once per heartbeat tick for anything queued out-of-band — saving each under DELIVERED_FILES_DIR (default ~/Downloads/Bolt). Downloading a file dequeues it server-side, so it's only ever handed out once; save() never overwrites an existing download, suffixing " (1)", " (2)", ... on a name collision. sanitize_filename() reduces a server-supplied name to its bare filename (Path(...).name), which is defense-in-depth against a delivered name that's secretly a path, since a per-user desk API key means the name isn't always coming from someone as trusted as the owner. Toggle off entirely with RECEIVE_FILES=false.

• audio/ — mic.py (energy-based VAD utterance capture, ported from the server repo's bolt_desk.py, plus flush() — see the note below on the pet hearing itself), wake_word.py (openWakeWord thunderbolt.onnx detection + NearMissLog for threshold tuning — see below), stt.py (Deepgram), tts.py (ElevenLabs, streaming by default — stream_pcm() + play_stream() start playback on the first chunk; chunks_to_int16() carries odd bytes across HTTP chunk boundaries, without which everything after the first split sample plays as static — falling back to whole-clip PCM then offline pyttsx3; every entry point takes an optional voice_id overriding ELEVENLABS_VOICE_ID, and model_for() picks the multilingual model whenever there's an override or non-ASCII text, since the default eleven_flash_v2 is English-only and would read either as garbled phonetic English rather than failing), barge_in.py (two detectors behind one reset()/check() shape, chosen by BARGE_IN_MODE via make_detector: wake (default) scores every frame with the same openWakeWord model the idle listener uses, so only the wake phrase cuts playback; energy is the original N-consecutive-loud-frames rule, threshold ~4x the VAD one because the mic hears the pet's own voice. Wake mode shares _default_model with the idle listener — the two never run concurrently — and reset()s it on detection so the tail of one reply can't count toward the next). Each accepts an injectable stream/model/protocol so tests don't need real audio hardware or a display.

• pet_actions.py — petctl pseudo-commands (petctl move top-left, petctl emote wave, say/wander/nap, the screen verbs jump/monitors/read, and voice reset). The desk API has no "move the pet" payload type and this repo can't change the server, so these ride the existing shell-command relay: controller._handle_command parses them and they never reach subprocess; anything else is a real shell command exactly as before. Pure parsing; the UI half is PetWindow.apply_action. Note jump's target is not validated here — which monitors exist is a runtime fact this pure module doesn't have, so the spec passes through to monitors.resolve(). Query verbs (monitors, read) are answered in _handle_command rather than by pet_actions.describe(), because their output is the point: it goes back up the tool-result relay for Bolt to use in his reply — as is voice reset, which reports what it dropped since the server can't see which voice is in use. voice only ever resets: picking one is the server's job (speak_as, which it already knows how to use), so a petctl voice <name> attempt is an error pointing back at that marker.

• self_restart.py — petctl self_restart, the pet restarting itself so Bolt can see a code change he just made instead of waiting for a human to restart it. Three problems shape it, and all three are the interesting part. (1) The restart can't happen inline: killing the process mid-turn would drop the HTTP tool relay before the result was posted, leaving the server to wait out its timeout on a turn that can never finish — so the command only arms it (controller._arm_self_restart) and controller._maybe_self_restart fires it after the reply is spoken, the same "only between turns" rule the updater follows. (2) A broken edit must not be fatal, so preflight() imports the package in a subprocess before arming — this process holds the old modules, so an in-process import would pass on a file that no longer parses — and a SyntaxError comes back as the command's output, in the same turn, with the pet still running. (3) The reason has to outlive the process, so it's written to ~/.cache/bolt-pet/restart_context.json (never inside the repo Bolt is editing) and read on the way back up by controller._report_self_restart, which posts it to the server as an ordinary turn — that's what makes "restart and check the sprites load" finish as a spoken sentence rather than a silence. check_loop_guard refuses after SELF_RESTART_MAX restarts in SELF_RESTART_WINDOW_SECONDS, so an edit-restart-crash cycle stops itself. Off switch: SELF_RESTART=false.

• dialogue.py — dialoguectl pseudo-commands: a multi-voice scene through ElevenLabs' Text to Dialogue endpoint (audio/tts. synthesize_dialogue), checked in _handle_command between petctl and filectl. Same single-line-JSON wire format as filectl and for the same reason (the server's command marker captures only up to the next newline), and it accepts the ElevenLabs field names (inputs/voice_id) as well as its own (lines/voice) because the model has read that API and copying its shape is the obvious thing to try. Voices are named (DIALOGUE_VOICES maps names to ids) rather than pasted as raw ids, and self resolves to whatever voice the pet is speaking with right now — including a speak_as pick — so Bolt sounds like himself in his own scenes. The API's limits (10 distinct voices, ~2000 characters) are enforced before the request so a mistake comes back up the tool-result relay as a sentence Bolt can act on rather than an HTTP 422 he can't see. Unlike the normal reply path there is no streaming variant, so a scene is whole-clip: controller._play_dialogue plays it with the same bubble, transcript and barge-in handling a spoken reply gets, and returns to THINKING afterwards (not IDLE) because the server is still waiting on the tool result — that leg is why state.py allows TALKING -> THINKING.

• file_ops.py — filectl pseudo-commands, checked in _handle_command right after petctl and before falling through to a real shell command. Executing arbitrary commands already worked via the shell relay (run_local_command — see server_client.py below); what filectl adds is a reliable way to do the read/write/edit/list slice of that, since getting the model to hand-roll a shell heredoc for multi-line content full of quotes/$/backticks is failure-prone — and list exists as its own op (rather than relying on the model shelling out to ls/dir) because this project is cross-platform and the model shouldn't have to guess which listing command applies on Windows vs. Linux vs. macOS; one glob-based op (pattern, default *; recursive for rglob instead of glob) covers all three. Wire format is filectl <json> where <json> is a single-line compact JSON object — {"op": "list"|"read"|"write"|"edit", "path": ..., ...} — not a multi-line marker block (an earlier design): the server relays this as the argument to the ordinary command tool marker, and that marker's extractor (ai/agents/default.py in the main repo) only captures up to the next newline, so anything genuinely multi-line silently got truncated no matter how the prompt worded it. JSON sidesteps that for free — json.dumps already encodes embedded newlines as the two characters \n, not a real line break, so multi-line file content still fits on the one physical line the extractor sees. edit requires the old text to match exactly once — same discipline as this project's own code-editing tool — and raises rather than guessing if it's missing or ambiguous. This doesn't expand what the server can do to this machine (a relayed shell command could already overwrite anything the desktop user can write — see the security notes below); it's a safer path to the same capability. Pure parsing (parse) is separated from the filesystem I/O (execute), matching pet_actions.py's parse/describe split.

• relay_json.py — the JSON parser both filectl and dialoguectl use instead of json.loads, because their payload is hand-typed by a model into a tool marker and fails in a small, repeatable set of ways (stray quote after a bare literal, trailing comma, single or smart quotes, Python True/False, a markdown fence). Strict parsing already cost a live turn: the call was rejected, the model re-sent the identical line, was rejected again, and then told the user "I'll check now" without ever calling anything. So loads() tries strict first, then applies named, individually-narrow repairs and accepts one only if the result parses — and on total failure raises RelayJsonError carrying a caret pointed at the offending character, since a model can act on a pointed-at fragment but not on "Expecting ',' delimiter: char 74". Two conventions matter for any new relayed-JSON command: repairs are never silent — parse stashes them on the action as _repairs and describe appends relay_json.repair_note(...) to the tool result, so the model is told it sent something broken while it still has the turn — and new repairs go in the _REPAIRS tuple ordered cheapest/safest first. Tested inside tests/test_file_ops.py, not a file of its own.

• screen_context.py — active-window title (xprop/xdotool, Win32, osascript) appended to each utterance via context_for(), plus is_fullscreen_active() for do-not-disturb. Text only — the desk API takes no images. Every probe is best-effort and returns None/False rather than raising; the parsing is split into pure functions that are tested without a display server.

• monitors.py — the screen layout, and resolving petctl jump targets (a 1-based number, a name, next/prev/primary/other, or a direction like left/up worked out from the actual geometry). Pure — no Qt, no subprocess. The monitor list is published by the UI (PetWindow.publish_monitors builds it from QGuiApplication.screens() and emits it over a queued signal to controller.set_monitors), because the controller and the window must agree on what "monitor 2" means: enumerating with xrandr on one side and Qt's screen list on the other gives different orderings on the same machine, and Bolt would announce one screen and land on another. Qt is the single source of truth; Monitor.index is 0-based and .number is the 1-based value used in every string a human or the model sees. The controller resolves a jump to a concrete index before emitting it, so the window can't re-resolve against a different list.

• screen_text.py — OCR, so Bolt can read what's on a monitor (petctl read [n|here|all]). Pull, not push: nothing captures on its own — the server has to ask, and the text goes back as that command's output. That's deliberate; OCR of a 4K screen costs a second or two that would otherwise be added to every utterance, and screen contents leaving the machine should be a visible decision rather than a constant. Capture needs mss (X11/Win32/macOS, not Wayland), recognition needs Tesseract or RapidOCR; both are optional and soft-fail with a reason the way hotkey.py does, and read_monitor() never raises because its return value is command output. Engine selection takes injected probes so it's testable wherever.

• quiet.py — quiet-hours spec parsing (23:00-08:00, wraps midnight, comma-separated). Napping suppresses proactive noise and wandering only; wake word / click / push-to-talk still work.

• notifications.py — Linux/D-Bus notification bridge: tails dbus-monitor, parses Notify calls (pure iter_notifications()), filters and rate-limits them (NotificationGate), and the controller forwards survivors through converse(). Off by default — each one is a round trip. Note where the queue between the two threads lives: notifications arrive on the watcher thread and are forwarded from the heartbeat, which doesn't run while the pet is napping — so they accumulate overnight. The controller's queue is therefore a bounded deque stamped on arrival, and the drain discards anything older than NOTIFICATION_MAX_AGE_SECONDS rather than reading a nine-hour-old backlog out at 8am. A drain that stops early (a nap starting mid-loop, or the server going down) re-queues what it didn't forward instead of dropping it, which the original swap-and-return did silently.

• sudo_askpass.py — makes server-relayed sudo usable from a process with no terminal, by pointing sudo's SUDO_ASKPASS at a GUI helper and rewriting bare sudo to sudo -A (add_askpass_flag, a conservative regex that skips anything already carrying a flag and anything inside quotes). Prefers a real askpass binary and falls back to generating a zenity/kdialog wrapper in ~/.cache/bolt-pet/askpass.sh. Resolution order is injectable (is_executable/which) so it's testable on a machine with a different set installed. See the security notes — the dialog is the boundary.

• updater.py — self-update from the Gitea releases API. Polls <UPDATE_REPO_API>/releases/latest for a tag newer than bolt_pet.__version__ and moves the checkout to it with git fetch --tags + git checkout tags/<tag>, so "downloading an update" is just git and rolling back is one command. Three safety rules: a dirty working tree is skipped, never stashed (silently discarding your work-in-progress beats running an old version); everything after the checkout — dependency install, then an import smoke test in a subprocess (this process still has the old modules loaded, so importing in-process would prove nothing) — is guarded, and any failure rolls back to the exact ref that was live before, branch name or SHA; and the restart only happens once the new code imports, so a broken release costs a log line rather than a pet that won't start. Git goes through an injectable run(args) -> (code, output) callable so apply/rollback is unit-tested against a fake git; version comparison and release parsing are pure. controller._maybe_update drives it from the wake-listener tick (so the pet is IDLE and between turns by construction) and the actual os.execv happens in ui/app.py after app.exec() returns — that ordering is what guarantees the mic is released before the new process opens it.

• history.py — rolling transcript (HISTORY_LIMIT turns) behind the tray's History window and click-to-copy on the bubble.

• hotkey.py — global push-to-talk via pynput; soft-fails with a logged reason (Wayland, missing package, macOS permissions) since the wake word is the primary trigger.

• speech_text.py — sanitizes server replies before they're heard/shown. for_speech() (called inside tts.speak(), so every path to the speakers is covered) strips markdown, emoji, URLs and stray symbols the voice would read literally ("asterisk asterisk"), turns bullet lists into full sentences, and words a few symbols (& → "and"), abbreviations the voice would spell out letter by letter (e.g. → "for example", etc. → "and so on") and a long option's leading -- (heard as "dash dash force"; the single hyphen has to survive for "bolt-pet"). for_display() is the looser version for the speech bubble — markdown syntax gone, emoji kept. is_question() decides whether a reply leaves the pet waiting on an answer: it tests the spoken form (so a '?' inside a stripped code block or URL doesn't count) and a '?' anywhere counts. That last part was once trailing-only, on the theory that "What time is it? It's 7:15." isn't awaiting a reply — true of that sentence and wrong more often, since Bolt routinely asks and then keeps talking ("Want me to fix it? I'd start with the config"), which is the case that actually costs you a wake word. The asymmetry is the argument: an unwanted extra listen ends itself on VAD_GRACE_SECONDS of silence, a missed one makes you start over. controller._should_follow_up uses it to keep listening without the wake word, capped by FOLLOW_UP_MAX_TURNS so a server that ends every reply with a question can't loop forever off mic noise. Pure string logic, no Qt/audio imports.

• intents.py — the handful of utterances answered without the server. "stop", "come here", "go to sleep", "say that again", "use your normal voice" are commands to the body, and routing them through the desk API costs two to four seconds and three network hops to make the pet walk left — and only works if the server's prompt happens to advertise the matching petctl verb (which is why voice reset needs a block in ai/desk_api.py's pet prompt; see the Voices section). Recognising the phrase here removes both the latency and that coupling. The design problem is not stealing real requests, and three rules cover it: whole-utterance exact match after normalisation (so "stop" is an intent and "stop the docker container" is a question for Bolt), a closed table with nothing arguable in it, and never on a follow-up turn — if Bolt just asked you something your answer is his, and swallowing "never mind" locally would leave the server holding a question it never got an answer to. Both sides of the comparison go through normalize() (the table is canonicalised at import, and _build() refuses to build one where two intents claim the same normalised phrase, or where a phrase reduces to "" and would match pure filler like "hey bolt"). Actions come back in the same shape pet_actions.parse produces, so PetWindow.apply_action needs no new vocabulary; the effects live in controller._handle_local_intent. Off switch: LOCAL_INTENTS=false.

• ui/ — app.py wires QApplication + PetWindow + PetTray + the history/tuner windows + the push-to-talk hotkey + the controller thread together; pet_window.py is the frameless/translucent/always-on-top sprite window + speech bubble (non-square frames are centered in the square PET_SIZE window, see paintEvent), and also owns:

  • wandering — a ~30fps timer walks the window toward a random on-screen target every PET_WANDER_INTERVAL_SECONDS (randomized), suppressed whenever the pet is non-IDLE, napping, dragged, or has a bubble up. A commanded petctl move overrides all of that except the drag.
  • the walk cycle — while actually travelling, _animation_key() swaps the state animation for the side-view walk/ frames (not a PetState — see the sprites README). It is stepped by distance travelled (_WALK_PIXELS_PER_FRAME), never by the animation timer, so the planted paw tracks backwards at exactly the speed the window moves forwards; _advance_frame deliberately no-ops while walking so the two can't double-step it. The art is drawn facing right and _oriented() mirrors it (cached per frame) when heading left. No walk/ art → falls back to the old coded bob rather than a placeholder blob.
  • emotes — emote_transform() is pure maths (dx, dy, rotation, scale from a 0..1 progress) kept out of paintEvent so the curves are unit tested; every emote must return to the identity transform at progress 1.0 or the pet ends up permanently askew.
  • shaped input / click-through — PET_SHAPED_INPUT masks the window to the sprite's opaque pixels so the square window's transparent corners stop eating clicks (mask rebuilt only when the frame changes, and pinned to the resting position so a bob/spin doesn't thrash it); PET_CLICK_THROUGH makes the pet ignore the mouse entirely.
  • edge snapping (PET_EDGE_SNAP) after a drag or a stroll, and nap dimming (set_napping). sprite.py loads assets/sprites/<state>/*.png (filename-sorted, looping — the art is generated by scripts/generate_bolt_sprites.py, a Pillow drawing of Bolt as a shepherd pup; edit the script and re-run it rather than the committed PNGs, see assets/sprites/README.md) and falls back to a procedurally-drawn placeholder blob per state if a folder has no frames. It also loads EXTRA_ANIMATIONS — currently just walk/ — keyed by name rather than by PetState, with has() reporting whether a key is backed by real art so callers can decline a placeholder instead of trotting a blob across the desktop; tray.py is the system tray menu (talk now / mute / nap / wander / click-through / history / wake-word tuning / use-default-voice / quit) — the pet window has no title bar or taskbar entry; history_window.py and wake_tuner.py are the two dialogs it opens.

Voices (the server's speak_as)

Ask Bolt to talk like someone else, or in another language, and the server does the picking: its desk-only voice_search marker browses the ElevenLabs voice library, and speak_as: <voice_id> on the final reply tags that reply with the chosen voice (adding a Voice Library pick to the ElevenLabs account first, so the id is usable by the time it reaches us). Nothing about that is this repo's to decide — all the client owes it is actually speaking in the voice it was handed: converse() returns it on Reply, _apply_voice() records it, and _speak() passes it to tts.speak(voice_id=...).

Two things are decided here, though, because the server can't:

• The voice sticks (VOICE_STICKY, default on). The server tags one reply and strips the marker before storing the turn, so it never sees the id again — "keep talking like that" would send it searching for a voice all over again, and it'd likely land on a different one. Holding the id client-side is what makes the rest of the conversation stay in that voice. An untagged reply therefore never changes the voice; only a new speak_as, VOICE_STICKY=false, or a reset does.
• There's a way back. Since the server was never told Bolt's own voice id, it can't ask for it back with speak_as — so reverting is local: the tray's Use default voice entry (enabled only while a picked voice is in use, kept in sync by the voice_changed signal), a restart, or petctl voice reset, which is what lets Bolt honour "go back to your normal voice" out loud. That last one needs the server's pet prompt block (ai/desk_api.py, pet_tools) to mention the verb, or the model never emits it — the desk API's prompt is where petctl is advertised.

Wake-word detection

audio/wake_word.py uses a custom-trained openWakeWord model, thunderbolt.onnx (ships in the project root), the same way the server repo's desk_client/bolt_desk.py uses bolt.onnx for "hey bolt" — same runtime (openWakeWord, ONNX inference framework), same per-frame predict()/reset() loop. Every mic frame is scored; any class score at or above WAKE_WORD_THRESHOLD (default 0.5, in .env) counts as a detection. Swap WAKE_MODEL_FILE to point at a differently-trained .onnx model to change the wake phrase — everything downstream (STT, server call, TTS) is unaffected.

openwakeword's Model.reset() is not enough to forget a detection. It clears the prediction buffer only; the rolling audio window the classifier actually scores lives in model.preprocessor (raw_data_buffer — 10s of raw audio — plus melspectrogram_buffer and a ~120-frame feature_buffer) and AudioFeatures has no reset method at all. So after a detection the wake phrase is still in the window, and the next frame fed to the model re-fires on it. Symptom when this bites: the pet cuts itself off a word into every reply, because wake-mode barge-in resumes feeding the model and instantly matches the "thunderbolt" that started the turn. wake_word.hard_reset(model) restores the preprocessor to its as-constructed (silence) state and is what both listen_for_wake_word and WakeWordBargeIn.reset() call — use it, not reset(), anywhere a detection needs to be genuinely forgotten. The blank state is cached on the preprocessor object (not in an id()-keyed dict — CPython reuses ids after GC), since rebuilding it costs an ONNX pass over 10s of silence.

The threshold is tunable at runtime: the tray's Wake word tuning… window (ui/wake_tuner.py) shows the peak score seen and a rolling list of near misses (frames within WAKE_NEAR_MISS_MARGIN below the threshold — i.e. the times it nearly heard you), and its slider is read per frame because listen_for_wake_word accepts a callable threshold. Set the threshold just under the peak you can hit reliably, then persist it in .env.

The mic keeps recording while nothing is reading it

Same family of bug as the openwakeword one above, one layer down: PortAudio captures into a ring buffer continuously, so audio from a stretch where the pipeline thread was busy elsewhere is still queued when the next read happens. It bites in exactly one place. At the end of a reply that asked you something, _speak sets _talk_now and the next turn starts recording immediately — with the tail of the pet's own TTS sitting in that buffer, above the VAD threshold. The VAD takes it for the start of your answer, Deepgram transcribes it, and Bolt is handed his own last sentence as if you had said it. With barge-in on the detector was draining the stream during playback so the window is small; with BARGE_IN=false nothing drains it at all.

mic.flush(stream) drops what's buffered, and _speak calls it on the follow-up branch only. That placement is the whole correctness argument — flushing is only safe where the buffer is known to hold nothing you said: playback ran to completion, so if you had spoken, barge-in would have cut it and taken the interrupted branch instead. Never flush before a wake-triggered recording, where the rest of "thunderbolt, what time is it" is legitimately queued and dropping it clips the request. A single call is bounded by max_seconds so it can't chase a stream filling as fast as it drains, and it no-ops on a stream with no read_available (i.e. every fake stream in tests).

Testing conventions

tests/ covers pure logic only (state machine, wake-word scoring loop, mic VAD, HTTP client against mocks) — nothing there needs real audio hardware or a display. Modules under test are written to accept fake streams/models/ on_command callables specifically to keep tests hardware-free; follow that pattern (inject a Protocol-typed collaborator) rather than mocking at the sounddevice/openwakeword import boundary when adding new testable logic. test_controller.py, test_controller_features.py, test_wander.py and test_pet_window_features.py need a QApplication, which segfaults without a display unless run with QT_QPA_PLATFORM=offscreen.

Newer subsystems follow the same rule — the testable part is separated from the part that needs hardware: dbus-monitor output is parsed by a pure iter_notifications(lines), xprop output by pure parse_xprop_* functions, HTTP chunk reassembly by chunks_to_int16, emote motion by emote_transform. Tests that touch the controller monkeypatch screen_context.context_for / is_fullscreen_active, otherwise they shell out to xprop on a headless box.

Security notes

The server can relay a shell command back to this machine to execute as the desktop user (see server_client.run_local_command). This is intentional ("full desktop control" for things like "open firefox" or disk checks) and matches the trust model of the server repo's other desk clients. Keep DESK_API_KEY private and don't expose the desk API port to the open internet.

file_ops.py's filectl read/write/edit pseudo-commands ride that same relay and are bound by the same trust model — no path is off-limits beyond normal filesystem permissions for the desktop user, exactly like a relayed cat/sed/rm already isn't. They don't grant the server anything a shell command couldn't already do; they just make the read/write/edit path reliable instead of relying on the model getting shell quoting right.

sudo_askpass.py widens that further, by design: with SUDO_ASKPASS_PROMPT on (the default), a relayed bare sudo is rewritten to sudo -A and the password is collected in a desktop dialog, so commands can escalate to root instead of hanging on a tty the pet doesn't have. The dialog is the security boundary — it's the only thing between the server deciding to run sudo and it running, so the prompt is deliberately not suppressible per-command and those commands get their own longer timeout (SUDO_COMMAND_TIMEOUT_SECONDS) rather than being made non-interactive. Set SUDO_ASKPASS_PROMPT=false to take the capability away entirely; sudo commands then fail. Note that sudo -n / sudo -A / sudo -u … in a relayed command are never rewritten, so an explicit non-interactive sudo stays non-interactive.

Don't run the pet as root. It needs no privileges of its own, PortAudio can't reach the user's PipeWire socket from a root session (raw ALSA devices reject the 16 kHz capture rate — paInvalidSampleRate), and every relayed command would run unconstrained.

Several features widen what leaves this machine, all switchable in .env: SCREEN_CONTEXT appends the focused window's title to each utterance (titles often contain file paths, document names, or subject lines), MONITOR_CONTEXT appends the screen layout (sizes and names only — no contents), and NOTIFICATION_BRIDGE (off by default) forwards matching desktop notifications to the server. None of those send screenshots or notification contents you haven't matched with NOTIFICATION_FILTER.

SCREEN_TEXT is the biggest of them: petctl read OCRs a whole monitor and sends the recognised text to the server — everything visible, not just the focused window. Two things keep it honest. It's pull-only: no capture happens unless the server explicitly asks, so it can't leak in the background the way a per-turn annotation would, and each read is logged. And it is strictly not a new capability — the shell relay could already run a screenshot tool and pipe it through OCR — it just makes a thing the trust model already allowed reliable, bounded (SCREEN_TEXT_MAX_CHARS) and visible. It is nonetheless far easier to reach for than the shell route, so if that trade isn't one you want, SCREEN_TEXT=false removes it and petctl read starts reporting that it's disabled. Capture is mss-based and therefore silently unavailable on Wayland.