TheMajesticNetwork/Bolt-Console
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 2 Wiki Activity

Bolt Console: desktop app with chat list / new chat, dictation, file tools

Browse Source 
Add a chat list and server-minted new chats on top of the desk API's
GET/POST /desk/chats:

- server_client.list_chats() / create_chat(), same shape and error
  handling as list_outbox_files().
- Switching is config.set_session(id): a plain reassignment of the
  SESSION_ID every call reads, persisted to session.json. The window
  refuses to switch mid-turn rather than locking.
- gui/chat_list_window.py: a Chats dialog (Ctrl+K) with a QListWidget,
  populated on open, with Refresh and New Chat. Network calls run as
  QRunnables on the global QThreadPool and come back through Signals,
  never from a slot. The header New chat (Ctrl+N) uses the same path.
- Switching clears the local transcript; there is no endpoint for a
  past chat's messages. Bolt's memory is per user, so it carries over.
- Terminal: /chats, /chat <n|id>, and /new (server-minted id, local
  fallback).

This commit also carries the uncommitted work since 54030a7 that turned
this Bolt-Pet fork into Bolt Console: the pet, wake word, TTS and screen
features removed; a Claude Code-style terminal client; the PySide6
window with tool cards, approvals and diffs; mic dictation through the
server's /desk/stt relay; filectl/JSON repairs; and detection of
invented tool results.

Build output (build/, dist/, Release/, *.spec) is now ignored.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
themajesticmagician
2026-09-25 19:50:10 -06:00
co-authored by Claude Opus 5.5
parent 54030a7b31
commit 3977804ae3
144 changed files with 5477 additions and 14736 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+99 -567
View File
@@ -4,587 +4,119 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## What this is
A desktop pet (PySide6 window) that is a voice/click UI on top of an external
Bolt server's desk API — same brain, memory, tools, and persona as that
server's Discord bot and Linux desk client. This repo has **no import
dependency** on the server repo; it's a standalone HTTP client configured via
its own `.env`.
A PySide6 desktop app (default) and a terminal REPL (`--cli`), both Claude Code-style, on top of an
external Bolt server's desk API. It uses the same
brain, memory, tools and persona as the server's Discord bot and its other desk clients. There is
**no import dependency** on the server repo: it's a standalone HTTP client configured through its
own `.env`. It was forked from Bolt-Pet with the desktop pet, wake word, STT/TTS, screen
context/OCR, notifications, auto-updater and self-restart removed.
Pipeline: `mic → openWakeWord ("thunderbolt", on-device) / push-to-talk /
click → record utterance → the server's own /desk/stt (same relay the
Android app uses — no local Deepgram account) → + active-window + screen-
layout context → POST /desk/converse → [server may relay a shell command to
run on this machine, or a `petctl` pseudo-command that moves/emotes the pet,
jumps it to another monitor, reads a screen's text back, or plays a
multi-voice scene instead] → reply (optionally tagged with a voice the
server picked for it) → the server's own /desk/tts, streaming (or offline
pyttsx3 fallback) → speakers`, with the pet sprite/speech bubble reflecting
state throughout, and playback interruptible by talking over it (barge-in).
Because a relayed command's output goes back up the tool-result relay before
the final reply, a `petctl read` mid-turn means Bolt can look at a monitor and
then talk about what's on it in the same answer.
Side channels that let the pet act between turns: the heartbeat (proactive
announcements), the desktop notification bridge, and autonomous wandering —
all suppressed while it's napping (quiet hours / fullscreen DND).
Pipeline: `typed line → + cwd/OS note → POST /desk/converse_stream (or /desk/converse) → [server
relays a command: filectl file op, or a shell command, after y/n approval] → /desk/tool_result →
… → reply printed`.
## Commands
```bash
# Setup + run (creates .venv and installs requirements.txt on first run)
./run.sh # macOS/Linux
run.bat # Windows
# Preflight: is this install actually going to work? (config, mic, keys, sprites…)
python -m bolt_pet --doctor # shallow: no network, no mic
python -m bolt_pet --doctor --deep # contacts the server and opens the microphone
# Run tests (no pytest config file — tests self-insert repo root via sys.path).
# QT_QPA_PLATFORM=offscreen avoids a QApplication segfault on headless/no-display hosts.
QT_QPA_PLATFORM=offscreen .venv/bin/pytest tests/
.venv/bin/pytest tests/test_state.py::test_happy_path_transitions # single test
# Redraw the pet's sprite frames (the committed PNGs are this script's output)
python scripts/generate_bolt_sprites.py # --out /tmp/x to preview first
# Convert a grid sprite sheet into the per-frame-PNG convention sprite.py expects
python scripts/slice_spritesheet.py path/to/sheet.png assets/sprites/idle --cols 6 --rows 1
run.bat / run-cli.bat / ./run.sh [--cli] # window / terminal; creates .venv; keeps the caller's cwd
python -m bolt_console --doctor [--deep]
python -m bolt_console -p "message" # one-shot
.venv/Scripts/python -m pytest tests # (.venv/bin on Linux/macOS)
```
There is no lint/build step configured beyond pytest. `cp .env.example .env`
and fill in `BOLT_SERVER_URL` / `DESK_API_KEY` before running — without them
the controller logs a missing-config message and exits its thread instead of
starting. That's also all STT and normal-reply TTS need now (both go through
the server); `ELEVENLABS_VOICE_ID` picks the voice, and `ELEVENLABS_API_KEY`
is only for the one feature with no server endpoint — multi-voice
`dialoguectl` scenes, see dialogue.py below.
Runtime dependencies: `requests`, plus `PySide6` for the window only. `bolt_console.gui` is imported
lazily, so `--cli` works without Qt. The local venv is Python 3.9, so no `match`, no
runtime `X | Y` unions, and no builtin generics outside annotations.
## Architecture
## Architecture (`bolt_console/`)
- **`config.py`** — loads `.env` from the project root (not via python-dotenv;
a small hand-rolled parser matching the server repo's `desk_client/bolt_desk.py`
convention) into module-level constants. Everything else reads config from
here, never `os.environ` directly.
- **`state.py`** — `PetStateMachine`, pure logic with no Qt/audio imports (kept
that way deliberately for cheap unit testing). Enforces a transition table;
notably `IDLE -> TALKING` is legal directly (no LISTENING/THINKING leg)
because the heartbeat can make the pet speak proactively/unprompted.
- **`controller.py`** — `PetController(QObject)`, the pipeline orchestrator.
Runs on a background `QThread` (wired in `ui/app.py`) so audio I/O/network
never blocks the Qt event loop; communicates with the UI only through Qt
signals (`state_changed`, `said`, `log`, `action`, `napping`), never touches
a `QWidget` directly. Also drives the periodic heartbeat
(`_maybe_heartbeat`, gated by `HEARTBEAT_INTERVAL_SECONDS`) which lets the
server push proactive spoken announcements between user turns, and on the
same tick re-evaluates nap state, checks for delivered files, and drains
queued desktop notifications. It owns the live wake-word threshold
(`wake_threshold()` is passed to `listen_for_wake_word` as a *callable* so
the tray slider takes effect mid-listen) and the conversation `history`.
- **`server_client.py`** — HTTP client for the desk API, dependency-free
beyond `requests` so it's easy to mock in tests. `converse()` loops relaying
server-issued shell commands (`run_local_command`, executed via
`subprocess.run(shell=True)` as the desktop user, 30s default timeout) via
`/desk/tool_result` until the server sends a final `reply` (capped at
`_MAX_RELAY_HOPS`). This is the same "full desktop control" trust model as
the server repo's other desk clients — commands only ever originate from
the user's own voice/click requests in their own session. A final reply is
returned as a `Reply(text, voice_id, voice_name)` rather than a bare string,
because the server can tag it with a voice — see "Voices" below.
`list_outbox_files`
/ `download_outbox_file` hit the same `/desk/files` and `/desk/files/<id>`
endpoints the server's `deliver_files` tool queues onto — see `file_delivery.py`.
- **`file_delivery.py`** — the filesystem half of receiving files the server
queues via its `deliver_files` tool (`ai/desk_api.py` in the main tmn-api
repo — "send me that report" during a conversation spools the matched
workspace files, zipping multiple into one, onto the session's outbox).
`controller._check_deliveries` lists `/desk/files` and downloads anything
queued — right after a conversation/notification turn (the common case)
and once per heartbeat tick for anything queued out-of-band — saving each
under `DELIVERED_FILES_DIR` (default `~/Downloads/Bolt`). Downloading a
file dequeues it server-side, so it's only ever handed out once; `save()`
never overwrites an existing download, suffixing `" (1)"`, `" (2)"`, ... on
a name collision. `sanitize_filename()` reduces a server-supplied name to
its bare filename (`Path(...).name`), which is defense-in-depth against a
delivered name that's secretly a path, since a per-user desk API key means
the name isn't always coming from someone as trusted as the owner. Toggle
off entirely with `RECEIVE_FILES=false`.
- **`audio/`** — `mic.py` (energy-based VAD utterance capture, ported from the
server repo's `bolt_desk.py`), `wake_word.py` (openWakeWord `thunderbolt.onnx`
detection + `NearMissLog` for threshold tuning — see below), `stt.py` +
`stt_stream.py` (the server's own `/desk/stt` websocket relay — no local
Deepgram account; see below), `tts.py` (the server's own `/desk/tts`,
streaming by default — `stream_pcm()` + `play_stream()` start playback on
the first chunk; `chunks_to_int16()` carries odd bytes across HTTP chunk
boundaries, without which everything after the first split sample plays as
static — falling back to whole-clip PCM then offline `pyttsx3`; every entry
point takes an optional `voice_id` overriding `ELEVENLABS_VOICE_ID` — which
model to synthesize with is the server's call now, not this client's;
see `synthesize_dialogue()` further down for the one path that's still
ElevenLabs-direct), `barge_in.py` (two detectors behind one
`reset()`/`check()` shape, chosen by `BARGE_IN_MODE` via `make_detector`:
**wake** (default) scores every frame with the same openWakeWord model the
idle listener uses, so only the wake phrase cuts playback; **energy** is the
original N-consecutive-loud-frames rule, threshold ~4x the VAD one because
the mic hears the pet's own voice. Wake mode shares `_default_model` with
the idle listener — the two never run concurrently — and `reset()`s it on
detection so the tail of one reply can't count toward the next). Each
accepts an injectable stream/model/protocol so tests don't need real audio
hardware or a display.
- **`pet_actions.py`** — `petctl` pseudo-commands (`petctl move top-left`,
`petctl emote wave`, `say`/`wander`/`nap`, the screen verbs
`jump`/`monitors`/`read`, and `voice reset`). The desk API has no "move the pet" payload type
and this repo can't change the server, so these ride the existing
shell-command relay: `controller._handle_command` parses them and they never
reach `subprocess`; anything else is a real shell command exactly as before.
Pure parsing; the UI half is `PetWindow.apply_action`. Note `jump`'s target
is *not* validated here — which monitors exist is a runtime fact this pure
module doesn't have, so the spec passes through to `monitors.resolve()`.
Query verbs (`monitors`, `read`) are answered in `_handle_command` rather
than by `pet_actions.describe()`, because their output *is* the point: it
goes back up the tool-result relay for Bolt to use in his reply — as is
`voice reset`, which reports what it dropped since the server can't see
which voice is in use. `voice` only ever resets: picking one is the
server's job (`speak_as`, which it already knows how to use), so a
`petctl voice <name>` attempt is an error pointing back at that marker.
- **`self_restart.py`** — `petctl self_restart`, the pet restarting itself so
Bolt can *see* a code change he just made instead of waiting for a human to
restart it. Three problems shape it, and all three are the interesting part.
(1) The restart can't happen inline: killing the process mid-turn would drop
the HTTP tool relay before the result was posted, leaving the server to wait
out its timeout on a turn that can never finish — so the command only
*arms* it (`controller._arm_self_restart`) and
`controller._maybe_self_restart` fires it after the reply is spoken, the
same "only between turns" rule the updater follows. (2) A broken edit must
not be fatal, so `preflight()` imports the package in a **subprocess**
before arming — this process holds the old modules, so an in-process import
would pass on a file that no longer parses — and a SyntaxError comes back as
the command's output, in the same turn, with the pet still running. (3) The
reason has to outlive the process, so it's written to
`~/.cache/bolt-pet/restart_context.json` (never inside the repo Bolt is
editing) and read on the way back up by `controller._report_self_restart`,
which posts it to the server as an ordinary turn — that's what makes
"restart and check the sprites load" finish as a spoken sentence rather than
a silence. `check_loop_guard` refuses after `SELF_RESTART_MAX` restarts in
`SELF_RESTART_WINDOW_SECONDS`, so an edit-restart-crash cycle stops itself.
Off switch: `SELF_RESTART=false`.
- **`dialogue.py`** — `dialoguectl` pseudo-commands: a multi-voice *scene*
through ElevenLabs' Text to Dialogue endpoint (`audio/tts.
synthesize_dialogue`), checked in `_handle_command` between petctl and
filectl. Same single-line-JSON wire format as filectl and for the same
reason (the server's `command` marker captures only up to the next
newline), and it accepts the ElevenLabs field names (`inputs`/`voice_id`)
as well as its own (`lines`/`voice`) because the model has read that API
and copying its shape is the obvious thing to try. Voices are *named*
(`DIALOGUE_VOICES` maps names to ids) rather than pasted as raw ids, and
`self` resolves to whatever voice the pet is speaking with right now —
including a `speak_as` pick — so Bolt sounds like himself in his own
scenes. The API's limits (10 distinct voices, ~2000 characters) are
enforced *before* the request so a mistake comes back up the tool-result
relay as a sentence Bolt can act on rather than an HTTP 422 he can't see.
Unlike the normal reply path there is no streaming variant, so a scene is
whole-clip: `controller._play_dialogue` plays it with the same bubble,
transcript and barge-in handling a spoken reply gets, and returns to
THINKING afterwards (not IDLE) because the server is still waiting on the
tool result — that leg is why `state.py` allows TALKING -> THINKING.
- **`file_ops.py`** — `filectl` pseudo-commands, checked in `_handle_command`
right after petctl and before falling through to a real shell command.
Executing arbitrary commands already worked via the shell relay
(`run_local_command` — see `server_client.py` below); what filectl adds is
a *reliable* way to do the read/write/edit/list slice of that, since
getting the model to hand-roll a shell heredoc for multi-line content full
of quotes/`$`/backticks is failure-prone — and `list` exists as its own op
(rather than relying on the model shelling out to `ls`/`dir`) because this
project is cross-platform and the model shouldn't have to guess which
listing command applies on Windows vs. Linux vs. macOS; one glob-based op
(`pattern`, default `*`; `recursive` for `rglob` instead of `glob`) covers
all three. Wire format is `filectl <json>` where `<json>` is a
**single-line** compact JSON object —
`{"op": "list"|"read"|"write"|"edit", "path": ..., ...}` — not a multi-line
marker block (an earlier design): the server relays this as the argument
to the ordinary `command` tool marker, and that marker's extractor
(`ai/agents/default.py` in the main repo) only captures up to the next
newline, so anything genuinely multi-line silently got truncated no matter
how the prompt worded it. JSON sidesteps that for free — `json.dumps`
already encodes embedded newlines as the two characters `\n`, not a real
line break, so multi-line file content still fits on the one physical line
the extractor sees. `edit` requires the old text to match exactly once —
same discipline as this project's own code-editing tool — and raises
rather than guessing if it's missing or ambiguous. This doesn't expand
what the server can do to this machine (a relayed shell command could
already overwrite anything the desktop user can write — see the security
notes below); it's a safer path to the same capability. Pure parsing
(`parse`) is separated from the filesystem I/O (`execute`), matching
pet_actions.py's parse/describe split.
- **`screen_context.py`** — active-window title (xprop/xdotool, Win32,
osascript) appended to each utterance via `context_for()`, plus
`is_fullscreen_active()` for do-not-disturb. Text only — the desk API takes
no images. Every probe is best-effort and returns None/False rather than
raising; the parsing is split into pure functions that are tested without a
display server.
- **`monitors.py`** — the screen layout, and resolving `petctl jump` targets
(a 1-based number, a name, `next`/`prev`/`primary`/`other`, or a direction
like `left`/`up` worked out from the actual geometry). Pure — no Qt, no
subprocess. The monitor list is *published by the UI*
(`PetWindow.publish_monitors` builds it from `QGuiApplication.screens()` and
emits it over a queued signal to `controller.set_monitors`), because the
controller and the window must agree on what "monitor 2" means: enumerating
with `xrandr` on one side and Qt's screen list on the other gives different
orderings on the same machine, and Bolt would announce one screen and land
on another. Qt is the single source of truth; `Monitor.index` is 0-based and
`.number` is the 1-based value used in every string a human or the model
sees. The controller resolves a jump to a concrete index *before* emitting
it, so the window can't re-resolve against a different list.
- **`screen_text.py`** — OCR, so Bolt can read what's on a monitor
(`petctl read [n|here|all]`). **Pull, not push**: nothing captures on its own
— the server has to ask, and the text goes back as that command's output.
That's deliberate; OCR of a 4K screen costs a second or two that would
otherwise be added to *every* utterance, and screen contents leaving the
machine should be a visible decision rather than a constant. Capture needs
`mss` (X11/Win32/macOS, **not** Wayland), recognition needs Tesseract or
RapidOCR; both are optional and soft-fail with a reason the way `hotkey.py`
does, and `read_monitor()` never raises because its return value is command
output. Engine selection takes injected probes so it's testable wherever.
- **`quiet.py`** — quiet-hours spec parsing (`23:00-08:00`, wraps midnight,
comma-separated). Napping suppresses *proactive* noise and wandering only;
wake word / click / push-to-talk still work.
- **`notifications.py`** — Linux/D-Bus notification bridge. Two latency/loss bugs
fixed 2026-08-02, both in `controller._maybe_heartbeat`: draining was wired to the
60s heartbeat interval rather than the ~1.2s wake tick, and a heartbeat that landed
mid-conversation stamped its own clock *before* checking — burning the slot and
waiting another full interval, repeatedly, which is how a notification could go
unspoken for five or ten minutes. Draining now runs on every tick while IDLE, and the
heartbeat clock only advances when the heartbeat actually runs. Separately, the rate
limit used to *drop* notifications inside its window (a second text a minute later was
silently lost); the filter still gates at queue time but the limit is gone — a burst
is **batched into one turn** instead, same single round trip, no lost messages, capped
by `_MAX_PENDING_NOTIFICATIONS`.
- **`notifications.py` internals** — tails
`dbus-monitor`, parses Notify calls (pure `iter_notifications()`), filters
and rate-limits them (`NotificationGate`), and the controller forwards
survivors through `converse()`. Off by default — each one is a round trip.
- **`sudo_askpass.py`** — makes server-relayed `sudo` usable from a process
with no terminal, by pointing sudo's `SUDO_ASKPASS` at a GUI helper and
rewriting bare `sudo` to `sudo -A` (`add_askpass_flag`, a conservative regex
that skips anything already carrying a flag and anything inside quotes).
Prefers a real askpass binary and falls back to generating a
zenity/kdialog wrapper in `~/.cache/bolt-pet/askpass.sh`. Resolution order
is injectable (`is_executable`/`which`) so it's testable on a machine with a
different set installed. See the security notes — the dialog is the boundary.
- **`updater.py`** — self-update from the Gitea releases API. Polls
`<UPDATE_REPO_API>/releases/latest` for a tag newer than
`bolt_pet.__version__` and moves the checkout to it with
`git fetch --tags` + `git checkout tags/<tag>`, so "downloading an update"
is just git and rolling back is one command. Three safety rules: a **dirty
working tree is skipped, never stashed** (silently discarding your
work-in-progress beats running an old version); everything after the
checkout — dependency install, then an **import smoke test in a
subprocess** (this process still has the old modules loaded, so importing
in-process would prove nothing) — is guarded, and any failure rolls back to
the exact ref that was live before, branch name or SHA; and the restart only
happens once the new code imports, so a broken release costs a log line
rather than a pet that won't start. Git goes through an injectable
`run(args) -> (code, output)` callable so apply/rollback is unit-tested
against a fake git; version comparison and release parsing are pure.
`controller._maybe_update` drives it from the wake-listener tick (so the pet
is IDLE and between turns by construction) and the actual `os.execv` happens
in `ui/app.py` *after* `app.exec()` returns — that ordering is what
guarantees the mic is released before the new process opens it.
- **`history.py`** — rolling transcript (`HISTORY_LIMIT` turns) behind the
tray's History window and click-to-copy on the bubble.
- **`hotkey.py`** — global push-to-talk via `pynput`; soft-fails with a logged
reason (Wayland, missing package, macOS permissions) since the wake word is
the primary trigger.
- **`audio/stt_stream.py`** / **`audio/stt.py`** — speech-to-text via the
server's `/desk/stt` websocket relay (audio up, Deepgram's JSON messages
down untouched — the same relay the Android app uses; no local Deepgram
account or API key). `stt_stream.py` is the opportunistic optimisation:
the one-shot path waits for the utterance to end, then sends the whole
clip and waits again; that second wait is dead time that grows with how
long you spoke, and streaming removes it — `record_utterance(on_frame=...)`
hands each captured frame to a `StreamingTranscriber` as it's captured, so
by the time the VAD decides you stopped the transcript is essentially
already there. `stt.py`'s `transcribe()` is the *guaranteed* fallback for
when that didn't produce anything: since there's no separate REST endpoint
server-side, it opens the exact same relay via `stt_stream.connect()` and
just feeds the whole buffered utterance in one go — deliberately
*unconditional*, not gated by `STT_STREAMING`/`available()` the way the
opportunistic path is, since there's nothing left to fall back to if that
connection fails. Three more deliberate limits on the streaming half:
`open()` returning **None is an ordinary outcome** (no websocket-client, no
network, streaming turned off) because the full audio is still buffered
and `controller._transcribe` just falls back to `stt.transcribe()`; the
**local VAD still decides when you stopped** rather than Deepgram's
endpointing, since barge-in, follow-up listening and the grace period are
all built on it and coupling them to the network is not a first-pass
change; and the socket is **per-utterance**, because holding one open
across an idle pet bills for silence and dies on the first blip.
Off: `STT_STREAMING=false`.
- **Streamed replies** — `server_client.converse_stream()` reads NDJSON from the desk
API's `/desk/converse_stream` and speaks each sentence as it arrives
(`controller._speak_stream_chunk`), so the wait is time-to-first-sentence instead of
the whole model call. `Reply.spoken` marks a reply whose sentences were already said:
the text is still carried, because the follow-up rule needs to see whether it ended on
a question, but it must not be read out again. A stream that fails *before* anything
was spoken falls back to `converse()` invisibly; one that fails after ends the turn
quietly rather than repeating the first half. Off: `STREAMING_REPLIES=false`.
- **`speech.py`** — everything the pet says, and the policy differences between
kinds of saying. There were four near-copies of this in `controller.py` (a
reply, a holding line, a streamed sentence, a dialogue scene), each repeating
the same dance — transition state, show the bubble, maybe record history,
reset barge-in, call TTS, reset barge-in *again*, resume state, decide
whether to keep the mic open — and they had already drifted apart: one forgot
to arm barge-in, another skipped the follow-up rule. Now the dance is
`Speaker.say()` and the differences are data on a frozen `Utterance`
(`record`, `resume`, `hold_talking`, `follow_up`, `interruptible`), built by
the four classmethods `reply`/`holding`/`stream_chunk`/`scene`. Notable
policies: a holding line is **not** recorded (it's filler; the transcript
should keep the answer) and **not** interruptible (cutting off "give me a
sec" strands the tool already running), and it resumes the state it
interrupted rather than dropping to IDLE, because the turn isn't over. A
streamed chunk stays TALKING so the sprite doesn't flicker between sentences.
Collaborators are injected, so all of this is tested without Qt, audio or a
real state machine. Two subtleties that are bugs waiting to happen: the
barge-in detector is read through a **callable**, not held (it's built after
the Speaker — it needs the mic stream — and swapped when the mode changes; two
copies drifting apart is invisible until the wake model starts hearing the
pet), and `last_detail` is captured *before* the post-playback reset, since
reading it after means every interruption reports zeroed counters.
`follow_up_decision()` is the mic-open rule as a pure function.
- **Lip-sync** — the mouth is driven by the audio, not a timer.
`audio/tts.level_of(frame)` reduces a PCM frame to 0..1 loudness on a **sqrt
curve** (speech sits well below peak most of the time, so a linear map leaves
the mouth barely open during normal talking), `envelope()` does the same for a
whole clip, and playback calls the `on_level` hook per chunk. That travels
`Speaker` → `controller.mouth` (a Signal) → `PetWindow.set_mouth`, and
`_mouth_frame()` indexes the talking frames directly — which works because
`generate_bolt_sprites.py` draws them as an **openness ramp** (closed first,
widest last) rather than an arbitrary loop. Levels going stale
(`_MOUTH_STALE_SECONDS`) hands control back to the ordinary animation, so
offline TTS — which has no envelope — degrades to the timed loop instead of
freezing the mouth mid-syllable.
- **`window_state.py`** — where the pet was left, so it starts there.
`~/.cache/bolt-pet/window.json`, atomic write on drag-end, every function
swallows its own errors (a corrupt state file must mean the default corner,
never a pet that won't start). `is_visible_on()` re-validates against the
*current* screen layout on load, because the common case for a stale position
is exactly the dangerous one: the pet was last on a monitor that is now
unplugged, and restoring it faithfully puts it somewhere unreachable. Takes
plain rectangles rather than importing Qt. Off: `PET_REMEMBER_POSITION=false`.
- **`doctor.py`** — `python -m bolt_pet --doctor`, a preflight, written after a
week of debugging things one command would have shown: a venv whose python
was a zero-byte file, an OCR engine never installed, a wrong proxy header.
Twelve independent checks, each reporting ok / warn (degraded but working) /
fail, and each saying **what to do about it** — `screen reading: warn` is
useless alone, `apt install tesseract-ocr` is the whole point. Nothing raises:
a doctor that crashes on a broken install is diagnosing the wrong patient, so
`run()` catches per-check and a failed check becomes a FAIL row rather than a
traceback. Shallow by default (no network, no mic) since it's the first thing
you reach for when the network is what's broken; `--deep` actually contacts
the server and opens the microphone.
- **`speech_text.py`** — sanitizes server replies before they're heard/shown.
`for_speech()` (called inside `tts.speak()`, so every path to the speakers is
covered) strips markdown, emoji, URLs and stray symbols the voice would read
literally ("asterisk asterisk"), turns bullet lists into full sentences, and
words a few symbols (`&` → "and"). `for_display()` is the looser version for
the speech bubble — markdown syntax gone, emoji kept. `is_question()` decides
whether a reply leaves the pet waiting on an answer: it tests the *spoken*
form (so a '?' inside a stripped code block or URL doesn't count) and only a
trailing one counts, since a question asked in passing isn't awaiting a
reply. `speech.follow_up_decision` uses it to keep listening without the
wake word, capped by `FOLLOW_UP_MAX_TURNS` so a server that ends every reply
with a question can't loop forever off mic noise. Pure string logic, no
Qt/audio imports.
- **`ui/`** — `app.py` wires `QApplication` + `PetWindow` + `PetTray` + the
history/tuner windows + the push-to-talk hotkey + the controller thread
together; `pet_window.py` is the frameless/translucent/always-on-top sprite
window + speech bubble (non-square frames are centered in the square
`PET_SIZE` window, see `paintEvent`), and also owns:
- **wandering** — a ~30fps timer walks the window toward a random on-screen
target every `PET_WANDER_INTERVAL_SECONDS` (randomized), suppressed
whenever the pet is non-IDLE, napping, dragged, or has a bubble up. A
commanded `petctl move` overrides all of that except the drag.
- **the walk cycle** — while actually travelling, `_animation_key()` swaps
the state animation for the side-view `walk/` frames (not a `PetState` —
see the sprites README). It is stepped by *distance travelled*
(`_WALK_PIXELS_PER_FRAME`), never by the animation timer, so the planted
paw tracks backwards at exactly the speed the window moves forwards;
`_advance_frame` deliberately no-ops while walking so the two can't
double-step it. The art is drawn facing right and `_oriented()` mirrors it
(cached per frame) when heading left. No `walk/` art → falls back to the
old coded bob rather than a placeholder blob.
- **emotes** — `emote_transform()` is pure maths (dx, dy, rotation, scale
from a 0..1 progress) kept out of `paintEvent` so the curves are unit
tested; every emote must return to the identity transform at progress 1.0
or the pet ends up permanently askew.
- **shaped input / click-through** — `PET_SHAPED_INPUT` masks the window to
the sprite's opaque pixels so the square window's transparent corners stop
eating clicks (mask rebuilt only when the frame changes, and pinned to the
resting position so a bob/spin doesn't thrash it); `PET_CLICK_THROUGH`
makes the pet ignore the mouse entirely.
- **edge snapping** (`PET_EDGE_SNAP`) after a drag or a stroll, and **nap
dimming** (`set_napping`).
`sprite.py` loads `assets/sprites/<state>/*.png` (filename-sorted, looping —
the art is *generated* by `scripts/generate_bolt_sprites.py`, a Pillow
drawing of Bolt as a shepherd pup; edit the script and re-run it rather than
the committed PNGs, see `assets/sprites/README.md`) and falls
back to a procedurally-drawn placeholder blob per state if a folder has no
frames. It also loads `EXTRA_ANIMATIONS` — currently just `walk/` — keyed by
name rather than by `PetState`, with `has()` reporting whether a key is
backed by real art so callers can decline a placeholder instead of trotting
a blob across the desktop; `tray.py` is the system tray menu (talk now / mute / nap / wander /
click-through / history / wake-word tuning / use-default-voice / quit) — the
pet window has no title bar or taskbar entry; `history_window.py` and
`wake_tuner.py` are the two dialogs it opens.
- **`config.py`**: a small hand-written `.env` parser feeding module constants. Nothing else reads
`os.environ`. `SESSION_ID` defaults to `pet-console-<host>`. The `pet-` prefix is deliberate: the
server advertises `filectl` to pet sessions, and `-console` keeps this a separate conversation
from a Bolt-Pet on the same machine. `new_session()` (New chat / `/new`) rotates to
`<base>-<timestamp>` and remembers it in `~/.cache/bolt-console/session.json`. `DESK_SESSION_ID`
in `.env` overrides the remembered id. Tests must monkeypatch `config.STATE_FILE`.
- **`server_client.py`**: the desk protocol. `_relay()` is the shared command → tool_result loop.
`converse_stream()` hands each `say` sentence over as it arrives and finishes a tool call through
`_relay`. It raises `ServerError` only if nothing was shown yet, which lets the caller fall back to
`converse()` invisibly. `Reply.streamed` means "already on screen, don't print again".
`run_local_command` runs in `os.getcwd()` with stdin closed and `CREATE_NO_WINDOW`, because under
pythonw every cmd.exe would otherwise flash a console.
- **`tools.py`**: `Tools`, the `on_command` callable. It routes `filectl` to `file_ops`, answers
`petctl`/`dialoguectl` with `PET_UNAVAILABLE` (the server may still emit them), and treats
anything else as a shell command. Approval: reads and lists never ask. Shell, write and edit ask
via `ui.confirm` unless `mode == "auto"`, and "always" switches the session to auto. A decline
returns `DECLINED` to the model rather than raising, so the turn still completes.
- **`file_ops.py`**: `filectl {json}` list/read/write/edit. It is a **single-line** JSON wire format
because the server's `command` marker only captures to the first newline. `edit` must match
exactly once. Parsing (`parse`) is separate from I/O (`execute`).
- **`relay_json.py`**: lenient JSON for model-typed payloads. It tries strict parsing first, then
named repairs (fences, smart quotes, invalid backslash escapes, stray quotes, trailing commas,
Python literals, single quotes), and accepts a repair only if the result parses. Repairs are
always reported back to the model, never silent. The backslash repair only touches *invalid*
escapes, so `C:\temp` still parses as a tab (that's ambiguous, so it's left alone).
- **`ui.py`**: `Terminal` handles plain ANSI rendering, the spinner thread, tool-call/result lines,
diffs, `confirm` and `read_message`. All output goes through `_emit` under the same lock the
spinner draws under. The spinner redraws by erasing the current line, so output is always written
as whole lines (see `assistant_chunk`). Glyphs fall back to ASCII on non-UTF-8 streams.
- **`console.py`**: `Session` (one turn, delivered-file download, slash commands, `!` local shell,
the loop) and `run()` (argv). Collaborators are injected so tests can drive it.
- **`gui/`**: the window. `bridge.Bridge` implements the same surface as `ui.Terminal`
(tool_call, tool_result, confirm, assistant_chunk, …) with each call emitting a Qt signal, so the
window reuses `console.Session` and `Tools` unchanged. `window.TurnRunner` runs `Session.send` on a
QThread. `Bridge.confirm` blocks the worker on an Event until a ToolCard's Allow/Deny calls
`answer()`, and `close()` releases it as "no". `window.MainWindow` reacts only to Bridge signals.
`widgets.py` holds the transcript pieces (ToolCard, AssistantMessage, Composer, …). `theme.py` holds
the colours and the one stylesheet (widgets set object names and dynamic properties for it, then
`repolish()`). `icons.py` draws every icon with QPainter, so there are no asset files.
`markdown_html` swaps Qt's fixed font for code (Courier New on Windows) with the app's mono font.
- **`dictation.py`**: the mic button. It streams 16 kHz int16 to the server's `/desk/stt`
websocket (Bolt-Pet's route: the server relays to Deepgram) and buffers every frame. If the live
socket never opened or died, `_fallback()` sends the whole buffer once from an empty transcript,
so a half-live session's words don't come back twice. A healthy socket is never re-sent, even
when it heard nothing, because that would pay twice. Stops on click, on `silence_seconds` after
speech, or on `max_seconds`. No Qt and no numpy (the array module does RMS), so it's identical
in the Linux single file. The mic and socket are injected. The window turns the callbacks into
`DictationSignals`, and the composer is read-only while a recording rewrites it. PySide6 6.10
needs `QTextCursor.MoveOperation.End`, not `cursor.End`.
- **Chats**: `server_client.list_chats()`/`create_chat()` wrap `GET`/`POST /desk/chats` (the server
mints `chat-…` ids; a chat is only listed once used). Switching is `config.set_session(id)`: a
plain reassignment of the `SESSION_ID` every call reads, persisted to `session.json`, and
deliberately not locked. The window refuses to switch mid-turn instead. `gui/chat_list_window.py`
runs list/create as `ChatTask`s on `QThreadPool.globalInstance()` and hands results back through
`ChatSignals`, never calling the network from a slot. The header New chat uses the same pool.
`Session.new_conversation()` (terminal `/new`) is blocking and falls back to a local id if the
server can't mint one. `chats.py` holds the Qt-free row helpers (the preview drops the
`[Bolt Console: …]` and `[on screen right now: …]` notes).
- **`file_delivery.py`**: saves files the server queues via `deliver_files`. The name is reduced to
its basename and never overwrites an existing file. `Session.check_deliveries` runs after every
turn.
- **`doctor.py`**: independent checks, each reporting ok/warn/fail plus what to do about it. A check
never raises.
### Voices (the server's `speak_as`)
## Testing conventions
Ask Bolt to talk like someone else, or in another language, and the *server*
does the picking: its desk-only `voice_search` marker browses the ElevenLabs
voice library, and `speak_as: <voice_id>` on the final reply tags that reply
with the chosen voice (adding a Voice Library pick to the ElevenLabs account
first, so the id is usable by the time it reaches us). Nothing about that is
this repo's to decide — all the client owes it is actually speaking in the
voice it was handed: `converse()` returns it on `Reply`, `_apply_voice()`
records it, and `_speak()` passes it to `tts.speak(voice_id=...)`.
`tests/test_gui.py` sets `QT_QPA_PLATFORM=offscreen` itself and drives real turns through
`MainWindow` and its worker thread against the loopback server (`pump_until` processes events).
Offscreen Qt has **no font database**, so check the look by rendering with the real platform plus
`WA_DontShowOnScreen` and `grab()`, not offscreen.
Two things are decided *here*, though, because the server can't:
Tests cover logic with injected fakes (`FakeUI`, `FakeShell`, a `Terminal` writing to `StringIO`
with a scripted `input_fn`). `tests/test_console.py` is the end-to-end file: a real threaded HTTP
server on a loopback port driven through the real `server_client`, `Tools` and `Terminal`. Add a
case there when a bug crosses module boundaries. Tests that embed a path in JSON must use
`json.dumps(str(path))`, because Windows backslashes are otherwise invalid escapes.
- **The voice sticks** (`VOICE_STICKY`, default on). The server tags one
reply and strips the marker before storing the turn, so it never sees the
id again — "keep talking like that" would send it searching for a voice all
over again, and it'd likely land on a different one. Holding the id
client-side is what makes the rest of the conversation stay in that voice.
An untagged reply therefore never *changes* the voice; only a new
`speak_as`, `VOICE_STICKY=false`, or a reset does.
- **There's a way back.** Since the server was never told Bolt's own voice
id, it can't ask for it back with `speak_as` — so reverting is local: the
tray's **Use default voice** entry (enabled only while a picked voice is
in use, kept in sync by the `voice_changed` signal), a restart, or
`petctl voice reset`, which is what lets Bolt honour "go back to your
normal voice" out loud. That last one needs the server's pet prompt block
(`ai/desk_api.py`, `pet_tools`) to mention the verb, or the model never
emits it — the desk API's prompt is where petctl is advertised.
## Fabricated tool results
### Wake-word detection
`audio/wake_word.py` uses a custom-trained openWakeWord model,
`thunderbolt.onnx` (ships in the project root), the same way the server
repo's `desk_client/bolt_desk.py` uses `bolt.onnx` for "hey bolt" — same
runtime (openWakeWord, ONNX inference framework), same per-frame
`predict()`/`reset()` loop. Every mic frame is scored; any class score at or
above `WAKE_WORD_THRESHOLD` (default `0.5`, in `.env`) counts as a
detection. Swap `WAKE_MODEL_FILE` to point at a differently-trained `.onnx`
model to change the wake phrase — everything downstream (STT, server call,
TTS) is unaffected.
**openwakeword's `Model.reset()` is not enough to forget a detection.** It
clears the *prediction* buffer only; the rolling audio window the classifier
actually scores lives in `model.preprocessor` (`raw_data_buffer` — 10s of raw
audio — plus `melspectrogram_buffer` and a ~120-frame `feature_buffer`) and
`AudioFeatures` has no reset method at all. So after a detection the wake
phrase is still in the window, and the next frame fed to the model re-fires on
it. Symptom when this bites: the pet cuts itself off a word into every reply,
because wake-mode barge-in resumes feeding the model and instantly matches the
"thunderbolt" that *started* the turn. `wake_word.hard_reset(model)` restores
the preprocessor to its as-constructed (silence) state and is what both
`listen_for_wake_word` and `WakeWordBargeIn.reset()` call — use it, not
`reset()`, anywhere a detection needs to be genuinely forgotten. The blank
state is cached on the preprocessor object (not in an `id()`-keyed dict —
CPython reuses ids after GC), since rebuilding it costs an ONNX pass over 10s
of silence.
The threshold is tunable at runtime: the tray's **Wake word tuning…** window
(`ui/wake_tuner.py`) shows the peak score seen and a rolling list of near
misses (frames within `WAKE_NEAR_MISS_MARGIN` *below* the threshold — i.e.
the times it nearly heard you), and its slider is read per frame because
`listen_for_wake_word` accepts a callable threshold. Set the threshold just
under the peak you can hit reliably, then persist it in `.env`.
### Testing conventions
`tests/` covers pure logic only (state machine, wake-word scoring loop, mic
VAD, HTTP client against mocks) — nothing there needs real audio hardware or
a display. Modules under test are written to accept fake streams/models/
`on_command` callables specifically to keep tests hardware-free; follow that
pattern (inject a `Protocol`-typed collaborator) rather than mocking at the
`sounddevice`/`openwakeword` import boundary when adding new testable logic.
`test_controller.py`, `test_controller_features.py`, `test_wander.py` and
`test_pet_window_features.py` need a `QApplication`, which segfaults without
a display unless run with `QT_QPA_PLATFORM=offscreen`.
Newer subsystems follow the same rule — the testable part is separated from
the part that needs hardware: dbus-monitor output is parsed by a pure
`iter_notifications(lines)`, xprop output by pure `parse_xprop_*` functions,
HTTP chunk reassembly by `chunks_to_int16`, emote motion by
`emote_transform`. Tests that touch the controller monkeypatch
`screen_context.context_for` / `is_fullscreen_active`, otherwise they shell
out to xprop on a headless box.
**`test_pipeline_smoke.py` is the exception, deliberately.** Unit tests inject
a fake at the seam they care about, and a run of production bugs — notifications
sitting unspoken for minutes, the pet saying things twice, `[laughing]` read
aloud, a device command arriving as prose — got through with every one of them
green, because each was an *interaction* between two individually-correct
units. So that file stands up a real threaded HTTP server on a loopback port,
speaks the desk protocol at it, and drives whole turns through the real
`server_client` (including NDJSON streaming), the real controller and the real
state machine, faking only the mic stream and the speakers. When a bug crosses
a module boundary, add the case there; when it lives inside one module, the
pure-function pattern above is still the cheaper test. It is also worth
mutation-checking a new case — break the source line it is meant to catch and
confirm it actually goes red.
The model sometimes writes tool output as prose instead of emitting a command. The tool marker
syntax in `console.annotate()` is load-bearing: vague wording ("use filectl") measurably caused
this against the live server, and spelling out `command: ...` fixed it. `console.fabricated()`
flags a reply shaped like tool output when `Tools.calls` didn't move, and the next message carries
`CORRECTION`. A polluted session keeps copying its own history, so the remedy is New chat.
## Security notes
The server can relay a shell command back to this machine to execute as the
desktop user (see `server_client.run_local_command`). This is intentional
("full desktop control" for things like "open firefox" or disk checks) and
matches the trust model of the server repo's other desk clients. Keep
`DESK_API_KEY` private and don't expose the desk API port to the open
internet.
`file_ops.py`'s `filectl` read/write/edit pseudo-commands ride that same
relay and are bound by the same trust model — no path is off-limits beyond
normal filesystem permissions for the desktop user, exactly like a relayed
`cat`/`sed`/`rm` already isn't. They don't grant the server anything a shell
command couldn't already do; they just make the read/write/edit path
reliable instead of relying on the model getting shell quoting right.
`sudo_askpass.py` widens that further, by design: with `SUDO_ASKPASS_PROMPT`
on (the default), a relayed bare `sudo` is rewritten to `sudo -A` and the
password is collected in a desktop dialog, so commands can escalate to root
instead of hanging on a tty the pet doesn't have. The dialog is the security
boundary — it's the only thing between the server deciding to run `sudo` and
it running, so the prompt is deliberately not suppressible per-command and
those commands get their own longer timeout (`SUDO_COMMAND_TIMEOUT_SECONDS`)
rather than being made non-interactive. Set `SUDO_ASKPASS_PROMPT=false` to
take the capability away entirely; sudo commands then fail. Note that
`sudo -n` / `sudo -A` / `sudo -u …` in a relayed command are never rewritten,
so an explicit non-interactive sudo stays non-interactive.
**Don't run the pet as root.** It needs no privileges of its own, PortAudio
can't reach the user's PipeWire socket from a root session (raw ALSA devices
reject the 16 kHz capture rate — `paInvalidSampleRate`), and every relayed
command would run unconstrained.
Several features widen what leaves this machine, all switchable in `.env`:
`SCREEN_CONTEXT` appends the focused window's *title* to each utterance
(titles often contain file paths, document names, or subject lines),
`MONITOR_CONTEXT` appends the screen layout (sizes and names only — no
contents), and `NOTIFICATION_BRIDGE` (off by default) forwards matching
desktop notifications to the server. None of those send screenshots or
notification contents you haven't matched with `NOTIFICATION_FILTER`.
`SCREEN_TEXT` is the biggest of them: `petctl read` OCRs a whole monitor and
sends the recognised text to the server — everything visible, not just the
focused window. Two things keep it honest. It's **pull-only**: no capture
happens unless the server explicitly asks, so it can't leak in the background
the way a per-turn annotation would, and each read is logged. And it is
strictly *not* a new capability — the shell relay could already run a
screenshot tool and pipe it through OCR — it just makes a thing the trust
model already allowed reliable, bounded (`SCREEN_TEXT_MAX_CHARS`) and
visible. It is nonetheless far easier to reach for than the shell route, so
if that trade isn't one you want, `SCREEN_TEXT=false` removes it and
`petctl read` starts reporting that it's disabled. Capture is `mss`-based and
therefore silently unavailable on Wayland.
Approved shell commands, writes and edits run as the user with no sandbox. `auto` mode removes the
prompt. That is the same "full desktop control" trust model as the server's other desk clients,
which is why `ask` is the default here. Keep `DESK_API_KEY` private.