feat: Enhance local command handling and introduce local intents

- Refactor `run_local_command` to manage subprocesses more effectively, ensuring child processes are terminated on timeout.
- Introduce `_terminate` function to handle process group termination and capture output.
- Implement `_command_output` to format command results with a character limit.
- Add local intent recognition in `intents.py` to handle commands like "stop", "go to sleep", and "come here" without server interaction.
- Normalize user input to match local intents while stripping filler words.
- Update tests to cover new local intent functionality and ensure proper command handling.
- Enhance speech processing to handle abbreviations and improve spoken output clarity.
This commit is contained in:
2026-08-05 18:31:02 -06:00
parent 8d4751d80f
commit 3ee67cb4d6
14 changed files with 1107 additions and 59 deletions
+97 -8
View File
@@ -79,12 +79,41 @@ logs a missing-config message and exits its thread instead of starting.
queued desktop notifications. It owns the live wake-word threshold
(`wake_threshold()` is passed to `listen_for_wake_word` as a *callable* so
the tray slider takes effect mid-listen) and the conversation `history`.
**One thread drives all of it, so failure containment is structural.** Every
entry point that can raise runs inside `_guarded(work, label)`, which logs and
forces the machine back to IDLE (the only state it's always safe to resume
from): the conversation turn, the heartbeat tick — which matters most, since
`on_tick` is the one place control returns to us during a listen that blocks
for minutes, and everything it drives touches the network or shells out — and
the post-restart report. `run()` wraps the lot in try/finally because
`finished` is what `ui/app.py` waits on to quit the thread and to run a
pending `os.execv`; an exception escaping `_loop` used to skip it, so the
failure mode of any bug below was "the pet goes deaf with the mic still open
and the tray won't quit" rather than "one turn failed". `_handle_command` has
the same shape for a different reason: it must **always return a string**,
because the server is blocked on `/desk/tool_result` while it runs and an
exception there means the relay never posts and the server sits out its own
timeout on a turn that can't finish — silent on both ends. Handed back as
command output instead, Bolt can read what broke and say so in the same turn.
- **`server_client.py`** — HTTP client for the desk API, dependency-free
beyond `requests` so it's easy to mock in tests. `converse()` loops relaying
server-issued shell commands (`run_local_command`, executed via
`subprocess.run(shell=True)` as the desktop user, 30s default timeout) via
server-issued shell commands (`run_local_command`, executed via a
`shell=True` `Popen` as the desktop user, 30s default timeout) via
`/desk/tool_result` until the server sends a final `reply` (capped at
`_MAX_RELAY_HOPS`). This is the same "full desktop control" trust model as
`_MAX_RELAY_HOPS` — exhausting which is reported as its own error, because
"unknown server response" sent everyone looking at the payload shape when what
happened is a model that kept calling tools and never answered).
`run_local_command` is `Popen` rather than `subprocess.run` for the timeout
path: the command is a shell, and `run()`'s timeout would kill only that
shell, leaving whatever it spawned (a build, a `tail -f`, an ffmpeg) alive for
the rest of the session with no parent watching — so the child gets its own
process group (`start_new_session`, POSIX) and a timeout SIGTERMs the group,
SIGKILLs it two seconds later, then drains the pipes *with its own timeout* so
a grandchild holding stdout can't turn a timeout into a hang. Whatever the
command printed before it hung is returned alongside the timeout notice, since
the last line usually says exactly what it was stuck waiting for. This is the
same "full desktop control" trust model as
the server repo's other desk clients — commands only ever originate from
the user's own voice/click requests in their own session. A final reply is
returned as a `Reply(text, voice_id, voice_name)` rather than a bare string,
@@ -108,7 +137,8 @@ logs a missing-config message and exits its thread instead of starting.
the name isn't always coming from someone as trusted as the owner. Toggle
off entirely with `RECEIVE_FILES=false`.
- **`audio/`** — `mic.py` (energy-based VAD utterance capture, ported from the
server repo's `bolt_desk.py`), `wake_word.py` (openWakeWord `thunderbolt.onnx`
server repo's `bolt_desk.py`, plus `flush()` — see the note below on the pet
hearing itself), `wake_word.py` (openWakeWord `thunderbolt.onnx`
detection + `NearMissLog` for threshold tuning — see below), `stt.py`
(Deepgram), `tts.py` (ElevenLabs, streaming by default — `stream_pcm()` +
`play_stream()` start playback on the first chunk; `chunks_to_int16()`
@@ -266,6 +296,14 @@ logs a missing-config message and exits its thread instead of starting.
`dbus-monitor`, parses Notify calls (pure `iter_notifications()`), filters
and rate-limits them (`NotificationGate`), and the controller forwards
survivors through `converse()`. Off by default — each one is a round trip.
Note where the queue between the two threads lives: notifications arrive on
the watcher thread and are forwarded from the heartbeat, which **doesn't run
while the pet is napping** — so they accumulate overnight. The controller's
queue is therefore a bounded `deque` stamped on arrival, and the drain
discards anything older than `NOTIFICATION_MAX_AGE_SECONDS` rather than
reading a nine-hour-old backlog out at 8am. A drain that stops early (a nap
starting mid-loop, or the server going down) re-queues what it didn't forward
instead of dropping it, which the original swap-and-return did silently.
- **`sudo_askpass.py`** — makes server-relayed `sudo` usable from a process
with no terminal, by pointing sudo's `SUDO_ASKPASS` at a GUI helper and
rewriting bare `sudo` to `sudo -A` (`add_askpass_flag`, a conservative regex
@@ -302,15 +340,43 @@ logs a missing-config message and exits its thread instead of starting.
`for_speech()` (called inside `tts.speak()`, so every path to the speakers is
covered) strips markdown, emoji, URLs and stray symbols the voice would read
literally ("asterisk asterisk"), turns bullet lists into full sentences, and
words a few symbols (`&` → "and"). `for_display()` is the looser version for
words a few symbols (`&` → "and"), abbreviations the voice would spell out
letter by letter (`e.g.` → "for example", `etc.` → "and so on") and a long
option's leading `--` (heard as "dash dash force"; the single hyphen has to
survive for "bolt-pet"). `for_display()` is the looser version for
the speech bubble — markdown syntax gone, emoji kept. `is_question()` decides
whether a reply leaves the pet waiting on an answer: it tests the *spoken*
form (so a '?' inside a stripped code block or URL doesn't count) and only a
trailing one counts, since a question asked in passing isn't awaiting a
reply. `controller._should_follow_up` uses it to keep listening without the
form (so a '?' inside a stripped code block or URL doesn't count) and a '?'
**anywhere** counts. That last part was once trailing-only, on the theory that
"What time is it? It's 7:15." isn't awaiting a reply — true of that sentence
and wrong more often, since Bolt routinely asks and then keeps talking ("Want
me to fix it? I'd start with the config"), which is the case that actually
costs you a wake word. The asymmetry is the argument: an unwanted extra listen
ends itself on `VAD_GRACE_SECONDS` of silence, a missed one makes you start
over. `controller._should_follow_up` uses it to keep listening without the
wake word, capped by `FOLLOW_UP_MAX_TURNS` so a server that ends every reply
with a question can't loop forever off mic noise. Pure string logic, no
Qt/audio imports.
- **`intents.py`** — the handful of utterances answered *without* the server.
"stop", "come here", "go to sleep", "say that again", "use your normal voice"
are commands to the body, and routing them through the desk API costs two to
four seconds and three network hops to make the pet walk left — and only works
if the server's prompt happens to advertise the matching `petctl` verb (which
is why `voice reset` needs a block in `ai/desk_api.py`'s pet prompt; see the
Voices section). Recognising the phrase here removes both the latency and that
coupling. The design problem is *not stealing real requests*, and three rules
cover it: whole-utterance exact match after normalisation (so "stop" is an
intent and "stop the docker container" is a question for Bolt), a closed table
with nothing arguable in it, and **never on a follow-up turn** — if Bolt just
asked you something your answer is his, and swallowing "never mind" locally
would leave the server holding a question it never got an answer to. Both
sides of the comparison go through `normalize()` (the table is canonicalised
at import, and `_build()` refuses to build one where two intents claim the
same normalised phrase, or where a phrase reduces to "" and would match pure
filler like "hey bolt"). Actions come back in the **same shape
`pet_actions.parse` produces**, so `PetWindow.apply_action` needs no new
vocabulary; the effects live in `controller._handle_local_intent`. Off switch:
`LOCAL_INTENTS=false`.
- **`ui/`** — `app.py` wires `QApplication` + `PetWindow` + `PetTray` + the
history/tuner windows + the push-to-talk hotkey + the controller thread
together; `pet_window.py` is the frameless/translucent/always-on-top sprite
@@ -417,6 +483,29 @@ the times it nearly heard you), and its slider is read per frame because
`listen_for_wake_word` accepts a callable threshold. Set the threshold just
under the peak you can hit reliably, then persist it in `.env`.
### The mic keeps recording while nothing is reading it
Same family of bug as the openwakeword one above, one layer down: PortAudio
captures into a ring buffer continuously, so audio from a stretch where the
pipeline thread was busy elsewhere is still queued when the next read happens.
It bites in exactly one place. At the end of a reply that asked you something,
`_speak` sets `_talk_now` and the next turn starts recording immediately — with
the tail of the pet's own TTS sitting in that buffer, above the VAD threshold.
The VAD takes it for the start of your answer, Deepgram transcribes it, and Bolt
is handed his own last sentence as if you had said it. With barge-in on the
detector was draining the stream during playback so the window is small; with
`BARGE_IN=false` nothing drains it at all.
`mic.flush(stream)` drops what's buffered, and `_speak` calls it on the
follow-up branch only. **That placement is the whole correctness argument**
flushing is only safe where the buffer is known to hold nothing *you* said:
playback ran to completion, so if you had spoken, barge-in would have cut it and
taken the interrupted branch instead. Never flush before a wake-triggered
recording, where the rest of "thunderbolt, what time is it" is legitimately
queued and dropping it clips the request. A single call is bounded by
`max_seconds` so it can't chase a stream filling as fast as it drains, and it
no-ops on a stream with no `read_available` (i.e. every fake stream in tests).
### Testing conventions
`tests/` covers pure logic only (state machine, wake-word scoring loop, mic