feat: Enhance local command handling and introduce local intents
- Refactor `run_local_command` to manage subprocesses more effectively, ensuring child processes are terminated on timeout. - Introduce `_terminate` function to handle process group termination and capture output. - Implement `_command_output` to format command results with a character limit. - Add local intent recognition in `intents.py` to handle commands like "stop", "go to sleep", and "come here" without server interaction. - Normalize user input to match local intents while stripping filler words. - Update tests to cover new local intent functionality and ensure proper command handling. - Enhance speech processing to handle abbreviations and improve spoken output clarity.
This commit is contained in:
@@ -79,12 +79,41 @@ logs a missing-config message and exits its thread instead of starting.
|
||||
queued desktop notifications. It owns the live wake-word threshold
|
||||
(`wake_threshold()` is passed to `listen_for_wake_word` as a *callable* so
|
||||
the tray slider takes effect mid-listen) and the conversation `history`.
|
||||
|
||||
**One thread drives all of it, so failure containment is structural.** Every
|
||||
entry point that can raise runs inside `_guarded(work, label)`, which logs and
|
||||
forces the machine back to IDLE (the only state it's always safe to resume
|
||||
from): the conversation turn, the heartbeat tick — which matters most, since
|
||||
`on_tick` is the one place control returns to us during a listen that blocks
|
||||
for minutes, and everything it drives touches the network or shells out — and
|
||||
the post-restart report. `run()` wraps the lot in try/finally because
|
||||
`finished` is what `ui/app.py` waits on to quit the thread and to run a
|
||||
pending `os.execv`; an exception escaping `_loop` used to skip it, so the
|
||||
failure mode of any bug below was "the pet goes deaf with the mic still open
|
||||
and the tray won't quit" rather than "one turn failed". `_handle_command` has
|
||||
the same shape for a different reason: it must **always return a string**,
|
||||
because the server is blocked on `/desk/tool_result` while it runs and an
|
||||
exception there means the relay never posts and the server sits out its own
|
||||
timeout on a turn that can't finish — silent on both ends. Handed back as
|
||||
command output instead, Bolt can read what broke and say so in the same turn.
|
||||
- **`server_client.py`** — HTTP client for the desk API, dependency-free
|
||||
beyond `requests` so it's easy to mock in tests. `converse()` loops relaying
|
||||
server-issued shell commands (`run_local_command`, executed via
|
||||
`subprocess.run(shell=True)` as the desktop user, 30s default timeout) via
|
||||
server-issued shell commands (`run_local_command`, executed via a
|
||||
`shell=True` `Popen` as the desktop user, 30s default timeout) via
|
||||
`/desk/tool_result` until the server sends a final `reply` (capped at
|
||||
`_MAX_RELAY_HOPS`). This is the same "full desktop control" trust model as
|
||||
`_MAX_RELAY_HOPS` — exhausting which is reported as its own error, because
|
||||
"unknown server response" sent everyone looking at the payload shape when what
|
||||
happened is a model that kept calling tools and never answered).
|
||||
`run_local_command` is `Popen` rather than `subprocess.run` for the timeout
|
||||
path: the command is a shell, and `run()`'s timeout would kill only that
|
||||
shell, leaving whatever it spawned (a build, a `tail -f`, an ffmpeg) alive for
|
||||
the rest of the session with no parent watching — so the child gets its own
|
||||
process group (`start_new_session`, POSIX) and a timeout SIGTERMs the group,
|
||||
SIGKILLs it two seconds later, then drains the pipes *with its own timeout* so
|
||||
a grandchild holding stdout can't turn a timeout into a hang. Whatever the
|
||||
command printed before it hung is returned alongside the timeout notice, since
|
||||
the last line usually says exactly what it was stuck waiting for. This is the
|
||||
same "full desktop control" trust model as
|
||||
the server repo's other desk clients — commands only ever originate from
|
||||
the user's own voice/click requests in their own session. A final reply is
|
||||
returned as a `Reply(text, voice_id, voice_name)` rather than a bare string,
|
||||
@@ -108,7 +137,8 @@ logs a missing-config message and exits its thread instead of starting.
|
||||
the name isn't always coming from someone as trusted as the owner. Toggle
|
||||
off entirely with `RECEIVE_FILES=false`.
|
||||
- **`audio/`** — `mic.py` (energy-based VAD utterance capture, ported from the
|
||||
server repo's `bolt_desk.py`), `wake_word.py` (openWakeWord `thunderbolt.onnx`
|
||||
server repo's `bolt_desk.py`, plus `flush()` — see the note below on the pet
|
||||
hearing itself), `wake_word.py` (openWakeWord `thunderbolt.onnx`
|
||||
detection + `NearMissLog` for threshold tuning — see below), `stt.py`
|
||||
(Deepgram), `tts.py` (ElevenLabs, streaming by default — `stream_pcm()` +
|
||||
`play_stream()` start playback on the first chunk; `chunks_to_int16()`
|
||||
@@ -266,6 +296,14 @@ logs a missing-config message and exits its thread instead of starting.
|
||||
`dbus-monitor`, parses Notify calls (pure `iter_notifications()`), filters
|
||||
and rate-limits them (`NotificationGate`), and the controller forwards
|
||||
survivors through `converse()`. Off by default — each one is a round trip.
|
||||
Note where the queue between the two threads lives: notifications arrive on
|
||||
the watcher thread and are forwarded from the heartbeat, which **doesn't run
|
||||
while the pet is napping** — so they accumulate overnight. The controller's
|
||||
queue is therefore a bounded `deque` stamped on arrival, and the drain
|
||||
discards anything older than `NOTIFICATION_MAX_AGE_SECONDS` rather than
|
||||
reading a nine-hour-old backlog out at 8am. A drain that stops early (a nap
|
||||
starting mid-loop, or the server going down) re-queues what it didn't forward
|
||||
instead of dropping it, which the original swap-and-return did silently.
|
||||
- **`sudo_askpass.py`** — makes server-relayed `sudo` usable from a process
|
||||
with no terminal, by pointing sudo's `SUDO_ASKPASS` at a GUI helper and
|
||||
rewriting bare `sudo` to `sudo -A` (`add_askpass_flag`, a conservative regex
|
||||
@@ -302,15 +340,43 @@ logs a missing-config message and exits its thread instead of starting.
|
||||
`for_speech()` (called inside `tts.speak()`, so every path to the speakers is
|
||||
covered) strips markdown, emoji, URLs and stray symbols the voice would read
|
||||
literally ("asterisk asterisk"), turns bullet lists into full sentences, and
|
||||
words a few symbols (`&` → "and"). `for_display()` is the looser version for
|
||||
words a few symbols (`&` → "and"), abbreviations the voice would spell out
|
||||
letter by letter (`e.g.` → "for example", `etc.` → "and so on") and a long
|
||||
option's leading `--` (heard as "dash dash force"; the single hyphen has to
|
||||
survive for "bolt-pet"). `for_display()` is the looser version for
|
||||
the speech bubble — markdown syntax gone, emoji kept. `is_question()` decides
|
||||
whether a reply leaves the pet waiting on an answer: it tests the *spoken*
|
||||
form (so a '?' inside a stripped code block or URL doesn't count) and only a
|
||||
trailing one counts, since a question asked in passing isn't awaiting a
|
||||
reply. `controller._should_follow_up` uses it to keep listening without the
|
||||
form (so a '?' inside a stripped code block or URL doesn't count) and a '?'
|
||||
**anywhere** counts. That last part was once trailing-only, on the theory that
|
||||
"What time is it? It's 7:15." isn't awaiting a reply — true of that sentence
|
||||
and wrong more often, since Bolt routinely asks and then keeps talking ("Want
|
||||
me to fix it? I'd start with the config"), which is the case that actually
|
||||
costs you a wake word. The asymmetry is the argument: an unwanted extra listen
|
||||
ends itself on `VAD_GRACE_SECONDS` of silence, a missed one makes you start
|
||||
over. `controller._should_follow_up` uses it to keep listening without the
|
||||
wake word, capped by `FOLLOW_UP_MAX_TURNS` so a server that ends every reply
|
||||
with a question can't loop forever off mic noise. Pure string logic, no
|
||||
Qt/audio imports.
|
||||
- **`intents.py`** — the handful of utterances answered *without* the server.
|
||||
"stop", "come here", "go to sleep", "say that again", "use your normal voice"
|
||||
are commands to the body, and routing them through the desk API costs two to
|
||||
four seconds and three network hops to make the pet walk left — and only works
|
||||
if the server's prompt happens to advertise the matching `petctl` verb (which
|
||||
is why `voice reset` needs a block in `ai/desk_api.py`'s pet prompt; see the
|
||||
Voices section). Recognising the phrase here removes both the latency and that
|
||||
coupling. The design problem is *not stealing real requests*, and three rules
|
||||
cover it: whole-utterance exact match after normalisation (so "stop" is an
|
||||
intent and "stop the docker container" is a question for Bolt), a closed table
|
||||
with nothing arguable in it, and **never on a follow-up turn** — if Bolt just
|
||||
asked you something your answer is his, and swallowing "never mind" locally
|
||||
would leave the server holding a question it never got an answer to. Both
|
||||
sides of the comparison go through `normalize()` (the table is canonicalised
|
||||
at import, and `_build()` refuses to build one where two intents claim the
|
||||
same normalised phrase, or where a phrase reduces to "" and would match pure
|
||||
filler like "hey bolt"). Actions come back in the **same shape
|
||||
`pet_actions.parse` produces**, so `PetWindow.apply_action` needs no new
|
||||
vocabulary; the effects live in `controller._handle_local_intent`. Off switch:
|
||||
`LOCAL_INTENTS=false`.
|
||||
- **`ui/`** — `app.py` wires `QApplication` + `PetWindow` + `PetTray` + the
|
||||
history/tuner windows + the push-to-talk hotkey + the controller thread
|
||||
together; `pet_window.py` is the frameless/translucent/always-on-top sprite
|
||||
@@ -417,6 +483,29 @@ the times it nearly heard you), and its slider is read per frame because
|
||||
`listen_for_wake_word` accepts a callable threshold. Set the threshold just
|
||||
under the peak you can hit reliably, then persist it in `.env`.
|
||||
|
||||
### The mic keeps recording while nothing is reading it
|
||||
|
||||
Same family of bug as the openwakeword one above, one layer down: PortAudio
|
||||
captures into a ring buffer continuously, so audio from a stretch where the
|
||||
pipeline thread was busy elsewhere is still queued when the next read happens.
|
||||
It bites in exactly one place. At the end of a reply that asked you something,
|
||||
`_speak` sets `_talk_now` and the next turn starts recording immediately — with
|
||||
the tail of the pet's own TTS sitting in that buffer, above the VAD threshold.
|
||||
The VAD takes it for the start of your answer, Deepgram transcribes it, and Bolt
|
||||
is handed his own last sentence as if you had said it. With barge-in on the
|
||||
detector was draining the stream during playback so the window is small; with
|
||||
`BARGE_IN=false` nothing drains it at all.
|
||||
|
||||
`mic.flush(stream)` drops what's buffered, and `_speak` calls it on the
|
||||
follow-up branch only. **That placement is the whole correctness argument** —
|
||||
flushing is only safe where the buffer is known to hold nothing *you* said:
|
||||
playback ran to completion, so if you had spoken, barge-in would have cut it and
|
||||
taken the interrupted branch instead. Never flush before a wake-triggered
|
||||
recording, where the rest of "thunderbolt, what time is it" is legitimately
|
||||
queued and dropping it clips the request. A single call is bounded by
|
||||
`max_seconds` so it can't chase a stream filling as fast as it drains, and it
|
||||
no-ops on a stream with no `read_available` (i.e. every fake stream in tests).
|
||||
|
||||
### Testing conventions
|
||||
|
||||
`tests/` covers pure logic only (state machine, wake-word scoring loop, mic
|
||||
|
||||
Reference in New Issue
Block a user