Add text-to-dialogue, self-restart capability, and misc updates

This commit is contained in:
2026-07-30 20:50:41 -06:00
parent 5b49670983
commit 96afc351ac
21 changed files with 1819 additions and 59 deletions
+90 -9
View File
@@ -14,7 +14,8 @@ Pipeline: `mic → openWakeWord ("thunderbolt", on-device) / push-to-talk /
click → record utterance → Deepgram STT → + active-window + screen-layout
context → POST /desk/converse → [server may relay a shell command to run on
this machine, or a `petctl` pseudo-command that moves/emotes the pet, jumps it
to another monitor, or reads a screen's text back instead] → reply →
to another monitor, reads a screen's text back, or plays a multi-voice scene
instead] → reply (optionally tagged with a voice the server picked for it) →
ElevenLabs streaming TTS (or offline pyttsx3 fallback) → speakers`, with the
pet sprite/speech bubble reflecting state throughout, and playback
interruptible by talking over it (barge-in).
@@ -79,7 +80,10 @@ logs a missing-config message and exits its thread instead of starting.
`/desk/tool_result` until the server sends a final `reply` (capped at
`_MAX_RELAY_HOPS`). This is the same "full desktop control" trust model as
the server repo's other desk clients — commands only ever originate from
the user's own voice/click requests in their own session. `list_outbox_files`
the user's own voice/click requests in their own session. A final reply is
returned as a `Reply(text, voice_id, voice_name)` rather than a bare string,
because the server can tag it with a voice — see "Voices" below.
`list_outbox_files`
/ `download_outbox_file` hit the same `/desk/files` and `/desk/files/<id>`
endpoints the server's `deliver_files` tool queues onto — see `file_delivery.py`.
- **`file_delivery.py`** — the filesystem half of receiving files the server
@@ -104,7 +108,11 @@ logs a missing-config message and exits its thread instead of starting.
`play_stream()` start playback on the first chunk; `chunks_to_int16()`
carries odd bytes across HTTP chunk boundaries, without which everything
after the first split sample plays as static — falling back to whole-clip
PCM then offline `pyttsx3`), `barge_in.py` (two detectors behind one
PCM then offline `pyttsx3`; every entry point takes an optional `voice_id`
overriding `ELEVENLABS_VOICE_ID`, and `model_for()` picks the multilingual
model whenever there's an override or non-ASCII text, since the default
`eleven_flash_v2` is English-only and would read either as garbled
phonetic English rather than failing), `barge_in.py` (two detectors behind one
`reset()`/`check()` shape, chosen by `BARGE_IN_MODE` via `make_detector`:
**wake** (default) scores every frame with the same openWakeWord model the
idle listener uses, so only the wake phrase cuts playback; **energy** is the
@@ -115,8 +123,8 @@ logs a missing-config message and exits its thread instead of starting.
accepts an injectable stream/model/protocol so tests don't need real audio
hardware or a display.
- **`pet_actions.py`** — `petctl` pseudo-commands (`petctl move top-left`,
`petctl emote wave`, `say`/`wander`/`nap`, plus the screen verbs
`jump`/`monitors`/`read`). The desk API has no "move the pet" payload type
`petctl emote wave`, `say`/`wander`/`nap`, the screen verbs
`jump`/`monitors`/`read`, and `voice reset`). The desk API has no "move the pet" payload type
and this repo can't change the server, so these ride the existing
shell-command relay: `controller._handle_command` parses them and they never
reach `subprocess`; anything else is a real shell command exactly as before.
@@ -125,7 +133,51 @@ logs a missing-config message and exits its thread instead of starting.
module doesn't have, so the spec passes through to `monitors.resolve()`.
Query verbs (`monitors`, `read`) are answered in `_handle_command` rather
than by `pet_actions.describe()`, because their output *is* the point: it
goes back up the tool-result relay for Bolt to use in his reply.
goes back up the tool-result relay for Bolt to use in his reply — as is
`voice reset`, which reports what it dropped since the server can't see
which voice is in use. `voice` only ever resets: picking one is the
server's job (`speak_as`, which it already knows how to use), so a
`petctl voice <name>` attempt is an error pointing back at that marker.
- **`self_restart.py`** — `petctl self_restart`, the pet restarting itself so
Bolt can *see* a code change he just made instead of waiting for a human to
restart it. Three problems shape it, and all three are the interesting part.
(1) The restart can't happen inline: killing the process mid-turn would drop
the HTTP tool relay before the result was posted, leaving the server to wait
out its timeout on a turn that can never finish — so the command only
*arms* it (`controller._arm_self_restart`) and
`controller._maybe_self_restart` fires it after the reply is spoken, the
same "only between turns" rule the updater follows. (2) A broken edit must
not be fatal, so `preflight()` imports the package in a **subprocess**
before arming — this process holds the old modules, so an in-process import
would pass on a file that no longer parses — and a SyntaxError comes back as
the command's output, in the same turn, with the pet still running. (3) The
reason has to outlive the process, so it's written to
`~/.cache/bolt-pet/restart_context.json` (never inside the repo Bolt is
editing) and read on the way back up by `controller._report_self_restart`,
which posts it to the server as an ordinary turn — that's what makes
"restart and check the sprites load" finish as a spoken sentence rather than
a silence. `check_loop_guard` refuses after `SELF_RESTART_MAX` restarts in
`SELF_RESTART_WINDOW_SECONDS`, so an edit-restart-crash cycle stops itself.
Off switch: `SELF_RESTART=false`.
- **`dialogue.py`** — `dialoguectl` pseudo-commands: a multi-voice *scene*
through ElevenLabs' Text to Dialogue endpoint (`audio/tts.
synthesize_dialogue`), checked in `_handle_command` between petctl and
filectl. Same single-line-JSON wire format as filectl and for the same
reason (the server's `command` marker captures only up to the next
newline), and it accepts the ElevenLabs field names (`inputs`/`voice_id`)
as well as its own (`lines`/`voice`) because the model has read that API
and copying its shape is the obvious thing to try. Voices are *named*
(`DIALOGUE_VOICES` maps names to ids) rather than pasted as raw ids, and
`self` resolves to whatever voice the pet is speaking with right now —
including a `speak_as` pick — so Bolt sounds like himself in his own
scenes. The API's limits (10 distinct voices, ~2000 characters) are
enforced *before* the request so a mistake comes back up the tool-result
relay as a sentence Bolt can act on rather than an HTTP 422 he can't see.
Unlike the normal reply path there is no streaming variant, so a scene is
whole-clip: `controller._play_dialogue` plays it with the same bubble,
transcript and barge-in handling a spoken reply gets, and returns to
THINKING afterwards (not IDLE) because the server is still waiting on the
tool result — that leg is why `state.py` allows TALKING -> THINKING.
- **`file_ops.py`** — `filectl` pseudo-commands, checked in `_handle_command`
right after petctl and before falling through to a real shell command.
Executing arbitrary commands already worked via the shell relay
@@ -274,9 +326,38 @@ logs a missing-config message and exits its thread instead of starting.
name rather than by `PetState`, with `has()` reporting whether a key is
backed by real art so callers can decline a placeholder instead of trotting
a blob across the desktop; `tray.py` is the system tray menu (talk now / mute / nap / wander /
click-through / history / wake-word tuning / quit) — the pet window has no
title bar or taskbar entry; `history_window.py` and `wake_tuner.py` are the
two dialogs it opens.
click-through / history / wake-word tuning / use-default-voice / quit) — the
pet window has no title bar or taskbar entry; `history_window.py` and
`wake_tuner.py` are the two dialogs it opens.
### Voices (the server's `speak_as`)
Ask Bolt to talk like someone else, or in another language, and the *server*
does the picking: its desk-only `voice_search` marker browses the ElevenLabs
voice library, and `speak_as: <voice_id>` on the final reply tags that reply
with the chosen voice (adding a Voice Library pick to the ElevenLabs account
first, so the id is usable by the time it reaches us). Nothing about that is
this repo's to decide — all the client owes it is actually speaking in the
voice it was handed: `converse()` returns it on `Reply`, `_apply_voice()`
records it, and `_speak()` passes it to `tts.speak(voice_id=...)`.
Two things are decided *here*, though, because the server can't:
- **The voice sticks** (`VOICE_STICKY`, default on). The server tags one
reply and strips the marker before storing the turn, so it never sees the
id again — "keep talking like that" would send it searching for a voice all
over again, and it'd likely land on a different one. Holding the id
client-side is what makes the rest of the conversation stay in that voice.
An untagged reply therefore never *changes* the voice; only a new
`speak_as`, `VOICE_STICKY=false`, or a reset does.
- **There's a way back.** Since the server was never told Bolt's own voice
id, it can't ask for it back with `speak_as` — so reverting is local: the
tray's **Use default voice** entry (enabled only while a picked voice is
in use, kept in sync by the `voice_changed` signal), a restart, or
`petctl voice reset`, which is what lets Bolt honour "go back to your
normal voice" out loud. That last one needs the server's pet prompt block
(`ai/desk_api.py`, `pet_tools`) to mention the verb, or the model never
emits it — the desk API's prompt is where petctl is advertised.
### Wake-word detection