Add text-to-dialogue, self-restart capability, and misc updates
This commit is contained in:
@@ -14,7 +14,8 @@ Pipeline: `mic → openWakeWord ("thunderbolt", on-device) / push-to-talk /
|
||||
click → record utterance → Deepgram STT → + active-window + screen-layout
|
||||
context → POST /desk/converse → [server may relay a shell command to run on
|
||||
this machine, or a `petctl` pseudo-command that moves/emotes the pet, jumps it
|
||||
to another monitor, or reads a screen's text back instead] → reply →
|
||||
to another monitor, reads a screen's text back, or plays a multi-voice scene
|
||||
instead] → reply (optionally tagged with a voice the server picked for it) →
|
||||
ElevenLabs streaming TTS (or offline pyttsx3 fallback) → speakers`, with the
|
||||
pet sprite/speech bubble reflecting state throughout, and playback
|
||||
interruptible by talking over it (barge-in).
|
||||
@@ -79,7 +80,10 @@ logs a missing-config message and exits its thread instead of starting.
|
||||
`/desk/tool_result` until the server sends a final `reply` (capped at
|
||||
`_MAX_RELAY_HOPS`). This is the same "full desktop control" trust model as
|
||||
the server repo's other desk clients — commands only ever originate from
|
||||
the user's own voice/click requests in their own session. `list_outbox_files`
|
||||
the user's own voice/click requests in their own session. A final reply is
|
||||
returned as a `Reply(text, voice_id, voice_name)` rather than a bare string,
|
||||
because the server can tag it with a voice — see "Voices" below.
|
||||
`list_outbox_files`
|
||||
/ `download_outbox_file` hit the same `/desk/files` and `/desk/files/<id>`
|
||||
endpoints the server's `deliver_files` tool queues onto — see `file_delivery.py`.
|
||||
- **`file_delivery.py`** — the filesystem half of receiving files the server
|
||||
@@ -104,7 +108,11 @@ logs a missing-config message and exits its thread instead of starting.
|
||||
`play_stream()` start playback on the first chunk; `chunks_to_int16()`
|
||||
carries odd bytes across HTTP chunk boundaries, without which everything
|
||||
after the first split sample plays as static — falling back to whole-clip
|
||||
PCM then offline `pyttsx3`), `barge_in.py` (two detectors behind one
|
||||
PCM then offline `pyttsx3`; every entry point takes an optional `voice_id`
|
||||
overriding `ELEVENLABS_VOICE_ID`, and `model_for()` picks the multilingual
|
||||
model whenever there's an override or non-ASCII text, since the default
|
||||
`eleven_flash_v2` is English-only and would read either as garbled
|
||||
phonetic English rather than failing), `barge_in.py` (two detectors behind one
|
||||
`reset()`/`check()` shape, chosen by `BARGE_IN_MODE` via `make_detector`:
|
||||
**wake** (default) scores every frame with the same openWakeWord model the
|
||||
idle listener uses, so only the wake phrase cuts playback; **energy** is the
|
||||
@@ -115,8 +123,8 @@ logs a missing-config message and exits its thread instead of starting.
|
||||
accepts an injectable stream/model/protocol so tests don't need real audio
|
||||
hardware or a display.
|
||||
- **`pet_actions.py`** — `petctl` pseudo-commands (`petctl move top-left`,
|
||||
`petctl emote wave`, `say`/`wander`/`nap`, plus the screen verbs
|
||||
`jump`/`monitors`/`read`). The desk API has no "move the pet" payload type
|
||||
`petctl emote wave`, `say`/`wander`/`nap`, the screen verbs
|
||||
`jump`/`monitors`/`read`, and `voice reset`). The desk API has no "move the pet" payload type
|
||||
and this repo can't change the server, so these ride the existing
|
||||
shell-command relay: `controller._handle_command` parses them and they never
|
||||
reach `subprocess`; anything else is a real shell command exactly as before.
|
||||
@@ -125,7 +133,51 @@ logs a missing-config message and exits its thread instead of starting.
|
||||
module doesn't have, so the spec passes through to `monitors.resolve()`.
|
||||
Query verbs (`monitors`, `read`) are answered in `_handle_command` rather
|
||||
than by `pet_actions.describe()`, because their output *is* the point: it
|
||||
goes back up the tool-result relay for Bolt to use in his reply.
|
||||
goes back up the tool-result relay for Bolt to use in his reply — as is
|
||||
`voice reset`, which reports what it dropped since the server can't see
|
||||
which voice is in use. `voice` only ever resets: picking one is the
|
||||
server's job (`speak_as`, which it already knows how to use), so a
|
||||
`petctl voice <name>` attempt is an error pointing back at that marker.
|
||||
- **`self_restart.py`** — `petctl self_restart`, the pet restarting itself so
|
||||
Bolt can *see* a code change he just made instead of waiting for a human to
|
||||
restart it. Three problems shape it, and all three are the interesting part.
|
||||
(1) The restart can't happen inline: killing the process mid-turn would drop
|
||||
the HTTP tool relay before the result was posted, leaving the server to wait
|
||||
out its timeout on a turn that can never finish — so the command only
|
||||
*arms* it (`controller._arm_self_restart`) and
|
||||
`controller._maybe_self_restart` fires it after the reply is spoken, the
|
||||
same "only between turns" rule the updater follows. (2) A broken edit must
|
||||
not be fatal, so `preflight()` imports the package in a **subprocess**
|
||||
before arming — this process holds the old modules, so an in-process import
|
||||
would pass on a file that no longer parses — and a SyntaxError comes back as
|
||||
the command's output, in the same turn, with the pet still running. (3) The
|
||||
reason has to outlive the process, so it's written to
|
||||
`~/.cache/bolt-pet/restart_context.json` (never inside the repo Bolt is
|
||||
editing) and read on the way back up by `controller._report_self_restart`,
|
||||
which posts it to the server as an ordinary turn — that's what makes
|
||||
"restart and check the sprites load" finish as a spoken sentence rather than
|
||||
a silence. `check_loop_guard` refuses after `SELF_RESTART_MAX` restarts in
|
||||
`SELF_RESTART_WINDOW_SECONDS`, so an edit-restart-crash cycle stops itself.
|
||||
Off switch: `SELF_RESTART=false`.
|
||||
- **`dialogue.py`** — `dialoguectl` pseudo-commands: a multi-voice *scene*
|
||||
through ElevenLabs' Text to Dialogue endpoint (`audio/tts.
|
||||
synthesize_dialogue`), checked in `_handle_command` between petctl and
|
||||
filectl. Same single-line-JSON wire format as filectl and for the same
|
||||
reason (the server's `command` marker captures only up to the next
|
||||
newline), and it accepts the ElevenLabs field names (`inputs`/`voice_id`)
|
||||
as well as its own (`lines`/`voice`) because the model has read that API
|
||||
and copying its shape is the obvious thing to try. Voices are *named*
|
||||
(`DIALOGUE_VOICES` maps names to ids) rather than pasted as raw ids, and
|
||||
`self` resolves to whatever voice the pet is speaking with right now —
|
||||
including a `speak_as` pick — so Bolt sounds like himself in his own
|
||||
scenes. The API's limits (10 distinct voices, ~2000 characters) are
|
||||
enforced *before* the request so a mistake comes back up the tool-result
|
||||
relay as a sentence Bolt can act on rather than an HTTP 422 he can't see.
|
||||
Unlike the normal reply path there is no streaming variant, so a scene is
|
||||
whole-clip: `controller._play_dialogue` plays it with the same bubble,
|
||||
transcript and barge-in handling a spoken reply gets, and returns to
|
||||
THINKING afterwards (not IDLE) because the server is still waiting on the
|
||||
tool result — that leg is why `state.py` allows TALKING -> THINKING.
|
||||
- **`file_ops.py`** — `filectl` pseudo-commands, checked in `_handle_command`
|
||||
right after petctl and before falling through to a real shell command.
|
||||
Executing arbitrary commands already worked via the shell relay
|
||||
@@ -274,9 +326,38 @@ logs a missing-config message and exits its thread instead of starting.
|
||||
name rather than by `PetState`, with `has()` reporting whether a key is
|
||||
backed by real art so callers can decline a placeholder instead of trotting
|
||||
a blob across the desktop; `tray.py` is the system tray menu (talk now / mute / nap / wander /
|
||||
click-through / history / wake-word tuning / quit) — the pet window has no
|
||||
title bar or taskbar entry; `history_window.py` and `wake_tuner.py` are the
|
||||
two dialogs it opens.
|
||||
click-through / history / wake-word tuning / use-default-voice / quit) — the
|
||||
pet window has no title bar or taskbar entry; `history_window.py` and
|
||||
`wake_tuner.py` are the two dialogs it opens.
|
||||
|
||||
### Voices (the server's `speak_as`)
|
||||
|
||||
Ask Bolt to talk like someone else, or in another language, and the *server*
|
||||
does the picking: its desk-only `voice_search` marker browses the ElevenLabs
|
||||
voice library, and `speak_as: <voice_id>` on the final reply tags that reply
|
||||
with the chosen voice (adding a Voice Library pick to the ElevenLabs account
|
||||
first, so the id is usable by the time it reaches us). Nothing about that is
|
||||
this repo's to decide — all the client owes it is actually speaking in the
|
||||
voice it was handed: `converse()` returns it on `Reply`, `_apply_voice()`
|
||||
records it, and `_speak()` passes it to `tts.speak(voice_id=...)`.
|
||||
|
||||
Two things are decided *here*, though, because the server can't:
|
||||
|
||||
- **The voice sticks** (`VOICE_STICKY`, default on). The server tags one
|
||||
reply and strips the marker before storing the turn, so it never sees the
|
||||
id again — "keep talking like that" would send it searching for a voice all
|
||||
over again, and it'd likely land on a different one. Holding the id
|
||||
client-side is what makes the rest of the conversation stay in that voice.
|
||||
An untagged reply therefore never *changes* the voice; only a new
|
||||
`speak_as`, `VOICE_STICKY=false`, or a reset does.
|
||||
- **There's a way back.** Since the server was never told Bolt's own voice
|
||||
id, it can't ask for it back with `speak_as` — so reverting is local: the
|
||||
tray's **Use default voice** entry (enabled only while a picked voice is
|
||||
in use, kept in sync by the `voice_changed` signal), a restart, or
|
||||
`petctl voice reset`, which is what lets Bolt honour "go back to your
|
||||
normal voice" out loud. That last one needs the server's pet prompt block
|
||||
(`ai/desk_api.py`, `pet_tools`) to mention the verb, or the model never
|
||||
emits it — the desk API's prompt is where petctl is advertised.
|
||||
|
||||
### Wake-word detection
|
||||
|
||||
|
||||
Reference in New Issue
Block a user