Update desktop app to android app capabilities.

This commit is contained in:
2026-09-13 16:23:52 -06:00
parent 3a0959f55d
commit 2a2cf38399
13 changed files with 485 additions and 206 deletions
+50 -33
View File
@@ -11,14 +11,15 @@ dependency** on the server repo; it's a standalone HTTP client configured via
its own `.env`.
Pipeline: `mic → openWakeWord ("thunderbolt", on-device) / push-to-talk /
click → record utterance → Deepgram STT → + active-window + screen-layout
context → POST /desk/converse → [server may relay a shell command to run on
this machine, or a `petctl` pseudo-command that moves/emotes the pet, jumps it
to another monitor, reads a screen's text back, or plays a multi-voice scene
instead] → reply (optionally tagged with a voice the server picked for it) →
ElevenLabs streaming TTS (or offline pyttsx3 fallback) → speakers`, with the
pet sprite/speech bubble reflecting state throughout, and playback
interruptible by talking over it (barge-in).
click → record utterance → the server's own /desk/stt (same relay the
Android app uses — no local Deepgram account) → + active-window + screen-
layout context → POST /desk/converse → [server may relay a shell command to
run on this machine, or a `petctl` pseudo-command that moves/emotes the pet,
jumps it to another monitor, reads a screen's text back, or plays a
multi-voice scene instead] → reply (optionally tagged with a voice the
server picked for it) → the server's own /desk/tts, streaming (or offline
pyttsx3 fallback) → speakers`, with the pet sprite/speech bubble reflecting
state throughout, and playback interruptible by talking over it (barge-in).
Because a relayed command's output goes back up the tool-result relay before
the final reply, a `petctl read` mid-turn means Bolt can look at a monitor and
@@ -52,9 +53,12 @@ python scripts/slice_spritesheet.py path/to/sheet.png assets/sprites/idle --cols
```
There is no lint/build step configured beyond pytest. `cp .env.example .env`
and fill in `BOLT_SERVER_URL` / `DESK_API_KEY` (+ `DEEPGRAM_API_KEY`,
`ELEVENLABS_API_KEY`) before running — without server config the controller
logs a missing-config message and exits its thread instead of starting.
and fill in `BOLT_SERVER_URL` / `DESK_API_KEY` before running — without them
the controller logs a missing-config message and exits its thread instead of
starting. That's also all STT and normal-reply TTS need now (both go through
the server); `ELEVENLABS_VOICE_ID` picks the voice, and `ELEVENLABS_API_KEY`
is only for the one feature with no server endpoint — multi-voice
`dialoguectl` scenes, see dialogue.py below.
## Architecture
@@ -107,16 +111,17 @@ logs a missing-config message and exits its thread instead of starting.
off entirely with `RECEIVE_FILES=false`.
- **`audio/`** — `mic.py` (energy-based VAD utterance capture, ported from the
server repo's `bolt_desk.py`), `wake_word.py` (openWakeWord `thunderbolt.onnx`
detection + `NearMissLog` for threshold tuning — see below), `stt.py`
(Deepgram), `tts.py` (ElevenLabs, streaming by default — `stream_pcm()` +
`play_stream()` start playback on the first chunk; `chunks_to_int16()`
carries odd bytes across HTTP chunk boundaries, without which everything
after the first split sample plays as static — falling back to whole-clip
PCM then offline `pyttsx3`; every entry point takes an optional `voice_id`
overriding `ELEVENLABS_VOICE_ID`, and `model_for()` picks the multilingual
model whenever there's an override or non-ASCII text, since the default
`eleven_flash_v2` is English-only and would read either as garbled
phonetic English rather than failing), `barge_in.py` (two detectors behind one
detection + `NearMissLog` for threshold tuning — see below), `stt.py` +
`stt_stream.py` (the server's own `/desk/stt` websocket relay — no local
Deepgram account; see below), `tts.py` (the server's own `/desk/tts`,
streaming by default — `stream_pcm()` + `play_stream()` start playback on
the first chunk; `chunks_to_int16()` carries odd bytes across HTTP chunk
boundaries, without which everything after the first split sample plays as
static — falling back to whole-clip PCM then offline `pyttsx3`; every entry
point takes an optional `voice_id` overriding `ELEVENLABS_VOICE_ID` — which
model to synthesize with is the server's call now, not this client's;
see `synthesize_dialogue()` further down for the one path that's still
ElevenLabs-direct), `barge_in.py` (two detectors behind one
`reset()`/`check()` shape, chosen by `BARGE_IN_MODE` via `make_detector`:
**wake** (default) scores every frame with the same openWakeWord model the
idle listener uses, so only the wake phrase cuts playback; **energy** is the
@@ -290,18 +295,30 @@ logs a missing-config message and exits its thread instead of starting.
- **`hotkey.py`** — global push-to-talk via `pynput`; soft-fails with a logged
reason (Wayland, missing package, macOS permissions) since the wake word is
the primary trigger.
- **`audio/stt_stream.py`** — streaming speech-to-text. The one-shot path waits for
the utterance to end, uploads the whole WAV, then waits again; that second wait is
dead time that grows with how long you spoke. Deepgram's live websocket removes it:
`record_utterance(on_frame=...)` hands each captured frame to a
`StreamingTranscriber`, so by the time the VAD decides you stopped the transcript is
essentially already there. Three deliberate limits: `open()` returning **None is an
ordinary outcome** (no websocket-client, no network, no key) because the full audio is
still buffered and `controller._transcribe` just falls back; the **local VAD still
decides when you stopped** rather than Deepgram's endpointing, since barge-in,
follow-up listening and the grace period are all built on it and coupling them to the
network is not a first-pass change; and the socket is **per-utterance**, because
holding one open across an idle pet bills for silence and dies on the first blip.
- **`audio/stt_stream.py`** / **`audio/stt.py`** — speech-to-text via the
server's `/desk/stt` websocket relay (audio up, Deepgram's JSON messages
down untouched — the same relay the Android app uses; no local Deepgram
account or API key). `stt_stream.py` is the opportunistic optimisation:
the one-shot path waits for the utterance to end, then sends the whole
clip and waits again; that second wait is dead time that grows with how
long you spoke, and streaming removes it — `record_utterance(on_frame=...)`
hands each captured frame to a `StreamingTranscriber` as it's captured, so
by the time the VAD decides you stopped the transcript is essentially
already there. `stt.py`'s `transcribe()` is the *guaranteed* fallback for
when that didn't produce anything: since there's no separate REST endpoint
server-side, it opens the exact same relay via `stt_stream.connect()` and
just feeds the whole buffered utterance in one go — deliberately
*unconditional*, not gated by `STT_STREAMING`/`available()` the way the
opportunistic path is, since there's nothing left to fall back to if that
connection fails. Three more deliberate limits on the streaming half:
`open()` returning **None is an ordinary outcome** (no websocket-client, no
network, streaming turned off) because the full audio is still buffered
and `controller._transcribe` just falls back to `stt.transcribe()`; the
**local VAD still decides when you stopped** rather than Deepgram's
endpointing, since barge-in, follow-up listening and the grace period are
all built on it and coupling them to the network is not a first-pass
change; and the socket is **per-utterance**, because holding one open
across an idle pet bills for silence and dies on the first blip.
Off: `STT_STREAMING=false`.
- **Streamed replies** — `server_client.converse_stream()` reads NDJSON from the desk
API's `/desk/converse_stream` and speaks each sentence as it arrives