Update desktop app to android app capabilities.
This commit is contained in:
@@ -11,14 +11,15 @@ dependency** on the server repo; it's a standalone HTTP client configured via
|
||||
its own `.env`.
|
||||
|
||||
Pipeline: `mic → openWakeWord ("thunderbolt", on-device) / push-to-talk /
|
||||
click → record utterance → Deepgram STT → + active-window + screen-layout
|
||||
context → POST /desk/converse → [server may relay a shell command to run on
|
||||
this machine, or a `petctl` pseudo-command that moves/emotes the pet, jumps it
|
||||
to another monitor, reads a screen's text back, or plays a multi-voice scene
|
||||
instead] → reply (optionally tagged with a voice the server picked for it) →
|
||||
ElevenLabs streaming TTS (or offline pyttsx3 fallback) → speakers`, with the
|
||||
pet sprite/speech bubble reflecting state throughout, and playback
|
||||
interruptible by talking over it (barge-in).
|
||||
click → record utterance → the server's own /desk/stt (same relay the
|
||||
Android app uses — no local Deepgram account) → + active-window + screen-
|
||||
layout context → POST /desk/converse → [server may relay a shell command to
|
||||
run on this machine, or a `petctl` pseudo-command that moves/emotes the pet,
|
||||
jumps it to another monitor, reads a screen's text back, or plays a
|
||||
multi-voice scene instead] → reply (optionally tagged with a voice the
|
||||
server picked for it) → the server's own /desk/tts, streaming (or offline
|
||||
pyttsx3 fallback) → speakers`, with the pet sprite/speech bubble reflecting
|
||||
state throughout, and playback interruptible by talking over it (barge-in).
|
||||
|
||||
Because a relayed command's output goes back up the tool-result relay before
|
||||
the final reply, a `petctl read` mid-turn means Bolt can look at a monitor and
|
||||
@@ -52,9 +53,12 @@ python scripts/slice_spritesheet.py path/to/sheet.png assets/sprites/idle --cols
|
||||
```
|
||||
|
||||
There is no lint/build step configured beyond pytest. `cp .env.example .env`
|
||||
and fill in `BOLT_SERVER_URL` / `DESK_API_KEY` (+ `DEEPGRAM_API_KEY`,
|
||||
`ELEVENLABS_API_KEY`) before running — without server config the controller
|
||||
logs a missing-config message and exits its thread instead of starting.
|
||||
and fill in `BOLT_SERVER_URL` / `DESK_API_KEY` before running — without them
|
||||
the controller logs a missing-config message and exits its thread instead of
|
||||
starting. That's also all STT and normal-reply TTS need now (both go through
|
||||
the server); `ELEVENLABS_VOICE_ID` picks the voice, and `ELEVENLABS_API_KEY`
|
||||
is only for the one feature with no server endpoint — multi-voice
|
||||
`dialoguectl` scenes, see dialogue.py below.
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -107,16 +111,17 @@ logs a missing-config message and exits its thread instead of starting.
|
||||
off entirely with `RECEIVE_FILES=false`.
|
||||
- **`audio/`** — `mic.py` (energy-based VAD utterance capture, ported from the
|
||||
server repo's `bolt_desk.py`), `wake_word.py` (openWakeWord `thunderbolt.onnx`
|
||||
detection + `NearMissLog` for threshold tuning — see below), `stt.py`
|
||||
(Deepgram), `tts.py` (ElevenLabs, streaming by default — `stream_pcm()` +
|
||||
`play_stream()` start playback on the first chunk; `chunks_to_int16()`
|
||||
carries odd bytes across HTTP chunk boundaries, without which everything
|
||||
after the first split sample plays as static — falling back to whole-clip
|
||||
PCM then offline `pyttsx3`; every entry point takes an optional `voice_id`
|
||||
overriding `ELEVENLABS_VOICE_ID`, and `model_for()` picks the multilingual
|
||||
model whenever there's an override or non-ASCII text, since the default
|
||||
`eleven_flash_v2` is English-only and would read either as garbled
|
||||
phonetic English rather than failing), `barge_in.py` (two detectors behind one
|
||||
detection + `NearMissLog` for threshold tuning — see below), `stt.py` +
|
||||
`stt_stream.py` (the server's own `/desk/stt` websocket relay — no local
|
||||
Deepgram account; see below), `tts.py` (the server's own `/desk/tts`,
|
||||
streaming by default — `stream_pcm()` + `play_stream()` start playback on
|
||||
the first chunk; `chunks_to_int16()` carries odd bytes across HTTP chunk
|
||||
boundaries, without which everything after the first split sample plays as
|
||||
static — falling back to whole-clip PCM then offline `pyttsx3`; every entry
|
||||
point takes an optional `voice_id` overriding `ELEVENLABS_VOICE_ID` — which
|
||||
model to synthesize with is the server's call now, not this client's;
|
||||
see `synthesize_dialogue()` further down for the one path that's still
|
||||
ElevenLabs-direct), `barge_in.py` (two detectors behind one
|
||||
`reset()`/`check()` shape, chosen by `BARGE_IN_MODE` via `make_detector`:
|
||||
**wake** (default) scores every frame with the same openWakeWord model the
|
||||
idle listener uses, so only the wake phrase cuts playback; **energy** is the
|
||||
@@ -290,18 +295,30 @@ logs a missing-config message and exits its thread instead of starting.
|
||||
- **`hotkey.py`** — global push-to-talk via `pynput`; soft-fails with a logged
|
||||
reason (Wayland, missing package, macOS permissions) since the wake word is
|
||||
the primary trigger.
|
||||
- **`audio/stt_stream.py`** — streaming speech-to-text. The one-shot path waits for
|
||||
the utterance to end, uploads the whole WAV, then waits again; that second wait is
|
||||
dead time that grows with how long you spoke. Deepgram's live websocket removes it:
|
||||
`record_utterance(on_frame=...)` hands each captured frame to a
|
||||
`StreamingTranscriber`, so by the time the VAD decides you stopped the transcript is
|
||||
essentially already there. Three deliberate limits: `open()` returning **None is an
|
||||
ordinary outcome** (no websocket-client, no network, no key) because the full audio is
|
||||
still buffered and `controller._transcribe` just falls back; the **local VAD still
|
||||
decides when you stopped** rather than Deepgram's endpointing, since barge-in,
|
||||
follow-up listening and the grace period are all built on it and coupling them to the
|
||||
network is not a first-pass change; and the socket is **per-utterance**, because
|
||||
holding one open across an idle pet bills for silence and dies on the first blip.
|
||||
- **`audio/stt_stream.py`** / **`audio/stt.py`** — speech-to-text via the
|
||||
server's `/desk/stt` websocket relay (audio up, Deepgram's JSON messages
|
||||
down untouched — the same relay the Android app uses; no local Deepgram
|
||||
account or API key). `stt_stream.py` is the opportunistic optimisation:
|
||||
the one-shot path waits for the utterance to end, then sends the whole
|
||||
clip and waits again; that second wait is dead time that grows with how
|
||||
long you spoke, and streaming removes it — `record_utterance(on_frame=...)`
|
||||
hands each captured frame to a `StreamingTranscriber` as it's captured, so
|
||||
by the time the VAD decides you stopped the transcript is essentially
|
||||
already there. `stt.py`'s `transcribe()` is the *guaranteed* fallback for
|
||||
when that didn't produce anything: since there's no separate REST endpoint
|
||||
server-side, it opens the exact same relay via `stt_stream.connect()` and
|
||||
just feeds the whole buffered utterance in one go — deliberately
|
||||
*unconditional*, not gated by `STT_STREAMING`/`available()` the way the
|
||||
opportunistic path is, since there's nothing left to fall back to if that
|
||||
connection fails. Three more deliberate limits on the streaming half:
|
||||
`open()` returning **None is an ordinary outcome** (no websocket-client, no
|
||||
network, streaming turned off) because the full audio is still buffered
|
||||
and `controller._transcribe` just falls back to `stt.transcribe()`; the
|
||||
**local VAD still decides when you stopped** rather than Deepgram's
|
||||
endpointing, since barge-in, follow-up listening and the grace period are
|
||||
all built on it and coupling them to the network is not a first-pass
|
||||
change; and the socket is **per-utterance**, because holding one open
|
||||
across an idle pet bills for silence and dies on the first blip.
|
||||
Off: `STT_STREAMING=false`.
|
||||
- **Streamed replies** — `server_client.converse_stream()` reads NDJSON from the desk
|
||||
API's `/desk/converse_stream` and speaks each sentence as it arrives
|
||||
|
||||
Reference in New Issue
Block a user