Update desktop app to android app capabilities.

This commit is contained in:
2026-09-13 16:23:52 -06:00
parent 3a0959f55d
commit 2a2cf38399
13 changed files with 485 additions and 206 deletions
+24 -17
View File
@@ -3,16 +3,19 @@
A little animated pet that lives on your desktop and is just a face on top
of your Bolt server — same brain, memory, tools, and persona as Discord
chat and the Linux desk client. It talks to `ai/desk_api.py` on the server
exactly the way `desk_client/bolt_desk.py` does; this project only adds the
on-screen pet and swaps Deepgram/ElevenLabs playback to be cross-platform
(no `mpv`/`ffplay`/`espeak-ng` subprocess calls — pure `sounddevice`).
exactly the way `desk_client/bolt_desk.py` does — including speech: STT and
TTS are the server's own `/desk/stt` and `/desk/tts`, the same endpoints the
Android app uses, so there's no separate Deepgram or ElevenLabs account to
set up for the pet to talk. Playback is cross-platform (no `mpv`/`ffplay`/
`espeak-ng` subprocess calls — pure `sounddevice`).
```
mic → wake-phrase spotter ("thunderbolt") / hotkey / click → record utterance
Deepgram STT (+ the focused window's title, for "what's this error?")
→ POST /desk/converse on your Bolt server → [server may relay a shell
command back to run on THIS machine, or a `petctl` command that moves
or emotes the pet] → reply → ElevenLabs streaming TTS → speakers
server-hosted STT (+ the focused window's title, for "what's this
error?") → POST /desk/converse on your Bolt server → [server may relay
a shell command back to run on THIS machine, or a `petctl` command
that moves or emotes the pet] → reply → server-hosted streaming TTS
→ speakers
→ shown in a speech bubble + the pet's sprite state (idle/listening/
thinking/talking) updates the whole time
```
@@ -36,10 +39,13 @@ all, so it can be copied anywhere and configured with its own `.env`.
- `BOLT_SERVER_URL` + `DESK_API_KEY` — same as `desk_client/.env` on the
server side. Use the server's master `DESK_API_KEY`, or mint yourself a
personal one via the desk-only `api_key_generate` marker (see the main
repo's `CLAUDE.md` → "Per-user API keys").
- `DEEPGRAM_API_KEY` for STT.
- `ELEVENLABS_API_KEY` + `ELEVENLABS_VOICE_ID` for TTS (optional — falls
back to offline TTS via `pyttsx3` if omitted or if a request fails).
repo's `CLAUDE.md` → "Per-user API keys"). That's it for STT — no
Deepgram account needed, it goes through the server's own `/desk/stt`.
- `ELEVENLABS_VOICE_ID` for TTS (optional — falls back to offline TTS via
`pyttsx3` if omitted or if the server call fails). No local ElevenLabs
API key needed for this either; `ELEVENLABS_API_KEY` is only for the
multi-voice `dialoguectl` scenes further down, the one feature the
server has no endpoint for.
3. Run it:
- macOS/Linux: `./run.sh`
- Windows: `run.bat`
@@ -96,9 +102,10 @@ Ask for a different voice — "use a clearer voice", "talk like a pirate", "say
that in Japanese" — and Bolt searches the ElevenLabs voice library on the
server, picks one, and tags his reply with it (`speak_as`); the pet is what
actually speaks in it. A Voice Library pick is added to your ElevenLabs
account automatically the first time it's used, and non-English replies (or
any picked voice) go through `ELEVENLABS_MULTILINGUAL_MODEL_ID` rather than
the English-only `eleven_flash_v2` default.
account automatically the first time it's used. The server also synthesizes
with one fixed, multilingual-capable model for every request now (not a
flash/multilingual switch per reply) — nothing to configure on this side,
and non-English text or a picked voice no longer needs special-casing here.
The new voice **stays on** for the rest of the conversation, because the
server tags a single reply and doesn't remember which voice it chose — so
@@ -188,9 +195,9 @@ bolt_pet/
audio/
mic.py input stream + energy-based VAD utterance capture
wake_word.py openWakeWord thunderbolt.onnx detection (see above)
stt.py Deepgram (one-shot)
stt_stream.py Deepgram live websocket — transcribes while you speak
tts.py ElevenLabs streaming PCM, offline pyttsx3 fallback,
stt.py server /desk/stt relay (one-shot: whole utterance at once)
stt_stream.py server /desk/stt live websocket — transcribes while you speak
tts.py server /desk/tts streaming PCM, offline pyttsx3 fallback,
plus the loudness envelope that drives the mouth
barge_in.py "you started talking" detector, to cut playback short
ui/