Update desktop app to android app capabilities.
This commit is contained in:
@@ -3,16 +3,19 @@
|
||||
A little animated pet that lives on your desktop and is just a face on top
|
||||
of your Bolt server — same brain, memory, tools, and persona as Discord
|
||||
chat and the Linux desk client. It talks to `ai/desk_api.py` on the server
|
||||
exactly the way `desk_client/bolt_desk.py` does; this project only adds the
|
||||
on-screen pet and swaps Deepgram/ElevenLabs playback to be cross-platform
|
||||
(no `mpv`/`ffplay`/`espeak-ng` subprocess calls — pure `sounddevice`).
|
||||
exactly the way `desk_client/bolt_desk.py` does — including speech: STT and
|
||||
TTS are the server's own `/desk/stt` and `/desk/tts`, the same endpoints the
|
||||
Android app uses, so there's no separate Deepgram or ElevenLabs account to
|
||||
set up for the pet to talk. Playback is cross-platform (no `mpv`/`ffplay`/
|
||||
`espeak-ng` subprocess calls — pure `sounddevice`).
|
||||
|
||||
```
|
||||
mic → wake-phrase spotter ("thunderbolt") / hotkey / click → record utterance
|
||||
→ Deepgram STT (+ the focused window's title, for "what's this error?")
|
||||
→ POST /desk/converse on your Bolt server → [server may relay a shell
|
||||
command back to run on THIS machine, or a `petctl` command that moves
|
||||
or emotes the pet] → reply → ElevenLabs streaming TTS → speakers
|
||||
→ server-hosted STT (+ the focused window's title, for "what's this
|
||||
error?") → POST /desk/converse on your Bolt server → [server may relay
|
||||
a shell command back to run on THIS machine, or a `petctl` command
|
||||
that moves or emotes the pet] → reply → server-hosted streaming TTS
|
||||
→ speakers
|
||||
→ shown in a speech bubble + the pet's sprite state (idle/listening/
|
||||
thinking/talking) updates the whole time
|
||||
```
|
||||
@@ -36,10 +39,13 @@ all, so it can be copied anywhere and configured with its own `.env`.
|
||||
- `BOLT_SERVER_URL` + `DESK_API_KEY` — same as `desk_client/.env` on the
|
||||
server side. Use the server's master `DESK_API_KEY`, or mint yourself a
|
||||
personal one via the desk-only `api_key_generate` marker (see the main
|
||||
repo's `CLAUDE.md` → "Per-user API keys").
|
||||
- `DEEPGRAM_API_KEY` for STT.
|
||||
- `ELEVENLABS_API_KEY` + `ELEVENLABS_VOICE_ID` for TTS (optional — falls
|
||||
back to offline TTS via `pyttsx3` if omitted or if a request fails).
|
||||
repo's `CLAUDE.md` → "Per-user API keys"). That's it for STT — no
|
||||
Deepgram account needed, it goes through the server's own `/desk/stt`.
|
||||
- `ELEVENLABS_VOICE_ID` for TTS (optional — falls back to offline TTS via
|
||||
`pyttsx3` if omitted or if the server call fails). No local ElevenLabs
|
||||
API key needed for this either; `ELEVENLABS_API_KEY` is only for the
|
||||
multi-voice `dialoguectl` scenes further down, the one feature the
|
||||
server has no endpoint for.
|
||||
3. Run it:
|
||||
- macOS/Linux: `./run.sh`
|
||||
- Windows: `run.bat`
|
||||
@@ -96,9 +102,10 @@ Ask for a different voice — "use a clearer voice", "talk like a pirate", "say
|
||||
that in Japanese" — and Bolt searches the ElevenLabs voice library on the
|
||||
server, picks one, and tags his reply with it (`speak_as`); the pet is what
|
||||
actually speaks in it. A Voice Library pick is added to your ElevenLabs
|
||||
account automatically the first time it's used, and non-English replies (or
|
||||
any picked voice) go through `ELEVENLABS_MULTILINGUAL_MODEL_ID` rather than
|
||||
the English-only `eleven_flash_v2` default.
|
||||
account automatically the first time it's used. The server also synthesizes
|
||||
with one fixed, multilingual-capable model for every request now (not a
|
||||
flash/multilingual switch per reply) — nothing to configure on this side,
|
||||
and non-English text or a picked voice no longer needs special-casing here.
|
||||
|
||||
The new voice **stays on** for the rest of the conversation, because the
|
||||
server tags a single reply and doesn't remember which voice it chose — so
|
||||
@@ -188,9 +195,9 @@ bolt_pet/
|
||||
audio/
|
||||
mic.py input stream + energy-based VAD utterance capture
|
||||
wake_word.py openWakeWord thunderbolt.onnx detection (see above)
|
||||
stt.py Deepgram (one-shot)
|
||||
stt_stream.py Deepgram live websocket — transcribes while you speak
|
||||
tts.py ElevenLabs streaming PCM, offline pyttsx3 fallback,
|
||||
stt.py server /desk/stt relay (one-shot: whole utterance at once)
|
||||
stt_stream.py server /desk/stt live websocket — transcribes while you speak
|
||||
tts.py server /desk/tts streaming PCM, offline pyttsx3 fallback,
|
||||
plus the loudness envelope that drives the mouth
|
||||
barge_in.py "you started talking" detector, to cut playback short
|
||||
ui/
|
||||
|
||||
Reference in New Issue
Block a user