Streaming replies and STT, amplitude lip-sync, one place for speaking
Latency: replies are spoken sentence-by-sentence off the desk API's NDJSON endpoint, so the wait is time-to-first-sentence rather than the whole model call, and Deepgram's live websocket transcribes while you're still talking instead of uploading the WAV afterwards. Both fall back invisibly — a stream that fails before anything was said drops to converse(), and a socket that never opens just means the old one-shot path. Speaking lived in four near-copies in the controller (a reply, a holding line, a streamed sentence, a dialogue scene) that had already drifted: one didn't arm barge-in, another skipped the follow-up rule. It's now speech.Speaker plus an Utterance describing the policy differences, with collaborators injected so the whole of it tests without Qt or audio. The mouth follows the audio rather than a timer: tts.level_of reduces each PCM frame to a 0..1 loudness on a sqrt curve (speech sits well below peak, and a linear map leaves the mouth barely open during normal talking) and that indexes the talking frames, which the sprite script now draws as an openness ramp. Offline pyttsx3 has no waveform, so stale levels hand control back to the timed loop instead of freezing the mouth mid-syllable. Also: the pet starts where you left it (ignoring positions on monitors that are no longer connected, since restoring those faithfully is how it ends up somewhere unreachable), and `python -m bolt_pet --doctor` is a preflight that says what to do about each problem rather than only what's wrong. tests/test_pipeline_smoke.py breaks the pure-logic rule on purpose. Every unit test passed all week while notifications sat unspoken for minutes, the pet said things twice and [laughing] got read aloud — each an interaction between two individually-correct units. It drives whole turns against a real HTTP server on a loopback port, faking only the mic and the speakers. It found a NameError in the paint path that would have fired on every repaint while talking. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -140,6 +140,33 @@ near misses — frames that scored just under the threshold — and the slider
|
||||
takes effect immediately, mid-listen. Set the threshold just below the peak
|
||||
you can hit reliably, then write it into `.env` as `WAKE_WORD_THRESHOLD`.
|
||||
|
||||
## Something not working?
|
||||
|
||||
```bash
|
||||
python -m bolt_pet --doctor
|
||||
```
|
||||
|
||||
Checks the things that make the pet look broken in ways that don't point at
|
||||
themselves — missing server config (the controller exits its thread at startup,
|
||||
so the pet appears alive and simply never answers), no input device, no OCR
|
||||
engine behind `petctl read`, a wake model that isn't where `.env` says, a
|
||||
silence timeout long enough to feel like lag. Each line says what to do about
|
||||
it, not just what's wrong. It doesn't touch the network or open the microphone
|
||||
unless you add `--deep`, so it's safe to run when the network is the suspect.
|
||||
|
||||
## Little things
|
||||
|
||||
The pet **remembers where you left it** — drag it somewhere deliberate and
|
||||
that's where it starts next time. If that position is on a monitor you've since
|
||||
unplugged it goes back to the default corner rather than restoring itself
|
||||
somewhere off-screen. `PET_REMEMBER_POSITION=false` to always start in the
|
||||
corner.
|
||||
|
||||
Its **mouth moves with the actual audio** rather than flapping on a timer: the
|
||||
PCM going to the speakers is reduced to a loudness per frame and that picks the
|
||||
talking sprite, so the pet shuts up when the voice pauses. Offline `pyttsx3`
|
||||
playback has no waveform to follow, so it falls back to the timed loop.
|
||||
|
||||
## Project layout
|
||||
|
||||
```
|
||||
@@ -148,18 +175,23 @@ bolt_pet/
|
||||
state.py PetState enum + a small transition-checked state machine
|
||||
server_client.py /desk/converse, /desk/tool_result, /desk/report_status
|
||||
controller.py the pipeline: wake word -> STT -> server -> TTS, on a QThread
|
||||
speech_text.py strips markdown/emoji/URLs so the voice never says "asterisk"
|
||||
speech.py what the pet says and how each kind of saying behaves
|
||||
speech_text.py strips markdown/emoji/URLs so the voice never says "asterisk"
|
||||
pet_actions.py petctl move/emote/say/wander/nap parsing
|
||||
screen_context.py active-window title + fullscreen detection
|
||||
quiet.py quiet-hours schedule
|
||||
notifications.py desktop notification bridge (Linux/D-Bus)
|
||||
history.py rolling conversation transcript
|
||||
hotkey.py global push-to-talk (pynput, optional)
|
||||
window_state.py remembers where you left the pet
|
||||
doctor.py `--doctor` preflight: is this install going to work?
|
||||
audio/
|
||||
mic.py input stream + energy-based VAD utterance capture
|
||||
wake_word.py openWakeWord thunderbolt.onnx detection (see above)
|
||||
stt.py Deepgram
|
||||
tts.py ElevenLabs streaming PCM, offline pyttsx3 fallback
|
||||
stt.py Deepgram (one-shot)
|
||||
stt_stream.py Deepgram live websocket — transcribes while you speak
|
||||
tts.py ElevenLabs streaming PCM, offline pyttsx3 fallback,
|
||||
plus the loudness envelope that drives the mouth
|
||||
barge_in.py "you started talking" detector, to cut playback short
|
||||
ui/
|
||||
app.py wires QApplication + window + tray + controller thread together
|
||||
@@ -172,8 +204,10 @@ bolt_pet/
|
||||
scripts/
|
||||
slice_spritesheet.py cuts a grid sprite sheet into the per-frame convention
|
||||
tests/ pure-logic unit tests (state machine, wake-phrase
|
||||
matching, HTTP client against mocks) — nothing here
|
||||
needs real audio hardware or a display
|
||||
matching, HTTP client against mocks) plus one
|
||||
end-to-end smoke test that drives whole turns against
|
||||
a real local HTTP server — nothing here needs real
|
||||
audio hardware or a display
|
||||
```
|
||||
|
||||
## Security notes
|
||||
@@ -197,7 +231,6 @@ notifications. No screenshots or images are ever sent.
|
||||
|
||||
## Known limitations / not-yet-done
|
||||
|
||||
- Pet screen position isn't persisted across restarts.
|
||||
- Wandering is a straight walk to a random point — no Shimeji-style physics,
|
||||
wall-climbing or falling.
|
||||
- Push-to-talk and the notification bridge are platform-limited: the hotkey
|
||||
|
||||
Reference in New Issue
Block a user