Streaming replies and STT, amplitude lip-sync, one place for speaking
Latency: replies are spoken sentence-by-sentence off the desk API's NDJSON endpoint, so the wait is time-to-first-sentence rather than the whole model call, and Deepgram's live websocket transcribes while you're still talking instead of uploading the WAV afterwards. Both fall back invisibly — a stream that fails before anything was said drops to converse(), and a socket that never opens just means the old one-shot path. Speaking lived in four near-copies in the controller (a reply, a holding line, a streamed sentence, a dialogue scene) that had already drifted: one didn't arm barge-in, another skipped the follow-up rule. It's now speech.Speaker plus an Utterance describing the policy differences, with collaborators injected so the whole of it tests without Qt or audio. The mouth follows the audio rather than a timer: tts.level_of reduces each PCM frame to a 0..1 loudness on a sqrt curve (speech sits well below peak, and a linear map leaves the mouth barely open during normal talking) and that indexes the talking frames, which the sprite script now draws as an openness ramp. Offline pyttsx3 has no waveform, so stale levels hand control back to the timed loop instead of freezing the mouth mid-syllable. Also: the pet starts where you left it (ignoring positions on monitors that are no longer connected, since restoring those faithfully is how it ends up somewhere unreachable), and `python -m bolt_pet --doctor` is a preflight that says what to do about each problem rather than only what's wrong. tests/test_pipeline_smoke.py breaks the pure-logic rule on purpose. Every unit test passed all week while notifications sat unspoken for minutes, the pet said things twice and [laughing] got read aloud — each an interaction between two individually-correct units. It drives whole turns against a real HTTP server on a loopback port, faking only the mic and the speakers. It found a NameError in the paint path that would have fired on every repaint while talking. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -58,6 +58,11 @@ WAKE_CHECK_INTERVAL_SECONDS = float(os.environ.get("WAKE_CHECK_INTERVAL_SECONDS"
|
||||
|
||||
DEEPGRAM_API_KEY = os.environ.get("DEEPGRAM_API_KEY", "")
|
||||
DEEPGRAM_MODEL = os.environ.get("DEEPGRAM_MODEL", "nova-3")
|
||||
# Transcribe *while* you talk instead of uploading the finished clip: frames go
|
||||
# up as they are captured, so the transcript is ready the moment the VAD says
|
||||
# you stopped. Needs `websocket-client`; falls back to the one-shot upload
|
||||
# whenever it can't connect, so turning it on can only help.
|
||||
STT_STREAMING = os.environ.get("STT_STREAMING", "true").lower() in ("1", "true", "yes", "on")
|
||||
|
||||
# ── TTS (ElevenLabs, requested as raw PCM so playback needs no external
|
||||
# player binary — cross-platform via sounddevice instead of shelling out to
|
||||
@@ -168,6 +173,13 @@ BARGE_IN_FRAMES = int(os.environ.get("BARGE_IN_FRAMES", "4")) # consecutive lou
|
||||
# waking the pet from idle (useful if Bolt's own voice trips the model).
|
||||
BARGE_IN_WAKE_THRESHOLD = float(os.environ.get("BARGE_IN_WAKE_THRESHOLD") or 0) or None
|
||||
|
||||
# ── streaming the reply ─────────────────────────────────────────────────────
|
||||
# Speak each sentence as the server produces it, instead of waiting out the
|
||||
# whole model call before the first word. Falls back to the ordinary
|
||||
# request/response path automatically if the server has no streaming endpoint
|
||||
# or the stream fails before anything has been spoken.
|
||||
STREAMING_REPLIES = os.environ.get("STREAMING_REPLIES", "true").lower() in ("1", "true", "yes", "on")
|
||||
|
||||
# ── streaming TTS ───────────────────────────────────────────────────────────
|
||||
# ElevenLabs' /stream endpoint + chunked playback: the pet starts talking
|
||||
# after the first PCM chunk instead of after the whole clip is synthesized.
|
||||
@@ -307,6 +319,10 @@ PET_WANDER_MARGIN = int(os.environ.get("PET_WANDER_MARGIN", "20")) # keep off s
|
||||
PET_SHAPED_INPUT = os.environ.get("PET_SHAPED_INPUT", "true").lower() in ("1", "true", "yes", "on")
|
||||
PET_CLICK_THROUGH = os.environ.get("PET_CLICK_THROUGH", "false").lower() in ("1", "true", "yes", "on")
|
||||
# Snap flush to a screen edge when dropped/parked within this many pixels of it.
|
||||
# Start where it was left rather than in the bottom-right corner. Ignored when
|
||||
# PET_START_X/Y pin it explicitly, and a saved position on a monitor that is no
|
||||
# longer plugged in is discarded rather than hiding the pet offscreen.
|
||||
PET_REMEMBER_POSITION = os.environ.get("PET_REMEMBER_POSITION", "true").lower() in ("1", "true", "yes", "on")
|
||||
PET_EDGE_SNAP = os.environ.get("PET_EDGE_SNAP", "true").lower() in ("1", "true", "yes", "on")
|
||||
PET_SNAP_MARGIN = int(os.environ.get("PET_SNAP_MARGIN", "48"))
|
||||
|
||||
|
||||
Reference in New Issue
Block a user