TheMajesticNetwork/Bolt-Pet
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 3 Wiki Activity

Streaming replies and STT, amplitude lip-sync, one place for speaking

Browse Source 
Latency: replies are spoken sentence-by-sentence off the desk API's NDJSON
endpoint, so the wait is time-to-first-sentence rather than the whole model
call, and Deepgram's live websocket transcribes while you're still talking
instead of uploading the WAV afterwards. Both fall back invisibly — a stream
that fails before anything was said drops to converse(), and a socket that
never opens just means the old one-shot path.

Speaking lived in four near-copies in the controller (a reply, a holding line,
a streamed sentence, a dialogue scene) that had already drifted: one didn't arm
barge-in, another skipped the follow-up rule. It's now speech.Speaker plus an
Utterance describing the policy differences, with collaborators injected so the
whole of it tests without Qt or audio.

The mouth follows the audio rather than a timer: tts.level_of reduces each PCM
frame to a 0..1 loudness on a sqrt curve (speech sits well below peak, and a
linear map leaves the mouth barely open during normal talking) and that indexes
the talking frames, which the sprite script now draws as an openness ramp.
Offline pyttsx3 has no waveform, so stale levels hand control back to the timed
loop instead of freezing the mouth mid-syllable.

Also: the pet starts where you left it (ignoring positions on monitors that are
no longer connected, since restoring those faithfully is how it ends up
somewhere unreachable), and `python -m bolt_pet --doctor` is a preflight that
says what to do about each problem rather than only what's wrong.

tests/test_pipeline_smoke.py breaks the pure-logic rule on purpose. Every unit
test passed all week while notifications sat unspoken for minutes, the pet said
things twice and [laughing] got read aloud — each an interaction between two
individually-correct units. It drives whole turns against a real HTTP server on
a loopback port, faking only the mic and the speakers. It found a NameError in
the paint path that would have fired on every repaint while talking.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Aaron Franks
2026-08-02 19:01:06 -06:00
parent c4e805defd
commit 3a0959f55d
55 changed files with 2796 additions and 151 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bolt_pet/audio/mic.py
+22
View File
@@ -37,6 +37,20 @@ def rms(frame: np.ndarray) -> float:
return float(np.sqrt(np.mean(frame.astype(np.float64) ** 2)))
def _emit(on_frame, frame) -> None:
"""Hand a frame to a listener without letting it break the capture.
Streaming STT is an optimisation riding along with recording; if its
socket dies mid-utterance the recording must carry on untouched, because
the one-shot fallback is about to need the full buffer."""
if on_frame is None:
return
try:
on_frame(frame)
except Exception:
pass
def record_utterance(
stream: AudioStream,
should_continue=lambda: True,
@@ -47,6 +61,7 @@ def record_utterance(
grace_s: float = None,
frame_len: int = config.FRAME_LEN,
sample_rate: int = config.SAMPLE_RATE,
on_frame=None,
) -> Optional[np.ndarray]:
"""Capture one utterance from *stream*: wait for speech to start, stop
after trailing silence. Returns None if nothing usable was heard.
@@ -55,6 +70,11 @@ def record_utterance(
(e.g. the pet window was closed) without needing threading primitives
baked into this function.
*on_frame*, if given, is called with each captured frame while speech is
in progress — that is how streaming STT transcribes as you talk rather
than after (see audio/stt_stream.py). It is fire-and-forget: this function
still returns the full buffer, so a failed stream costs nothing.
*grace_s* is how long to wait for speech to *begin* before giving up.
The controller stretches it for follow-up questions, where you're being
asked something and need a moment to think rather than having just said
@@ -83,10 +103,12 @@ def record_utterance(
if frame_rms >= rms_threshold:
started = True
frames.append(frame)
_emit(on_frame, frame)
elif waited > grace_frames:
return None # woke it up but said nothing
continue
frames.append(frame)
_emit(on_frame, frame)
if frame_rms < rms_threshold:
silence_frames += 1
if silence_frames >= silence_limit:
bolt_pet/audio/stt_stream.py
+181
View File
@@ -0,0 +1,181 @@
"""Streaming speech-to-text — transcribing *while* you talk, not after.
The one-shot path (`stt.transcribe`) waits for the utterance to finish, then
uploads the whole WAV and waits again. That second wait is dead time between
you stopping and the pet reacting, and it grows with the length of what you
said — a thirty-second question costs noticeably more than a five-second one.
Deepgram's live endpoint removes it: frames go up as they are captured, so by
the time the VAD decides you have stopped, the transcript is essentially
already there. Same model, same account, same accuracy — the difference is
purely when the work happens.
Design constraints that shaped this:
- **Failure must be invisible.** No websocket, no network, a mid-utterance
disconnect — all of it falls back to the one-shot path, which still has the
full audio buffered. Streaming is an optimisation, never a dependency, so
`open()` returning None is an ordinary outcome rather than an error.
- **The VAD still decides when you stopped.** Deepgram has its own endpointing
and using it would save more, but it would also move a decision the rest of
the pipeline is built around (barge-in, follow-up listening, the grace
period) into a remote service. Not worth coupling those to the network on
the first pass.
- **The socket is per-utterance.** Holding one open across an idle pet would
bill for silence and drop on the first network blip; opening one takes
~100ms, which is already inside the time it takes a person to start talking.
The websocket client is injectable, so the whole protocol — send frames, read
`is_final` transcripts, close, take the result — is tested without a network.
"""
from __future__ import annotations
import json
import logging
import threading
from typing import Callable, Optional
import numpy as np
from .. import config
logger = logging.getLogger("bolt_pet.stt_stream")
_ENDPOINT = (
"wss://api.deepgram.com/v1/listen"
"?encoding=linear16&channels=1&sample_rate={rate}&model={model}"
"&language=en&smart_format=true&interim_results=false"
)
def available() -> bool:
"""Whether streaming STT can even be attempted in this install."""
if not config.STT_STREAMING or not config.DEEPGRAM_API_KEY:
return False
try:
import websocket # noqa: F401 (websocket-client)
return True
except Exception:
return False
class StreamingTranscriber:
"""One utterance's worth of live transcription.
Usage mirrors how the capture loop already works — feed frames as they
arrive, then ask what was said:
session = StreamingTranscriber.open()
...
session.feed(frame) # per mic frame, non-blocking
text = session.finish() # after the VAD says you stopped
"""
def __init__(self, socket, *, sample_rate: int = None):
self._socket = socket
self._sample_rate = sample_rate or config.SAMPLE_RATE
self._transcript: list[str] = []
self._lock = threading.Lock()
self._closed = False
self._reader = threading.Thread(
target=self._read_loop, name="stt-stream-reader", daemon=True)
self._reader.start()
# -- lifecycle ----------------------------------------------------------
@classmethod
def open(cls, *, connect: Optional[Callable] = None,
sample_rate: int = None) -> Optional["StreamingTranscriber"]:
"""Connect, or return None if streaming isn't possible right now.
None is a normal outcome, not a failure: the caller keeps the audio and
falls back to the one-shot upload."""
if connect is None and not available():
return None
rate = sample_rate or config.SAMPLE_RATE
try:
if connect is not None:
socket = connect()
else:
import websocket
socket = websocket.create_connection(
_ENDPOINT.format(rate=rate, model=config.DEEPGRAM_MODEL),
header={"Authorization": f"Token {config.DEEPGRAM_API_KEY}"},
timeout=10,
)
return cls(socket, sample_rate=rate)
except Exception as exc:
logger.info("Streaming STT unavailable (%s) — using the one-shot path.", exc)
return None
def feed(self, frame: np.ndarray) -> None:
"""Send one captured frame. Never raises — a dead socket just means the
fallback will do the work."""
if self._closed:
return
try:
self._socket.send_binary(np.asarray(frame, dtype=np.int16).tobytes())
except Exception:
logger.debug("Streaming STT send failed; abandoning the stream", exc_info=True)
self._closed = True
def finish(self, timeout: float = 3.0) -> str:
"""Close the stream and return whatever was transcribed.
Deepgram flushes its final results after the close frame, so this waits
briefly for the reader — bounded, because a hung socket must not hold
up the reply."""
if not self._closed:
try:
self._socket.send(json.dumps({"type": "CloseStream"}))
except Exception:
pass
self._closed = True
self._reader.join(timeout=timeout)
try:
self._socket.close()
except Exception:
pass
with self._lock:
return " ".join(part for part in self._transcript if part).strip()
# -- the reader ---------------------------------------------------------
def _read_loop(self) -> None:
while not self._closed:
try:
message = self._socket.recv()
except Exception:
break
if not message:
break
text, is_final = self._parse(message)
if text and is_final:
with self._lock:
self._transcript.append(text)
@staticmethod
def _parse(message) -> tuple[str, bool]:
"""Pull (text, is_final) out of a Deepgram results frame.
Tolerant on purpose: anything unrecognised is ignored rather than
raising on the reader thread, where an exception would silently kill
transcription for the rest of the utterance."""
try:
if isinstance(message, bytes):
message = message.decode("utf-8", "ignore")
data = json.loads(message)
except (TypeError, ValueError):
return "", False
if not isinstance(data, dict):
return "", False
alternatives = (
((data.get("channel") or {}).get("alternatives") or [])
if data.get("type") in (None, "Results") else []
)
if not alternatives:
return "", False
text = str((alternatives[0] or {}).get("transcript") or "").strip()
return text, bool(data.get("is_final") or data.get("speech_final"))
bolt_pet/audio/tts.py
+64 -7
View File
@@ -14,6 +14,7 @@ fallback has no such concept and always sounds like itself.
from __future__ import annotations
import time
from typing import Iterable, Iterator, Optional
import numpy as np
@@ -162,32 +163,81 @@ def synthesize_dialogue(
return pcm, config.TTS_SAMPLE_RATE
def play_pcm(pcm: np.ndarray, sample_rate: int, blocking: bool = True, should_stop=None) -> bool:
# ── how loud is it right now ────────────────────────────────────────────────
# The PCM is already decoded here on its way to the speakers, so the amplitude
# envelope is free — and it is exactly what a mouth needs to move in time with
# speech. Throwing it away and animating the mouth on a timer instead is why
# most talking sprites look dubbed.
# int16 RMS that counts as "mouth fully open". Speech peaks around 8-12k;
# 6000 keeps normal talking in the upper half of the range without clipping
# every syllable to wide-open.
_LOUD_RMS = 6000.0
def level_of(frame: np.ndarray) -> float:
"""0..1 loudness for one chunk of PCM.
Square-rooted because perceived loudness is not linear in amplitude — a
linear mapping leaves the mouth barely moving through ordinary speech."""
if frame is None or len(frame) == 0:
return 0.0
rms = float(np.sqrt(np.mean(np.square(frame.astype(np.float32)))))
return float(min(1.0, (rms / _LOUD_RMS) ** 0.5))
def envelope(pcm: np.ndarray, sample_rate: int, fps: int = 30) -> list:
"""Per-frame loudness for a whole clip, for playback that isn't streamed."""
if pcm is None or len(pcm) == 0:
return []
window = max(1, int(sample_rate / max(1, fps)))
return [level_of(pcm[start:start + window]) for start in range(0, len(pcm), window)]
def play_pcm(pcm: np.ndarray, sample_rate: int, blocking: bool = True, should_stop=None,
on_level=None) -> bool:
"""Play a whole clip. Returns True if it finished, False if *should_stop*
(barge-in) cut it short. *should_stop* is polled while audio plays — each
poll consumes one mic frame, which is what paces this loop."""
import sounddevice as sd
levels = envelope(pcm, sample_rate) if on_level is not None else []
started = time.monotonic()
sd.play(pcm, samplerate=sample_rate, device=config.SPEAKER_DEVICE)
if not blocking:
return True
if should_stop is None:
if should_stop is None and on_level is None:
sd.wait()
return True
while True:
if levels:
# Indexed by elapsed time rather than by chunk, because this path
# hands the whole clip to the device at once and never sees it
# again — wall clock is the only position we have.
index = int((time.monotonic() - started) * 30)
if index < len(levels):
try:
on_level(levels[index])
except Exception:
levels = []
try:
if not sd.get_stream().active:
break
except Exception:
break # stream already torn down — playback is over
if should_stop():
if should_stop is not None and should_stop():
sd.stop()
return False
return True
def play_stream(chunks: Iterable[np.ndarray], sample_rate: int, should_stop=None) -> bool:
"""Play int16 chunks as they arrive. Returns False if interrupted."""
def play_stream(chunks: Iterable[np.ndarray], sample_rate: int, should_stop=None,
on_level=None) -> bool:
"""Play int16 chunks as they arrive. Returns False if interrupted.
*on_level* receives each chunk's loudness (0..1) just before it is written,
which is what drives the mouth: the sprite is animated by the same audio
the speakers are getting, not by a guess about how long a word takes."""
import sounddevice as sd
with sd.OutputStream(
@@ -199,6 +249,11 @@ def play_stream(chunks: Iterable[np.ndarray], sample_rate: int, should_stop=None
# now, not at the end of the buffered chunk.
out.abort()
return False
if on_level is not None:
try:
on_level(level_of(chunk))
except Exception:
on_level = None # a broken listener must not stop playback
out.write(chunk)
return True
@@ -213,7 +268,8 @@ def speak_offline(text: str) -> None:
engine.runAndWait()
def speak(text: str, on_error=None, should_stop=None, voice_id: Optional[str] = None) -> bool:
def speak(text: str, on_error=None, should_stop=None, voice_id: Optional[str] = None,
on_level=None) -> bool:
"""Speak *text*, preferring streaming ElevenLabs, then whole-clip
ElevenLabs, then offline TTS. *on_error*, if given, is called with the
exception when ElevenLabs fails (useful for logging) — a fallback still
@@ -233,13 +289,14 @@ def speak(text: str, on_error=None, should_stop=None, voice_id: Optional[str] =
stream_pcm(text, voice_id=voice_id),
config.TTS_SAMPLE_RATE,
should_stop=should_stop,
on_level=on_level,
)
except TtsError as exc:
if on_error is not None:
on_error(exc)
try:
pcm, sample_rate = synthesize_pcm(text, voice_id=voice_id)
return play_pcm(pcm, sample_rate, should_stop=should_stop)
return play_pcm(pcm, sample_rate, should_stop=should_stop, on_level=on_level)
except TtsError as exc:
if on_error is not None:
on_error(exc)