Streaming replies and STT, amplitude lip-sync, one place for speaking

Latency: replies are spoken sentence-by-sentence off the desk API's NDJSON
endpoint, so the wait is time-to-first-sentence rather than the whole model
call, and Deepgram's live websocket transcribes while you're still talking
instead of uploading the WAV afterwards. Both fall back invisibly — a stream
that fails before anything was said drops to converse(), and a socket that
never opens just means the old one-shot path.

Speaking lived in four near-copies in the controller (a reply, a holding line,
a streamed sentence, a dialogue scene) that had already drifted: one didn't arm
barge-in, another skipped the follow-up rule. It's now speech.Speaker plus an
Utterance describing the policy differences, with collaborators injected so the
whole of it tests without Qt or audio.

The mouth follows the audio rather than a timer: tts.level_of reduces each PCM
frame to a 0..1 loudness on a sqrt curve (speech sits well below peak, and a
linear map leaves the mouth barely open during normal talking) and that indexes
the talking frames, which the sprite script now draws as an openness ramp.
Offline pyttsx3 has no waveform, so stale levels hand control back to the timed
loop instead of freezing the mouth mid-syllable.

Also: the pet starts where you left it (ignoring positions on monitors that are
no longer connected, since restoring those faithfully is how it ends up
somewhere unreachable), and `python -m bolt_pet --doctor` is a preflight that
says what to do about each problem rather than only what's wrong.

tests/test_pipeline_smoke.py breaks the pure-logic rule on purpose. Every unit
test passed all week while notifications sat unspoken for minutes, the pet said
things twice and [laughing] got read aloud — each an interaction between two
individually-correct units. It drives whole turns against a real HTTP server on
a loopback port, faking only the mic and the speakers. It found a NameError in
the paint path that would have fired on every repaint while talking.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

bolt_pet/audio/tts.py

@@ -14,6 +14,7 @@ fallback has no such concept and always sounds like itself.
 
 from __future__ import annotations
 
+import time
 from typing import Iterable, Iterator, Optional
 
 import numpy as np
@@ -162,32 +163,81 @@ def synthesize_dialogue(
     return pcm, config.TTS_SAMPLE_RATE
 
 
-def play_pcm(pcm: np.ndarray, sample_rate: int, blocking: bool = True, should_stop=None) -> bool:
+# ── how loud is it right now ────────────────────────────────────────────────
+# The PCM is already decoded here on its way to the speakers, so the amplitude
+# envelope is free — and it is exactly what a mouth needs to move in time with
+# speech. Throwing it away and animating the mouth on a timer instead is why
+# most talking sprites look dubbed.
+
+# int16 RMS that counts as "mouth fully open". Speech peaks around 8-12k;
+# 6000 keeps normal talking in the upper half of the range without clipping
+# every syllable to wide-open.
+_LOUD_RMS = 6000.0
+
+
+def level_of(frame: np.ndarray) -> float:
+    """0..1 loudness for one chunk of PCM.
+
+    Square-rooted because perceived loudness is not linear in amplitude — a
+    linear mapping leaves the mouth barely moving through ordinary speech."""
+    if frame is None or len(frame) == 0:
+        return 0.0
+    rms = float(np.sqrt(np.mean(np.square(frame.astype(np.float32)))))
+    return float(min(1.0, (rms / _LOUD_RMS) ** 0.5))
+
+
+def envelope(pcm: np.ndarray, sample_rate: int, fps: int = 30) -> list:
+    """Per-frame loudness for a whole clip, for playback that isn't streamed."""
+    if pcm is None or len(pcm) == 0:
+        return []
+    window = max(1, int(sample_rate / max(1, fps)))
+    return [level_of(pcm[start:start + window]) for start in range(0, len(pcm), window)]
+
+
+def play_pcm(pcm: np.ndarray, sample_rate: int, blocking: bool = True, should_stop=None,
+             on_level=None) -> bool:
     """Play a whole clip. Returns True if it finished, False if *should_stop*
     (barge-in) cut it short. *should_stop* is polled while audio plays — each
     poll consumes one mic frame, which is what paces this loop."""
     import sounddevice as sd
 
+    levels = envelope(pcm, sample_rate) if on_level is not None else []
+    started = time.monotonic()
     sd.play(pcm, samplerate=sample_rate, device=config.SPEAKER_DEVICE)
     if not blocking:
         return True
-    if should_stop is None:
+    if should_stop is None and on_level is None:
         sd.wait()
         return True
     while True:
+        if levels:
+            # Indexed by elapsed time rather than by chunk, because this path
+            # hands the whole clip to the device at once and never sees it
+            # again — wall clock is the only position we have.
+            index = int((time.monotonic() - started) * 30)
+            if index < len(levels):
+                try:
+                    on_level(levels[index])
+                except Exception:
+                    levels = []
         try:
             if not sd.get_stream().active:
                 break
         except Exception:
             break  # stream already torn down — playback is over
-        if should_stop():
+        if should_stop is not None and should_stop():
             sd.stop()
             return False
     return True
 
 
-def play_stream(chunks: Iterable[np.ndarray], sample_rate: int, should_stop=None) -> bool:
-    """Play int16 chunks as they arrive. Returns False if interrupted."""
+def play_stream(chunks: Iterable[np.ndarray], sample_rate: int, should_stop=None,
+                on_level=None) -> bool:
+    """Play int16 chunks as they arrive. Returns False if interrupted.
+
+    *on_level* receives each chunk's loudness (0..1) just before it is written,
+    which is what drives the mouth: the sprite is animated by the same audio
+    the speakers are getting, not by a guess about how long a word takes."""
     import sounddevice as sd
 
     with sd.OutputStream(
@@ -199,6 +249,11 @@ def play_stream(chunks: Iterable[np.ndarray], sample_rate: int, should_stop=None
                 # now, not at the end of the buffered chunk.
                 out.abort()
                 return False
+            if on_level is not None:
+                try:
+                    on_level(level_of(chunk))
+                except Exception:
+                    on_level = None  # a broken listener must not stop playback
             out.write(chunk)
     return True
 
@@ -213,7 +268,8 @@ def speak_offline(text: str) -> None:
     engine.runAndWait()
 
 
-def speak(text: str, on_error=None, should_stop=None, voice_id: Optional[str] = None) -> bool:
+def speak(text: str, on_error=None, should_stop=None, voice_id: Optional[str] = None,
+          on_level=None) -> bool:
     """Speak *text*, preferring streaming ElevenLabs, then whole-clip
     ElevenLabs, then offline TTS. *on_error*, if given, is called with the
     exception when ElevenLabs fails (useful for logging) — a fallback still
@@ -233,13 +289,14 @@ def speak(text: str, on_error=None, should_stop=None, voice_id: Optional[str] =
                 stream_pcm(text, voice_id=voice_id),
                 config.TTS_SAMPLE_RATE,
                 should_stop=should_stop,
+                on_level=on_level,
             )
         except TtsError as exc:
             if on_error is not None:
                 on_error(exc)
     try:
         pcm, sample_rate = synthesize_pcm(text, voice_id=voice_id)
-        return play_pcm(pcm, sample_rate, should_stop=should_stop)
+        return play_pcm(pcm, sample_rate, should_stop=should_stop, on_level=on_level)
     except TtsError as exc:
         if on_error is not None:
             on_error(exc)