Streaming replies and STT, amplitude lip-sync, one place for speaking

Latency: replies are spoken sentence-by-sentence off the desk API's NDJSON
endpoint, so the wait is time-to-first-sentence rather than the whole model
call, and Deepgram's live websocket transcribes while you're still talking
instead of uploading the WAV afterwards. Both fall back invisibly — a stream
that fails before anything was said drops to converse(), and a socket that
never opens just means the old one-shot path.

Speaking lived in four near-copies in the controller (a reply, a holding line,
a streamed sentence, a dialogue scene) that had already drifted: one didn't arm
barge-in, another skipped the follow-up rule. It's now speech.Speaker plus an
Utterance describing the policy differences, with collaborators injected so the
whole of it tests without Qt or audio.

The mouth follows the audio rather than a timer: tts.level_of reduces each PCM
frame to a 0..1 loudness on a sqrt curve (speech sits well below peak, and a
linear map leaves the mouth barely open during normal talking) and that indexes
the talking frames, which the sprite script now draws as an openness ramp.
Offline pyttsx3 has no waveform, so stale levels hand control back to the timed
loop instead of freezing the mouth mid-syllable.

Also: the pet starts where you left it (ignoring positions on monitors that are
no longer connected, since restoring those faithfully is how it ends up
somewhere unreachable), and `python -m bolt_pet --doctor` is a preflight that
says what to do about each problem rather than only what's wrong.

tests/test_pipeline_smoke.py breaks the pure-logic rule on purpose. Every unit
test passed all week while notifications sat unspoken for minutes, the pet said
things twice and [laughing] got read aloud — each an interaction between two
individually-correct units. It drives whole turns against a real HTTP server on
a loopback port, faking only the mic and the speakers. It found a NameError in
the paint path that would have fired on every repaint while talking.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

tests/test_lipsync.py

@@ -0,0 +1,135 @@
+"""Amplitude → mouth openness, from PCM samples to the drawn frame.
+
+Two halves, tested separately because only one of them needs a display:
+`tts.level_of`/`envelope` turn samples into a 0..1 loudness, and
+`PetWindow._mouth_frame` turns that loudness into a frame index. They meet at
+the `mouth` signal, which the pipeline smoke test covers end to end.
+"""
+
+import sys
+import time
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from bolt_pet.audio import tts
+
+
+def frame(amplitude: int, samples: int = 480) -> np.ndarray:
+    return np.full(samples, amplitude, dtype=np.int16)
+
+
+# ── loudness ────────────────────────────────────────────────────────────────
+
+def test_silence_closes_the_mouth():
+    assert tts.level_of(np.zeros(480, dtype=np.int16)) == 0.0
+
+
+def test_a_loud_frame_opens_it_fully():
+    assert tts.level_of(frame(30000)) == 1.0
+
+
+def test_the_level_never_leaves_zero_to_one():
+    """It indexes a frame list; out of range is an IndexError on the UI thread."""
+    for amplitude in (0, 1, 500, 6000, 20000, 32767):
+        assert 0.0 <= tts.level_of(frame(amplitude)) <= 1.0
+
+
+def test_it_rises_with_amplitude():
+    quiet, middling, loud = (tts.level_of(frame(a)) for a in (800, 4000, 12000))
+    assert quiet < middling < loud
+
+
+def test_quiet_speech_still_moves_the_mouth_visibly():
+    """The sqrt curve is the whole point. Speech spends most of its time well
+    below peak, so a linear map leaves the mouth barely open for normal talking
+    and the pet looks like it's mumbling."""
+    assert tts.level_of(frame(1500)) > 0.15
+
+
+def test_an_empty_frame_is_silence_not_a_crash():
+    assert tts.level_of(np.zeros(0, dtype=np.int16)) == 0.0
+
+
+# ── the envelope of a whole clip ────────────────────────────────────────────
+
+def test_an_envelope_has_one_value_per_frame_at_the_requested_rate():
+    one_second = np.zeros(16000, dtype=np.int16)
+    assert len(tts.envelope(one_second, 16000, fps=30)) == pytest.approx(30, abs=1)
+
+
+def test_an_envelope_tracks_loud_and_quiet_stretches():
+    pcm = np.concatenate([np.zeros(8000, dtype=np.int16), frame(20000, 8000)])
+    levels = tts.envelope(pcm, 16000, fps=10)
+    assert max(levels[:4]) == 0.0      # the silent half
+    assert min(levels[-4:]) > 0.5      # the loud half
+
+
+def test_an_empty_clip_has_an_empty_envelope():
+    assert tts.envelope(np.zeros(0, dtype=np.int16), 16000) == []
+
+
+# ── the drawn frame ─────────────────────────────────────────────────────────
+
+@pytest.fixture
+def window():
+    """Needs a QApplication; run with QT_QPA_PLATFORM=offscreen."""
+    from PySide6.QtWidgets import QApplication
+
+    from bolt_pet.ui.pet_window import PetWindow
+
+    _app = QApplication.instance() or QApplication(["test"])
+    win = PetWindow()
+    yield win
+    win.close()
+
+
+class FakeAnimation:
+    """Stands in for sprite.Animation — _mouth_frame only wants `.frames`."""
+
+    def __init__(self, frames):
+        self.frames = list(frames)
+
+
+def test_the_talking_frame_follows_the_level(window):
+    """The talking frames are an openness ramp — closed first, widest last —
+    specifically so loudness can index them directly."""
+    animation = FakeAnimation(["closed", "a", "b", "c", "d", "open"])
+
+    assert window._mouth_frame(animation) is None      # no level set yet
+
+    window.set_mouth(0.0)
+    assert window._mouth_frame(animation) == "closed"
+
+    window.set_mouth(1.0)
+    assert window._mouth_frame(animation) == "open"
+
+    window.set_mouth(0.5)
+    assert window._mouth_frame(animation) not in ("closed", "open")
+
+
+def test_a_stale_level_gives_the_animation_back(window):
+    """If the audio thread stops sending levels — TTS died, the clip ended
+    without a final zero — the mouth must not freeze half-open forever. After a
+    moment it falls back to the ordinary looping animation."""
+    animation = FakeAnimation(["a", "b", "c"])
+    window.set_mouth(0.9)
+    assert window._mouth_frame(animation) is not None
+
+    window._mouth_at = time.monotonic() - 5.0
+    assert window._mouth_frame(animation) is None
+
+
+def test_a_single_frame_animation_falls_back_instead_of_indexing(window):
+    """Art with one talking frame can't lip-sync; it must not try."""
+    window.set_mouth(1.0)
+    assert window._mouth_frame(FakeAnimation(["only"])) is None
+    assert window._mouth_frame(FakeAnimation([])) is None
+
+
+def test_no_animation_at_all_is_handled(window):
+    window.set_mouth(0.5)
+    assert window._mouth_frame(None) is None