Streaming replies and STT, amplitude lip-sync, one place for speaking

Latency: replies are spoken sentence-by-sentence off the desk API's NDJSON
endpoint, so the wait is time-to-first-sentence rather than the whole model
call, and Deepgram's live websocket transcribes while you're still talking
instead of uploading the WAV afterwards. Both fall back invisibly — a stream
that fails before anything was said drops to converse(), and a socket that
never opens just means the old one-shot path.

Speaking lived in four near-copies in the controller (a reply, a holding line,
a streamed sentence, a dialogue scene) that had already drifted: one didn't arm
barge-in, another skipped the follow-up rule. It's now speech.Speaker plus an
Utterance describing the policy differences, with collaborators injected so the
whole of it tests without Qt or audio.

The mouth follows the audio rather than a timer: tts.level_of reduces each PCM
frame to a 0..1 loudness on a sqrt curve (speech sits well below peak, and a
linear map leaves the mouth barely open during normal talking) and that indexes
the talking frames, which the sprite script now draws as an openness ramp.
Offline pyttsx3 has no waveform, so stale levels hand control back to the timed
loop instead of freezing the mouth mid-syllable.

Also: the pet starts where you left it (ignoring positions on monitors that are
no longer connected, since restoring those faithfully is how it ends up
somewhere unreachable), and `python -m bolt_pet --doctor` is a preflight that
says what to do about each problem rather than only what's wrong.

tests/test_pipeline_smoke.py breaks the pure-logic rule on purpose. Every unit
test passed all week while notifications sat unspoken for minutes, the pet said
things twice and [laughing] got read aloud — each an interaction between two
individually-correct units. It drives whole turns against a real HTTP server on
a loopback port, faking only the mic and the speakers. It found a NameError in
the paint path that would have fired on every repaint while talking.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CLAUDE.md

@@ -35,6 +35,10 @@ all suppressed while it's napping (quiet hours / fullscreen DND).
 ./run.sh          # macOS/Linux
 run.bat           # Windows
 
+# Preflight: is this install actually going to work? (config, mic, keys, sprites…)
+python -m bolt_pet --doctor          # shallow: no network, no mic
+python -m bolt_pet --doctor --deep   # contacts the server and opens the microphone
+
 # Run tests (no pytest config file — tests self-insert repo root via sys.path).
 # QT_QPA_PLATFORM=offscreen avoids a QApplication segfault on headless/no-display hosts.
 QT_QPA_PLATFORM=offscreen .venv/bin/pytest tests/
@@ -239,7 +243,18 @@ logs a missing-config message and exits its thread instead of starting.
 - **`quiet.py`** — quiet-hours spec parsing (`23:00-08:00`, wraps midnight,
   comma-separated). Napping suppresses *proactive* noise and wandering only;
   wake word / click / push-to-talk still work.
-- **`notifications.py`** — Linux/D-Bus notification bridge: tails
+- **`notifications.py`** — Linux/D-Bus notification bridge. Two latency/loss bugs
+  fixed 2026-08-02, both in `controller._maybe_heartbeat`: draining was wired to the
+  60s heartbeat interval rather than the ~1.2s wake tick, and a heartbeat that landed
+  mid-conversation stamped its own clock *before* checking — burning the slot and
+  waiting another full interval, repeatedly, which is how a notification could go
+  unspoken for five or ten minutes. Draining now runs on every tick while IDLE, and the
+  heartbeat clock only advances when the heartbeat actually runs. Separately, the rate
+  limit used to *drop* notifications inside its window (a second text a minute later was
+  silently lost); the filter still gates at queue time but the limit is gone — a burst
+  is **batched into one turn** instead, same single round trip, no lost messages, capped
+  by `_MAX_PENDING_NOTIFICATIONS`.
+- **`notifications.py` internals** — tails
   `dbus-monitor`, parses Notify calls (pure `iter_notifications()`), filters
   and rate-limits them (`NotificationGate`), and the controller forwards
   survivors through `converse()`. Off by default — each one is a round trip.
@@ -275,6 +290,81 @@ logs a missing-config message and exits its thread instead of starting.
 - **`hotkey.py`** — global push-to-talk via `pynput`; soft-fails with a logged
   reason (Wayland, missing package, macOS permissions) since the wake word is
   the primary trigger.
+- **`audio/stt_stream.py`** — streaming speech-to-text. The one-shot path waits for
+  the utterance to end, uploads the whole WAV, then waits again; that second wait is
+  dead time that grows with how long you spoke. Deepgram's live websocket removes it:
+  `record_utterance(on_frame=...)` hands each captured frame to a
+  `StreamingTranscriber`, so by the time the VAD decides you stopped the transcript is
+  essentially already there. Three deliberate limits: `open()` returning **None is an
+  ordinary outcome** (no websocket-client, no network, no key) because the full audio is
+  still buffered and `controller._transcribe` just falls back; the **local VAD still
+  decides when you stopped** rather than Deepgram's endpointing, since barge-in,
+  follow-up listening and the grace period are all built on it and coupling them to the
+  network is not a first-pass change; and the socket is **per-utterance**, because
+  holding one open across an idle pet bills for silence and dies on the first blip.
+  Off: `STT_STREAMING=false`.
+- **Streamed replies** — `server_client.converse_stream()` reads NDJSON from the desk
+  API's `/desk/converse_stream` and speaks each sentence as it arrives
+  (`controller._speak_stream_chunk`), so the wait is time-to-first-sentence instead of
+  the whole model call. `Reply.spoken` marks a reply whose sentences were already said:
+  the text is still carried, because the follow-up rule needs to see whether it ended on
+  a question, but it must not be read out again. A stream that fails *before* anything
+  was spoken falls back to `converse()` invisibly; one that fails after ends the turn
+  quietly rather than repeating the first half. Off: `STREAMING_REPLIES=false`.
+- **`speech.py`** — everything the pet says, and the policy differences between
+  kinds of saying. There were four near-copies of this in `controller.py` (a
+  reply, a holding line, a streamed sentence, a dialogue scene), each repeating
+  the same dance — transition state, show the bubble, maybe record history,
+  reset barge-in, call TTS, reset barge-in *again*, resume state, decide
+  whether to keep the mic open — and they had already drifted apart: one forgot
+  to arm barge-in, another skipped the follow-up rule. Now the dance is
+  `Speaker.say()` and the differences are data on a frozen `Utterance`
+  (`record`, `resume`, `hold_talking`, `follow_up`, `interruptible`), built by
+  the four classmethods `reply`/`holding`/`stream_chunk`/`scene`. Notable
+  policies: a holding line is **not** recorded (it's filler; the transcript
+  should keep the answer) and **not** interruptible (cutting off "give me a
+  sec" strands the tool already running), and it resumes the state it
+  interrupted rather than dropping to IDLE, because the turn isn't over. A
+  streamed chunk stays TALKING so the sprite doesn't flicker between sentences.
+  Collaborators are injected, so all of this is tested without Qt, audio or a
+  real state machine. Two subtleties that are bugs waiting to happen: the
+  barge-in detector is read through a **callable**, not held (it's built after
+  the Speaker — it needs the mic stream — and swapped when the mode changes; two
+  copies drifting apart is invisible until the wake model starts hearing the
+  pet), and `last_detail` is captured *before* the post-playback reset, since
+  reading it after means every interruption reports zeroed counters.
+  `follow_up_decision()` is the mic-open rule as a pure function.
+- **Lip-sync** — the mouth is driven by the audio, not a timer.
+  `audio/tts.level_of(frame)` reduces a PCM frame to 0..1 loudness on a **sqrt
+  curve** (speech sits well below peak most of the time, so a linear map leaves
+  the mouth barely open during normal talking), `envelope()` does the same for a
+  whole clip, and playback calls the `on_level` hook per chunk. That travels
+  `Speaker` → `controller.mouth` (a Signal) → `PetWindow.set_mouth`, and
+  `_mouth_frame()` indexes the talking frames directly — which works because
+  `generate_bolt_sprites.py` draws them as an **openness ramp** (closed first,
+  widest last) rather than an arbitrary loop. Levels going stale
+  (`_MOUTH_STALE_SECONDS`) hands control back to the ordinary animation, so
+  offline TTS — which has no envelope — degrades to the timed loop instead of
+  freezing the mouth mid-syllable.
+- **`window_state.py`** — where the pet was left, so it starts there.
+  `~/.cache/bolt-pet/window.json`, atomic write on drag-end, every function
+  swallows its own errors (a corrupt state file must mean the default corner,
+  never a pet that won't start). `is_visible_on()` re-validates against the
+  *current* screen layout on load, because the common case for a stale position
+  is exactly the dangerous one: the pet was last on a monitor that is now
+  unplugged, and restoring it faithfully puts it somewhere unreachable. Takes
+  plain rectangles rather than importing Qt. Off: `PET_REMEMBER_POSITION=false`.
+- **`doctor.py`** — `python -m bolt_pet --doctor`, a preflight, written after a
+  week of debugging things one command would have shown: a venv whose python
+  was a zero-byte file, an OCR engine never installed, a wrong proxy header.
+  Twelve independent checks, each reporting ok / warn (degraded but working) /
+  fail, and each saying **what to do about it** — `screen reading: warn` is
+  useless alone, `apt install tesseract-ocr` is the whole point. Nothing raises:
+  a doctor that crashes on a broken install is diagnosing the wrong patient, so
+  `run()` catches per-check and a failed check becomes a FAIL row rather than a
+  traceback. Shallow by default (no network, no mic) since it's the first thing
+  you reach for when the network is what's broken; `--deep` actually contacts
+  the server and opens the microphone.
 - **`speech_text.py`** — sanitizes server replies before they're heard/shown.
   `for_speech()` (called inside `tts.speak()`, so every path to the speakers is
   covered) strips markdown, emoji, URLs and stray symbols the voice would read
@@ -284,7 +374,7 @@ logs a missing-config message and exits its thread instead of starting.
   whether a reply leaves the pet waiting on an answer: it tests the *spoken*
   form (so a '?' inside a stripped code block or URL doesn't count) and only a
   trailing one counts, since a question asked in passing isn't awaiting a
-  reply. `controller._should_follow_up` uses it to keep listening without the
+  reply. `speech.follow_up_decision` uses it to keep listening without the
   wake word, capped by `FOLLOW_UP_MAX_TURNS` so a server that ends every reply
   with a question can't loop forever off mic noise. Pure string logic, no
   Qt/audio imports.
@@ -414,6 +504,20 @@ HTTP chunk reassembly by `chunks_to_int16`, emote motion by
 `screen_context.context_for` / `is_fullscreen_active`, otherwise they shell
 out to xprop on a headless box.
 
+**`test_pipeline_smoke.py` is the exception, deliberately.** Unit tests inject
+a fake at the seam they care about, and a run of production bugs — notifications
+sitting unspoken for minutes, the pet saying things twice, `[laughing]` read
+aloud, a device command arriving as prose — got through with every one of them
+green, because each was an *interaction* between two individually-correct
+units. So that file stands up a real threaded HTTP server on a loopback port,
+speaks the desk protocol at it, and drives whole turns through the real
+`server_client` (including NDJSON streaming), the real controller and the real
+state machine, faking only the mic stream and the speakers. When a bug crosses
+a module boundary, add the case there; when it lives inside one module, the
+pure-function pattern above is still the cheaper test. It is also worth
+mutation-checking a new case — break the source line it is meant to catch and
+confirm it actually goes red.
+
 ## Security notes
 
 The server can relay a shell command back to this machine to execute as the