Streaming replies and STT, amplitude lip-sync, one place for speaking

Latency: replies are spoken sentence-by-sentence off the desk API's NDJSON
endpoint, so the wait is time-to-first-sentence rather than the whole model
call, and Deepgram's live websocket transcribes while you're still talking
instead of uploading the WAV afterwards. Both fall back invisibly — a stream
that fails before anything was said drops to converse(), and a socket that
never opens just means the old one-shot path.

Speaking lived in four near-copies in the controller (a reply, a holding line,
a streamed sentence, a dialogue scene) that had already drifted: one didn't arm
barge-in, another skipped the follow-up rule. It's now speech.Speaker plus an
Utterance describing the policy differences, with collaborators injected so the
whole of it tests without Qt or audio.

The mouth follows the audio rather than a timer: tts.level_of reduces each PCM
frame to a 0..1 loudness on a sqrt curve (speech sits well below peak, and a
linear map leaves the mouth barely open during normal talking) and that indexes
the talking frames, which the sprite script now draws as an openness ramp.
Offline pyttsx3 has no waveform, so stale levels hand control back to the timed
loop instead of freezing the mouth mid-syllable.

Also: the pet starts where you left it (ignoring positions on monitors that are
no longer connected, since restoring those faithfully is how it ends up
somewhere unreachable), and `python -m bolt_pet --doctor` is a preflight that
says what to do about each problem rather than only what's wrong.

tests/test_pipeline_smoke.py breaks the pure-logic rule on purpose. Every unit
test passed all week while notifications sat unspoken for minutes, the pet said
things twice and [laughing] got read aloud — each an interaction between two
individually-correct units. It drives whole turns against a real HTTP server on
a loopback port, faking only the mic and the speakers. It found a NameError in
the paint path that would have fired on every repaint while talking.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

bolt_pet/__main__.py

@@ -1,8 +1,15 @@
-"""Entry point: python -m bolt_pet"""
+"""Entry point: python -m bolt_pet [--doctor [--deep]]"""
 
 import sys
 
-from .ui.app import run
-
 if __name__ == "__main__":
+    if "--doctor" in sys.argv:
+        # Imported inside the branch, not at module scope: the doctor exists to
+        # diagnose installs where importing the UI would itself blow up.
+        from .doctor import main as doctor
+
+        sys.exit(doctor(sys.argv[1:]))
+
+    from .ui.app import run
+
     sys.exit(run())

bolt_pet/audio/mic.py

@@ -37,6 +37,20 @@ def rms(frame: np.ndarray) -> float:
     return float(np.sqrt(np.mean(frame.astype(np.float64) ** 2)))
 
 
+def _emit(on_frame, frame) -> None:
+    """Hand a frame to a listener without letting it break the capture.
+
+    Streaming STT is an optimisation riding along with recording; if its
+    socket dies mid-utterance the recording must carry on untouched, because
+    the one-shot fallback is about to need the full buffer."""
+    if on_frame is None:
+        return
+    try:
+        on_frame(frame)
+    except Exception:
+        pass
+
+
 def record_utterance(
     stream: AudioStream,
     should_continue=lambda: True,
@@ -47,6 +61,7 @@ def record_utterance(
     grace_s: float = None,
     frame_len: int = config.FRAME_LEN,
     sample_rate: int = config.SAMPLE_RATE,
+    on_frame=None,
 ) -> Optional[np.ndarray]:
     """Capture one utterance from *stream*: wait for speech to start, stop
     after trailing silence. Returns None if nothing usable was heard.
@@ -55,6 +70,11 @@ def record_utterance(
     (e.g. the pet window was closed) without needing threading primitives
     baked into this function.
 
+    *on_frame*, if given, is called with each captured frame while speech is
+    in progress — that is how streaming STT transcribes as you talk rather
+    than after (see audio/stt_stream.py). It is fire-and-forget: this function
+    still returns the full buffer, so a failed stream costs nothing.
+
     *grace_s* is how long to wait for speech to *begin* before giving up.
     The controller stretches it for follow-up questions, where you're being
     asked something and need a moment to think rather than having just said
@@ -83,10 +103,12 @@ def record_utterance(
             if frame_rms >= rms_threshold:
                 started = True
                 frames.append(frame)
+                _emit(on_frame, frame)
             elif waited > grace_frames:
                 return None  # woke it up but said nothing
             continue
         frames.append(frame)
+        _emit(on_frame, frame)
         if frame_rms < rms_threshold:
             silence_frames += 1
             if silence_frames >= silence_limit:

bolt_pet/audio/stt_stream.py

@@ -0,0 +1,181 @@
+"""Streaming speech-to-text — transcribing *while* you talk, not after.
+
+The one-shot path (`stt.transcribe`) waits for the utterance to finish, then
+uploads the whole WAV and waits again. That second wait is dead time between
+you stopping and the pet reacting, and it grows with the length of what you
+said — a thirty-second question costs noticeably more than a five-second one.
+
+Deepgram's live endpoint removes it: frames go up as they are captured, so by
+the time the VAD decides you have stopped, the transcript is essentially
+already there. Same model, same account, same accuracy — the difference is
+purely when the work happens.
+
+Design constraints that shaped this:
+
+- **Failure must be invisible.** No websocket, no network, a mid-utterance
+  disconnect — all of it falls back to the one-shot path, which still has the
+  full audio buffered. Streaming is an optimisation, never a dependency, so
+  `open()` returning None is an ordinary outcome rather than an error.
+- **The VAD still decides when you stopped.** Deepgram has its own endpointing
+  and using it would save more, but it would also move a decision the rest of
+  the pipeline is built around (barge-in, follow-up listening, the grace
+  period) into a remote service. Not worth coupling those to the network on
+  the first pass.
+- **The socket is per-utterance.** Holding one open across an idle pet would
+  bill for silence and drop on the first network blip; opening one takes
+  ~100ms, which is already inside the time it takes a person to start talking.
+
+The websocket client is injectable, so the whole protocol — send frames, read
+`is_final` transcripts, close, take the result — is tested without a network.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import threading
+from typing import Callable, Optional
+
+import numpy as np
+
+from .. import config
+
+logger = logging.getLogger("bolt_pet.stt_stream")
+
+_ENDPOINT = (
+    "wss://api.deepgram.com/v1/listen"
+    "?encoding=linear16&channels=1&sample_rate={rate}&model={model}"
+    "&language=en&smart_format=true&interim_results=false"
+)
+
+
+def available() -> bool:
+    """Whether streaming STT can even be attempted in this install."""
+    if not config.STT_STREAMING or not config.DEEPGRAM_API_KEY:
+        return False
+    try:
+        import websocket  # noqa: F401  (websocket-client)
+        return True
+    except Exception:
+        return False
+
+
+class StreamingTranscriber:
+    """One utterance's worth of live transcription.
+
+    Usage mirrors how the capture loop already works — feed frames as they
+    arrive, then ask what was said:
+
+        session = StreamingTranscriber.open()
+        ...
+        session.feed(frame)      # per mic frame, non-blocking
+        text = session.finish()  # after the VAD says you stopped
+    """
+
+    def __init__(self, socket, *, sample_rate: int = None):
+        self._socket = socket
+        self._sample_rate = sample_rate or config.SAMPLE_RATE
+        self._transcript: list[str] = []
+        self._lock = threading.Lock()
+        self._closed = False
+        self._reader = threading.Thread(
+            target=self._read_loop, name="stt-stream-reader", daemon=True)
+        self._reader.start()
+
+    # -- lifecycle ----------------------------------------------------------
+
+    @classmethod
+    def open(cls, *, connect: Optional[Callable] = None,
+             sample_rate: int = None) -> Optional["StreamingTranscriber"]:
+        """Connect, or return None if streaming isn't possible right now.
+
+        None is a normal outcome, not a failure: the caller keeps the audio and
+        falls back to the one-shot upload."""
+        if connect is None and not available():
+            return None
+        rate = sample_rate or config.SAMPLE_RATE
+        try:
+            if connect is not None:
+                socket = connect()
+            else:
+                import websocket
+
+                socket = websocket.create_connection(
+                    _ENDPOINT.format(rate=rate, model=config.DEEPGRAM_MODEL),
+                    header={"Authorization": f"Token {config.DEEPGRAM_API_KEY}"},
+                    timeout=10,
+                )
+            return cls(socket, sample_rate=rate)
+        except Exception as exc:
+            logger.info("Streaming STT unavailable (%s) — using the one-shot path.", exc)
+            return None
+
+    def feed(self, frame: np.ndarray) -> None:
+        """Send one captured frame. Never raises — a dead socket just means the
+        fallback will do the work."""
+        if self._closed:
+            return
+        try:
+            self._socket.send_binary(np.asarray(frame, dtype=np.int16).tobytes())
+        except Exception:
+            logger.debug("Streaming STT send failed; abandoning the stream", exc_info=True)
+            self._closed = True
+
+    def finish(self, timeout: float = 3.0) -> str:
+        """Close the stream and return whatever was transcribed.
+
+        Deepgram flushes its final results after the close frame, so this waits
+        briefly for the reader — bounded, because a hung socket must not hold
+        up the reply."""
+        if not self._closed:
+            try:
+                self._socket.send(json.dumps({"type": "CloseStream"}))
+            except Exception:
+                pass
+        self._closed = True
+        self._reader.join(timeout=timeout)
+        try:
+            self._socket.close()
+        except Exception:
+            pass
+        with self._lock:
+            return " ".join(part for part in self._transcript if part).strip()
+
+    # -- the reader ---------------------------------------------------------
+
+    def _read_loop(self) -> None:
+        while not self._closed:
+            try:
+                message = self._socket.recv()
+            except Exception:
+                break
+            if not message:
+                break
+            text, is_final = self._parse(message)
+            if text and is_final:
+                with self._lock:
+                    self._transcript.append(text)
+
+    @staticmethod
+    def _parse(message) -> tuple[str, bool]:
+        """Pull (text, is_final) out of a Deepgram results frame.
+
+        Tolerant on purpose: anything unrecognised is ignored rather than
+        raising on the reader thread, where an exception would silently kill
+        transcription for the rest of the utterance."""
+        try:
+            if isinstance(message, bytes):
+                message = message.decode("utf-8", "ignore")
+            data = json.loads(message)
+        except (TypeError, ValueError):
+            return "", False
+        if not isinstance(data, dict):
+            return "", False
+        alternatives = (
+            ((data.get("channel") or {}).get("alternatives") or [])
+            if data.get("type") in (None, "Results") else []
+        )
+        if not alternatives:
+            return "", False
+        text = str((alternatives[0] or {}).get("transcript") or "").strip()
+        return text, bool(data.get("is_final") or data.get("speech_final"))

bolt_pet/audio/tts.py

@@ -14,6 +14,7 @@ fallback has no such concept and always sounds like itself.
 
 from __future__ import annotations
 
+import time
 from typing import Iterable, Iterator, Optional
 
 import numpy as np
@@ -162,32 +163,81 @@ def synthesize_dialogue(
     return pcm, config.TTS_SAMPLE_RATE
 
 
-def play_pcm(pcm: np.ndarray, sample_rate: int, blocking: bool = True, should_stop=None) -> bool:
+# ── how loud is it right now ────────────────────────────────────────────────
+# The PCM is already decoded here on its way to the speakers, so the amplitude
+# envelope is free — and it is exactly what a mouth needs to move in time with
+# speech. Throwing it away and animating the mouth on a timer instead is why
+# most talking sprites look dubbed.
+
+# int16 RMS that counts as "mouth fully open". Speech peaks around 8-12k;
+# 6000 keeps normal talking in the upper half of the range without clipping
+# every syllable to wide-open.
+_LOUD_RMS = 6000.0
+
+
+def level_of(frame: np.ndarray) -> float:
+    """0..1 loudness for one chunk of PCM.
+
+    Square-rooted because perceived loudness is not linear in amplitude — a
+    linear mapping leaves the mouth barely moving through ordinary speech."""
+    if frame is None or len(frame) == 0:
+        return 0.0
+    rms = float(np.sqrt(np.mean(np.square(frame.astype(np.float32)))))
+    return float(min(1.0, (rms / _LOUD_RMS) ** 0.5))
+
+
+def envelope(pcm: np.ndarray, sample_rate: int, fps: int = 30) -> list:
+    """Per-frame loudness for a whole clip, for playback that isn't streamed."""
+    if pcm is None or len(pcm) == 0:
+        return []
+    window = max(1, int(sample_rate / max(1, fps)))
+    return [level_of(pcm[start:start + window]) for start in range(0, len(pcm), window)]
+
+
+def play_pcm(pcm: np.ndarray, sample_rate: int, blocking: bool = True, should_stop=None,
+             on_level=None) -> bool:
     """Play a whole clip. Returns True if it finished, False if *should_stop*
     (barge-in) cut it short. *should_stop* is polled while audio plays — each
     poll consumes one mic frame, which is what paces this loop."""
     import sounddevice as sd
 
+    levels = envelope(pcm, sample_rate) if on_level is not None else []
+    started = time.monotonic()
     sd.play(pcm, samplerate=sample_rate, device=config.SPEAKER_DEVICE)
     if not blocking:
         return True
-    if should_stop is None:
+    if should_stop is None and on_level is None:
         sd.wait()
         return True
     while True:
+        if levels:
+            # Indexed by elapsed time rather than by chunk, because this path
+            # hands the whole clip to the device at once and never sees it
+            # again — wall clock is the only position we have.
+            index = int((time.monotonic() - started) * 30)
+            if index < len(levels):
+                try:
+                    on_level(levels[index])
+                except Exception:
+                    levels = []
         try:
             if not sd.get_stream().active:
                 break
         except Exception:
             break  # stream already torn down — playback is over
-        if should_stop():
+        if should_stop is not None and should_stop():
             sd.stop()
             return False
     return True
 
 
-def play_stream(chunks: Iterable[np.ndarray], sample_rate: int, should_stop=None) -> bool:
-    """Play int16 chunks as they arrive. Returns False if interrupted."""
+def play_stream(chunks: Iterable[np.ndarray], sample_rate: int, should_stop=None,
+                on_level=None) -> bool:
+    """Play int16 chunks as they arrive. Returns False if interrupted.
+
+    *on_level* receives each chunk's loudness (0..1) just before it is written,
+    which is what drives the mouth: the sprite is animated by the same audio
+    the speakers are getting, not by a guess about how long a word takes."""
     import sounddevice as sd
 
     with sd.OutputStream(
@@ -199,6 +249,11 @@ def play_stream(chunks: Iterable[np.ndarray], sample_rate: int, should_stop=None
                 # now, not at the end of the buffered chunk.
                 out.abort()
                 return False
+            if on_level is not None:
+                try:
+                    on_level(level_of(chunk))
+                except Exception:
+                    on_level = None  # a broken listener must not stop playback
             out.write(chunk)
     return True
 
@@ -213,7 +268,8 @@ def speak_offline(text: str) -> None:
     engine.runAndWait()
 
 
-def speak(text: str, on_error=None, should_stop=None, voice_id: Optional[str] = None) -> bool:
+def speak(text: str, on_error=None, should_stop=None, voice_id: Optional[str] = None,
+          on_level=None) -> bool:
     """Speak *text*, preferring streaming ElevenLabs, then whole-clip
     ElevenLabs, then offline TTS. *on_error*, if given, is called with the
     exception when ElevenLabs fails (useful for logging) — a fallback still
@@ -233,13 +289,14 @@ def speak(text: str, on_error=None, should_stop=None, voice_id: Optional[str] =
                 stream_pcm(text, voice_id=voice_id),
                 config.TTS_SAMPLE_RATE,
                 should_stop=should_stop,
+                on_level=on_level,
             )
         except TtsError as exc:
             if on_error is not None:
                 on_error(exc)
     try:
         pcm, sample_rate = synthesize_pcm(text, voice_id=voice_id)
-        return play_pcm(pcm, sample_rate, should_stop=should_stop)
+        return play_pcm(pcm, sample_rate, should_stop=should_stop, on_level=on_level)
     except TtsError as exc:
         if on_error is not None:
             on_error(exc)

bolt_pet/config.py

@@ -58,6 +58,11 @@ WAKE_CHECK_INTERVAL_SECONDS = float(os.environ.get("WAKE_CHECK_INTERVAL_SECONDS"
 
 DEEPGRAM_API_KEY = os.environ.get("DEEPGRAM_API_KEY", "")
 DEEPGRAM_MODEL = os.environ.get("DEEPGRAM_MODEL", "nova-3")
+# Transcribe *while* you talk instead of uploading the finished clip: frames go
+# up as they are captured, so the transcript is ready the moment the VAD says
+# you stopped. Needs `websocket-client`; falls back to the one-shot upload
+# whenever it can't connect, so turning it on can only help.
+STT_STREAMING = os.environ.get("STT_STREAMING", "true").lower() in ("1", "true", "yes", "on")
 
 # ── TTS (ElevenLabs, requested as raw PCM so playback needs no external
 # player binary — cross-platform via sounddevice instead of shelling out to
@@ -168,6 +173,13 @@ BARGE_IN_FRAMES = int(os.environ.get("BARGE_IN_FRAMES", "4"))  # consecutive lou
 # waking the pet from idle (useful if Bolt's own voice trips the model).
 BARGE_IN_WAKE_THRESHOLD = float(os.environ.get("BARGE_IN_WAKE_THRESHOLD") or 0) or None
 
+# ── streaming the reply ─────────────────────────────────────────────────────
+# Speak each sentence as the server produces it, instead of waiting out the
+# whole model call before the first word. Falls back to the ordinary
+# request/response path automatically if the server has no streaming endpoint
+# or the stream fails before anything has been spoken.
+STREAMING_REPLIES = os.environ.get("STREAMING_REPLIES", "true").lower() in ("1", "true", "yes", "on")
+
 # ── streaming TTS ───────────────────────────────────────────────────────────
 # ElevenLabs' /stream endpoint + chunked playback: the pet starts talking
 # after the first PCM chunk instead of after the whole clip is synthesized.
@@ -307,6 +319,10 @@ PET_WANDER_MARGIN = int(os.environ.get("PET_WANDER_MARGIN", "20"))  # keep off s
 PET_SHAPED_INPUT = os.environ.get("PET_SHAPED_INPUT", "true").lower() in ("1", "true", "yes", "on")
 PET_CLICK_THROUGH = os.environ.get("PET_CLICK_THROUGH", "false").lower() in ("1", "true", "yes", "on")
 # Snap flush to a screen edge when dropped/parked within this many pixels of it.
+# Start where it was left rather than in the bottom-right corner. Ignored when
+# PET_START_X/Y pin it explicitly, and a saved position on a monitor that is no
+# longer plugged in is discarded rather than hiding the pet offscreen.
+PET_REMEMBER_POSITION = os.environ.get("PET_REMEMBER_POSITION", "true").lower() in ("1", "true", "yes", "on")
 PET_EDGE_SNAP = os.environ.get("PET_EDGE_SNAP", "true").lower() in ("1", "true", "yes", "on")
 PET_SNAP_MARGIN = int(os.environ.get("PET_SNAP_MARGIN", "48"))
 

bolt_pet/controller.py

@@ -20,15 +20,20 @@ from typing import Optional
 from PySide6.QtCore import QObject, Signal
 
 from . import (
-    config, dialogue as dialogue_mod, file_delivery, file_ops,
+    config, dialogue as dialogue_mod, speech, file_delivery, file_ops,
     history as history_mod, monitors as monitors_mod, notifications,
     pet_actions, quiet, screen_context, screen_text, self_restart,
     server_client, speech_text, updater,
 )
 from . import __version__
-from .audio import barge_in, mic, stt, tts, wake_word
+from .audio import barge_in, mic, stt, stt_stream, tts, wake_word
 from .state import PetState, PetStateMachine
 
+# A burst while the pet is busy is batched into one turn, but the queue still
+# needs a ceiling — a notification storm must not become an unbounded backlog
+# that gets read out minutes later.
+_MAX_PENDING_NOTIFICATIONS = 12
+
 # How often to re-check whether the pet should be napping. The fullscreen
 # probe shells out to xprop, so this deliberately isn't every heartbeat tick.
 _NAP_CHECK_INTERVAL_SECONDS = 10.0
@@ -41,6 +46,7 @@ class PetController(QObject):
     action = Signal(dict)  # parsed petctl action for the UI to perform
     napping = Signal(bool)  # quiet hours / fullscreen do-not-disturb
     voice_changed = Signal(str)  # name of the server-picked voice ("" = default)
+    mouth = Signal(float)  # 0..1 speech loudness, for lip-sync while talking
     restart_requested = Signal(str)  # version we just updated to
     finished = Signal()
 
@@ -77,6 +83,17 @@ class PetController(QObject):
         self._voice_name = ""
 
         self._barge_in: Optional[barge_in.BargeInDetector] = None
+        # Everything the pet says goes through here; see speech.py for why the
+        # four hand-rolled copies of this became one.
+        self._speaker = speech.Speaker(
+            state=self._state, tts=tts,
+            history=lambda text: self.history.add(history_mod.PET, text, time.time()),
+            on_said=self.said.emit, on_log=self.log.emit,
+            barge_in=lambda: self._barge_in,
+            detail_of=self._barge_in_detail,
+            voice_id=lambda: self._voice_id,
+            on_level=self.mouth.emit,
+        )
         self._napping = False
         self._nap_forced: Optional[bool] = None  # petctl nap on/off overrides the schedule
         self._last_nap_check = 0.0
@@ -249,21 +266,28 @@ class PetController(QObject):
             self._follow_ups = 0
 
         self._state.transition(PetState.LISTENING)
+        # Transcribe while they talk rather than after: frames go to Deepgram
+        # as they are captured, so the text is ready the moment the VAD says
+        # they stopped. None here just means the one-shot path will do it.
+        streamed = stt_stream.StreamingTranscriber.open()
         pcm = mic.record_utterance(
             self._stream,
             should_continue=self._should_continue,
             # Answering a question deserves longer than saying the wake word
             # on purpose does — you were just asked something.
             grace_s=config.FOLLOW_UP_GRACE_SECONDS if following_up else None,
+            on_frame=streamed.feed if streamed is not None else None,
         )
         if pcm is None:
+            if streamed is not None:
+                streamed.finish()
             self._follow_ups = 0  # silence ends the chain
             self._state.transition(PetState.IDLE)
             return
 
         self._state.transition(PetState.THINKING)
         try:
-            text = stt.transcribe(pcm)
+            text = self._transcribe(pcm, streamed)
         except stt.SttError as exc:
             self.log.emit(f"STT failed: {exc}")
             self._state.transition(PetState.ERROR)
@@ -278,9 +302,7 @@ class PetController(QObject):
         try:
             # What's focused right now rides along, so "what's this error?"
             # has a referent without you having to describe the window.
-            reply = server_client.converse(
-                self._with_context(text), on_command=self._handle_command
-            )
+            reply = self._ask_server(self._with_context(text))
         except server_client.ServerError as exc:
             self.log.emit(f"Server error: {exc}")
             self._state.transition(PetState.ERROR)
@@ -289,10 +311,34 @@ class PetController(QObject):
 
         self._check_deliveries()
         self._apply_voice(reply)
-        self._speak(reply.text)
+        if reply.spoken:
+            # Streamed: every sentence was spoken and logged as it arrived.
+            # The text still matters — a reply ending on a question should keep
+            # the mic open — but saying it again would repeat the whole answer.
+            self._after_speaking(reply.text, completed=True)
+        else:
+            self._speak(reply.text)
         self._state.transition(PetState.IDLE)
         self._maybe_self_restart()
 
+    def _transcribe(self, pcm, streamed) -> str:
+        """The transcript, from the live stream if it produced one.
+
+        The fallback is not a rare path to be tolerated — it is the safety net
+        that lets streaming be switched on at all. Whatever happened to the
+        socket, the full audio is still buffered here, so a failed stream costs
+        one ordinary upload and nothing else."""
+        if streamed is not None:
+            try:
+                text = streamed.finish()
+            except Exception:
+                self.log.emit("Streaming STT failed — falling back.")
+                text = ""
+            if text:
+                self.log.emit("(transcribed while you spoke)")
+                return text
+        return stt.transcribe(pcm)
+
     def _with_context(self, text: str) -> str:
         """Everything the server gets alongside what you actually said: the
         focused window title, and a one-line note about the screen layout so
@@ -492,6 +538,26 @@ class PetController(QObject):
             self._speak(reply.text)
         self._state.force(PetState.IDLE)
 
+    def _ask_server(self, text: str):
+        """One turn with the server, streamed when possible.
+
+        Streaming speaks each sentence as it is generated, so the wait is
+        time-to-first-sentence rather than the whole model call. It falls back
+        to the ordinary request/response path when the server has no streaming
+        endpoint, or when a stream dies *before* anything was spoken — after
+        that, retrying would say the first half twice."""
+        if config.STREAMING_REPLIES:
+            try:
+                return server_client.converse_stream(
+                    text, on_say=self._speak_stream_chunk,
+                    on_command=self._handle_command,
+                )
+            except server_client.ServerError as exc:
+                self.log.emit(f"Streaming unavailable ({exc}) — using the plain path.")
+        return server_client.converse(
+            text, on_command=self._handle_command, on_say=self._speak_holding,
+        )
+
     def _play_dialogue(self, scene: dict) -> str:
         """Play a `dialoguectl` scene and report back up the relay.
 
@@ -525,20 +591,8 @@ class PetController(QObject):
             # scene or fix the voice, and try again inside the same turn.
             return f"[dialogue] couldn't synthesize it: {exc}"
 
-        resume = self._state.state
-        self._state.transition(PetState.TALKING)
-        self.said.emit(speech_text.for_display(text))
-        self.history.add(history_mod.PET, text, time.time())
-
-        should_stop = None
-        if self._barge_in is not None:
-            self._barge_in.reset()
-            should_stop = self._barge_in.check
-        completed = tts.play_pcm(pcm, sample_rate, should_stop=should_stop)
-        if self._barge_in is not None:
-            self._barge_in.reset()  # the pet's own voices are in the wake window
-        if resume in (PetState.THINKING, PetState.IDLE):
-            self._state.transition(resume)
+        completed = self._speaker.say_pcm(
+            speech.Utterance.scene(text), pcm=pcm, sample_rate=sample_rate)
 
         if not completed:
             return dialogue_mod.describe(scene) + " (interrupted — they talked over it)"
@@ -569,32 +623,30 @@ class PetController(QObject):
             self.reset_voice()
 
     def _speak(self, text: str) -> None:
-        self._state.transition(PetState.TALKING)
-        # Bubble gets the markdown stripped but emoji kept (it can't render
-        # **bold** but draws emoji fine); tts.speak() does its own, stricter
-        # sanitizing for the voice.
-        self.said.emit(speech_text.for_display(text))
-        self.log.emit(f"Bolt: {text}")
-        self.history.add(history_mod.PET, text, time.time())
+        """The answer: transcript, bubble, and the follow-up rule."""
+        completed = self._speaker.say(speech.Utterance.reply(text))
+        self._after_speaking(text, completed=completed,
+                             detail=self._speaker.last_detail)
 
-        should_stop = None
-        if self._barge_in is not None:
-            self._barge_in.reset()
-            should_stop = self._barge_in.check
-        completed = tts.speak(
-            text,
-            on_error=lambda exc: self.log.emit(f"TTS failed: {exc}"),
-            should_stop=should_stop,
-            voice_id=self._voice_id or None,
-        )
-        # Read the scoring history *before* resetting, or the log reports the
-        # blank counters instead of what actually fired.
-        detail = self._barge_in_detail()
-        if self._barge_in is not None:
-            # Playback fed the pet's own voice into the wake model's rolling
-            # window. Clear it before the idle listener starts scoring again,
-            # or Bolt's last sentence is still in there being re-scored.
-            self._barge_in.reset()
+    def _speak_holding(self, text: str) -> None:
+        """"Give me a sec" while a tool runs — filler, so no transcript, and it
+        resumes the state it interrupted because the turn isn't over."""
+        self._speaker.say(speech.Utterance.holding(text))
+
+    def _speak_stream_chunk(self, text: str) -> None:
+        """One sentence of a streamed answer, spoken the moment it arrives."""
+        completed = self._speaker.say(speech.Utterance.stream_chunk(text))
+        if not completed:
+            self.log.emit("Interrupted — listening.")
+            self._follow_ups = 0
+            self._talk_now.set()
+
+    def _after_speaking(self, text: str, *, completed: bool, detail: str = "") -> None:
+        """What happens once an answer has been said, however it was said.
+
+        Shared by the plain and streamed paths: a streamed reply is spoken
+        sentence by sentence, but it still has to obey the same rules about
+        keeping the mic open when it ended on a question."""
         if not completed:
             # You talked over it — take that as the start of the next turn
             # rather than making you say the wake word again.
@@ -614,20 +666,22 @@ class PetController(QObject):
     def _should_follow_up(self, text: str) -> bool:
         """Whether *text* leaves the pet waiting on an answer.
 
-        Muted is excluded because mute means "don't listen to me" — an
-        automatic turn would walk straight past it. Napping isn't: quiet
-        hours suppress the pet *starting* something, and a question is only
-        ever asked in reply to you."""
-        if not config.FOLLOW_UP_LISTEN or self._muted:
+        The rule itself — question, cap, off switch — is
+        `speech.follow_up_decision`, because it is a rule with an off-by-one in
+        it and deserves a test that needs no audio. What stays here is the part
+        that is genuinely the controller's: mute, and the log line. Muted is
+        excluded because mute means "don't listen to me" and an automatic turn
+        would walk straight past it. Napping isn't: quiet hours suppress the
+        pet *starting* something, and a question is only ever asked in reply to
+        you."""
+        if self._muted:
             return False
-        if not speech_text.is_question(text):
-            return False
-        # Only worth mentioning the cap on a reply that would otherwise have
-        # kept listening, or it fires on every statement the pet makes.
-        if config.FOLLOW_UP_MAX_TURNS > 0 and self._follow_ups >= config.FOLLOW_UP_MAX_TURNS:
+        keep, why = speech.follow_up_decision(text, completed=True, follow_ups=self._follow_ups)
+        if not keep and "cap" in why:
+            # Only worth mentioning the cap on a reply that would otherwise have
+            # kept listening, or it fires on every statement the pet makes.
             self.log.emit("Follow-up limit reached — say the wake word to keep going.")
-            return False
-        return True
+        return keep
 
     def _barge_in_detail(self) -> str:
         """Why the interruption fired, for the log. How far into playback it
@@ -685,33 +739,54 @@ class PetController(QObject):
 
     def _queue_notification(self, notification: notifications.Notification) -> None:
         """Called on the watcher thread — just queue it; forwarding happens on
-        the pipeline thread where it can't collide with a live conversation."""
-        if not self._notification_gate.should_forward(notification, time.monotonic()):
+        the pipeline thread where it can't collide with a live conversation.
+
+        Only the *filter* applies here. The rate limit used to as well, which
+        meant a second message arriving inside the window was silently thrown
+        away — with NOTIFICATION_MIN_INTERVAL_SECONDS=60, two texts a minute
+        apart and you only ever heard about one of them. Losing a message from
+        a person to save a round trip is the wrong trade; they are batched at
+        the far end instead, which costs the same one round trip and keeps
+        them all."""
+        if not self._notification_gate.matches(notification):
             return
         with self._notification_lock:
+            if len(self._pending_notifications) >= _MAX_PENDING_NOTIFICATIONS:
+                self._pending_notifications.pop(0)  # bound it; oldest goes first
             self._pending_notifications.append(notification)
 
     def _drain_notifications(self) -> None:
+        """Forward everything waiting as ONE turn.
+
+        Batching is what makes it safe to keep every notification: five that
+        arrived while the pet was mid-conversation become one message and one
+        round trip, instead of five separate interruptions queued up to fire
+        back to back."""
         with self._notification_lock:
             pending, self._pending_notifications = self._pending_notifications, []
+        if not pending or not self._running or self._napping:
+            return
         for notification in pending:
-            if not self._running or self._napping:
-                return
             self.log.emit(f"Notification: {notification.as_text()}")
             self.history.add(history_mod.SYSTEM, notification.as_text(), time.time())
-            try:
-                reply = server_client.converse(
-                    f"[desktop notification] {notification.as_text()}",
-                    on_command=self._handle_command,
-                )
-            except server_client.ServerError as exc:
-                self.log.emit(f"Couldn't forward notification: {exc}")
-                return
-            self._check_deliveries()
-            self._apply_voice(reply)
-            if reply.text.strip():
-                self._speak(reply.text)
-                self._state.transition(PetState.IDLE)
+
+        if len(pending) == 1:
+            message = f"[desktop notification] {pending[0].as_text()}"
+        else:
+            lines = "\n".join(f"- {n.as_text()}" for n in pending)
+            message = f"[{len(pending)} desktop notifications]\n{lines}"
+        try:
+            reply = server_client.converse(
+                message, on_command=self._handle_command, on_say=self._speak_holding,
+            )
+        except server_client.ServerError as exc:
+            self.log.emit(f"Couldn't forward notification: {exc}")
+            return
+        self._check_deliveries()
+        self._apply_voice(reply)
+        if reply.text.strip() and not reply.spoken:
+            self._speak(reply.text)
+        self._state.transition(PetState.IDLE)
 
     # ── file delivery ────────────────────────────────────────────────────
 
@@ -784,20 +859,36 @@ class PetController(QObject):
     # ── heartbeat ────────────────────────────────────────────────────────
 
     def _maybe_heartbeat(self) -> None:
+        """Called from the wake listener's tick (~every WAKE_CHECK_INTERVAL_SECONDS).
+
+        Two different cadences live here, and conflating them was costing
+        minutes. A *notification* is an event that already happened — it should
+        go out as soon as the pet is free, which is the next tick. The
+        *heartbeat* is a poll, and polling the server every 1.2s would be
+        absurd, so it stays on its own interval."""
         self._refresh_nap_state()
         self._maybe_update()
         if self._update_pending:
             return  # on the way out — don't start a conversation now
+
+        # Notifications: every tick, not every heartbeat.
+        if self._state.state == PetState.IDLE and not self._napping:
+            self._drain_notifications()
+
         now = time.monotonic()
         if now - self._last_heartbeat < config.HEARTBEAT_INTERVAL_SECONDS:
             return
-        self._last_heartbeat = now
         if self._state.state != PetState.IDLE:
+            # Mid-conversation. Do NOT stamp the clock — an earlier version
+            # did, so a heartbeat that landed while the pet was talking burned
+            # its slot and waited another full interval. With follow-up
+            # listening that could repeat for several cycles, which is why a
+            # notification could sit unspoken for five or ten minutes.
             return
         if self._napping:
             return  # quiet hours: still answers when spoken to, just doesn't start
+        self._last_heartbeat = now
         self._check_deliveries()
-        self._drain_notifications()
         if self._state.state != PetState.IDLE:
             return
         try:

bolt_pet/dialogue.py

@@ -224,4 +224,12 @@ def describe(action: dict, *, played: bool = True) -> str:
     return (
         f"[dialogue] played {len(lines)} line{'s' if len(lines) != 1 else ''} "
         f"in {len(voices)} voice{'s' if len(voices) != 1 else ''}: {', '.join(voices)}{note}"
+        # The scene was spoken out loud before this result got back to the
+        # server, and the model has no other way to know that. Without saying
+        # so it writes a final reply summarising what the user just heard, and
+        # the pet says the same thing twice in a row (observed 2026-07-31).
+        # An instruction delivered here, at the moment it applies, lands far
+        # better than a rule buried in a long system prompt.
+        "\nThe user HEARD this already. Do not repeat, summarise or narrate it "
+        "in your reply — answer with at most one short line, or nothing new."
     )

bolt_pet/doctor.py

@@ -0,0 +1,272 @@
+"""`python -m bolt_pet --doctor` — is this install actually going to work?
+
+Written after a week of debugging things a preflight would have shown in one
+command: a missing port publish, a wrong reverse-proxy header, a venv whose
+python was a zero-byte file, an OCR engine that was never installed. Every one
+of those presented as "the pet is being weird" and took a conversation to find.
+
+Each check is independent and reports one of three things — ok, a warning
+(works, but degraded), or a failure (this will not do what you expect) — and
+says *what to do about it* rather than just what is wrong. Nothing here raises:
+a doctor that crashes on a broken install is diagnosing the wrong patient.
+
+Deliberately does not open the mic or call a paid API by default. It checks
+that the door is unlocked, not that the room is furnished; `--deep` is there
+when you want it to actually knock.
+"""
+
+from __future__ import annotations
+
+import importlib
+import shutil
+import sys
+from dataclasses import dataclass
+from typing import Callable, Optional
+
+from . import config
+
+OK, WARN, FAIL = "ok", "warn", "fail"
+
+_MARKS = {OK: "  ok  ", WARN: " warn ", FAIL: " FAIL "}
+
+
+@dataclass
+class Check:
+    name: str
+    status: str
+    detail: str = ""
+    fix: str = ""
+
+    def line(self) -> str:
+        text = f"[{_MARKS[self.status]}] {self.name:22} {self.detail}"
+        if self.fix and self.status != OK:
+            text += f"\n{'':32}→ {self.fix}"
+        return text
+
+
+def _module(name: str) -> bool:
+    try:
+        importlib.import_module(name)
+        return True
+    except Exception:
+        return False
+
+
+# ── the checks ──────────────────────────────────────────────────────────────
+
+def check_config() -> Check:
+    missing = config.missing_config()
+    if missing:
+        return Check("server config", FAIL, f"missing {', '.join(missing)}",
+                     "set them in .env — without these the controller exits at startup")
+    return Check("server config", OK, f"{config.SERVER_URL} as {config.SESSION_ID}")
+
+
+def check_server(deep: bool = False) -> Check:
+    if not config.SERVER_URL:
+        return Check("server", FAIL, "no BOLT_SERVER_URL",
+                     "cp .env.example .env and set BOLT_SERVER_URL to your Bolt server")
+    if not deep:
+        return Check("server", OK, f"{config.SERVER_URL} (not contacted; --deep to try)")
+    try:
+        from . import server_client
+
+        health = server_client.check_health(timeout=8)
+        return Check("server", OK, f"reachable — {health}")
+    except Exception as exc:
+        return Check("server", FAIL, f"unreachable: {exc}",
+                     "check the URL, the key, and that the container is up")
+
+
+def check_streaming_endpoint(deep: bool = False) -> Check:
+    """The streamed-reply endpoint is newer than some deployed servers."""
+    if not config.STREAMING_REPLIES:
+        return Check("streamed replies", WARN, "disabled (STREAMING_REPLIES=false)")
+    if not deep:
+        return Check("streamed replies", OK, "enabled (endpoint not probed)")
+    try:
+        import requests
+
+        response = requests.post(
+            f"{config.SERVER_URL}/desk/converse_stream",
+            json={"session_id": config.SESSION_ID, "text": ""},
+            headers={"X-Desk-Api-Key": config.API_KEY}, timeout=8, stream=True,
+        )
+        if response.status_code == 404:
+            return Check("streamed replies", WARN, "server has no /desk/converse_stream",
+                         "update the server, or set STREAMING_REPLIES=false to skip the probe")
+        return Check("streamed replies", OK, f"endpoint answered {response.status_code}")
+    except Exception as exc:
+        return Check("streamed replies", WARN, f"probe failed: {exc}")
+
+
+def check_microphone(deep: bool = False) -> Check:
+    if not _module("sounddevice"):
+        return Check("microphone", FAIL, "sounddevice is not installed",
+                     "pip install -r requirements.txt")
+    try:
+        import sounddevice as sd
+
+        devices = [d for d in sd.query_devices() if d.get("max_input_channels", 0) > 0]
+        if not devices:
+            return Check("microphone", FAIL, "no input devices",
+                         "on Linux check PipeWire/PulseAudio is running as your user")
+        chosen = config.MIC_DEVICE or "system default"
+        if not deep:
+            return Check("microphone", OK, f"{len(devices)} input device(s), using {chosen}")
+        with sd.InputStream(samplerate=config.SAMPLE_RATE, channels=1, dtype="int16",
+                            device=config.MIC_DEVICE, blocksize=config.FRAME_LEN):
+            pass
+        return Check("microphone", OK, f"opened at {config.SAMPLE_RATE} Hz ({chosen})")
+    except Exception as exc:
+        return Check("microphone", FAIL, f"could not open: {exc}",
+                     "don't run the pet as root — PortAudio can't reach your PipeWire socket")
+
+
+def check_wake_model() -> Check:
+    from pathlib import Path
+
+    path = Path(config.WAKE_MODEL_PATH)
+    if not path.exists():
+        return Check("wake word", FAIL, f"{path.name} is missing",
+                     "it ships in the project root; check WAKE_MODEL_FILE")
+    if not _module("openwakeword"):
+        return Check("wake word", FAIL, "openwakeword is not installed",
+                     "pip install -r requirements.txt")
+    return Check("wake word", OK, f"{path.name}, threshold {config.WAKE_WORD_THRESHOLD}")
+
+
+def check_stt() -> Check:
+    if not config.DEEPGRAM_API_KEY:
+        return Check("speech-to-text", FAIL, "no DEEPGRAM_API_KEY",
+                     "set it in .env — nothing you say can be transcribed without it")
+    if config.STT_STREAMING and not _module("websocket"):
+        return Check("speech-to-text", WARN, "streaming on, but websocket-client is missing",
+                     "pip install websocket-client — it falls back to one-shot uploads")
+    mode = "streaming" if config.STT_STREAMING else "one-shot"
+    return Check("speech-to-text", OK, f"Deepgram {config.DEEPGRAM_MODEL}, {mode}")
+
+
+def check_tts() -> Check:
+    if not (config.ELEVENLABS_API_KEY and config.ELEVENLABS_VOICE_ID):
+        if _module("pyttsx3"):
+            return Check("text-to-speech", WARN, "no ElevenLabs key/voice — offline voice only",
+                         "set ELEVENLABS_API_KEY and ELEVENLABS_VOICE_ID for the real voice")
+        return Check("text-to-speech", FAIL, "no ElevenLabs config and no pyttsx3 fallback")
+    return Check("text-to-speech", OK,
+                 f"ElevenLabs {config.ELEVENLABS_MODEL_ID}, voice …{config.ELEVENLABS_VOICE_ID[-6:]}")
+
+
+def check_dialogue() -> Check:
+    if not config.DIALOGUE:
+        return Check("multi-voice scenes", WARN, "disabled (DIALOGUE=false)")
+    from . import dialogue
+
+    cast = dialogue.parse_voice_map(config.DIALOGUE_VOICES)
+    if not cast:
+        return Check("multi-voice scenes", WARN, "no cast configured — only 'self' works",
+                     'set DIALOGUE_VOICES=narrator:<id>,villain:<id>')
+    return Check("multi-voice scenes", OK, f"{len(cast)} voice(s): {', '.join(sorted(cast))}")
+
+
+def check_screen_text() -> Check:
+    if not config.SCREEN_TEXT:
+        return Check("screen reading", WARN, "disabled (SCREEN_TEXT=false)")
+    if not _module("mss"):
+        return Check("screen reading", WARN, "mss is not installed — petctl read will decline",
+                     "pip install mss (and note it cannot capture on Wayland)")
+    engine = "pytesseract" if _module("pytesseract") else (
+        "rapidocr" if _module("rapidocr_onnxruntime") else "")
+    if not engine:
+        return Check("screen reading", WARN, "no OCR engine",
+                     "pip install pytesseract && apt install tesseract-ocr, "
+                     "or pip install rapidocr-onnxruntime")
+    if engine == "pytesseract" and not shutil.which("tesseract"):
+        return Check("screen reading", WARN, "pytesseract is installed but tesseract is not",
+                     "apt install tesseract-ocr")
+    return Check("screen reading", OK, f"mss + {engine}")
+
+
+def check_hotkey() -> Check:
+    if not _module("pynput"):
+        return Check("push-to-talk", WARN, "pynput is not installed",
+                     "the wake word still works; pip install pynput for the hotkey")
+    import os
+
+    if os.environ.get("WAYLAND_DISPLAY") and not os.environ.get("DISPLAY"):
+        return Check("push-to-talk", WARN, "Wayland session — global hotkeys usually blocked",
+                     "use the wake word, or click the pet")
+    return Check("push-to-talk", OK, config.PUSH_TO_TALK_HOTKEY)
+
+
+def check_sprites() -> Check:
+    from pathlib import Path
+
+    root = Path(__file__).resolve().parent / "assets" / "sprites"
+    if not root.exists():
+        return Check("sprites", WARN, "no art — the placeholder blob will be drawn",
+                     "python scripts/generate_bolt_sprites.py")
+    counts = {d.name: len(list(d.glob("*.png"))) for d in sorted(root.iterdir()) if d.is_dir()}
+    empty = [name for name, count in counts.items() if count == 0]
+    if empty:
+        return Check("sprites", WARN, f"no frames for: {', '.join(empty)}",
+                     "python scripts/generate_bolt_sprites.py")
+    return Check("sprites", OK, ", ".join(f"{n} {c}" for n, c in counts.items()))
+
+
+def check_latency() -> Check:
+    """The setting most likely to make it feel slow, and the least obvious."""
+    silence = config.SILENCE_END_SEC
+    if silence >= 1.5:
+        return Check("turn latency", WARN, f"VAD_SILENCE_END_SEC={silence:g}s of dead air per turn",
+                     "0.8-1.0 feels markedly snappier; it is pure wait before anything starts")
+    return Check("turn latency", OK, f"silence timeout {silence:g}s, "
+                 f"streaming {'on' if config.STREAMING_REPLIES else 'off'}")
+
+
+CHECKS: tuple[tuple[str, Callable], ...] = (
+    ("config", check_config),
+    ("server", check_server),
+    ("stream", check_streaming_endpoint),
+    ("mic", check_microphone),
+    ("wake", check_wake_model),
+    ("stt", check_stt),
+    ("tts", check_tts),
+    ("dialogue", check_dialogue),
+    ("screen", check_screen_text),
+    ("hotkey", check_hotkey),
+    ("sprites", check_sprites),
+    ("latency", check_latency),
+)
+
+
+def run(deep: bool = False) -> list[Check]:
+    results = []
+    for _name, check in CHECKS:
+        try:
+            try:
+                results.append(check(deep))
+            except TypeError:
+                results.append(check())
+        except Exception as exc:  # a broken check must not hide the others
+            results.append(Check(_name, FAIL, f"the check itself failed: {exc}"))
+    return results
+
+
+def main(argv: Optional[list] = None) -> int:
+    argv = list(argv if argv is not None else sys.argv[1:])
+    deep = "--deep" in argv
+    print(f"Bolt pet preflight{' (deep: contacting the server and opening the mic)' if deep else ''}\n")
+    results = run(deep=deep)
+    for check in results:
+        print(check.line())
+    failures = [c for c in results if c.status == FAIL]
+    warnings = [c for c in results if c.status == WARN]
+    print()
+    if failures:
+        print(f"{len(failures)} problem(s) will stop this working. Fix those first.")
+    elif warnings:
+        print(f"Ready. {len(warnings)} thing(s) degraded but working.")
+    else:
+        print("Everything checks out.")
+    return 1 if failures else 0

bolt_pet/server_client.py

@@ -19,6 +19,7 @@ Kept dependency-free beyond `requests` so it's easy to unit test with mocks.
 
 from __future__ import annotations
 
+import json
 import subprocess
 from pathlib import Path
 from typing import Callable, NamedTuple, Optional
@@ -43,6 +44,10 @@ class Reply(NamedTuple):
     text: str
     voice_id: str = ""
     voice_name: str = ""
+    # True when the sentences were already spoken as they streamed in. The
+    # text is still carried — the follow-up rule needs to see whether the
+    # answer ended on a question — it just must not be read out again.
+    spoken: bool = False
 
 
 def _headers() -> dict:
@@ -90,12 +95,20 @@ def converse(
     text: str,
     on_command: Callable[[str], str] = run_local_command,
     timeout: float = 120.0,
+    on_say: Optional[Callable[[str], None]] = None,
 ) -> Reply:
     """Send one turn of conversation to the desk API, relaying any commands
     the server sends back until it produces a final reply.
 
     *on_command* is injectable for tests; defaults to actually running the
     command locally (matching bolt_desk.py's behavior).
+
+    *on_say* is called with a short holding line ("give me a sec") when the
+    server sends one alongside a command. It is the difference between silence
+    and an answer while a tool runs: the model's acknowledgement used to be
+    discarded server-side, so the whole round trip was dead air and the model
+    then repeated itself in the final reply. Optional, so an older pet against
+    a newer server simply stays quiet as before.
     """
     headers = _headers()
     try:
@@ -111,6 +124,13 @@ def converse(
     for _ in range(_MAX_RELAY_HOPS):
         if payload.get("type") != "command":
             break
+        holding = str(payload.get("say") or "").strip()
+        if holding and on_say is not None:
+            # Spoken *before* the command runs — that is the whole point.
+            try:
+                on_say(holding)
+            except Exception:
+                pass  # a failed acknowledgement must not cost the tool call
         output = on_command(str(payload.get("command") or ""))
         try:
             response = requests.post(
@@ -135,6 +155,104 @@ def converse(
     raise ServerError(str(payload.get("error") or "unknown server response"))
 
 
+def converse_stream(
+    text: str,
+    on_say: Callable[[str], None],
+    on_command: Callable[[str], str] = run_local_command,
+    timeout: float = 180.0,
+) -> Reply:
+    """Same turn as converse(), but speaking each sentence as it arrives.
+
+    Without this the pet waits out the *entire* model call before a single
+    word is heard; with it the wait is time-to-first-sentence, which on a
+    multi-sentence answer is most of the difference.
+
+    Falls back by raising ServerError before anything has been spoken — the
+    caller then retries the ordinary path and the user never finds out. Once a
+    sentence *has* been spoken there is no going back, so late failures end the
+    turn with whatever was said rather than repeating it.
+    """
+    spoke_anything = False
+    try:
+        response = requests.post(
+            f"{config.SERVER_URL}/desk/converse_stream",
+            json={"session_id": config.SESSION_ID, "text": text},
+            headers=_headers(), timeout=timeout, stream=True,
+        )
+        response.raise_for_status()
+        for raw in response.iter_lines(decode_unicode=True):
+            if not raw:
+                continue
+            try:
+                event = json.loads(raw)
+            except (TypeError, ValueError):
+                continue
+            kind = str(event.get("type") or "")
+
+            if kind == "say":
+                line = str(event.get("text") or "").strip()
+                if line:
+                    spoke_anything = True
+                    on_say(line)
+            elif kind == "command":
+                # The tool loop is request/response, so the rest of the turn
+                # finishes through the ordinary relay rather than inside the
+                # stream — one protocol for tools, not two.
+                response.close()
+                return _finish_relay(event, on_command, on_say)
+            elif kind == "reply":
+                return Reply(
+                    text=str(event.get("text") or ""),
+                    voice_id=str(event.get("voice_id") or ""),
+                    voice_name=str(event.get("voice_name") or ""),
+                    spoken=bool(event.get("already_spoken")),
+                )
+            elif kind == "error":
+                raise ServerError(str(event.get("error") or "stream failed"))
+    except ServerError:
+        raise
+    except Exception as exc:
+        if spoke_anything:
+            # Half a reply is out loud already; ending quietly beats saying it
+            # all again through the fallback path.
+            return Reply(text="", spoken=True)
+        raise ServerError(f"streaming failed: {exc}") from exc
+    raise ServerError("stream ended without a reply")
+
+
+def _finish_relay(event: dict, on_command, on_say) -> Reply:
+    """Run the tool the stream handed over, then continue the classic relay."""
+    payload = dict(event)
+    headers = _headers()
+    for _ in range(_MAX_RELAY_HOPS):
+        if payload.get("type") != "command":
+            break
+        holding = str(payload.get("say") or "").strip()
+        if holding and on_say is not None:
+            try:
+                on_say(holding)
+            except Exception:
+                pass
+        output = on_command(str(payload.get("command") or ""))
+        try:
+            response = requests.post(
+                f"{config.SERVER_URL}/desk/tool_result",
+                json={"session_id": config.SESSION_ID,
+                      "token": payload.get("token"), "output": output},
+                headers=headers, timeout=180,
+            )
+            payload = response.json()
+        except Exception as exc:
+            raise ServerError(f"couldn't reach the server during tool relay: {exc}") from exc
+    if payload.get("type") == "reply":
+        return Reply(
+            text=str(payload.get("text") or ""),
+            voice_id=str(payload.get("voice_id") or ""),
+            voice_name=str(payload.get("voice_name") or ""),
+        )
+    raise ServerError(str(payload.get("error") or "unknown server response"))
+
+
 def list_outbox_files(timeout: float = 15.0) -> list:
     """Files the server has queued for this session via its deliver_files
     tool (e.g. "send me that report" during a conversation) — each entry has

bolt_pet/speech.py

@@ -0,0 +1,194 @@
+"""Everything the pet says, and the policy differences between kinds of saying.
+
+There used to be four of these in controller.py — a reply, a holding line, a
+streamed sentence, a dialogue scene — each written when its feature was built,
+each repeating the same dance: transition state, show the bubble, maybe record
+history, reset barge-in, call TTS, reset barge-in again, resume state, decide
+whether to keep the mic open. Only the *policy* differed, and the copies had
+already started to drift: one forgot to arm barge-in, another logged a
+different prefix, a third skipped the follow-up rule.
+
+So the dance lives here once, and the differences are data:
+
+    reply        the answer. Transcript, bubble, follow-up rule, ends IDLE.
+    holding      "give me a sec" while a tool runs. No transcript — it is
+                 filler, and the transcript should keep the answer. Resumes
+                 whatever state it interrupted, because the turn isn't over.
+    stream       one sentence of a streamed answer. Transcript and bubble like
+                 a reply, but stays TALKING so the sprite doesn't flicker
+                 between sentences, and the follow-up rule waits for the last.
+    scene        a dialoguectl take. Transcript (the user heard it), resumes
+                 mid-turn like a holding line.
+
+The controller keeps the pipeline; this keeps the rules about talking.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Callable, Optional
+
+from . import config, speech_text
+from .state import PetState
+
+
+@dataclass(frozen=True)
+class Utterance:
+    """One thing to say, and how saying it should behave."""
+
+    text: str
+    record: bool = True         # goes in the transcript, or is it filler?
+    resume: bool = False        # return to the state it interrupted (mid-turn)
+    hold_talking: bool = False  # stay TALKING afterwards (more is coming)
+    follow_up: bool = True      # may leave the mic open if it ends on a question
+    interruptible: bool = True  # arm barge-in for this one
+    log_prefix: str = "Bolt"
+
+    @classmethod
+    def reply(cls, text: str) -> "Utterance":
+        return cls(text)
+
+    @classmethod
+    def holding(cls, text: str) -> "Utterance":
+        # Filler: no transcript, no follow-up, and not interruptible — cutting
+        # off "give me a sec" would strand the tool that is already running.
+        return cls(text, record=False, resume=True, follow_up=False,
+                   interruptible=False, log_prefix="Bolt (holding)")
+
+    @classmethod
+    def stream_chunk(cls, text: str) -> "Utterance":
+        return cls(text, hold_talking=True, follow_up=False)
+
+    @classmethod
+    def scene(cls, text: str) -> "Utterance":
+        return cls(text, resume=True, follow_up=False)
+
+
+class Speaker:
+    """Says things on behalf of the controller.
+
+    Collaborators are passed in rather than reached for, so the whole of
+    speaking is testable without Qt, audio hardware or a state machine: hand it
+    fakes and assert on what came out."""
+
+    def __init__(self, *, state, tts, history=None, on_said=None, on_log=None,
+                 barge_in: Callable[[], object] = lambda: None,
+                 voice_id: Callable[[], str] = lambda: "",
+                 detail_of: Callable[[], str] = lambda: "",
+                 on_level: Optional[Callable[[float], None]] = None):
+        self._state = state
+        self._tts = tts
+        self._history = history
+        self._on_said = on_said or (lambda _text: None)
+        self._on_log = on_log or (lambda _msg: None)
+        # Read through a callable rather than held: the detector is built after
+        # the speaker (it needs the mic stream), replaced when barge-in mode
+        # changes, and swapped by tests. Two copies of it drifting apart is a
+        # bug nobody notices until the wake model starts hearing the pet.
+        self._barge_in_of = barge_in
+        self._voice_id = voice_id
+        # What actually fired, captured BEFORE the post-playback reset. Read it
+        # afterwards and every interruption reports 0.000 at frame 0 — which
+        # looks like hard evidence and is nothing of the sort.
+        self._detail_of = detail_of
+        self.last_detail = ""
+        self._on_level = on_level
+
+    @property
+    def _barge_in(self):
+        try:
+            return self._barge_in_of()
+        except Exception:
+            return None
+
+    def say(self, utterance: Utterance) -> bool:
+        """Speak it. Returns False if it was interrupted.
+
+        The one place that knows the order these steps go in — which is the
+        point, because getting that order wrong is invisible until the wake
+        model starts hearing the pet's own voice."""
+        line = speech_text.for_display(utterance.text)
+        if not line:
+            return True
+
+        resume_state = self._state.state
+        if self._state.state != PetState.TALKING:
+            self._state.transition(PetState.TALKING)
+        self._on_said(line)
+        self._on_log(f"{utterance.log_prefix}: {line}")
+        if utterance.record and self._history is not None:
+            self._history(utterance.text)
+
+        should_stop = None
+        if self._barge_in is not None and utterance.interruptible:
+            self._barge_in.reset()
+            should_stop = self._barge_in.check
+
+        completed = self._tts.speak(
+            utterance.text,
+            on_error=lambda exc: self._on_log(f"TTS failed: {exc}"),
+            should_stop=should_stop,
+            voice_id=self._voice_id() or None,
+            on_level=self._on_level,
+        )
+        self.last_detail = self._detail_of()
+        if self._barge_in is not None:
+            # Playback fed the pet's own voice into the wake model's rolling
+            # window. Clear it before the idle listener scores again, or the
+            # last sentence is still in there being re-heard.
+            self._barge_in.reset()
+        if self._on_level is not None:
+            self._on_level(0.0)  # mouth closed; nothing is playing now
+
+        if utterance.resume and resume_state in (PetState.THINKING, PetState.IDLE):
+            self._state.transition(resume_state)
+        return bool(completed)
+
+
+    def say_pcm(self, utterance: Utterance, *, pcm, sample_rate: int) -> bool:
+        """Speak audio that is already synthesized — a dialoguectl scene.
+
+        Same policy, same barge-in handling, same mouth; only the source of
+        the samples differs. Sharing this is why a scene can be talked over
+        exactly like an ordinary reply."""
+        line = speech_text.for_display(utterance.text)
+        resume_state = self._state.state
+        if self._state.state != PetState.TALKING:
+            self._state.transition(PetState.TALKING)
+        if line:
+            self._on_said(line)
+        self._on_log(f"{utterance.log_prefix}: {line}")
+        if utterance.record and self._history is not None:
+            self._history(utterance.text)
+
+        should_stop = None
+        if self._barge_in is not None and utterance.interruptible:
+            self._barge_in.reset()
+            should_stop = self._barge_in.check
+        completed = self._tts.play_pcm(
+            pcm, sample_rate, should_stop=should_stop, on_level=self._on_level)
+        self.last_detail = self._detail_of()
+        if self._barge_in is not None:
+            self._barge_in.reset()
+        if self._on_level is not None:
+            self._on_level(0.0)
+        if utterance.resume and resume_state in (PetState.THINKING, PetState.IDLE):
+            self._state.transition(resume_state)
+        return bool(completed)
+
+
+def follow_up_decision(text: str, *, completed: bool, follow_ups: int) -> tuple[bool, str]:
+    """Whether to keep listening after speaking, and why.
+
+    Split out as a pure function because it is a *rule* with an off-by-one cap
+    in it, and rules with counters deserve a test that doesn't need audio."""
+    if not completed:
+        return True, "interrupted"
+    if not speech_text.is_question(text):
+        return False, ""
+    cap = config.FOLLOW_UP_MAX_TURNS
+    if not config.FOLLOW_UP_LISTEN:
+        return False, ""
+    if cap > 0 and follow_ups >= cap:
+        return False, f"follow-up cap ({cap}) reached"
+    return True, "question"

bolt_pet/speech_text.py

@@ -97,10 +97,34 @@ def _bullets_to_sentences(text: str) -> str:
     return " ".join(line if line[-1] in ".!?:,;" else line + "." for line in lines)
 
 
+# ElevenLabs v3 delivery tags — "[laughing]", "[whispering]", "[sighs]". They
+# are instructions to the *dialogue* model (see dialogue.py), and only inside a
+# dialoguectl scene. Left in an ordinary reply they reach eleven_flash_v2,
+# which has no idea what they mean and simply reads the word: observed live
+# 2026-07-31, the pet announcing "laughing That one came through clean".
+#
+# Matched narrowly — a bracketed adverb/gerund, or one of the noise words that
+# aren't either — so real bracketed text ("[1]", "[see the docs]") survives.
+_AUDIO_TAG_RE = re.compile(
+    r"\[\s*(?:"
+    r"[a-z]+(?:ly|ing)"
+    r"|laughs?|chuckles?|sighs?|exhales?|inhales?|gulps?|pause|beat"
+    r"|clears throat|shouts?|whispers?|cries|sings?|gasps?|snorts?"
+    r"|sarcastic|excited|curious|nervous|angry|sad|happy|deadpan|flat"
+    r")\s*\]",
+    re.IGNORECASE,
+)
+
+
+def strip_audio_tags(text: str) -> str:
+    """Remove v3 delivery tags from text destined for the ordinary voice."""
+    return _AUDIO_TAG_RE.sub(" ", str(text or ""))
+
+
 def for_speech(text: str) -> str:
     """Plain prose for the TTS engine: no markdown, no emoji, no bare URLs,
     no stray symbols that would be read out character by character."""
-    text = (text or "").strip()
+    text = strip_audio_tags((text or "").strip())
     if not text:
         return ""
     for symbol, spoken in _PRE_SPOKEN_SYMBOLS.items():
@@ -136,8 +160,11 @@ def is_question(text: str) -> bool:
 
 def for_display(text: str) -> str:
     """What the speech bubble shows: markdown syntax removed (the bubble
-    can't render it) but emoji and layout-ish punctuation left alone."""
-    text = (text or "").strip()
+    can't render it) but emoji and layout-ish punctuation left alone.
+
+    Delivery tags go too — the voice no longer says them, so showing them
+    would caption a laugh nobody heard."""
+    text = strip_audio_tags((text or "").strip())
     if not text:
         return ""
     text = _strip_markdown(text, keep_emoji=True)

bolt_pet/ui/app.py

@@ -41,6 +41,8 @@ def run() -> int:
     thread.started.connect(controller.run)
     controller.state_changed.connect(lambda value: window.set_state(PetState(value)))
     controller.said.connect(window.say)
+    # Lip-sync: the loudness of the audio actually going to the speakers.
+    controller.mouth.connect(window.set_mouth)
     controller.log.connect(_log)
     controller.action.connect(window.apply_action)  # petctl move/emote/say/...
     controller.finished.connect(thread.quit)

bolt_pet/ui/pet_window.py

@@ -19,7 +19,7 @@ from PySide6.QtGui import (
 )
 from PySide6.QtWidgets import QApplication, QWidget
 
-from .. import config
+from .. import config, window_state
 from ..monitors import Monitor
 from ..state import PetState
 from .sprite import WALK, SpriteSet
@@ -36,6 +36,11 @@ _NAP_OPACITY = 0.35
 # and the feet skate whenever PET_WANDER_SPEED doesn't happen to match the fps.
 # Eight frames at 13px is a ~104px stride cycle, a bit under the pet's width.
 _WALK_PIXELS_PER_FRAME = 13.0
+# How long a loudness level stays believable. The audio thread sends one per
+# ~30ms while a clip plays; if they stop arriving (offline TTS has no envelope,
+# or playback died) the mouth must not stay frozen mid-syllable, so after this
+# long the ordinary looping animation takes back over.
+_MOUTH_STALE_SECONDS = 0.35
 
 
 def emote_transform(emote: str, progress: float) -> tuple[float, float, float, float]:
@@ -213,6 +218,9 @@ class PetWindow(QWidget):
         self._commanded_move = False  # a petctl move — happens even mid-conversation
         self._next_wander_at = 0.0
         self._bob_offset = 0
+        # Lip-sync: loudness of what is playing right now, and when it arrived.
+        self._mouth_level: Optional[float] = None
+        self._mouth_at = 0.0
         self._bob_phase = 0.0
         self._walking = False
         self._facing = 1  # +1 right, -1 left; the walk art is drawn facing right
@@ -249,6 +257,19 @@ class PetWindow(QWidget):
             y = int(config.PET_START_Y) if config.PET_START_Y else None
         except ValueError:
             x = y = None
+        if x is None and y is None and config.PET_REMEMBER_POSITION:
+            remembered = window_state.load()
+            # Only if it still lands on a screen that exists — the usual reason
+            # a saved position is stale is that the monitor it was on has been
+            # unplugged, and restoring it faithfully would hide the pet.
+            if remembered is not None:
+                rectangles = [
+                    (s.availableGeometry().left(), s.availableGeometry().top(),
+                     s.availableGeometry().right(), s.availableGeometry().bottom())
+                    for s in QApplication.screens()
+                ]
+                if window_state.is_visible_on(*remembered, self.width(), rectangles):
+                    x, y = remembered
         if geo is not None:
             x = geo.right() - self.width() - 40 if x is None else x
             y = geo.bottom() - self.height() - 60 if y is None else y
@@ -688,6 +709,34 @@ class PetWindow(QWidget):
 
     # ── animation ────────────────────────────────────────────────────────
 
+    def set_mouth(self, level: float) -> None:
+        """How loud the pet is *right now* (0..1), straight off the PCM going
+        to the speakers (audio/tts.level_of).
+
+        The talking frames are ordered by mouth openness, so this indexes them
+        directly: the mouth moves with the actual waveform instead of flapping
+        on a timer, which is the difference between a talking sprite and a
+        dubbed one."""
+        self._mouth_level = max(0.0, min(1.0, float(level)))
+        self._mouth_at = time.monotonic()
+        if self._current_state == PetState.TALKING:
+            self.update()
+
+    def _mouth_frame(self, animation) -> Optional[QPixmap]:
+        """The frame matching the current loudness, or None to use the timer.
+
+        Falls back the moment the levels go stale — offline TTS has no
+        envelope, and a mouth frozen mid-syllable is worse than a timed loop."""
+        if self._mouth_level is None or animation is None:
+            return None
+        if time.monotonic() - self._mouth_at > _MOUTH_STALE_SECONDS:
+            return None
+        frames = animation.frames
+        if len(frames) < 2:
+            return None
+        index = int(round(self._mouth_level * (len(frames) - 1)))
+        return frames[max(0, min(len(frames) - 1, index))]
+
     def _advance_frame(self) -> None:
         # While walking the cycle is stepped by _advance_walk from distance
         # travelled; letting this timer also advance it would double-step it
@@ -702,7 +751,12 @@ class PetWindow(QWidget):
         painter.setRenderHint(QPainter.Antialiasing)
         painter.setRenderHint(QPainter.SmoothPixmapTransform)
         key = self._animation_key()
-        pixmap: Optional[QPixmap] = self.sprites.get(key).current()
+        animation = self.sprites.get(key)
+        pixmap: Optional[QPixmap] = None
+        if key == PetState.TALKING:
+            pixmap = self._mouth_frame(animation)
+        if pixmap is None:
+            pixmap = animation.current()
         if key == WALK:
             pixmap = self._oriented(pixmap)
         if pixmap is None:
@@ -759,6 +813,9 @@ class PetWindow(QWidget):
         was_click = not self._dragged
         self._drag_offset = None
         self._press_pos = None
+        if not was_click and config.PET_REMEMBER_POSITION:
+            position = self.geometry().topLeft()
+            window_state.save(position.x(), position.y())
         if was_click:
             self.talk_requested.emit()
         else:

bolt_pet/window_state.py

@@ -0,0 +1,75 @@
+"""Where the pet was left, so it starts there next time.
+
+Listed in the README as a known limitation: drag it somewhere deliberate, and
+the next launch puts it back in the bottom-right corner. For something that
+lives on your desktop all day that is a small daily annoyance, and it is a
+config write on drag-end.
+
+Lives in the cache dir rather than the repo, next to the restart context: it
+is per-machine state about this install, not something that belongs in a git
+diff. Every function swallows its own errors — a corrupt or unwritable state
+file must never stop the pet from starting, it just means the default corner.
+
+Positions are validated against the *current* screen layout on load, because
+the common case for a stale position is exactly the case where it is
+dangerous: the pet was last on a monitor that is now unplugged, and restoring
+it faithfully would put it somewhere you cannot see or reach.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import os
+import tempfile
+from pathlib import Path
+from typing import Optional
+
+logger = logging.getLogger("bolt_pet.window_state")
+
+DEFAULT_PATH = Path.home() / ".cache" / "bolt-pet" / "window.json"
+
+
+def load(path: Optional[Path] = None) -> Optional[tuple[int, int]]:
+    """The saved position, or None if there isn't a usable one."""
+    try:
+        data = json.loads(Path(path or DEFAULT_PATH).read_text(encoding="utf-8"))
+        return int(data["x"]), int(data["y"])
+    except (FileNotFoundError, json.JSONDecodeError, KeyError, TypeError, ValueError, OSError):
+        return None
+
+
+def save(x: int, y: int, path: Optional[Path] = None) -> None:
+    """Remember where it is now. Atomic, so a crash mid-write can't leave a
+    half-file that makes the next start fall back to the corner."""
+    target = Path(path or DEFAULT_PATH)
+    try:
+        target.parent.mkdir(parents=True, exist_ok=True)
+        descriptor, temp_path = tempfile.mkstemp(dir=target.parent, prefix=".window_", suffix=".tmp")
+        try:
+            with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
+                json.dump({"x": int(x), "y": int(y)}, handle)
+            os.replace(temp_path, target)
+        except BaseException:
+            try:
+                os.unlink(temp_path)
+            except OSError:
+                pass
+            raise
+    except Exception:
+        logger.debug("Could not save the window position", exc_info=True)
+
+
+def is_visible_on(x: int, y: int, size: int, rectangles) -> bool:
+    """Whether that position still lands on a screen that exists.
+
+    *rectangles* are (left, top, right, bottom) tuples — the caller's job,
+    because this module has no business importing Qt. Requires a real overlap
+    rather than a touching edge, so a pet saved flush against the boundary of a
+    monitor that has since been unplugged is not counted as reachable."""
+    for left, top, right, bottom in rectangles:
+        overlap_x = min(x + size, right) - max(x, left)
+        overlap_y = min(y + size, bottom) - max(y, top)
+        if overlap_x > size * 0.25 and overlap_y > size * 0.25:
+            return True
+    return False