TheMajesticNetwork/Bolt-Pet
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 3 Wiki Activity

Streaming replies and STT, amplitude lip-sync, one place for speaking

Browse Source 
Latency: replies are spoken sentence-by-sentence off the desk API's NDJSON
endpoint, so the wait is time-to-first-sentence rather than the whole model
call, and Deepgram's live websocket transcribes while you're still talking
instead of uploading the WAV afterwards. Both fall back invisibly — a stream
that fails before anything was said drops to converse(), and a socket that
never opens just means the old one-shot path.

Speaking lived in four near-copies in the controller (a reply, a holding line,
a streamed sentence, a dialogue scene) that had already drifted: one didn't arm
barge-in, another skipped the follow-up rule. It's now speech.Speaker plus an
Utterance describing the policy differences, with collaborators injected so the
whole of it tests without Qt or audio.

The mouth follows the audio rather than a timer: tts.level_of reduces each PCM
frame to a 0..1 loudness on a sqrt curve (speech sits well below peak, and a
linear map leaves the mouth barely open during normal talking) and that indexes
the talking frames, which the sprite script now draws as an openness ramp.
Offline pyttsx3 has no waveform, so stale levels hand control back to the timed
loop instead of freezing the mouth mid-syllable.

Also: the pet starts where you left it (ignoring positions on monitors that are
no longer connected, since restoring those faithfully is how it ends up
somewhere unreachable), and `python -m bolt_pet --doctor` is a preflight that
says what to do about each problem rather than only what's wrong.

tests/test_pipeline_smoke.py breaks the pure-logic rule on purpose. Every unit
test passed all week while notifications sat unspoken for minutes, the pet said
things twice and [laughing] got read aloud — each an interaction between two
individually-correct units. It drives whole turns against a real HTTP server on
a loopback port, faking only the mic and the speakers. It found a NameError in
the paint path that would have fired on every repaint while talking.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Aaron Franks
2026-08-02 19:01:06 -06:00
parent c4e805defd
commit 3a0959f55d
55 changed files with 2796 additions and 151 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bolt_pet/controller.py
+168 -77
View File
@@ -20,15 +20,20 @@ from typing import Optional
from PySide6.QtCore import QObject, Signal
from . import (
config, dialogue as dialogue_mod, file_delivery, file_ops,
config, dialogue as dialogue_mod, speech, file_delivery, file_ops,
history as history_mod, monitors as monitors_mod, notifications,
pet_actions, quiet, screen_context, screen_text, self_restart,
server_client, speech_text, updater,
)
from . import __version__
from .audio import barge_in, mic, stt, tts, wake_word
from .audio import barge_in, mic, stt, stt_stream, tts, wake_word
from .state import PetState, PetStateMachine
# A burst while the pet is busy is batched into one turn, but the queue still
# needs a ceiling — a notification storm must not become an unbounded backlog
# that gets read out minutes later.
_MAX_PENDING_NOTIFICATIONS = 12
# How often to re-check whether the pet should be napping. The fullscreen
# probe shells out to xprop, so this deliberately isn't every heartbeat tick.
_NAP_CHECK_INTERVAL_SECONDS = 10.0
@@ -41,6 +46,7 @@ class PetController(QObject):
action = Signal(dict) # parsed petctl action for the UI to perform
napping = Signal(bool) # quiet hours / fullscreen do-not-disturb
voice_changed = Signal(str) # name of the server-picked voice ("" = default)
mouth = Signal(float) # 0..1 speech loudness, for lip-sync while talking
restart_requested = Signal(str) # version we just updated to
finished = Signal()
@@ -77,6 +83,17 @@ class PetController(QObject):
self._voice_name = ""
self._barge_in: Optional[barge_in.BargeInDetector] = None
# Everything the pet says goes through here; see speech.py for why the
# four hand-rolled copies of this became one.
self._speaker = speech.Speaker(
state=self._state, tts=tts,
history=lambda text: self.history.add(history_mod.PET, text, time.time()),
on_said=self.said.emit, on_log=self.log.emit,
barge_in=lambda: self._barge_in,
detail_of=self._barge_in_detail,
voice_id=lambda: self._voice_id,
on_level=self.mouth.emit,
)
self._napping = False
self._nap_forced: Optional[bool] = None # petctl nap on/off overrides the schedule
self._last_nap_check = 0.0
@@ -249,21 +266,28 @@ class PetController(QObject):
self._follow_ups = 0
self._state.transition(PetState.LISTENING)
# Transcribe while they talk rather than after: frames go to Deepgram
# as they are captured, so the text is ready the moment the VAD says
# they stopped. None here just means the one-shot path will do it.
streamed = stt_stream.StreamingTranscriber.open()
pcm = mic.record_utterance(
self._stream,
should_continue=self._should_continue,
# Answering a question deserves longer than saying the wake word
# on purpose does — you were just asked something.
grace_s=config.FOLLOW_UP_GRACE_SECONDS if following_up else None,
on_frame=streamed.feed if streamed is not None else None,
)
if pcm is None:
if streamed is not None:
streamed.finish()
self._follow_ups = 0 # silence ends the chain
self._state.transition(PetState.IDLE)
return
self._state.transition(PetState.THINKING)
try:
text = stt.transcribe(pcm)
text = self._transcribe(pcm, streamed)
except stt.SttError as exc:
self.log.emit(f"STT failed: {exc}")
self._state.transition(PetState.ERROR)
@@ -278,9 +302,7 @@ class PetController(QObject):
try:
# What's focused right now rides along, so "what's this error?"
# has a referent without you having to describe the window.
reply = server_client.converse(
self._with_context(text), on_command=self._handle_command
)
reply = self._ask_server(self._with_context(text))
except server_client.ServerError as exc:
self.log.emit(f"Server error: {exc}")
self._state.transition(PetState.ERROR)
@@ -289,10 +311,34 @@ class PetController(QObject):
self._check_deliveries()
self._apply_voice(reply)
self._speak(reply.text)
if reply.spoken:
# Streamed: every sentence was spoken and logged as it arrived.
# The text still matters — a reply ending on a question should keep
# the mic open — but saying it again would repeat the whole answer.
self._after_speaking(reply.text, completed=True)
else:
self._speak(reply.text)
self._state.transition(PetState.IDLE)
self._maybe_self_restart()
def _transcribe(self, pcm, streamed) -> str:
"""The transcript, from the live stream if it produced one.
The fallback is not a rare path to be tolerated — it is the safety net
that lets streaming be switched on at all. Whatever happened to the
socket, the full audio is still buffered here, so a failed stream costs
one ordinary upload and nothing else."""
if streamed is not None:
try:
text = streamed.finish()
except Exception:
self.log.emit("Streaming STT failed — falling back.")
text = ""
if text:
self.log.emit("(transcribed while you spoke)")
return text
return stt.transcribe(pcm)
def _with_context(self, text: str) -> str:
"""Everything the server gets alongside what you actually said: the
focused window title, and a one-line note about the screen layout so
@@ -492,6 +538,26 @@ class PetController(QObject):
self._speak(reply.text)
self._state.force(PetState.IDLE)
def _ask_server(self, text: str):
"""One turn with the server, streamed when possible.
Streaming speaks each sentence as it is generated, so the wait is
time-to-first-sentence rather than the whole model call. It falls back
to the ordinary request/response path when the server has no streaming
endpoint, or when a stream dies *before* anything was spoken — after
that, retrying would say the first half twice."""
if config.STREAMING_REPLIES:
try:
return server_client.converse_stream(
text, on_say=self._speak_stream_chunk,
on_command=self._handle_command,
)
except server_client.ServerError as exc:
self.log.emit(f"Streaming unavailable ({exc}) — using the plain path.")
return server_client.converse(
text, on_command=self._handle_command, on_say=self._speak_holding,
)
def _play_dialogue(self, scene: dict) -> str:
"""Play a `dialoguectl` scene and report back up the relay.
@@ -525,20 +591,8 @@ class PetController(QObject):
# scene or fix the voice, and try again inside the same turn.
return f"[dialogue] couldn't synthesize it: {exc}"
resume = self._state.state
self._state.transition(PetState.TALKING)
self.said.emit(speech_text.for_display(text))
self.history.add(history_mod.PET, text, time.time())
should_stop = None
if self._barge_in is not None:
self._barge_in.reset()
should_stop = self._barge_in.check
completed = tts.play_pcm(pcm, sample_rate, should_stop=should_stop)
if self._barge_in is not None:
self._barge_in.reset() # the pet's own voices are in the wake window
if resume in (PetState.THINKING, PetState.IDLE):
self._state.transition(resume)
completed = self._speaker.say_pcm(
speech.Utterance.scene(text), pcm=pcm, sample_rate=sample_rate)
if not completed:
return dialogue_mod.describe(scene) + " (interrupted — they talked over it)"
@@ -569,32 +623,30 @@ class PetController(QObject):
self.reset_voice()
def _speak(self, text: str) -> None:
self._state.transition(PetState.TALKING)
# Bubble gets the markdown stripped but emoji kept (it can't render
# **bold** but draws emoji fine); tts.speak() does its own, stricter
# sanitizing for the voice.
self.said.emit(speech_text.for_display(text))
self.log.emit(f"Bolt: {text}")
self.history.add(history_mod.PET, text, time.time())
"""The answer: transcript, bubble, and the follow-up rule."""
completed = self._speaker.say(speech.Utterance.reply(text))
self._after_speaking(text, completed=completed,
detail=self._speaker.last_detail)
should_stop = None
if self._barge_in is not None:
self._barge_in.reset()
should_stop = self._barge_in.check
completed = tts.speak(
text,
on_error=lambda exc: self.log.emit(f"TTS failed: {exc}"),
should_stop=should_stop,
voice_id=self._voice_id or None,
)
# Read the scoring history *before* resetting, or the log reports the
# blank counters instead of what actually fired.
detail = self._barge_in_detail()
if self._barge_in is not None:
# Playback fed the pet's own voice into the wake model's rolling
# window. Clear it before the idle listener starts scoring again,
# or Bolt's last sentence is still in there being re-scored.
self._barge_in.reset()
def _speak_holding(self, text: str) -> None:
""""Give me a sec" while a tool runs — filler, so no transcript, and it
resumes the state it interrupted because the turn isn't over."""
self._speaker.say(speech.Utterance.holding(text))
def _speak_stream_chunk(self, text: str) -> None:
"""One sentence of a streamed answer, spoken the moment it arrives."""
completed = self._speaker.say(speech.Utterance.stream_chunk(text))
if not completed:
self.log.emit("Interrupted — listening.")
self._follow_ups = 0
self._talk_now.set()
def _after_speaking(self, text: str, *, completed: bool, detail: str = "") -> None:
"""What happens once an answer has been said, however it was said.
Shared by the plain and streamed paths: a streamed reply is spoken
sentence by sentence, but it still has to obey the same rules about
keeping the mic open when it ended on a question."""
if not completed:
# You talked over it — take that as the start of the next turn
# rather than making you say the wake word again.
@@ -614,20 +666,22 @@ class PetController(QObject):
def _should_follow_up(self, text: str) -> bool:
"""Whether *text* leaves the pet waiting on an answer.
Muted is excluded because mute means "don't listen to me" — an
automatic turn would walk straight past it. Napping isn't: quiet
hours suppress the pet *starting* something, and a question is only
ever asked in reply to you."""
if not config.FOLLOW_UP_LISTEN or self._muted:
The rule itself — question, cap, off switch — is
`speech.follow_up_decision`, because it is a rule with an off-by-one in
it and deserves a test that needs no audio. What stays here is the part
that is genuinely the controller's: mute, and the log line. Muted is
excluded because mute means "don't listen to me" and an automatic turn
would walk straight past it. Napping isn't: quiet hours suppress the
pet *starting* something, and a question is only ever asked in reply to
you."""
if self._muted:
return False
if not speech_text.is_question(text):
return False
# Only worth mentioning the cap on a reply that would otherwise have
# kept listening, or it fires on every statement the pet makes.
if config.FOLLOW_UP_MAX_TURNS > 0 and self._follow_ups >= config.FOLLOW_UP_MAX_TURNS:
keep, why = speech.follow_up_decision(text, completed=True, follow_ups=self._follow_ups)
if not keep and "cap" in why:
# Only worth mentioning the cap on a reply that would otherwise have
# kept listening, or it fires on every statement the pet makes.
self.log.emit("Follow-up limit reached — say the wake word to keep going.")
return False
return True
return keep
def _barge_in_detail(self) -> str:
"""Why the interruption fired, for the log. How far into playback it
@@ -685,33 +739,54 @@ class PetController(QObject):
def _queue_notification(self, notification: notifications.Notification) -> None:
"""Called on the watcher thread — just queue it; forwarding happens on
the pipeline thread where it can't collide with a live conversation."""
if not self._notification_gate.should_forward(notification, time.monotonic()):
the pipeline thread where it can't collide with a live conversation.
Only the *filter* applies here. The rate limit used to as well, which
meant a second message arriving inside the window was silently thrown
away — with NOTIFICATION_MIN_INTERVAL_SECONDS=60, two texts a minute
apart and you only ever heard about one of them. Losing a message from
a person to save a round trip is the wrong trade; they are batched at
the far end instead, which costs the same one round trip and keeps
them all."""
if not self._notification_gate.matches(notification):
return
with self._notification_lock:
if len(self._pending_notifications) >= _MAX_PENDING_NOTIFICATIONS:
self._pending_notifications.pop(0) # bound it; oldest goes first
self._pending_notifications.append(notification)
def _drain_notifications(self) -> None:
"""Forward everything waiting as ONE turn.
Batching is what makes it safe to keep every notification: five that
arrived while the pet was mid-conversation become one message and one
round trip, instead of five separate interruptions queued up to fire
back to back."""
with self._notification_lock:
pending, self._pending_notifications = self._pending_notifications, []
if not pending or not self._running or self._napping:
return
for notification in pending:
if not self._running or self._napping:
return
self.log.emit(f"Notification: {notification.as_text()}")
self.history.add(history_mod.SYSTEM, notification.as_text(), time.time())
try:
reply = server_client.converse(
f"[desktop notification] {notification.as_text()}",
on_command=self._handle_command,
)
except server_client.ServerError as exc:
self.log.emit(f"Couldn't forward notification: {exc}")
return
self._check_deliveries()
self._apply_voice(reply)
if reply.text.strip():
self._speak(reply.text)
self._state.transition(PetState.IDLE)
if len(pending) == 1:
message = f"[desktop notification] {pending[0].as_text()}"
else:
lines = "\n".join(f"- {n.as_text()}" for n in pending)
message = f"[{len(pending)} desktop notifications]\n{lines}"
try:
reply = server_client.converse(
message, on_command=self._handle_command, on_say=self._speak_holding,
)
except server_client.ServerError as exc:
self.log.emit(f"Couldn't forward notification: {exc}")
return
self._check_deliveries()
self._apply_voice(reply)
if reply.text.strip() and not reply.spoken:
self._speak(reply.text)
self._state.transition(PetState.IDLE)
# ── file delivery ────────────────────────────────────────────────────
@@ -784,20 +859,36 @@ class PetController(QObject):
# ── heartbeat ────────────────────────────────────────────────────────
def _maybe_heartbeat(self) -> None:
"""Called from the wake listener's tick (~every WAKE_CHECK_INTERVAL_SECONDS).
Two different cadences live here, and conflating them was costing
minutes. A *notification* is an event that already happened — it should
go out as soon as the pet is free, which is the next tick. The
*heartbeat* is a poll, and polling the server every 1.2s would be
absurd, so it stays on its own interval."""
self._refresh_nap_state()
self._maybe_update()
if self._update_pending:
return # on the way out — don't start a conversation now
# Notifications: every tick, not every heartbeat.
if self._state.state == PetState.IDLE and not self._napping:
self._drain_notifications()
now = time.monotonic()
if now - self._last_heartbeat < config.HEARTBEAT_INTERVAL_SECONDS:
return
self._last_heartbeat = now
if self._state.state != PetState.IDLE:
# Mid-conversation. Do NOT stamp the clock — an earlier version
# did, so a heartbeat that landed while the pet was talking burned
# its slot and waited another full interval. With follow-up
# listening that could repeat for several cycles, which is why a
# notification could sit unspoken for five or ten minutes.
return
if self._napping:
return # quiet hours: still answers when spoken to, just doesn't start
self._last_heartbeat = now
self._check_deliveries()
self._drain_notifications()
if self._state.state != PetState.IDLE:
return
try: