TheMajesticNetwork/Bolt-Pet
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 3 Wiki Activity

Streaming replies and STT, amplitude lip-sync, one place for speaking

Browse Source 
Latency: replies are spoken sentence-by-sentence off the desk API's NDJSON
endpoint, so the wait is time-to-first-sentence rather than the whole model
call, and Deepgram's live websocket transcribes while you're still talking
instead of uploading the WAV afterwards. Both fall back invisibly — a stream
that fails before anything was said drops to converse(), and a socket that
never opens just means the old one-shot path.

Speaking lived in four near-copies in the controller (a reply, a holding line,
a streamed sentence, a dialogue scene) that had already drifted: one didn't arm
barge-in, another skipped the follow-up rule. It's now speech.Speaker plus an
Utterance describing the policy differences, with collaborators injected so the
whole of it tests without Qt or audio.

The mouth follows the audio rather than a timer: tts.level_of reduces each PCM
frame to a 0..1 loudness on a sqrt curve (speech sits well below peak, and a
linear map leaves the mouth barely open during normal talking) and that indexes
the talking frames, which the sprite script now draws as an openness ramp.
Offline pyttsx3 has no waveform, so stale levels hand control back to the timed
loop instead of freezing the mouth mid-syllable.

Also: the pet starts where you left it (ignoring positions on monitors that are
no longer connected, since restoring those faithfully is how it ends up
somewhere unreachable), and `python -m bolt_pet --doctor` is a preflight that
says what to do about each problem rather than only what's wrong.

tests/test_pipeline_smoke.py breaks the pure-logic rule on purpose. Every unit
test passed all week while notifications sat unspoken for minutes, the pet said
things twice and [laughing] got read aloud — each an interaction between two
individually-correct units. It drives whole turns against a real HTTP server on
a loopback port, faking only the mic and the speakers. It found a NameError in
the paint path that would have fired on every repaint while talking.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Aaron Franks
2026-08-02 19:01:06 -06:00
parent c4e805defd
commit 3a0959f55d
55 changed files with 2796 additions and 151 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bolt_pet/server_client.py
+118
View File
@@ -19,6 +19,7 @@ Kept dependency-free beyond `requests` so it's easy to unit test with mocks.
from __future__ import annotations
import json
import subprocess
from pathlib import Path
from typing import Callable, NamedTuple, Optional
@@ -43,6 +44,10 @@ class Reply(NamedTuple):
text: str
voice_id: str = ""
voice_name: str = ""
# True when the sentences were already spoken as they streamed in. The
# text is still carried — the follow-up rule needs to see whether the
# answer ended on a question — it just must not be read out again.
spoken: bool = False
def _headers() -> dict:
@@ -90,12 +95,20 @@ def converse(
text: str,
on_command: Callable[[str], str] = run_local_command,
timeout: float = 120.0,
on_say: Optional[Callable[[str], None]] = None,
) -> Reply:
"""Send one turn of conversation to the desk API, relaying any commands
the server sends back until it produces a final reply.
*on_command* is injectable for tests; defaults to actually running the
command locally (matching bolt_desk.py's behavior).
*on_say* is called with a short holding line ("give me a sec") when the
server sends one alongside a command. It is the difference between silence
and an answer while a tool runs: the model's acknowledgement used to be
discarded server-side, so the whole round trip was dead air and the model
then repeated itself in the final reply. Optional, so an older pet against
a newer server simply stays quiet as before.
"""
headers = _headers()
try:
@@ -111,6 +124,13 @@ def converse(
for _ in range(_MAX_RELAY_HOPS):
if payload.get("type") != "command":
break
holding = str(payload.get("say") or "").strip()
if holding and on_say is not None:
# Spoken *before* the command runs — that is the whole point.
try:
on_say(holding)
except Exception:
pass # a failed acknowledgement must not cost the tool call
output = on_command(str(payload.get("command") or ""))
try:
response = requests.post(
@@ -135,6 +155,104 @@ def converse(
raise ServerError(str(payload.get("error") or "unknown server response"))
def converse_stream(
text: str,
on_say: Callable[[str], None],
on_command: Callable[[str], str] = run_local_command,
timeout: float = 180.0,
) -> Reply:
"""Same turn as converse(), but speaking each sentence as it arrives.
Without this the pet waits out the *entire* model call before a single
word is heard; with it the wait is time-to-first-sentence, which on a
multi-sentence answer is most of the difference.
Falls back by raising ServerError before anything has been spoken — the
caller then retries the ordinary path and the user never finds out. Once a
sentence *has* been spoken there is no going back, so late failures end the
turn with whatever was said rather than repeating it.
"""
spoke_anything = False
try:
response = requests.post(
f"{config.SERVER_URL}/desk/converse_stream",
json={"session_id": config.SESSION_ID, "text": text},
headers=_headers(), timeout=timeout, stream=True,
)
response.raise_for_status()
for raw in response.iter_lines(decode_unicode=True):
if not raw:
continue
try:
event = json.loads(raw)
except (TypeError, ValueError):
continue
kind = str(event.get("type") or "")
if kind == "say":
line = str(event.get("text") or "").strip()
if line:
spoke_anything = True
on_say(line)
elif kind == "command":
# The tool loop is request/response, so the rest of the turn
# finishes through the ordinary relay rather than inside the
# stream — one protocol for tools, not two.
response.close()
return _finish_relay(event, on_command, on_say)
elif kind == "reply":
return Reply(
text=str(event.get("text") or ""),
voice_id=str(event.get("voice_id") or ""),
voice_name=str(event.get("voice_name") or ""),
spoken=bool(event.get("already_spoken")),
)
elif kind == "error":
raise ServerError(str(event.get("error") or "stream failed"))
except ServerError:
raise
except Exception as exc:
if spoke_anything:
# Half a reply is out loud already; ending quietly beats saying it
# all again through the fallback path.
return Reply(text="", spoken=True)
raise ServerError(f"streaming failed: {exc}") from exc
raise ServerError("stream ended without a reply")
def _finish_relay(event: dict, on_command, on_say) -> Reply:
"""Run the tool the stream handed over, then continue the classic relay."""
payload = dict(event)
headers = _headers()
for _ in range(_MAX_RELAY_HOPS):
if payload.get("type") != "command":
break
holding = str(payload.get("say") or "").strip()
if holding and on_say is not None:
try:
on_say(holding)
except Exception:
pass
output = on_command(str(payload.get("command") or ""))
try:
response = requests.post(
f"{config.SERVER_URL}/desk/tool_result",
json={"session_id": config.SESSION_ID,
"token": payload.get("token"), "output": output},
headers=headers, timeout=180,
)
payload = response.json()
except Exception as exc:
raise ServerError(f"couldn't reach the server during tool relay: {exc}") from exc
if payload.get("type") == "reply":
return Reply(
text=str(payload.get("text") or ""),
voice_id=str(payload.get("voice_id") or ""),
voice_name=str(payload.get("voice_name") or ""),
)
raise ServerError(str(payload.get("error") or "unknown server response"))
def list_outbox_files(timeout: float = 15.0) -> list:
"""Files the server has queued for this session via its deliver_files
tool (e.g. "send me that report" during a conversation) — each entry has