Files
Bolt-Pet/bolt_pet/speech_text.py
T
themajesticmagician 3a0959f55d Streaming replies and STT, amplitude lip-sync, one place for speaking
Latency: replies are spoken sentence-by-sentence off the desk API's NDJSON
endpoint, so the wait is time-to-first-sentence rather than the whole model
call, and Deepgram's live websocket transcribes while you're still talking
instead of uploading the WAV afterwards. Both fall back invisibly — a stream
that fails before anything was said drops to converse(), and a socket that
never opens just means the old one-shot path.

Speaking lived in four near-copies in the controller (a reply, a holding line,
a streamed sentence, a dialogue scene) that had already drifted: one didn't arm
barge-in, another skipped the follow-up rule. It's now speech.Speaker plus an
Utterance describing the policy differences, with collaborators injected so the
whole of it tests without Qt or audio.

The mouth follows the audio rather than a timer: tts.level_of reduces each PCM
frame to a 0..1 loudness on a sqrt curve (speech sits well below peak, and a
linear map leaves the mouth barely open during normal talking) and that indexes
the talking frames, which the sprite script now draws as an openness ramp.
Offline pyttsx3 has no waveform, so stale levels hand control back to the timed
loop instead of freezing the mouth mid-syllable.

Also: the pet starts where you left it (ignoring positions on monitors that are
no longer connected, since restoring those faithfully is how it ends up
somewhere unreachable), and `python -m bolt_pet --doctor` is a preflight that
says what to do about each problem rather than only what's wrong.

tests/test_pipeline_smoke.py breaks the pure-logic rule on purpose. Every unit
test passed all week while notifications sat unspoken for minutes, the pet said
things twice and [laughing] got read aloud — each an interaction between two
individually-correct units. It drives whole turns against a real HTTP server on
a loopback port, faking only the mic and the speakers. It found a NameError in
the paint path that would have fired on every repaint while talking.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 19:01:06 -06:00

174 lines
7.0 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Turn a server reply into something worth *hearing*.
The server's persona writes for a chat window: markdown emphasis, bullet
lists, emoji, bare URLs. A TTS voice reads those literally ("asterisk
asterisk OS colon", "https colon slash slash..."), so everything spoken goes
through for_speech() first. Pure string logic, no Qt/audio imports — cheap to
unit test (see tests/test_speech_text.py).
for_display() is the lighter sibling used for the speech bubble: it drops the
markdown *syntax* but keeps emoji and punctuation, since those render fine.
"""
from __future__ import annotations
import re
# Pictographs, symbols, flags, dingbats, arrows, box drawing, variation
# selectors, ZWJ — anything a voice would either skip or read as a name
# ("black right-pointing triangle").
_EMOJI = re.compile(
"["
"\U0001F000-\U0001FAFF" # emoji / pictographs / symbols blocks
"\U00002190-\U000021FF" # arrows
"\U00002300-\U000023FF" # misc technical (⌘ ⏱ …)
"\U000025A0-\U000027BF" # geometric shapes, misc symbols, dingbats
"\U00002B00-\U00002BFF" # extra arrows / shapes
"\U0000FE00-\U0000FE0F" # variation selectors
"\U0001F1E6-\U0001F1FF" # regional indicators (flags)
"\U0000200D" # zero-width joiner
"]+",
flags=re.UNICODE,
)
# Characters that are markup or decoration rather than speech. Kept out of
# the spoken text entirely; ordinary punctuation (. , ! ? ; : ' " ( ) -) is
# preserved because it shapes prosody.
_UNSPEAKABLE = re.compile(r"[*_#`~^|<>\\{}\[\]/=+@©®™•·–—]+")
_FENCED_CODE = re.compile(r"```.*?```", re.DOTALL)
_INLINE_CODE = re.compile(r"`([^`]*)`")
_MD_IMAGE = re.compile(r"!\[([^\]]*)\]\([^)]*\)")
_MD_LINK = re.compile(r"\[([^\]]+)\]\([^)]*\)")
_URL = re.compile(r"\b(?:https?://|www\.)\S+")
_HEADING = re.compile(r"^\s{0,3}#{1,6}\s*", re.MULTILINE)
_BLOCKQUOTE = re.compile(r"^\s{0,3}>\s?", re.MULTILINE)
_RULE = re.compile(r"^\s*([-*_=])(?:\s*\1){2,}\s*$", re.MULTILINE)
_BULLET = re.compile(r"^\s*(?:[-*+•·]|\d+[.)])\s+", re.MULTILINE)
_EMPHASIS = re.compile(r"(\*{1,3}|_{1,3})(\S(?:.*?\S)?)\1", re.DOTALL)
_TABLE_PIPE = re.compile(r"[ \t]*\|[ \t]*")
# Symbols worth saying out loud rather than dropping — a bare "&" read as
# nothing turns "R&D" into "RD". Split in two because the non-ASCII ones sit
# inside the arrow/symbol blocks _EMOJI strips, so they have to be worded
# before that pass; the ASCII ones must wait until *after* markdown parsing
# (an "=" turned into " equals " would stop a "====" rule matching _RULE).
_PRE_SPOKEN_SYMBOLS = {
"→": " to ",
"×": " by ",
"°": " degrees ",
"≈": " about ",
}
_SPOKEN_SYMBOLS = {
"&": " and ",
"%": " percent ",
"@": " at ",
"+": " plus ",
"=": " equals ",
}
_MULTI_SPACE = re.compile(r"[ \t]+")
_MULTI_PUNCT = re.compile(r"(?:\s*\.){2,}")
def _strip_markdown(text: str, *, keep_emoji: bool) -> str:
text = _FENCED_CODE.sub(" (code) ", text)
text = _INLINE_CODE.sub(r"\1", text)
text = _MD_IMAGE.sub(r"\1", text)
text = _MD_LINK.sub(r"\1", text)
text = _RULE.sub("", text)
text = _HEADING.sub("", text)
text = _BLOCKQUOTE.sub("", text)
text = _EMPHASIS.sub(r"\2", text)
if not keep_emoji:
text = _EMOJI.sub(" ", text)
return text
def _bullets_to_sentences(text: str) -> str:
"""A read-aloud list needs pauses where the eye would see line breaks,
otherwise "OS Linux Uptime 1 day CPU load moderate" runs together."""
lines = [_BULLET.sub("", line).strip() for line in text.splitlines()]
lines = [line for line in lines if line]
if len(lines) < 2:
return lines[0] if lines else ""
# A bullet like "OS: Linux" ends mid-thought — give the voice a full stop
# so consecutive items don't slur into one run-on sentence.
return " ".join(line if line[-1] in ".!?:,;" else line + "." for line in lines)
# ElevenLabs v3 delivery tags — "[laughing]", "[whispering]", "[sighs]". They
# are instructions to the *dialogue* model (see dialogue.py), and only inside a
# dialoguectl scene. Left in an ordinary reply they reach eleven_flash_v2,
# which has no idea what they mean and simply reads the word: observed live
# 2026-07-31, the pet announcing "laughing That one came through clean".
#
# Matched narrowly — a bracketed adverb/gerund, or one of the noise words that
# aren't either — so real bracketed text ("[1]", "[see the docs]") survives.
_AUDIO_TAG_RE = re.compile(
r"\[\s*(?:"
r"[a-z]+(?:ly|ing)"
r"|laughs?|chuckles?|sighs?|exhales?|inhales?|gulps?|pause|beat"
r"|clears throat|shouts?|whispers?|cries|sings?|gasps?|snorts?"
r"|sarcastic|excited|curious|nervous|angry|sad|happy|deadpan|flat"
r")\s*\]",
re.IGNORECASE,
)
def strip_audio_tags(text: str) -> str:
"""Remove v3 delivery tags from text destined for the ordinary voice."""
return _AUDIO_TAG_RE.sub(" ", str(text or ""))
def for_speech(text: str) -> str:
"""Plain prose for the TTS engine: no markdown, no emoji, no bare URLs,
no stray symbols that would be read out character by character."""
text = strip_audio_tags((text or "").strip())
if not text:
return ""
for symbol, spoken in _PRE_SPOKEN_SYMBOLS.items():
text = text.replace(symbol, spoken)
text = _strip_markdown(text, keep_emoji=False)
text = _URL.sub(" link ", text)
text = _TABLE_PIPE.sub(", ", text)
text = _bullets_to_sentences(text)
for symbol, spoken in _SPOKEN_SYMBOLS.items():
text = text.replace(symbol, spoken)
text = _UNSPEAKABLE.sub(" ", text)
text = _MULTI_PUNCT.sub(".", text)
text = _MULTI_SPACE.sub(" ", text)
text = re.sub(r"\s+([.,!?;:])", r"\1", text)
return text.strip()
def is_question(text: str) -> bool:
"""True if the reply *ends* by asking the user something — the cue for
the pet to keep listening instead of making you say the wake word again.
Deliberately only looks at the end. A reply that asks something in
passing ("What time is it? It's 7:15.") isn't waiting on an answer,
whereas one that finishes on a question mark is. The test runs on the
spoken form, so a '?' that only exists inside a stripped code block or a
URL doesn't count, and trailing decoration (emoji, quotes, brackets) is
peeled off first so "Ready to go? 🚀" still reads as a question."""
spoken = for_speech(text)
while spoken and not (spoken[-1].isalnum() or spoken[-1] == "?"):
spoken = spoken[:-1]
return spoken.endswith("?")
def for_display(text: str) -> str:
"""What the speech bubble shows: markdown syntax removed (the bubble
can't render it) but emoji and layout-ish punctuation left alone.
Delivery tags go too — the voice no longer says them, so showing them
would caption a laugh nobody heard."""
text = strip_audio_tags((text or "").strip())
if not text:
return ""
text = _strip_markdown(text, keep_emoji=True)
lines = [_BULLET.sub("• ", line).strip() for line in text.splitlines()]
text = " ".join(line for line in lines if line)
return _MULTI_SPACE.sub(" ", text).strip()