TheMajesticNetwork/Bolt-Console
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 2 Wiki Activity

Bolt Console: desktop app with chat list / new chat, dictation, file tools

Browse Source 
Add a chat list and server-minted new chats on top of the desk API's
GET/POST /desk/chats:

- server_client.list_chats() / create_chat(), same shape and error
  handling as list_outbox_files().
- Switching is config.set_session(id): a plain reassignment of the
  SESSION_ID every call reads, persisted to session.json. The window
  refuses to switch mid-turn rather than locking.
- gui/chat_list_window.py: a Chats dialog (Ctrl+K) with a QListWidget,
  populated on open, with Refresh and New Chat. Network calls run as
  QRunnables on the global QThreadPool and come back through Signals,
  never from a slot. The header New chat (Ctrl+N) uses the same path.
- Switching clears the local transcript; there is no endpoint for a
  past chat's messages. Bolt's memory is per user, so it carries over.
- Terminal: /chats, /chat <n|id>, and /new (server-minted id, local
  fallback).

This commit also carries the uncommitted work since 54030a7 that turned
this Bolt-Pet fork into Bolt Console: the pet, wake word, TTS and screen
features removed; a Claude Code-style terminal client; the PySide6
window with tool cards, approvals and diffs; mic dictation through the
server's /desk/stt relay; filectl/JSON repairs; and detection of
invented tool results.

Build output (build/, dist/, Release/, *.spec) is now ignored.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
themajesticmagician
2026-09-25 19:50:10 -06:00
co-authored by Claude Opus 5.5
parent 54030a7b31
commit 3977804ae3
144 changed files with 5477 additions and 14736 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+101 -229
View File
@@ -1,250 +1,122 @@
# Bolt Desktop Pet
# Bolt Console
A little animated pet that lives on your desktop and is just a face on top
of your Bolt server — same brain, memory, tools, and persona as Discord
chat and the Linux desk client. It talks to `ai/desk_api.py` on the server
exactly the way `desk_client/bolt_desk.py` does — including speech: STT and
TTS are the server's own `/desk/stt` and `/desk/tts`, the same endpoints the
Android app uses, so there's no separate Deepgram or ElevenLabs account to
set up for the pet to talk. Playback is cross-platform (no `mpv`/`ffplay`/
`espeak-ng` subprocess calls — pure `sounddevice`).
A desktop app for Bolt, in the style of Claude Code (with a terminal version too). You type, Bolt
answers, and when he needs to look at or change something on this machine you see each step as it
happens: reading files, listing folders, editing, writing, running commands. Changes wait for your
**Allow** on the step itself, and edits show a diff first.
It talks to the same Bolt server as Bolt-Pet, the Discord bot and the phone app, so it has the same
brain, memory, tools and persona. What it drops is the desktop pet around it: no sprite, no wake
word, no microphone, no speech, no screen reading.
The window is PySide6 with its own dark theme. Header: working folder (click to change) and an
**Ask first / Auto** switch. Below that: the conversation, with each tool call as a card that you can
click to expand, and a composer at the bottom (Enter sends, Shift+Enter adds a new line, Ctrl+L
clears, Ctrl+O picks a folder).
**Chats** (Ctrl+K) lists every conversation you've had with Bolt, from any device (console, pet,
phone), newest first. Open one to continue it. **New chat** (Ctrl+N, or the button in the Chats
dialog) asks the server for a fresh chat: new conversation history, but Bolt still remembers you,
because his memory belongs to you, not to a chat. Switching clears the window's transcript, since
the server has no endpoint for an old chat's messages, although Bolt still has them. The current
chat is remembered across restarts. The terminal version has `/chats`, `/chat <n>` and `/new`.
**Dictation:** the mic button next to Send (or Ctrl+M) records you and types what you say into the box
as you talk. Click again or pause for 3 seconds to stop, and Esc cancels. The text isn't sent until you
press Enter, so you can fix it first. It uses the same speech-to-text as Bolt-Pet: the Bolt server's
`/desk/stt` relay to Deepgram, so you don't need a Deepgram account. The server does count it against
your credits like a chat turn. `MIC_DEVICE` in `.env` picks the microphone.
The terminal version (`run-cli.bat` / `./run.sh --cli`) looks like this:
```
mic → wake-phrase spotter ("thunderbolt") / hotkey / click → record utterance
→ server-hosted STT (+ the focused window's title, for "what's this
error?") → POST /desk/converse on your Bolt server → [server may relay
a shell command back to run on THIS machine, or a `petctl` command
that moves or emotes the pet] → reply → server-hosted streaming TTS
→ speakers
→ shown in a speech bubble + the pet's sprite state (idle/listening/
thinking/talking) updates the whole time
> what's taking up space in this repo?
● List(C:\work\site, recursive)
⎿ assets/
assets/hero.mp4 48211332B
... +212 lines
● Shell(du -sh assets/* | sort -h | tail -5)
Run this command? [y]es / [n]o / [a]lways this session y
⎿ 48M assets/hero.mp4
...
● Most of it is one file: assets/hero.mp4 at 48 MB. ...
```
Nothing is sent to the server until the wake phrase fires, you press the
push-to-talk hotkey, or you click the pet — plus, if you turn them on, the
heartbeat and the desktop-notification bridge.
## Why a separate project instead of living in the tmn-api repo
This runs on your desktop machine, not the server — same relationship as
`desk_client/` (Linux) or the Android app, both of which are just clients of
the desk API over HTTP. It has no import dependency on the server repo at
all, so it can be copied anywhere and configured with its own `.env`.
## Setup
1. Copy this whole `bolt-pet/` folder to the machine you want the pet to run
on (if that isn't already this machine).
2. `cp .env.example .env` and fill in:
- `BOLT_SERVER_URL` + `DESK_API_KEY` — same as `desk_client/.env` on the
server side. Use the server's master `DESK_API_KEY`, or mint yourself a
personal one via the desk-only `api_key_generate` marker (see the main
repo's `CLAUDE.md` → "Per-user API keys"). That's it for STT — no
Deepgram account needed, it goes through the server's own `/desk/stt`.
- `ELEVENLABS_VOICE_ID` for TTS (optional — falls back to offline TTS via
`pyttsx3` if omitted or if the server call fails). No local ElevenLabs
API key needed for this either; `ELEVENLABS_API_KEY` is only for the
multi-voice `dialoguectl` scenes further down, the one feature the
server has no endpoint for.
3. Run it:
- macOS/Linux: `./run.sh`
- Windows: `run.bat`
```bash
cp .env.example .env # set BOLT_SERVER_URL and DESK_API_KEY
run.bat # Windows: the window
run-cli.bat # Windows: the terminal version
./run.sh [--cli] # macOS/Linux
```
Both scripts create a local `.venv` and install `requirements.txt` on
first run. On Linux you'll also need system packages for audio:
`sudo apt install libportaudio2 espeak-ng`.
The pet appears near the bottom-right of your screen. It wanders off on its
own now and then; drag it anywhere and it tucks itself flush against a nearby
screen edge. That position isn't saved across restarts (see Known
limitations).
## Talking to it
- Say **"thunderbolt"** — detected fully on-device by a custom-trained
openWakeWord model (`thunderbolt.onnx`, ships in the project root), the
same way the server repo's `desk_client/bolt_desk.py` detects "hey bolt"
with `bolt.onnx`. Matches the `DEFAULT_WAKE_WORD` already used for Bolt's
Discord voice channels, so it's the same word everywhere.
- Or press **Ctrl+Alt+Space** (`PUSH_TO_TALK_HOTKEY`) from anywhere — useful
in a noisy room where the wake word misfires. Needs `pynput` and a session
that allows global key hooks; most Wayland sessions don't, in which case it
logs why at startup and everything else still works.
- Or just **click the pet** once (a drag doesn't count as a click).
- **Talk over it** to cut a long answer short — the mic stays live while it
speaks, and barging in starts your next turn immediately (`BARGE_IN`).
- Right-click the tray icon for **Talk now**, **Mute mic**, **Nap**,
**Wander around**, **Click through the pet**, **History…**, **Wake word
tuning…**, **Use default voice** and **Quit** — the pet window itself has
no title bar or taskbar entry.
- **Click the speech bubble** to copy what it just said; the tray's
**History…** window keeps the last `HISTORY_LIMIT` turns.
## What it does on its own
- **Wanders** the desktop while idle (`PET_WANDER`), stands still while
listening/thinking/talking or while a bubble is up.
- **Moves and emotes on command.** Bolt can relay `petctl move top-left`,
`petctl emote wave|hop|spin|nod|shake`, `petctl say ...`, `petctl wander
on|off`, `petctl nap on|off`, `petctl voice reset`, and `dialoguectl` for a
multi-voice scene. These are intercepted here and never reach a shell.
- **Naps** during `QUIET_HOURS` (e.g. `23:00-08:00`) or while a fullscreen
app is focused (`DND_ON_FULLSCREEN`) — it dims, stops wandering, and makes
no proactive noise. It still answers when you speak to it.
- **Reacts to desktop notifications** if you turn on `NOTIFICATION_BRIDGE`
(Linux/D-Bus) and set a `NOTIFICATION_FILTER` regex — matching
notifications get forwarded to the server, so it can tell you the deploy
went green. Off by default: each one costs a round trip.
## Speaking in another voice
Ask for a different voice — "use a clearer voice", "talk like a pirate", "say
that in Japanese" — and Bolt searches the ElevenLabs voice library on the
server, picks one, and tags his reply with it (`speak_as`); the pet is what
actually speaks in it. A Voice Library pick is added to your ElevenLabs
account automatically the first time it's used. The server also synthesizes
with one fixed, multilingual-capable model for every request now (not a
flash/multilingual switch per reply) — nothing to configure on this side,
and non-English text or a picked voice no longer needs special-casing here.
The new voice **stays on** for the rest of the conversation, because the
server tags a single reply and doesn't remember which voice it chose — so
"keep talking like that" would otherwise send it hunting for a voice again.
To get his own voice back: ask him ("use your normal voice" — he relays
`petctl voice reset`), use **Use default voice** in the tray menu (greyed
out unless a picked voice is active), or restart the pet. Set
`VOICE_STICKY=false` in `.env` if you'd rather each pick lasted exactly one
reply.
## Multi-voice dialogue
Ask for a scene — "do the argument between the two of them", "read that back
as a radio play" — and Bolt can relay a `dialoguectl` command that the pet
renders through ElevenLabs' Text to Dialogue endpoint: several voices in one
take, with delivery tags the v3 model acts on (`[cheerfully]`, `[whispering]`,
`[stuttering]`). One request per scene, so the voices actually react to each
other instead of sounding like clips glued together.
Name the cast in `.env` (`DIALOGUE_VOICES=narrator:9BWts…,villain:IKne3…`);
the name `self` always means whatever voice the pet is currently using, so
Bolt sounds like himself in his own scenes — including after a `speak_as`
switch. Scenes show up in the speech bubble with the tags stripped, count as
normal speech for the transcript, and can be talked over like any other reply.
`DIALOGUE=false` turns the whole thing off on this device.
## Wake-word detection
`bolt_pet/audio/wake_word.py` feeds every mic frame into `thunderbolt.onnx`
via the openWakeWord runtime (ONNX inference) and treats any class score at
or above `WAKE_WORD_THRESHOLD` (default `0.5`) as a detection — the exact
same per-frame `predict()`/`reset()` pattern as `desk_client/bolt_desk.py`'s
main loop. Point `WAKE_MODEL_FILE` in `.env` at a different `.onnx` model to
change the wake phrase later without touching any other code.
If it keeps ignoring you (or firing at the TV), open **Wake word tuning…**
from the tray: it shows the peak score while you talk and a rolling list of
near misses — frames that scored just under the threshold — and the slider
takes effect immediately, mid-listen. Set the threshold just below the peak
you can hit reliably, then write it into `.env` as `WAKE_WORD_THRESHOLD`.
## Something not working?
The launchers create `.venv` on first run. Dependencies: `requests`, plus `PySide6` for the window
(the terminal version doesn't import Qt). Bolt works in the
directory you **launch from**, so you can put the launcher on your PATH and start it inside whatever
project you want him to work on.
```bash
python -m bolt_pet --doctor
run.bat --auto # don't ask before commands, writes and edits
run-cli.bat -p "summarise README.md" # one message, print the reply, exit
run-cli.bat --doctor [--deep] # check the install (--deep contacts the server)
```
Checks the things that make the pet look broken in ways that don't point at
themselves — missing server config (the controller exits its thread at startup,
so the pet appears alive and simply never answers), no input device, no OCR
engine behind `petctl read`, a wake model that isn't where `.env` says, a
silence timeout long enough to feel like lag. Each line says what to do about
it, not just what's wrong. It doesn't touch the network or open the microphone
unless you add `--deep`, so it's safe to run when the network is the suspect.
## Using the terminal version
## Little things
| Input | What it does |
|---|---|
| any text | sent to Bolt |
| `!<command>` | runs a shell command yourself. Nothing is sent to Bolt |
| `/cd <dir>`, `/pwd` | change or show the working directory |
| `/ask`, `/auto` | turn approval prompts on or off for this session |
| `/status` | server, session, approval mode, working directory |
| `/files` | download anything Bolt has queued for you |
| `/new` | start a fresh chat (same as New chat in the window) |
| `/chats`, `/chat <n>` | list your chats from every device; switch to number n (or a session id) |
| `/clear` | clear the screen. Bolt's memory lives on the server and is kept |
| `/help`, `/exit` | |
| line ending in `\` | keep typing on the next line |
The pet **remembers where you left it** — drag it somewhere deliberate and
that's where it starts next time. If that position is on a monitor you've since
unplugged it goes back to the default corner rather than restoring itself
somewhere off-screen. `PET_REMEMBER_POSITION=false` to always start in the
corner.
Ctrl+C stops the spinner and gives the prompt back. The server may still finish that turn on its
side. At an approval prompt, Ctrl+C means no.
Its **mouth moves with the actual audio** rather than flapping on a timer: the
PCM going to the speakers is reduced to a loudness per frame and that picks the
talking sprite, so the pet shuts up when the voice pauses. Offline `pyttsx3`
playback has no waveform to follow, so it falls back to the timed loop.
## How Bolt reaches this machine
## Project layout
The desk API has one tool channel: the server sends back a `command` string, the client runs it and
posts the output to `/desk/tool_result`, and this repeats until a final reply. The console decides
what each string means (`bolt_console/tools.py`):
```
bolt_pet/
config.py .env loading (same pattern as desk_client/bolt_desk.py)
state.py PetState enum + a small transition-checked state machine
server_client.py /desk/converse, /desk/tool_result, /desk/report_status
controller.py the pipeline: wake word -> STT -> server -> TTS, on a QThread
speech.py what the pet says and how each kind of saying behaves
speech_text.py strips markdown/emoji/URLs so the voice never says "asterisk"
pet_actions.py petctl move/emote/say/wander/nap parsing
screen_context.py active-window title + fullscreen detection
quiet.py quiet-hours schedule
notifications.py desktop notification bridge (Linux/D-Bus)
history.py rolling conversation transcript
hotkey.py global push-to-talk (pynput, optional)
window_state.py remembers where you left the pet
doctor.py `--doctor` preflight: is this install going to work?
audio/
mic.py input stream + energy-based VAD utterance capture
wake_word.py openWakeWord thunderbolt.onnx detection (see above)
stt.py server /desk/stt relay (one-shot: whole utterance at once)
stt_stream.py server /desk/stt live websocket — transcribes while you speak
tts.py server /desk/tts streaming PCM, offline pyttsx3 fallback,
plus the loudness envelope that drives the mouth
barge_in.py "you started talking" detector, to cut playback short
ui/
app.py wires QApplication + window + tray + controller thread together
pet_window.py frameless/translucent/always-on-top sprite window + speech bubble
sprite.py frame animation loader (see assets/sprites/README.md)
tray.py system tray menu
history_window.py conversation scrollback (copyable)
wake_tuner.py live wake-word threshold + near-miss log
assets/sprites/ Kenney robot-pack art (CC0) — see assets/sprites/README.md
scripts/
slice_spritesheet.py cuts a grid sprite sheet into the per-frame convention
tests/ pure-logic unit tests (state machine, wake-phrase
matching, HTTP client against mocks) plus one
end-to-end smoke test that drives whole turns against
a real local HTTP server — nothing here needs real
audio hardware or a display
- **`filectl {json}`**: list, read, write or edit a file (`file_ops.py`). This is what makes file
editing reliable, because the model never has to hand-quote a shell heredoc. Edits must match
exactly once. Common JSON slips, such as a raw `C:\Users\...` path, are repaired and the model is
told about the repair.
- **`petctl …` / `dialoguectl …`**: the pet's body and voice. These are not run. Bolt is told
there's no pet here.
- **anything else**: a real shell command, run in the working directory, with a
`COMMAND_TIMEOUT_SECONDS` limit (default 120s).
Reads and listings never ask. Shell commands, writes and edits ask first in `ask` mode (the
default). Edits show a diff before you answer.
## Security
Anything you approve runs as your user, with your permissions. In `auto` mode that means anything
the server asks for. Keep `DESK_API_KEY` private and don't expose the desk API to the open internet.
Each message includes your working directory and OS name (`CONSOLE_CONTEXT=false` to stop that).
## Development
```bash
QT_QPA_PLATFORM=offscreen .venv/bin/pytest tests/ # Windows: .venv\Scripts\pytest tests
```
## Security notes
`tests/test_gui.py` drives turns through the real window and its worker thread against the same
loopback server.
Same as `desk_client/bolt_desk.py`: the server can relay a shell command
back to this machine ("full desktop control" — "open firefox", "how full is
my disk", etc.), which this client executes as your desktop user with a
30-second timeout (`COMMAND_TIMEOUT_SECONDS`). That's the same trust model
as the Linux desk client and the Android app — commands only ever originate
from your own voice/click requests in your own session. Keep `DESK_API_KEY`
private; don't expose the desk API port to the open internet.
`tests/test_console.py` runs whole turns against a real HTTP server on a loopback port. Only the
keyboard is faked. Everything else takes its collaborators as arguments, so no test needs a real
terminal or server.
`petctl` commands (move/emote/say/wander/nap) are handled inside the pet and
never reach a shell, so that channel can't run anything.
Two features widen what leaves this machine, both off-switchable in `.env`:
`SCREEN_CONTEXT=true` (default) appends the focused window's *title* to what
you say — titles often contain file paths, document names or email subjects —
and `NOTIFICATION_BRIDGE=false` (default) can forward matching desktop
notifications. No screenshots or images are ever sent.
## Known limitations / not-yet-done
- Wandering is a straight walk to a random point — no Shimeji-style physics,
wall-climbing or falling.
- Push-to-talk and the notification bridge are platform-limited: the hotkey
needs a session that allows global key hooks (most Wayland setups don't),
and the notification bridge is Linux/D-Bus only.
- Barge-in listens through the same mic that hears the pet's own voice. It
wants headphones or a decent gap between speaker and mic; if playback
interrupts itself, raise `BARGE_IN_RMS_THRESHOLD` or set `BARGE_IN=false`.
- Screen context is the window *title* only — the desk API takes text, so
there's no screenshot understanding.
This project started as a copy of Bolt-Pet. See that repo for the voice/desktop-pet version.