- A final is stored under the channel, speaker name and time fixed when its end was decided. Decoding takes a while; a move, rename or late joiner meanwhile used to change where and when it was said, and who heard it. - At start the worker closes presence a killed run left open (HistoryStore.close_open_presence), so a person who is gone no longer hears new speech after a restart. - run_reply_engine(legacy_memory=False) skips v1's single-user memory (warm profile, extractor, diary, graph, digest); the v2 brain uses it so another user's profile or diary never reaches the prompt. - While a brain reply runs, stdout and stderr are muted (also helper threads such as the web fetch pool); the worker logs through voice_worker/log.py to the original stderr. - Live check: Ollama's /v1 fixes the context at 2,048 tokens and cut the prompt in earlier runs; with an 8k tag the chat-record question was answered 4 of 4 and nothing reached the output. Noted in llm_contexts and the eval. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
16 KiB
Voice worker spec (v2 M1)
The voice worker is the Python process that turns many Discord speakers' audio into text and the
assistant's reply text into speech. The Electron app's Discord voice core talks to it over one local
WebSocket. Design background lives in docs/v2/voice-and-llm.md and docs/v2/architecture.md.
Principles
- Local and private. The server binds to a loopback address only and refuses any other host. Every
connection must present the token the app generated; the token comes from the environment
(
JARVIS_VOICE_TOKEN), never from the command line, so it does not show in process listings. Logs carry timings, sizes and counts, never transcript or reply text. - One shared model. A single faster-whisper model serves every speaker. Speakers that need transcription at the same moment go through the model in one batch.
- Finals before partials. A finished utterance is always transcribed before any live caption.
- Language-agnostic. Nothing in the worker matches words. The STT language is a setting (
nullmeans detect per utterance); sentence splitting uses punctuation only. - Engines are pluggable. STT, VAD and TTS sit behind small interfaces so the logic is tested with fakes and the real models are tested separately.
Running
JARVIS_VOICE_TOKEN=<secret> python -m voice_worker --host 127.0.0.1 --port 8771 \
[--stt-model large-v3-turbo] [--stt-device cuda] [--stt-compute int8_float16] \
[--melo-python <path to the MeloTTS venv python>] [--melo-device cuda] [--melo-speed 1.5]
[--history <history.sqlite>] [--brain]
Run from src/ (or with src/ on PYTHONPATH) in a venv built from voice_worker/requirements.txt. MeloTTS needs its own Python 3.11 venv
(transformers 4.27), so it runs as a child process (voice_worker/melo_child.py) under
--melo-python. Without --melo-python the worker has no TTS and answers tts requests with an
error. Both models load and warm up once at start; hello_ok reports whether each is ready.
Protocol
Text frames are JSON objects with a type. Binary frames start with a one-byte kind.
Handshake
The first frame must be hello within 5 seconds, otherwise the worker closes with code 4400. A wrong
token closes with 4401.
-> {"type": "hello", "token": "...",
"audio": {"rate": 48000, "channels": 2}, // incoming PCM format: rate 16000|48000, channels 1|2
"tts": {"rate": 48000, "channels": 2}, // outgoing PCM format (any rate, channels 1|2)
"stt": { ...settings, see config... }} // optional
<- {"type": "hello_ok", "stt_ready": true, "tts_ready": true,
"audio": {...}, "tts": {...}, "stt": { ...effective settings... }}
STT
-> binary 0x01 | u8 n | speaker id (n bytes UTF-8) | PCM16LE samples in the hello "audio" format
-> {"type": "speaking_end", "speaker": "..."} // Discord speaking stopped for this speaker
-> {"type": "speaker_leave", "speaker": "..."} // the speaker left the voice channel
-> {"type": "config", ...any STT settings...} // takes effect immediately
<- {"type": "speech_start", "speaker": "...", "utt": 3}
<- {"type": "partial", "speaker": "...", "utt": 3, "text": "..."}
<- {"type": "final", "speaker": "...", "utt": 3, "text": "...", "end_reason": "silence",
"audio_ms": 2310, "wait_ms": 160, "stt_ms": 120}
speaker_leave ends an open utterance as speaking_end does (its final is still sent) and frees the
speaker's state. A session holds at most 32 speakers at once, so the app sends it whenever someone
leaves; a speaker who comes back starts again at utt 1.
utt counts utterances per speaker. final always closes an utterance that sent speech_start, even
when the text is empty (noise that Whisper rejected). wait_ms is from the moment the end was decided
to the moment the final was sent; stt_ms is the model time of the batch it went through.
STT settings (all optional in hello and config):
| Setting | Default | Meaning |
|---|---|---|
partials |
true |
Send live captions. false drops queued partials and stops new ones; finals continue. The app turns it off while the LLM answers during a game. |
partial_interval_ms |
400 | At most one partial per speaker per interval, only when audio grew. |
end_silence_ms |
300 | Trailing VAD silence that ends an utterance. Range 100-2000. |
final_beam |
2 | Beam size for finals (partials always use 1). Range 1-5. |
language |
"ko" |
Whisper language code, or null to detect per utterance. |
hotwords |
[] |
Words (wake words, names) passed to Whisper as a hint. The hint alone made Whisper write the wake word on laughter, so with hotwords every clip is also decoded without them and that plain decode decides (see Hallucination gate). |
Utterance rules
Each speaker's audio is mixed to mono, resampled to 16 kHz with a streaming filter, and run through its own streaming Silero VAD (512-sample windows, state kept between frames).
- Start: the first window with speech probability at or above 0.5. The 200 ms before it are kept.
- speech_start is sent once the utterance holds 250 ms of clearly voiced windows (probability at or above 0.5). An utterance that ends before that is dropped without any message and never reaches the model: a click, a cough, or noise.
- Speech continues while windows stay at or above 0.35 (hysteresis); a quieter window counts as silence. Windows between 0.35 and 0.5 keep an utterance open but are not evidence of a voice, so noise the VAD half-believes (fans, typing) never adds up to an utterance.
- End, with
end_reason:silence: trailing silence reachedend_silence_ms.speaking_end: the app sentspeaking_end. Discord stops sending packets when a user stops talking, so the worker cannot wait for silence that never arrives.idle: no audio arrived forend_silence_ms(a missedspeaking_end).max_length: the utterance reached 28 seconds (Whisper's window). The next audio starts a new utterance.
- The final clip keeps at most 200 ms of trailing silence.
- Loudness equalisation before transcription (partials and finals), so a quiet and a loud speaker reach the model alike: rumble below 60 Hz is removed, the level is measured as the RMS of the voiced windows only (silence, a click or a knock do not set the gain), the clip is brought to -20 dBFS with the gain kept between -20 and +40 dB, and peaks above 0.8 are soft-limited below 0.98 instead of clipping. Measured: speech from -60 dBFS to clipped shouting was transcribed equally well, and quiet speech under loud clicks or knocks better than with peak normalisation.
Scheduling
One thread owns the STT model and loops:
- If any finals are waiting, take all of them (up to 16) and transcribe in one batch with
final_beam. - Otherwise take up to 2 waiting partials and transcribe them in one batch with beam 1. A speaker has at most one waiting partial; a newer request replaces the older one and goes to the back of the line, so speakers take turns.
Whisper's encoder always runs on a full 30-second window, so its cost grows with every clip in a batch (measured on an RTX 5050: 0.2 s for one clip, 2.1 s for ten). A large partial batch would make a final wait seconds behind it; two clips keep that wait to about one clip's time. With many people talking, captions fall behind and only the newest audio per speaker is transcribed.
Batches are grouped by decode settings (beam, language, hotwords). A partial whose utterance has already ended, or that is older than the last partial sent, is dropped.
Hallucination gate
Whisper writes text for voice-like sounds without words (coughs, laughter, a crying baby) and, prompted with a hint, the hint itself. A transcript is kept only when all of these hold, else the text is empty:
- It has letters or digits (not only punctuation).
- Its mean log-probability per token is at least -0.7. This is the decoder's own per-token mean; the
avg_logprobused by faster-whisper divides by one token more, which lifted a three-token hallucination from -0.73 to -0.55 and let it through. - Its compression ratio (UTF-8 bytes over zlib bytes) is at most 2.4, which catches repetition loops.
- With hotwords: the clip is decoded once with them and once without, from the same encoder pass. The plain decode must pass the checks above, otherwise nothing was said. The hinted text is used only when it stays within half the plain text's letters of it and contains every hotword exactly as often as the plain one (ignoring case, spacing and punctuation, no fuzzy matching), so the hint can never add, drop or create a wake word: not from laughter, not after a real sentence, not from a one-letter wake word, and not by turning a similar word ("서비스", or the near miss "자비수") into it. Otherwise the plain text is used; matching near misses is left to call detection.
no_speech_prob is not used: large-v3-turbo reports 0.0 for silence and noise alike.
Measured (2026-11-11, RTX 5050) with 54 real noise recordings from ESC-50 (typing, mouse, breathing,
coughing, laughing, crying, vacuum, ...), each at its own level and at -25 dBFS: of the 108 noise-only
clips streamed through the worker with the wake word hint on, 103 never reached the model (the VAD
rule above) and the 5 that did (coughs, laughter, crying) all came out empty
(test_real_noise_recordings_produce_no_text with JARVIS_NOISE_DIR). Over 89 mixes of speech with that noise
(SNR 20, 10 and 0 dB, plus clean) the mean character error rate was 4.8%. Left over: one
"thank you" written for laughter just before a sentence (mean log-probability -0.70, right at the
line), two short words lost in loud noise, and one word repeated three times at SNR 0. A stricter
line (-0.6) would lose more real words than it saves.
TTS
-> {"type": "tts", "id": "r1", "text": "...", "more": false} // more=true: more text follows for r1
-> {"type": "tts_cancel", "id": "r1"}
<- binary 0x02 | u8 n | id (n bytes UTF-8) | u32 LE seq | PCM16LE in the hello "tts" format
<- {"type": "tts_done", "id": "r1", "chunks": 3, "cancelled": false, "failed": false}
Text is split into sentences on punctuation only: a run of newlines, CJK terminators (。!?…), or
ASCII terminators followed by whitespace, a closing quote or bracket, or the end of the text, so
decimals, versions and URLs stay whole. Fragments under 5 characters are merged into a neighbour.
With more=true the worker synthesises every complete sentence at once and holds the unfinished tail
until more text or more=false arrives (a short last fragment is then spoken on its own, because the
sentence before it has already gone out). One audio chunk is sent per sentence, in order, as soon as it
is ready. Requests are served first come, first served by one TTS engine. tts_cancel drops the
request's unsent sentences, and tts_done follows with cancelled: true. A sentence that fails to
synthesise is reported with error (code: "tts_failed") and the request ends with failed: true.
History and brain (M3)
The worker also hosts the brain (the existing jarvis reply engine) and the conversation history
store (src/jarvis/memory/history.spec.md). Start it with --history <file> to keep history and
--brain to answer (--brain needs --history; it loads the jarvis settings, whose
ollama_base_url points at Strata's /v1). Without them the messages below are accepted and
ignored, except reply, which answers error brain_unavailable.
The app tells the worker where Jarvis is and who is there. Speaker IDs are Discord user IDs, the same strings as in audio frames. Times are taken by the worker on arrival, except chat messages, which carry Discord's own time.
-> {"type": "voice_join", "channel": "<voice channel id>"} // Jarvis joined (or moved to) this channel
-> {"type": "voice_leave"} // Jarvis left the voice channel
-> {"type": "member_join", "user": "<id>", "name": "<display name>"} // also for everyone already there
-> {"type": "member_leave", "user": "<id>"}
-> {"type": "chat_message", "channel": "...", "message_id": "...", "author": "<id>", "name": "...",
"text": "...", "ts": 1762830000.5, "to": "<id>"} // to: only on Jarvis's own answer to someone
-> {"type": "chat_edit", "channel": "...", "message_id": "...", "text": "..."}
-> {"type": "chat_delete", "channel": "...", "message_id": "..."}
-> {"type": "reply", "id": "r1", "user": "<id>", "text": "...", "via": "voice", "speak": true}
<- {"type": "reply_text", "id": "r1", "text": "...", "failed": false}
- At start the worker closes every presence interval a previous run left open (it may have been killed); the app then reports who is there.
voice_joincloses the presence of everyone in the previous channel and starts a new one;voice_leaveand the end of the session close it, because nothing said after that is heard.member_joinopens the user's presence in the current channel and remembers their name;member_leavecloses it. Members outside a voice channel are ignored.- Every non-empty
finalis stored as avoicerecord under the channel, the speaker's name (the user ID untilmember_joinnames them) and the time of the moment its end was decided, all fixed then: decoding takes a while, and a move, a rename or a late joiner meanwhile must not change where or when it was said, or who heard it. A final said outside a voice channel is not stored. - Chat messages are stored only as given; the app sends only registered channels.
replyasks the brain to answeruser'stext. The worker builds the history context for that user (their scope only) and runs the reply engine on a worker thread, one reply at a time.reply_textcarries the answer (failed: truewith empty text when the brain gave none). A voice answer (via: "voice") is stored as areplyrecord to the user in the current channel; a chat answer is stored when the app reports the sent message aschat_messagewithto. Withspeak: truethe answer is also spoken through the TTS path under the same id (audio frames andtts_done), andtts_cancelwith that id stops it. Each user keeps their own short-term dialogue memory (tool carry-over) across replies. The brain runs the reply engine with v1's single-user memory off (legacy_memory=False: no warm profile, diary or graph), because several people share this assistant and only the asker's scoped history may reach the prompt.- While a reply runs, the process's stdout and stderr are muted: the v1 engine and its tools print
questions, answers, tool arguments and memory text, also from helper threads. The worker's own log
(
voice_worker/log.py) writes to the original stderr and keeps working. - Memory settings, optional in
helloas"memory": {...}and changeable with{"type": "memory_config", ...}:chat(true),others(true),off(false),since_days(null: whole history),recent_minutes(30),recent_records(20),matches(5),budget_tokens(4000). - Logs carry IDs, counts and timings only, never record, question or answer text.
Errors
{"type": "error", "code": "...", "message": "...", "id": "..."}. Codes: bad_message (unknown type,
bad JSON, bad field), bad_frame (malformed binary frame), tts_unavailable, tts_failed,
brain_unavailable. Errors
never close the connection after the handshake.
Debug logging
debug_log(..., "voice_worker") at: connection accepted or refused, settings changed, speaker left, utterance end
(speaker, reason, length), each STT batch (kind, size, model time), each TTS sentence (length, time),
cancel, and engine failures.