Files
javis_bot/src/voice_worker/voice_worker.spec.md
javis-bot 46e4e75d05 feat(voice_worker): a short voice prompt and one judgement before each v2 answer
- compact_prompt: the v2 persona in a tenth of v1's words, answers as short as
  possible, the full tool guidance only on turns with a data tool. A greeting's
  reply prompt went from about 4,500 to 1,400 tokens on Strata.
- intent_judge: the tool router judges whether the turn needs tools; the
  planner runs only when it does. prepare_intent runs the pick ahead on a
  partial transcript (worker message prepare), and the reply reuses it.
- Without stream_browser the screen-share tools are not offered.
- The time context carries the 12-hour form (10:37 was read as pm).
- A <tool_call> block written as the reply is parsed as a call, not spoken.

Evals (Qwen3.6 on Strata, EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1):
221 passed, 4 failed (known failures); new evals/test_voice_reply_length.py
(v2 answers 234 chars against v1's 416) and test_reply_language_default.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:25:15 +09:00

22 KiB

Voice worker spec (v2 M1)

The voice worker is the Python process that turns many Discord speakers' audio into text and the assistant's reply text into speech. The Electron app's Discord voice core talks to it over one local WebSocket. Design background lives in docs/v2/voice-and-llm.md and docs/v2/architecture.md.

Principles

  • Local and private. The server binds to a loopback address only and refuses any other host. Every connection must present the token the app generated; the token comes from the environment (JARVIS_VOICE_TOKEN), never from the command line, so it does not show in process listings. Logs carry timings, sizes and counts, never transcript or reply text.
  • One shared model. A single faster-whisper model serves every speaker. Speakers that need transcription at the same moment go through the model in one batch.
  • Finals before partials. A finished utterance is always transcribed before any live caption.
  • Language-agnostic. Nothing in the worker matches words. The STT language is a setting (null means detect per utterance); sentence splitting uses punctuation only.
  • Engines are pluggable. STT, VAD and TTS sit behind small interfaces so the logic is tested with fakes and the real models are tested separately.

Running

JARVIS_VOICE_TOKEN=<secret> python -m voice_worker --host 127.0.0.1 --port 8771 \
    [--stt-model large-v3-turbo] [--stt-device cuda] [--stt-compute int8_float16] \
    [--melo-python <path to the MeloTTS venv python>] [--melo-device cuda] [--melo-speed 1.5]
    [--history <history.sqlite>] [--brain]

Run from src/ (or with src/ on PYTHONPATH) in a venv built from voice_worker/requirements.txt. MeloTTS needs its own Python 3.11 venv (transformers 4.27), so it runs as a child process (voice_worker/melo_child.py) under --melo-python. Without --melo-python the worker has no TTS and answers tts requests with an error. Both models load and warm up once at start; hello_ok reports whether each is ready.

Protocol

Text frames are JSON objects with a type. Binary frames start with a one-byte kind.

Handshake

The first frame must be hello within 5 seconds, otherwise the worker closes with code 4400. A wrong token closes with 4401.

-> {"type": "hello", "token": "...",
    "audio": {"rate": 48000, "channels": 2},        // incoming PCM format: rate 16000|48000, channels 1|2
    "tts": {"rate": 48000, "channels": 2},          // outgoing PCM format (any rate, channels 1|2)
    "stt": { ...settings, see config... }}           // optional
<- {"type": "hello_ok", "stt_ready": true, "tts_ready": true,
    "audio": {...}, "tts": {...}, "stt": { ...effective settings... }}

STT

-> binary  0x01 | u8 n | speaker id (n bytes UTF-8) | PCM16LE samples in the hello "audio" format
-> {"type": "speaking_end", "speaker": "..."}        // Discord speaking stopped for this speaker
-> {"type": "speaker_leave", "speaker": "..."}       // the speaker left the voice channel
-> {"type": "config", ...any STT settings...}       // takes effect immediately
<- {"type": "speech_start", "speaker": "...", "utt": 3}
<- {"type": "partial", "speaker": "...", "utt": 3, "text": "..."}
<- {"type": "final", "speaker": "...", "utt": 3, "text": "...", "end_reason": "silence",
    "audio_ms": 2310, "wait_ms": 160, "stt_ms": 120}

speaker_leave ends an open utterance as speaking_end does (its final is still sent) and frees the speaker's state. A session holds at most 32 speakers at once, so the app sends it whenever someone leaves; a speaker who comes back starts again at utt 1.

utt counts utterances per speaker. final always closes an utterance that sent speech_start, even when the text is empty (noise that Whisper rejected). wait_ms is from the moment the end was decided to the moment the final was sent; stt_ms is the model time of the batch it went through.

STT settings (all optional in hello and config):

Setting Default Meaning
partials true Send live captions. false drops queued partials and stops new ones; finals continue. The app turns it off while the LLM answers during a game.
partial_interval_ms 400 At most one partial per speaker per interval, only when audio grew.
end_silence_ms 300 Trailing VAD silence that ends an utterance. Range 100-2000.
final_beam 2 Beam size for finals (partials always use 1). Range 1-5.
language "ko" Whisper language code, or null to detect per utterance.
hotwords [] Words (wake words, names) passed to Whisper as a hint. The hint alone made Whisper write the wake word on laughter, so with hotwords every clip is also decoded without them and that plain decode decides (see Hallucination gate).

Utterance rules

Each speaker's audio is mixed to mono, resampled to 16 kHz with a streaming filter, and run through its own streaming Silero VAD (512-sample windows, state kept between frames).

  • Start: the first window with speech probability at or above 0.5. The 200 ms before it are kept.
  • speech_start is sent once the utterance holds 250 ms of clearly voiced windows (probability at or above 0.5). An utterance that ends before that is dropped without any message and never reaches the model: a click, a cough, or noise.
  • Speech continues while windows stay at or above 0.35 (hysteresis); a quieter window counts as silence. Windows between 0.35 and 0.5 keep an utterance open but are not evidence of a voice, so noise the VAD half-believes (fans, typing) never adds up to an utterance.
  • End, with end_reason:
    • silence: trailing silence reached end_silence_ms.
    • speaking_end: the app sent speaking_end. Discord stops sending packets when a user stops talking, so the worker cannot wait for silence that never arrives.
    • idle: no audio arrived for end_silence_ms (a missed speaking_end).
    • max_length: the utterance reached 28 seconds (Whisper's window). The next audio starts a new utterance.
  • The final clip keeps at most 200 ms of trailing silence.
  • Loudness equalisation before transcription (partials and finals), so a quiet and a loud speaker reach the model alike: rumble below 60 Hz is removed, the level is measured as the RMS of the voiced windows only (silence, a click or a knock do not set the gain), the clip is brought to -20 dBFS with the gain kept between -20 and +40 dB, and peaks above 0.8 are soft-limited below 0.98 instead of clipping. Measured: speech from -60 dBFS to clipped shouting was transcribed equally well, and quiet speech under loud clicks or knocks better than with peak normalisation.

Scheduling

One thread owns the STT model and loops:

  1. If any finals are waiting, take all of them (up to 16) and transcribe in one batch with final_beam.
  2. Otherwise take up to 2 waiting partials and transcribe them in one batch with beam 1. A speaker has at most one waiting partial; a newer request replaces the older one and goes to the back of the line, so speakers take turns.

Whisper's encoder always runs on a full 30-second window, so its cost grows with every clip in a batch (measured on an RTX 5050: 0.2 s for one clip, 2.1 s for ten). A large partial batch would make a final wait seconds behind it; two clips keep that wait to about one clip's time. With many people talking, captions fall behind and only the newest audio per speaker is transcribed.

Batches are grouped by decode settings (beam, language, hotwords). A partial whose utterance has already ended, or that is older than the last partial sent, is dropped.

Hallucination gate

Whisper writes text for voice-like sounds without words (coughs, laughter, a crying baby) and, prompted with a hint, the hint itself. A transcript is kept only when all of these hold, else the text is empty:

  • It has letters or digits (not only punctuation).
  • Its mean log-probability per token is at least -0.7. This is the decoder's own per-token mean; the avg_logprob used by faster-whisper divides by one token more, which lifted a three-token hallucination from -0.73 to -0.55 and let it through.
  • Its compression ratio (UTF-8 bytes over zlib bytes) is at most 2.4, which catches repetition loops.
  • With hotwords: the clip is decoded once with them and once without, from the same encoder pass. The plain decode must pass the checks above, otherwise nothing was said. The hinted text is used only when it stays within half the plain text's letters of it and contains every hotword exactly as often as the plain one (ignoring case, spacing and punctuation, no fuzzy matching), so the hint can never add, drop or create a wake word: not from laughter, not after a real sentence, not from a one-letter wake word, and not by turning a similar word ("서비스", or the near miss "자비수") into it. Otherwise the plain text is used; matching near misses is left to call detection.

no_speech_prob is not used: large-v3-turbo reports 0.0 for silence and noise alike.

Measured (2026-11-11, RTX 5050) with 54 real noise recordings from ESC-50 (typing, mouse, breathing, coughing, laughing, crying, vacuum, ...), each at its own level and at -25 dBFS: of the 108 noise-only clips streamed through the worker with the wake word hint on, 103 never reached the model (the VAD rule above) and the 5 that did (coughs, laughter, crying) all came out empty (test_real_noise_recordings_produce_no_text with JARVIS_NOISE_DIR). Over 89 mixes of speech with that noise (SNR 20, 10 and 0 dB, plus clean) the mean character error rate was 4.8%. Left over: one "thank you" written for laughter just before a sentence (mean log-probability -0.70, right at the line), two short words lost in loud noise, and one word repeated three times at SNR 0. A stricter line (-0.6) would lose more real words than it saves.

TTS

-> {"type": "tts", "id": "r1", "text": "...", "more": false}   // more=true: more text follows for r1
-> {"type": "tts_cancel", "id": "r1"}
<- binary  0x02 | u8 n | id (n bytes UTF-8) | u32 LE seq | PCM16LE in the hello "tts" format
<- {"type": "tts_done", "id": "r1", "chunks": 3, "cancelled": false, "failed": false}

Text is split into sentences on punctuation only: a run of newlines, CJK terminators (。!?…), or ASCII terminators followed by whitespace, a closing quote or bracket, or the end of the text, so decimals, versions and URLs stay whole. Fragments under 5 characters are merged into a neighbour. With more=true the worker synthesises every complete sentence at once and holds the unfinished tail until more text or more=false arrives (a short last fragment is then spoken on its own, because the sentence before it has already gone out). One audio chunk is sent per sentence, in order, as soon as it is ready. Requests are served first come, first served by one TTS engine. tts_cancel drops the request's unsent sentences, and tts_done follows with cancelled: true. A sentence that fails to synthesise is reported with error (code: "tts_failed") and the request ends with failed: true.

History and brain (M3)

The worker also hosts the brain (the existing jarvis reply engine) and the conversation history store (src/jarvis/memory/history.spec.md). Start it with --history <file> to keep history and --brain to answer (--brain needs --history; it loads the jarvis settings, whose ollama_base_url points at Strata's /v1). Without them the messages below are accepted and ignored, except reply, which answers error brain_unavailable.

The app tells the worker where Jarvis is and who is there. Speaker IDs are Discord user IDs, the same strings as in audio frames. Times are taken by the worker on arrival, except chat messages, which carry Discord's own time.

-> {"type": "voice_join", "channel": "<voice channel id>"}   // Jarvis joined (or moved to) this channel
-> {"type": "voice_leave"}                                   // Jarvis left the voice channel
-> {"type": "member_join", "user": "<id>", "name": "<display name>"}   // also for everyone already there
-> {"type": "member_leave", "user": "<id>"}
-> {"type": "chat_message", "channel": "...", "message_id": "...", "author": "<id>", "name": "...",
    "text": "...", "ts": 1762830000.5, "to": "<id>"}   // to: only on Jarvis's own answer to someone
-> {"type": "chat_edit", "channel": "...", "message_id": "...", "text": "..."}
-> {"type": "chat_delete", "channel": "...", "message_id": "..."}
-> {"type": "reply", "id": "r1", "user": "<id>", "text": "...", "via": "voice", "speak": true}
<- {"type": "reply_text", "id": "r1", "text": "...", "failed": false}
-> {"type": "reply_delivered", "id": "c1", "ok": true}   // a chat answer was posted (ok: false: it was not)
-> {"type": "prepare", "user": "<id>", "text": "..."}   // a call still being said; nothing comes back
-> {"type": "addressed", "id": "q1", "user": "<id>", "text": "..."}
<- {"type": "addressed_result", "id": "q1", "addressed": true, "failed": false}
-> {"type": "rank", "id": "k1", "candidates": [{"id": "7", "user": "<id>", "text": "..."}]}
<- {"type": "rank_result", "id": "k1", "order": ["7"], "failed": false}
  • At start the worker closes every presence interval a previous run left open (it may have been killed); the app then reports who is there.
  • voice_join closes the presence of everyone in the previous channel and starts a new one; voice_leave and the end of the session close it, because nothing said after that is heard. member_join opens the user's presence in the current channel and remembers their name; member_leave closes it. Members outside a voice channel are ignored.
  • Every non-empty final is stored as a voice record under the channel, the speaker's name (the user ID until member_join names them) and the time of the moment its end was decided, all fixed then: decoding takes a while, and a move, a rename or a late joiner meanwhile must not change where or when it was said, or who heard it. A final said outside a voice channel is not stored.
  • Chat messages are stored only as given; the app sends only registered channels.
  • reply asks the brain to answer user's text. The worker builds the history context for that user (their scope only) and runs the reply engine on a worker thread, one voice reply and one chat reply at a time: the two kinds have separate lanes, so a voice answer never waits behind another person's chat answer (app/src/conversation.spec.md, chat calls). One person's replies stay one at a time across both lanes, in the order asked, so their dialogue memory and delete confirmations ("the very next message") follow that order. reply_text carries the answer (failed: true with empty text when the brain gave none). A voice answer (via: "voice") is stored as a reply record to the user in the current channel; a chat answer is stored when the app reports the sent message as chat_message with to. With speak: true the answer is also spoken through the TTS path under the same id (audio frames and tts_done), and tts_cancel with that id stops it. A tts_cancel that arrives while the brain is still answering means the answer is not spoken: reply_text still comes, then tts_done with cancelled: true and no audio. Each user keeps their own short-term dialogue memory (tool carry-over) across replies. The brain runs the reply engine with v1's single-user memory off (legacy_memory=False: no warm profile, diary or graph), because several people share this assistant and only the asker's scoped history may reach the prompt. It answers in Korean by default (--reply-language, default Korean; '' answers in the asker's language), whatever language the question was heard or written in, and in another language only when the asker explicitly asks for it (just that answer, or from then on, as they said). Tools get the ISO code --tool-language (default ko). v1's tts_engine setting (an English-only Piper voice by default) does not lock replies to English, because v2 speaks through the worker's TTS. An output_language lock in the jarvis settings still wins. Eval: evals/test_reply_language_default.py. STT (language "ko") and TTS (--melo-language KR) are Korean by default too. Two reply options keep the time to the first word short (Strata reads every prompt from the start, about 0.36 s per 2,048 tokens, and each LLM call before the answer adds its whole time): the short voice persona (compact_prompt=True: the same butler in a tenth of v1's words, without v1's single-user memory and on-screen browser rules, answers as short as possible, the full tool guidance only on turns that offer a data tool), and the intent judge (intent_judge=True: the tool router judges whether the turn needs tools and which, and the planner runs only when it does; planner.spec.md, "Intent judge (v2)"). v1's screen-share browser (stream_browser) is off: it needs v1's Node bot (M6 brings the v2 one), so its tools are not offered.
  • prepare tells the brain that user is saying a call whose partial transcript is text (the app sends it while the speaker is still talking, app/src/conversation.spec.md). The brain runs the intent judge's tool pick for it on its own thread, with the asker's dialogue memory and the tool catalogue the reply will have. A reply from that user whose text is about the same (letters and digits, 85% alike), within 30 s and with the dialogue memory unchanged, uses that pick instead of making its own, waiting up to 5 s for one still being made. Each prepared judgement is used at most once. One prepare per user runs at a time: while it runs only the newest text waits, a text about the same as the one last prepared is dropped, and a reply from that user drops the waiting text. Nothing is sent back; without a brain, or with one that cannot prepare, it is ignored. The asker is set in jarvis.reply.turn_state for the turn, so tools that keep personal data (meals) store and read it under that person only.
  • addressed and rank are the conversation's questions (app/src/conversation.spec.md), answered on their own threads, so they never wait behind a running reply. addressed: was text said to the assistant? The session's hotwords are given to the judge as the assistant's names. rank: the candidate ids most important first (ids the brain left out follow in their order, unknown ids are dropped). A judgement the brain could not make, or that raised, comes back failed: true (addressed: false, empty order). Both are one short non-thinking call to the chat model with the heard text fenced as data (voice_worker/judges.py, eval evals/test_voice_judges.py). Without a brain they answer error brain_unavailable with the id.
  • While a reply runs, the process's stdout and stderr are muted: the v1 engine and its tools print questions, answers, tool arguments and memory text, also from helper threads. The worker's own log (voice_worker/log.py) writes to the original stderr and keeps working.
  • Chat channel tool (src/jarvis/tools/builtin/chat_channels.spec.md): the app sends the registered channels, and the brain's tool asks the app to act. Reads and searches come from the history store.
    -> {"type": "chat_channels", "bot": "<bot user id>", "channels": [{"id": "...", "name": "...",
        "description": "...", "read": true, "send": true, "edit": true, "react": true,
        "delete": false, "pin": false}]}
    <- {"type": "chat_action", "id": "a1", "action": "send|edit|react|delete|pin|unpin",
        "channel": "...", "message_id": "...", "text": "...", "emoji": "..."}
    -> {"type": "chat_action_result", "id": "a1", "ok": true, "message_ids": ["..."]}
    -> {"type": "chat_action_result", "id": "a1", "ok": false, "error": "..."}
    
    Action ids are unique across connections and worker runs, and an answer whose id is not waiting on this connection is ignored, so a late answer meant for an earlier connection never completes a newer action. With no registered channels the tool is not offered. An action waits up to 15 s for its result (then it fails), and a closed session fails every waiting action. Each reply counts as one turn per person, so a delete is confirmed only in that person's next reply. An answer reaches the person when its speech ends with tts_done neither failed nor cancelled (speak: true), when the app reports it posted with reply_delivered ok: true (via: "chat"), or at reply_text for an unspoken voice answer. It never reaches them when the brain gave none, its speech was stopped, failed or had no TTS (tts_unavailable), the app reports ok: false, no report comes within 30 s, or the session ends first. A reply sent before the person's previous answer reached them cannot answer it, and an answer that never reached them asked them nothing: in both cases a turn number is skipped, so a delete asked in that previous answer lapses.
  • Memory settings, optional in hello as "memory": {...} and changeable with {"type": "memory_config", ...}: chat (true), others (true), off (false), since_days (null: whole history), recent_minutes (30), recent_records (20), matches (5), budget_tokens (4000).
  • Logs carry IDs, counts and timings only, never record, question or answer text.

Errors

{"type": "error", "code": "...", "message": "...", "id": "..."}. Codes: bad_message (unknown type, bad JSON, bad field), bad_frame (malformed binary frame), tts_unavailable, tts_failed, brain_unavailable. Errors never close the connection after the handshake.

Debug logging

debug_log(..., "voice_worker") at: connection accepted or refused, settings changed, speaker left, utterance end (speaker, reason, length), each STT batch (kind, size, model time), each TTS sentence (length, time), cancel, and engine failures.