Each worker run's log sink keeps the values masked when it started (its secrets in use), so a line still being logged after a stop gave up waiting, and after the next start replaced the values in use, is still masked. The debug echo on stderr is masked the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
20 KiB
Discord voice core spec (v2 M2)
The Discord voice core is the part of the Windows app (Rust; the GUI arrives in M5 as a Tauri 2 shell)
that connects the bot to Discord: it joins a voice channel, hears each person separately, streams their
audio to the voice worker for transcription, plays the worker's speech back into the channel, and reads
and writes the registered chat channels. It has no assistant logic. It reports what it heard and does
what it is told; the brain (M3) and call detection and response policies (M4) sit on top of it.
Design background: docs/v2/architecture.md, docs/v2/conversation.md, docs/v2/privacy.md. The
worker side of the link is src/voice_worker/voice_worker.spec.md.
Principles
- Only where the bot is. The bot hears only the voice channel it is in and reads only the chat channels registered in settings with reading on. Every join posts the transcription notice in the voice channel's chat.
- Chat is data. Messages read from chat channels are handed on as data, never acted on as commands. Text the bot sends never pings anyone (empty allowed mentions).
- No text in logs. Logs carry ids, sizes, counts and timings, never what anyone said or wrote, or
what the bot says. Debug logs (
debug_log!, on withJARVIS_DEBUG=1) go at joins and leaves, command use, finals, speech requests and stops, chat events and sends, and worker connection changes. - Language-agnostic. Nothing in the core matches words. Wake words go to the STT as a hint; call
detection (
conversation.spec.md) is the only place they are matched. - Testable without Discord. The pieces with rules (receive window, playback queue, speaker limit, chat access and splitting, settings, worker link, worker process) are plain types tested with fakes; the Discord glue is tested on real Discord by an opt-in live test.
Layout
app/
├── Cargo.toml # crate jarvis-app: library + the jarvis-app binary
├── src/main.rs # jarvis-app <data folder>: runs the services until Ctrl+C / SIGTERM
├── src/services.rs # settings -> voice worker process -> worker client -> Discord core
├── src/discord.rs # Discord client (serenity), voice (songbird), slash command, chat channels
├── src/receive.rs # per-speaker reorder window and Opus decoding
├── src/playback.rs # speech queue (TTS requests -> sentence clips -> player)
├── src/earcon.rs # the earcon sound, made in code
├── src/speakers.rs # who is transcribed right now (max speakers)
├── src/chat.rs # chat channel access rules and message splitting
├── src/worker_client.rs # WebSocket client for the voice worker
├── src/worker_process.rs # starts and restarts the Python voice worker
├── src/frames.rs # binary frame layout shared with voice_worker/protocol.py
├── src/settings.rs # settings.json loading and validation
└── src/log.rs # debug_log! / error_log!
Starting
start_services(data_dir) reads <data_dir>/config/settings.json, starts the voice worker process,
connects to it once it prints READY, and starts the Discord core. The bot token comes from the
stored secrets (config/secrets.dat, admin.spec.md), else from JARVIS_DISCORD_TOKEN; the binary
takes the data folder from the first argument or JARVIS_DATA_DIR, installs the log hub (daily files
in logs/) and stops every service on Ctrl+C or SIGTERM.
Settings
<data folder>/config/settings.json. Every key is optional; missing keys take the defaults. A bad
value stops the app with a message naming the key. Unknown keys are ignored. Discord IDs are JSON
strings (numbers would lose precision).
| Key | Default | Meaning |
|---|---|---|
discord.guildId |
null |
The one server. The slash command is registered there; commands and channels from other servers are ignored. |
discord.voiceChannelId |
null |
Join this voice channel on start ("자동 입장"). |
discord.commandName |
"자비스" |
Slash command name (Discord rule: 1-32 lowercase letters, digits, -, _). |
discord.ephemeralReplies |
true |
Command replies only the caller sees. |
voice.maxSpeakers |
10 |
Most people transcribed at once, 1-10. |
voice.listenToBots |
false |
Also transcribe other bots (music bots are ignored by default). |
voice.joinNotice |
"자비스가 이 음성 채널에 들어왔습니다. 이 채널의 대화는 자비스가 받아 적습니다." | Posted in the voice channel's chat on every join. Must not be empty. |
voice.wakeWords |
["자비스"] |
Sent to the worker as STT hotwords, and the voice wake words of call detection. |
voice.stt |
{} |
Other worker STT settings (partials, end_silence_ms, final_beam, language, ...); the worker validates them. |
chatChannels |
[] |
{id, name, description, read (true), send (true), edit (true), react (true), delete (false), pin (false), answer ("wakeWord"), syncDiscord (true)} per channel; answer is in conversation.spec.md, syncDiscord (the channel's Discord name and topic follow name and description) in admin.spec.md. |
voice.tts.speed |
1.5 |
How fast the voice speaks (MeloTTS speed, 0.5-2.0), passed to the worker as --melo-speed when it speaks (--melo-python) and its arguments set no speed. MeloTTS's Korean model has one voice, so there is no voice choice. |
voiceWorker.python |
python3 (python.exe on Windows) |
Python of the worker venv. |
voiceWorker.cwd |
python-src next to the app's executable when it is there (the Windows build), else the repo's src/ |
Folder holding the voice_worker package. |
voiceWorker.args |
[] |
Extra worker arguments, e.g. ["--melo-python", "<MeloTTS venv python>"]. |
memory |
{} |
The worker's memory settings (voice_worker.spec.md, "Memory settings"), sent in its hello; the settings screens check them (admin.spec.md). |
The call and response sections (call detection and response policies, M4) are in
conversation.spec.md.
Voice worker process
The app starts python -m voice_worker --host 127.0.0.1 --port <port> <args> in voiceWorker.cwd
(no console window on Windows). Each launch makes a new random token (32 bytes) and passes it only in
JARVIS_VOICE_TOKEN. The port is a free loopback port chosen once at start. If the worker exits, it is
started again on the same port with the same token after 1 s, doubling to at most 30 s while it keeps
failing (reset once a run printed READY). Stopping kills it; MeloTTS's child process ends when its
input closes. A stop (and a restart) first logs everything the exited worker wrote, waiting at most
5 s (a child of the worker may hold its output open): a stop returns only after its last lines were
masked with the secrets that run was started with (admin.spec.md, "Logs"). Each run's lines are
masked with the values masked when it started too, so a line still being logged after that wait
(and after the next start replaced the values in use) stays masked.
Worker link
One WebSocket to the worker, hello with the token, the Discord PCM format both ways
(48 kHz stereo), and the STT settings (voice.stt with hotwords = voice.wakeWords). If the
connection drops, the client reconnects with backoff (0.5 s doubling to 5 s) and says hello again with
the same settings, including changes made since with configure(). While disconnected, audio frames
are dropped (not queued), and every TTS request in flight ends as failed, so playback moves on. A TTS
request the worker refuses (tts_unavailable) also ends as failed. TTS audio for request ids the client
did not open is ignored.
History reporting (M3)
The core tells the worker what its history store needs (voice_worker.spec.md, "History and brain"):
voice_joinwith the channel after every join or move (also when someone else moves the bot), andmember_joinwith the display name for everyone already in the channel that the bot hears (people; bots only withvoice.listenToBots). Again whenever the worker link becomes ready, because messages sent while disconnected are dropped.member_join/member_leaveas heard people enter and leave the channel.voice_leavewhenever a session ends (leave, move, disconnect).- For every message in a readable registered channel, the bot's own included:
chat_messagewith Discord's time; on the bot's own message that replies to someone,tois that person. Edits (chat_edit), deletes and bulk deletes (chat_delete) follow. Chat sent while the worker is disconnected is not recorded. - Finals are recorded by the worker itself; the core does not echo them.
Voice
One voice session at a time, in the configured server. Joining another channel ends the current
session (Left with moved). Within the same server the existing songbird connection is pointed at
the new channel and its event handlers are replaced: leaving and joining at once races the gateway,
and the late "left" update cancelled the new join (seen on real Discord).
- Join (slash command
join, auto-join, orjoin(channel_id)): connect through songbird with DAVE end-to-end encryption, not deafened, then postvoice.joinNoticein the voice channel's chat. A notice that cannot be posted (no permission) is logged and the bot stays. - Hearing: songbird only decrypts (transport and DAVE). Its own Opus decoding is not used: in
songbird 0.6.1 it decodes the payload without trimming what DAVE decryption left at its end, which
garbled, repeated or dropped about a third of the frames (measured with a test tone that changes
pitch every 20 ms). The core takes each decrypted RTP packet with songbird's exact payload bounds:
- Packets go through a per-speaker reorder window in sequence-number order. A missing packet is waited for until two later packets have arrived (about 40 ms), then rebuilt from the next packet's Opus in-band FEC (the last missing frame) or concealed (any earlier ones). Late and duplicate packets are dropped; a jump of 64 or more starts the window over.
- Frames are decoded with libopus to 48 kHz stereo PCM and sent to the worker in 20 ms frames under the user's ID. Decoding goes on for people who are not heard, so their decoder stays continuous.
- The bot's own audio is never heard; other bots are skipped unless
voice.listenToBots. - At most
voice.maxSpeakerspeople are transcribed at once. Someone who starts while the limit is reached is not heard until their next start after a place frees up. - No packet for 100 ms ends a person's speaking: what their window still holds is played out, then
speaking_endis sent. - When a person leaves the channel (gateway voice state or songbird's client disconnect), the bot
sends
speaker_leave, so their utterance finishes and the worker frees their place.
- Events for the layers above:
SpeechStart,PartialandFinalwith the speaker's ID, server display name and voice session. EachJoinedcarries a new session number (from 1); an utterance belongs to the session Jarvis was in when itsspeech_startcame (none outside a session, and none once that session ended or the worker restarted), and itsPartialandFinalcarry that number even when they arrive later;SpeakingStoppedwith the speaker's ID when theirspeaking_endis sent;SpeakStartandSpeakEnd;Stoppedwhen all speech is stopped (stop(), the slash command or the session ending), sent before theSpeakEnds it causes;JoinedandLeft(with a reason:command,moved,disconnected,shutdown);AddressedandRankedwith the brain's answers toask_addressedandask_rank.WorkerLostwhen the voice worker connection ends: the finals of utterances in progress will not come, and the restarted worker numbers utterances afresh.AnswerAsked { id, user, ended_at, final_at }whenanswer(user, text, heard)queued speechidfor a request whose speech ended and whose final arrived at those times (the dashboard times answers by it,admin.spec.md).voice_members()lists the people in Jarvis's voice channel that it hears (ID and display name), the same set reported to the worker. - Speaking:
say(text, more)queues a TTS request and returns its id (Nonewhen not in a voice channel);continue_text(id, text, more)adds text to an open request. Each sentence the worker sends becomes one clip (a 48 kHz stereo WAV played through songbird without resampling); clips play in sentence order as soon as they arrive, and requests play first come, first served. A sentence that failed is skipped and the rest still play.SpeakEndreports how many sentences played and whether the request was cancelled or failed. - Answering (M4):
answer(user, text)queues a request likesaywhose speech the worker makes from the brain's answer (replywithspeak, under the request's id) and returns its id. No answer from the brain, a refusedreply(brain_unavailable) or a lost worker ends it as failed.cancel(id)stops one request (its clip if playing, the worker's unsent sentences, or the speech of an answer still being written) and the rest of the queue goes on.ask_addressed(id, user, text)andask_rank(id, candidates)pass the conversation's questions to the brain.prepare(user, text)sendsprepare(a call still being said) and expects nothing back. - Earcon (M4):
earcon()queues the app's own earcon clip (app/src/earcon.rs, 48 kHz stereo, peak 0.3 of full scale) as a request of its own: no worker round trip,SpeakStart/SpeakEndlike any request, dropped by a stop. None when not in a voice channel. - Stop (slash command
stop, orstop()): the current clip stops at the next 20 ms mix, every queued request is dropped, and unfinished requests are cancelled at the worker; each ends withcancelled: true. Stopping by voice ("자비스, 멈춰") is call detection's stop (conversation.spec.md), which does the same. - Leave (slash command
leave, orleave()): stop speaking, sendspeaker_leavefor everyone heard, and disconnect. When songbird's driver gives up the connection (kicked, channel deleted, long network loss) the session ends withdisconnected. A driver disconnect without a reason is one the core asked for (leave or move) and is ignored, and a disconnect only ends the session its handler was attached to, so an old connection's late event never ends a newer session. When someone moves the bot to another channel, the session follows it.
Slash command
One guild command, /<commandName> with subcommands join (the caller's voice channel), leave
and stop, registered on every start with a create-or-update by name, so the app's other commands in
the server are left alone. A failed registration (for example a bot invited without the
applications.commands scope) is logged and the bot runs on without it. Usable only in the configured
server. Replies are in Korean, ephemeral by default. Renaming commandName leaves the old command
registered until it is removed (M5 setup).
Chat channels
- The bot asks for the message intents only when some registered channel has reading on. Reading needs the Message Content intent, which is checked in the application's flags before login. When it is off, the bot still logs in, voice works, and an error explains how to turn it on; chat reads then fail with a clear error instead of returning empty messages.
- Live: every new message in a readable registered channel, except the bot's own, is emitted as a
Chatevent: channel, message and author IDs, author display name, whether the author is a bot, the text, and the time. - History:
read_chat(channel_id, limit)returns the latest 1-100 messages, oldest first, withfrom_selfmarking the bot's own. - Send:
send_chat(channel_id, text)sends to a registered channel with sending on, split at Discord's 2,000-character limit at the last newline or space (never inside a character), and returns the message ids. - Answer (M4):
send_chat_reply(channel_id, message_id, text)sends like Send, with the first part as a Discord reply to the message; nobody is pinged, not even its author.chat_answer(id, user, text)asks the brain for a chat answer (replywithvia: "chat"); its text comes asReplyText.reply_delivered(id, ok)tells the worker whether that answer was posted (reply_delivered), so a delete asked in an answer the person never saw cannot be confirmed (voice_worker.spec.md). - A channel that is not registered, or has the needed switch off, is refused with an access error.
- Chat tool actions (M3,
chat_channels.spec.md): when the worker link is ready and after login, the core sendschat_channels(every registered channel with its switches, and the bot's user ID). Achat_actionfrom the worker runschat_action()and answerschat_action_resultwith the message ids or the error text:send(as Send above),edit(only the bot's own messages, one message long, pings nobody),react(a Unicode or custom emoji),delete,pin,unpin. Each needs its channel switch. Discord's own refusals come back as errors, for example a voice channel's chat cannot pin.
Tests
cargo test: unit tests for the receive window (with a tagging fake and real libopus), playback order, gaps, stop and cancel, the earcon clip (before an answer, dropped by a stop; its sound short, silent at both ends, below its peak), speaker limit, chat access and splitting, settings, frames; the worker link against a fake server (hello, events, reconnect, failed TTS); the worker process with a stand-invoice_workerpackage run by a real Python (token only in the environment, restart on the same port, stop, the last output logged before a stop returns, a line logged after the stop gave up masked with its run's secrets).JARVIS_VOICE_PYTHON=<venv python> cargo test --test worker_protocolruns the worker link and playback against the real Python worker server with fake engines (tests/fixtures/fake_engine_worker.py).tests/live_discord.rs(opt-in,JARVIS_LIVE_DISCORD=1, see the file header): two bots in one voice channel with the real worker on the GPU. It refuses to run if a person is in the channel or either bot is already in a voice channel, and cleans up even when a step fails. It checks command registration, transcription of recorded sentences played into the channel, speech the other bot hears and the STT reads back, stop, chat events, history, an edit and a delete, sending without pings, moving to a second voice channel and back (the session survives each move and only the two moves end one), the speaker leaving and coming back three times (each a DAVE group change), and leaving. The worker runs with--history; at the end the store must hold the speaker's finals under its display name and channel, its chat message with the edit and without the deleted one, Jarvis's own messages, a presence interval per join, and no presence left open. The chat tool's actions run on the channel too: editing its own message, refusing to edit the speaker's, a reaction, pin (Discord refuses it in a voice channel's chat, which must come back as an error), delete, and refusing an unregistered channel. It needs a second voice channel of the server (JARVIS_LIVE_VOICE_CHANNEL_2) and refuses if a person is in either. Measured on this server's RTX 5050 (2026-11-11): five sentences all heard (character error rate 0% for three, 5% and 12.5% for two), 0.71-0.90 s from the end of speech to the final, Jarvis's speech read back at 3.4-6.9%, sound gone 0.03-0.08 s after stop, and the sentence heard exactly after each of three rejoins. With history (2026-11-11): 8 voice records under the speaker's name, the edited chat message only, 3 of Jarvis's own messages, 5 presence intervals, none open.
Known risks
- songbird issue #310 (open): receive can stall after a DAVE (MLS) group change on some joins. The live test covers four changes (the second bot joining, then leaving and rejoining three times) without a stall; a channel where many people come and go over hours still needs a longer check before M4.