Files
javis_bot/app/src/discord_core.spec.md
javis-bot b88e20134f feat(app): a call still being said lets the brain get ready
A partial transcript that call detection already reads as a request is sent
to the worker as prepare, so the intent judge runs while the speaker talks.
The live round trip now times the earcon and the answer's own sound apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:25:15 +09:00

18 KiB

Discord voice core spec (v2 M2)

The Discord voice core is the part of the Windows app (Rust; the GUI arrives in M5 as a Tauri 2 shell) that connects the bot to Discord: it joins a voice channel, hears each person separately, streams their audio to the voice worker for transcription, plays the worker's speech back into the channel, and reads and writes the registered chat channels. It has no assistant logic. It reports what it heard and does what it is told; the brain (M3) and call detection and response policies (M4) sit on top of it. Design background: docs/v2/architecture.md, docs/v2/conversation.md, docs/v2/privacy.md. The worker side of the link is src/voice_worker/voice_worker.spec.md.

Principles

  • Only where the bot is. The bot hears only the voice channel it is in and reads only the chat channels registered in settings with reading on. Every join posts the transcription notice in the voice channel's chat.
  • Chat is data. Messages read from chat channels are handed on as data, never acted on as commands. Text the bot sends never pings anyone (empty allowed mentions).
  • No text in logs. Logs carry ids, sizes, counts and timings, never what anyone said or wrote, or what the bot says. Debug logs (debug_log!, on with JARVIS_DEBUG=1) go at joins and leaves, command use, finals, speech requests and stops, chat events and sends, and worker connection changes.
  • Language-agnostic. Nothing in the core matches words. Wake words go to the STT as a hint; call detection (conversation.spec.md) is the only place they are matched.
  • Testable without Discord. The pieces with rules (receive window, playback queue, speaker limit, chat access and splitting, settings, worker link, worker process) are plain types tested with fakes; the Discord glue is tested on real Discord by an opt-in live test.

Layout

app/
├── Cargo.toml             # crate jarvis-app: library + the jarvis-app binary
├── src/main.rs            # jarvis-app <data folder>: runs the services until Ctrl+C / SIGTERM
├── src/services.rs        # settings -> voice worker process -> worker client -> Discord core
├── src/discord.rs         # Discord client (serenity), voice (songbird), slash command, chat channels
├── src/receive.rs         # per-speaker reorder window and Opus decoding
├── src/playback.rs        # speech queue (TTS requests -> sentence clips -> player)
├── src/earcon.rs          # the earcon sound, made in code
├── src/speakers.rs        # who is transcribed right now (max speakers)
├── src/chat.rs            # chat channel access rules and message splitting
├── src/worker_client.rs   # WebSocket client for the voice worker
├── src/worker_process.rs  # starts and restarts the Python voice worker
├── src/frames.rs          # binary frame layout shared with voice_worker/protocol.py
├── src/settings.rs        # settings.json loading and validation
└── src/log.rs             # debug_log! / error_log!

Starting

start_services(data_dir) reads <data_dir>/config/settings.json, starts the voice worker process, connects to it once it prints READY, and starts the Discord core. Until the setup wizard (M5) stores it encrypted, the bot token comes from JARVIS_DISCORD_TOKEN and the data folder from the first argument or JARVIS_DATA_DIR. The binary stops every service on Ctrl+C or SIGTERM.

Settings

<data folder>/config/settings.json. Every key is optional; missing keys take the defaults. A bad value stops the app with a message naming the key. Unknown keys are ignored. Discord IDs are JSON strings (numbers would lose precision).

Key Default Meaning
discord.guildId null The one server. The slash command is registered there; commands and channels from other servers are ignored.
discord.voiceChannelId null Join this voice channel on start ("자동 입장").
discord.commandName "자비스" Slash command name (Discord rule: 1-32 lowercase letters, digits, -, _).
discord.ephemeralReplies true Command replies only the caller sees.
voice.maxSpeakers 10 Most people transcribed at once, 1-10.
voice.listenToBots false Also transcribe other bots (music bots are ignored by default).
voice.joinNotice "자비스가 이 음성 채널에 들어왔습니다. 이 채널의 대화는 자비스가 받아 적습니다." Posted in the voice channel's chat on every join. Must not be empty.
voice.wakeWords ["자비스"] Sent to the worker as STT hotwords, and the voice wake words of call detection.
voice.stt {} Other worker STT settings (partials, end_silence_ms, final_beam, language, ...); the worker validates them.
chatChannels [] {id, name, description, read (true), send (true), edit (true), react (true), delete (false), pin (false), answer ("wakeWord")} per channel; answer is in conversation.spec.md.
voiceWorker.python python3 (python.exe on Windows) Python of the worker venv.
voiceWorker.cwd the repo's src/ (development builds) Folder holding the voice_worker package.
voiceWorker.args [] Extra worker arguments, e.g. ["--melo-python", "<MeloTTS venv python>"].

The call and response sections (call detection and response policies, M4) are in conversation.spec.md.

Voice worker process

The app starts python -m voice_worker --host 127.0.0.1 --port <port> <args> in voiceWorker.cwd (no console window on Windows). Each launch makes a new random token (32 bytes) and passes it only in JARVIS_VOICE_TOKEN. The port is a free loopback port chosen once at start. If the worker exits, it is started again on the same port with the same token after 1 s, doubling to at most 30 s while it keeps failing (reset once a run printed READY). Stopping kills it; MeloTTS's child process ends when its input closes.

One WebSocket to the worker, hello with the token, the Discord PCM format both ways (48 kHz stereo), and the STT settings (voice.stt with hotwords = voice.wakeWords). If the connection drops, the client reconnects with backoff (0.5 s doubling to 5 s) and says hello again with the same settings, including changes made since with configure(). While disconnected, audio frames are dropped (not queued), and every TTS request in flight ends as failed, so playback moves on. A TTS request the worker refuses (tts_unavailable) also ends as failed. TTS audio for request ids the client did not open is ignored.

History reporting (M3)

The core tells the worker what its history store needs (voice_worker.spec.md, "History and brain"):

  • voice_join with the channel after every join or move (also when someone else moves the bot), and member_join with the display name for everyone already in the channel that the bot hears (people; bots only with voice.listenToBots). Again whenever the worker link becomes ready, because messages sent while disconnected are dropped.
  • member_join / member_leave as heard people enter and leave the channel.
  • voice_leave whenever a session ends (leave, move, disconnect).
  • For every message in a readable registered channel, the bot's own included: chat_message with Discord's time; on the bot's own message that replies to someone, to is that person. Edits (chat_edit), deletes and bulk deletes (chat_delete) follow. Chat sent while the worker is disconnected is not recorded.
  • Finals are recorded by the worker itself; the core does not echo them.

Voice

One voice session at a time, in the configured server. Joining another channel ends the current session (Left with moved). Within the same server the existing songbird connection is pointed at the new channel and its event handlers are replaced: leaving and joining at once races the gateway, and the late "left" update cancelled the new join (seen on real Discord).

  • Join (slash command join, auto-join, or join(channel_id)): connect through songbird with DAVE end-to-end encryption, not deafened, then post voice.joinNotice in the voice channel's chat. A notice that cannot be posted (no permission) is logged and the bot stays.
  • Hearing: songbird only decrypts (transport and DAVE). Its own Opus decoding is not used: in songbird 0.6.1 it decodes the payload without trimming what DAVE decryption left at its end, which garbled, repeated or dropped about a third of the frames (measured with a test tone that changes pitch every 20 ms). The core takes each decrypted RTP packet with songbird's exact payload bounds:
    • Packets go through a per-speaker reorder window in sequence-number order. A missing packet is waited for until two later packets have arrived (about 40 ms), then rebuilt from the next packet's Opus in-band FEC (the last missing frame) or concealed (any earlier ones). Late and duplicate packets are dropped; a jump of 64 or more starts the window over.
    • Frames are decoded with libopus to 48 kHz stereo PCM and sent to the worker in 20 ms frames under the user's ID. Decoding goes on for people who are not heard, so their decoder stays continuous.
    • The bot's own audio is never heard; other bots are skipped unless voice.listenToBots.
    • At most voice.maxSpeakers people are transcribed at once. Someone who starts while the limit is reached is not heard until their next start after a place frees up.
    • No packet for 100 ms ends a person's speaking: what their window still holds is played out, then speaking_end is sent.
    • When a person leaves the channel (gateway voice state or songbird's client disconnect), the bot sends speaker_leave, so their utterance finishes and the worker frees their place.
  • Events for the layers above: SpeechStart, Partial and Final with the speaker's ID and server display name; SpeakingStopped with the speaker's ID when their speaking_end is sent; SpeakStart and SpeakEnd; Stopped when all speech is stopped (stop(), the slash command or the session ending), sent before the SpeakEnds it causes; Joined and Left (with a reason: command, moved, disconnected, shutdown); Addressed and Ranked with the brain's answers to ask_addressed and ask_rank. WorkerLost when the voice worker connection ends: the finals of utterances in progress will not come, and the restarted worker numbers utterances afresh.
  • Speaking: say(text, more) queues a TTS request and returns its id (None when not in a voice channel); continue_text(id, text, more) adds text to an open request. Each sentence the worker sends becomes one clip (a 48 kHz stereo WAV played through songbird without resampling); clips play in sentence order as soon as they arrive, and requests play first come, first served. A sentence that failed is skipped and the rest still play. SpeakEnd reports how many sentences played and whether the request was cancelled or failed.
  • Answering (M4): answer(user, text) queues a request like say whose speech the worker makes from the brain's answer (reply with speak, under the request's id) and returns its id. No answer from the brain, a refused reply (brain_unavailable) or a lost worker ends it as failed. cancel(id) stops one request (its clip if playing, the worker's unsent sentences, or the speech of an answer still being written) and the rest of the queue goes on. ask_addressed(id, user, text) and ask_rank(id, candidates) pass the conversation's questions to the brain. prepare(user, text) sends prepare (a call still being said) and expects nothing back.
  • Earcon (M4): earcon() queues the app's own earcon clip (app/src/earcon.rs, 48 kHz stereo, peak 0.3 of full scale) as a request of its own: no worker round trip, SpeakStart/SpeakEnd like any request, dropped by a stop. None when not in a voice channel.
  • Stop (slash command stop, or stop()): the current clip stops at the next 20 ms mix, every queued request is dropped, and unfinished requests are cancelled at the worker; each ends with cancelled: true. Stopping by voice ("자비스, 멈춰") is call detection's stop (conversation.spec.md), which does the same.
  • Leave (slash command leave, or leave()): stop speaking, send speaker_leave for everyone heard, and disconnect. When songbird's driver gives up the connection (kicked, channel deleted, long network loss) the session ends with disconnected. A driver disconnect without a reason is one the core asked for (leave or move) and is ignored, and a disconnect only ends the session its handler was attached to, so an old connection's late event never ends a newer session. When someone moves the bot to another channel, the session follows it.

Slash command

One guild command, /<commandName> with subcommands join (the caller's voice channel), leave and stop, registered on every start with a create-or-update by name, so the app's other commands in the server are left alone. A failed registration (for example a bot invited without the applications.commands scope) is logged and the bot runs on without it. Usable only in the configured server. Replies are in Korean, ephemeral by default. Renaming commandName leaves the old command registered until it is removed (M5 setup).

Chat channels

  • The bot asks for the message intents only when some registered channel has reading on. Reading needs the Message Content intent, which is checked in the application's flags before login. When it is off, the bot still logs in, voice works, and an error explains how to turn it on; chat reads then fail with a clear error instead of returning empty messages.
  • Live: every new message in a readable registered channel, except the bot's own, is emitted as a Chat event: channel, message and author IDs, author display name, whether the author is a bot, the text, and the time.
  • History: read_chat(channel_id, limit) returns the latest 1-100 messages, oldest first, with from_self marking the bot's own.
  • Send: send_chat(channel_id, text) sends to a registered channel with sending on, split at Discord's 2,000-character limit at the last newline or space (never inside a character), and returns the message ids.
  • Answer (M4): send_chat_reply(channel_id, message_id, text) sends like Send, with the first part as a Discord reply to the message; nobody is pinged, not even its author. chat_answer(id, user, text) asks the brain for a chat answer (reply with via: "chat"); its text comes as ReplyText. reply_delivered(id, ok) tells the worker whether that answer was posted (reply_delivered), so a delete asked in an answer the person never saw cannot be confirmed (voice_worker.spec.md).
  • A channel that is not registered, or has the needed switch off, is refused with an access error.
  • Chat tool actions (M3, chat_channels.spec.md): when the worker link is ready and after login, the core sends chat_channels (every registered channel with its switches, and the bot's user ID). A chat_action from the worker runs chat_action() and answers chat_action_result with the message ids or the error text: send (as Send above), edit (only the bot's own messages, one message long, pings nobody), react (a Unicode or custom emoji), delete, pin, unpin. Each needs its channel switch. Discord's own refusals come back as errors, for example a voice channel's chat cannot pin.

Tests

  • cargo test: unit tests for the receive window (with a tagging fake and real libopus), playback order, gaps, stop and cancel, the earcon clip (before an answer, dropped by a stop; its sound short, silent at both ends, below its peak), speaker limit, chat access and splitting, settings, frames; the worker link against a fake server (hello, events, reconnect, failed TTS); the worker process with a stand-in voice_worker package run by a real Python (token only in the environment, restart on the same port, stop).
  • JARVIS_VOICE_PYTHON=<venv python> cargo test --test worker_protocol runs the worker link and playback against the real Python worker server with fake engines (tests/fixtures/fake_engine_worker.py).
  • tests/live_discord.rs (opt-in, JARVIS_LIVE_DISCORD=1, see the file header): two bots in one voice channel with the real worker on the GPU. It refuses to run if a person is in the channel or either bot is already in a voice channel, and cleans up even when a step fails. It checks command registration, transcription of recorded sentences played into the channel, speech the other bot hears and the STT reads back, stop, chat events, history, an edit and a delete, sending without pings, moving to a second voice channel and back (the session survives each move and only the two moves end one), the speaker leaving and coming back three times (each a DAVE group change), and leaving. The worker runs with --history; at the end the store must hold the speaker's finals under its display name and channel, its chat message with the edit and without the deleted one, Jarvis's own messages, a presence interval per join, and no presence left open. The chat tool's actions run on the channel too: editing its own message, refusing to edit the speaker's, a reaction, pin (Discord refuses it in a voice channel's chat, which must come back as an error), delete, and refusing an unregistered channel. It needs a second voice channel of the server (JARVIS_LIVE_VOICE_CHANNEL_2) and refuses if a person is in either. Measured on this server's RTX 5050 (2026-11-11): five sentences all heard (character error rate 0% for three, 5% and 12.5% for two), 0.71-0.90 s from the end of speech to the final, Jarvis's speech read back at 3.4-6.9%, sound gone 0.03-0.08 s after stop, and the sentence heard exactly after each of three rejoins. With history (2026-11-11): 8 voice records under the speaker's name, the edited chat message only, 3 of Jarvis's own messages, 5 presence intervals, none open.

Known risks

  • songbird issue #310 (open): receive can stall after a DAVE (MLS) group change on some joins. The live test covers four changes (the second bot joining, then leaving and rejoining three times) without a stall; a channel where many people come and go over hours still needs a longer check before M4.