Files
javis_bot/src/voice_worker/voice_worker.spec.md
javis-bot 282f114e60 feat(weather): KMA current weather inside Korea with the app's stored key
The 기능 screen's 기상청 날씨 card stores the KMA API hub key encrypted
(kmaApiKey, masked in logs). On start the app passes it to the worker in
JARVIS_KMA_API_KEY; the worker takes it out of its environment before any
child process starts. Inside Korea getWeather reads 초단기실황 and the
coming hour's sky from 초단기예보, and falls back to open-meteo, then
met.no, on any KMA failure. The key is in KMA's URLs, so failures are
logged by their kind only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:05:31 +09:00

38 KiB

Voice worker spec (v2 M1)

The voice worker is the Python process that turns many Discord speakers' audio into text and the assistant's reply text into speech. The Electron app's Discord voice core talks to it over one local WebSocket. Design background lives in docs/v2/voice-and-llm.md and docs/v2/architecture.md.

Principles

  • Local and private. The server binds to a loopback address only and refuses any other host. Every connection must present the token the app generated; the token comes from the environment (JARVIS_VOICE_TOKEN), never from the command line, so it does not show in process listings. Logs carry timings, sizes and counts, never transcript or reply text.
  • One shared model. A single faster-whisper model serves every speaker. Speakers that need transcription at the same moment go through the model in one batch.
  • Finals before partials. A finished utterance is always transcribed before any live caption.
  • Language-agnostic. Nothing in the worker matches words. The STT language is a setting (null means detect per utterance); sentence splitting uses punctuation only.
  • Engines are pluggable. STT, VAD and TTS sit behind small interfaces so the logic is tested with fakes and the real models are tested separately.

Running

JARVIS_VOICE_TOKEN=<secret> python -m voice_worker --host 127.0.0.1 --port 8771 \
    [--stt-model large-v3-turbo] [--stt-device cuda] [--stt-compute int8_float16] \
    [--melo-python <path to the MeloTTS venv python>] [--melo-device cuda] [--melo-speed 1.5]
    [--history <history.sqlite>] [--brain]

Run from src/ (or with src/ on PYTHONPATH) in a venv built from voice_worker/requirements.txt. MeloTTS needs its own Python 3.11 venv (transformers 4.27), so it runs as a child process (voice_worker/melo_child.py) under --melo-python. Without --melo-python the worker has no TTS and answers tts requests with an error. Both models load and warm up once at start; hello_ok reports whether each is ready.

Protocol

Text frames are JSON objects with a type. Binary frames start with a one-byte kind.

Handshake

The first frame must be hello within 5 seconds, otherwise the worker closes with code 4400. A wrong token closes with 4401.

-> {"type": "hello", "token": "...",
    "audio": {"rate": 48000, "channels": 2},        // incoming PCM format: rate 16000|48000, channels 1|2
    "tts": {"rate": 48000, "channels": 2},          // outgoing PCM format (any rate, channels 1|2)
    "stt": { ...settings, see config... }}           // optional
<- {"type": "hello_ok", "stt_ready": true, "tts_ready": true,
    "audio": {...}, "tts": {...}, "stt": { ...effective settings... }}

STT

-> binary  0x01 | u8 n | speaker id (n bytes UTF-8) | PCM16LE samples in the hello "audio" format
-> {"type": "speaking_end", "speaker": "..."}        // Discord speaking stopped for this speaker
-> {"type": "speaker_leave", "speaker": "..."}       // the speaker left the voice channel
-> {"type": "config", ...any STT settings...}       // takes effect immediately
<- {"type": "speech_start", "speaker": "...", "utt": 3}
<- {"type": "partial", "speaker": "...", "utt": 3, "text": "..."}
<- {"type": "final", "speaker": "...", "utt": 3, "text": "...", "end_reason": "silence",
    "audio_ms": 2310, "wait_ms": 160, "stt_ms": 120}

speaker_leave ends an open utterance as speaking_end does (its final is still sent) and frees the speaker's state. A session holds at most 32 speakers at once, so the app sends it whenever someone leaves; a speaker who comes back starts again at utt 1.

utt counts utterances per speaker. final always closes an utterance that sent speech_start, even when the text is empty (noise that Whisper rejected). wait_ms is from the moment the end was decided to the moment the final was sent; stt_ms is the model time of the batch it went through.

STT settings (all optional in hello and config):

Setting Default Meaning
partials true Send live captions. false drops queued partials and stops new ones; finals continue. The app turns it off while the LLM answers during a game.
partial_interval_ms 400 At most one partial per speaker per interval, only when audio grew.
end_silence_ms 300 Trailing VAD silence that ends an utterance. Range 100-2000.
final_beam 2 Beam size for finals (partials always use 1). Range 1-5.
language "ko" Whisper language code, or null to detect per utterance.
hotwords [] Words (wake words, names) passed to Whisper as a hint. The hint alone made Whisper write the wake word on laughter, so with hotwords every clip is also decoded without them and that plain decode decides (see Hallucination gate).

Utterance rules

Each speaker's audio is mixed to mono, resampled to 16 kHz with a streaming filter, and run through its own streaming Silero VAD (512-sample windows, state kept between frames).

  • Start: the first window with speech probability at or above 0.5. The 200 ms before it are kept.
  • speech_start is sent once the utterance holds 250 ms of clearly voiced windows (probability at or above 0.5). An utterance that ends before that is dropped without any message and never reaches the model: a click, a cough, or noise.
  • Speech continues while windows stay at or above 0.35 (hysteresis); a quieter window counts as silence. Windows between 0.35 and 0.5 keep an utterance open but are not evidence of a voice, so noise the VAD half-believes (fans, typing) never adds up to an utterance.
  • End, with end_reason:
    • silence: trailing silence reached end_silence_ms.
    • speaking_end: the app sent speaking_end. Discord stops sending packets when a user stops talking, so the worker cannot wait for silence that never arrives.
    • idle: no audio arrived for end_silence_ms (a missed speaking_end).
    • max_length: the utterance reached 28 seconds (Whisper's window). The next audio starts a new utterance.
  • The final clip keeps at most 200 ms of trailing silence.
  • Loudness equalisation before transcription (partials and finals), so a quiet and a loud speaker reach the model alike: rumble below 60 Hz is removed, the level is measured as the RMS of the voiced windows only (silence, a click or a knock do not set the gain), the clip is brought to -20 dBFS with the gain kept between -20 and +40 dB, and peaks above 0.8 are soft-limited below 0.98 instead of clipping. Measured: speech from -60 dBFS to clipped shouting was transcribed equally well, and quiet speech under loud clicks or knocks better than with peak normalisation.

Scheduling

One thread owns the STT model and loops:

  1. If any finals are waiting, take all of them (up to 16) and transcribe in one batch with final_beam.
  2. Otherwise take up to 2 waiting partials and transcribe them in one batch with beam 1. A speaker has at most one waiting partial; a newer request replaces the older one and goes to the back of the line, so speakers take turns.

Whisper's encoder always runs on a full 30-second window, so its cost grows with every clip in a batch (measured on an RTX 5050: 0.2 s for one clip, 2.1 s for ten). A large partial batch would make a final wait seconds behind it; two clips keep that wait to about one clip's time. With many people talking, captions fall behind and only the newest audio per speaker is transcribed.

Batches are grouped by decode settings (beam, language, hotwords). A partial whose utterance has already ended, or that is older than the last partial sent, is dropped. In a game, partials wait while an answer is being written ("LLM switches"); finals never wait.

Hallucination gate

Whisper writes text for voice-like sounds without words (coughs, laughter, a crying baby) and, prompted with a hint, the hint itself. A transcript is kept only when all of these hold, else the text is empty:

  • It has letters or digits (not only punctuation).
  • Its mean log-probability per token is at least -0.7. This is the decoder's own per-token mean; the avg_logprob used by faster-whisper divides by one token more, which lifted a three-token hallucination from -0.73 to -0.55 and let it through.
  • Its compression ratio (UTF-8 bytes over zlib bytes) is at most 2.4, which catches repetition loops.
  • With hotwords: the clip is decoded once with them and once without, from the same encoder pass. The plain decode must pass the checks above, otherwise nothing was said. The hinted text is used only when it stays within half the plain text's letters of it and contains every hotword exactly as often as the plain one (ignoring case, spacing and punctuation, no fuzzy matching), so the hint can never add, drop or create a wake word: not from laughter, not after a real sentence, not from a one-letter wake word, and not by turning a similar word ("서비스", or the near miss "자비수") into it. Otherwise the plain text is used; matching near misses is left to call detection.

no_speech_prob is not used: large-v3-turbo reports 0.0 for silence and noise alike.

Measured (2026-11-11, RTX 5050) with 54 real noise recordings from ESC-50 (typing, mouse, breathing, coughing, laughing, crying, vacuum, ...), each at its own level and at -25 dBFS: of the 108 noise-only clips streamed through the worker with the wake word hint on, 103 never reached the model (the VAD rule above) and the 5 that did (coughs, laughter, crying) all came out empty (test_real_noise_recordings_produce_no_text with JARVIS_NOISE_DIR). Over 89 mixes of speech with that noise (SNR 20, 10 and 0 dB, plus clean) the mean character error rate was 4.8%. Left over: one "thank you" written for laughter just before a sentence (mean log-probability -0.70, right at the line), two short words lost in loud noise, and one word repeated three times at SNR 0. A stricter line (-0.6) would lose more real words than it saves.

TTS

-> {"type": "tts", "id": "r1", "text": "...", "more": false}   // more=true: more text follows for r1
-> {"type": "tts_cancel", "id": "r1"}
-> {"type": "tts_speed", "speed": 1.8}                         // 0.5-2.0, from the next sentence on
-> {"type": "tts_played", "id": "r1", "played": 1, "cancelled": true, "last_ms": 1040} // how much the app played (Recordings)
<- binary  0x02 | u8 n | id (n bytes UTF-8) | u32 LE seq | PCM16LE in the hello "tts" format
<- {"type": "tts_done", "id": "r1", "chunks": 3, "cancelled": false, "failed": false}

Text is split into sentences on punctuation only: a run of newlines, CJK terminators (。!?…), or ASCII terminators followed by whitespace, a closing quote or bracket, or the end of the text, so decimals, versions and URLs stay whole. Fragments under 5 characters are merged into a neighbour. With more=true the worker synthesises every complete sentence at once and holds the unfinished tail until more text or more=false arrives (a short last fragment is then spoken on its own, because the sentence before it has already gone out). One audio chunk is sent per sentence, in order, as soon as it is ready. Requests are served first come, first served by one TTS engine. tts_cancel drops the request's unsent sentences, and tts_done follows with cancelled: true. A sentence that fails to synthesise is reported with error (code: "tts_failed") and the request ends with failed: true.

Speed starts at --melo-speed and tts_speed changes it for every sentence made after it (a speed outside 0.5-2.0 or not a number is bad_message and the last good one stays). MeloTTS itself speaks up to 1.25: its own speed shortens every sound, and past that it drops them (five Korean sentences, three renders each, read back by Whisper large-v3: at 1.5 sentence-final "요" and first syllables went missing, 4.6% character errors, 9.6% at 1.75, against 1.2% at 1.0). Faster speech is made at 1.25 and shortened with WSOLA, a time stretch that keeps the pitch (30 ms half-overlapping Hann frames, each shifted within 5 ms to where it best continues the one before): 2.6% at 1.5 and 1.9% at 1.75. The stretch costs naturalness the more it does (UTMOS at 1.5: 3.56, against 3.13 when all of it is stretched from 1.0), so it does only the part the model cannot; up to 1.25 nothing is stretched. About 15 ms per 4 s of speech.

History and brain (M3)

The worker also hosts the brain (the existing jarvis reply engine) and the conversation history store (src/jarvis/memory/history.spec.md). Start it with --history <file> to keep history and --brain to answer (--brain needs --history; it loads the jarvis settings, whose ollama_base_url points at Strata's /v1). Without them the messages below are accepted and ignored, except reply, which answers error brain_unavailable.

The app tells the worker where Jarvis is and who is there. Speaker IDs are Discord user IDs, the same strings as in audio frames. Times are taken by the worker on arrival, except chat messages, which carry Discord's own time.

-> {"type": "voice_join", "channel": "<voice channel id>"}   // Jarvis joined (or moved to) this channel
-> {"type": "voice_leave"}                                   // Jarvis left the voice channel
-> {"type": "member_join", "user": "<id>", "name": "<display name>"}   // also for everyone already there
-> {"type": "member_leave", "user": "<id>"}
-> {"type": "chat_message", "channel": "...", "message_id": "...", "author": "<id>", "name": "...",
    "text": "...", "ts": 1762830000.5, "to": "<id>"}   // to: only on Jarvis's own answer to someone
-> {"type": "chat_edit", "channel": "...", "message_id": "...", "text": "..."}
-> {"type": "chat_delete", "channel": "...", "message_id": "..."}
-> {"type": "reply", "id": "r1", "user": "<id>", "text": "...", "via": "voice", "speak": true}
                                                        // via chat: optional "channel" it was written in;
                                                        // optional "notes": the asker's standing instructions
<- {"type": "reply_text", "id": "r1", "text": "...", "failed": false}
-> {"type": "reply_delivered", "id": "c1", "ok": true}   // a chat answer was posted (ok: false: it was not)
-> {"type": "prepare", "user": "<id>", "text": "..."}   // a call still being said; nothing comes back
-> {"type": "addressed", "id": "q1", "user": "<id>", "text": "...", "people": 2, "callers": 1}
<- {"type": "addressed_result", "id": "q1", "addressed": true, "failed": false}
-> {"type": "rank", "id": "k1", "candidates": [{"id": "7", "user": "<id>", "text": "..."}]}
<- {"type": "rank_result", "id": "k1", "order": ["7"], "failed": false}
  • At start the worker closes every presence interval a previous run left open (it may have been killed); the app then reports who is there.
  • voice_join closes the presence of everyone in the previous channel and starts a new one; voice_leave and the end of the session close it, because nothing said after that is heard. member_join opens the user's presence in the current channel and remembers their name; member_leave closes it. Members outside a voice channel are ignored.
  • Every non-empty final is stored as a voice record under the channel, the speaker's name (the user ID until member_join names them) and the time of the moment its end was decided, all fixed then: decoding takes a while, and a move, a rename or a late joiner meanwhile must not change where or when it was said, or who heard it. A final said outside a voice channel is not stored.
  • Chat messages are stored only as given; the app sends only registered channels.
  • reply asks the brain to answer user's text. The worker builds the history context for that user (their scope only) and runs the reply engine on a worker thread, one voice reply and one chat reply at a time: the two kinds have separate lanes, so a voice answer never waits behind another person's chat answer (app/src/conversation.spec.md, chat calls). One person's replies stay one at a time across both lanes, in the order asked, so their dialogue memory and delete confirmations ("the very next message") follow that order. reply_text carries the answer (failed: true with empty text when the brain gave none). A voice answer (via: "voice") is stored as a reply record to the user in the current channel; a chat answer is stored when the app reports the sent message as chat_message with to. With speak: true the answer is also spoken through the TTS path under the same id (audio frames and tts_done), and tts_cancel with that id stops it. A tts_cancel that arrives while the brain is still answering means the answer is not spoken: reply_text still comes, then tts_done with cancelled: true and no audio. Every reply, a task's too and one that failed, leaves a trace of how it was made (history.spec.md, "Traces"): the history lines and number of notes it was given, then what the engine wrote down. It is stored as soon as the request arrives (before it waits for the asker's previous answer) and again as each step is added, then ends with the answer, or with none when the reply fails, the session closes or the reply is cancelled; a trace that cannot be stored never stops the answer. At start the worker ends the traces a killed run left unfinished and removes those older than 30 days. Each user keeps their own short-term dialogue memory (tool carry-over) across replies. The brain runs the reply engine with v1's single-user memory off (legacy_memory=False: no warm profile, diary or graph), because several people share this assistant and only the asker's scoped history may reach the prompt. It answers in Korean by default (--reply-language, default Korean; '' answers in the asker's language), whatever language the question was heard or written in, and in another language only when the asker explicitly asks for it (just that answer, or from then on, as they said). Tools get the ISO code --tool-language (default ko). v1's tts_engine setting (an English-only Piper voice by default) does not lock replies to English, because v2 speaks through the worker's TTS. An output_language lock in the jarvis settings still wins. Eval: evals/test_reply_language_default.py. STT (language "ko") and TTS (--melo-language KR) are Korean by default too. Two reply options keep the time to the first word short (Strata reads every prompt from the start, about 0.36 s per 2,048 tokens, and each LLM call before the answer adds its whole time): the short voice persona (compact_prompt=True: the same butler in a tenth of v1's words, without v1's single-user memory and on-screen browser rules, answers as short as possible, v1's tool guidance in short only on turns that offer a data tool), and the intent judge (intent_judge=True: the tool router judges whether the turn needs tools and which, and the planner runs only when it does and the pick is not a lone one-step tool; planner.spec.md, "Intent judge (v2)"). v1's screen-share browser (stream_browser) is off: it needs v1's Node bot (M6 brings the v2 one), so its tools are not offered.
  • Tools: the brain offers and runs no built-in tool in the jarvis settings' disabled_tools (the app writes the tools of the features turned off on its 기능 screen, app/src/admin.spec.md, "Features and MCP servers"), and never v1's host tools localFiles (the PC owner's home folder: read, write, delete) and screenshot (the PC's screen): in v2 anyone in the voice channel or a registered chat channel asks, so they stay off until M7 brings screen viewing with its own permissions. The jarvis settings' mcps are connected on a thread of their own when the brain starts, so a slow server does not hold up the voice; their tools are offered once discovered. Discovery starts the servers, so it runs outside the tool cache's lock: a turn reading the cached tools never waits for a slow server, and a new conversation refreshes the MCP tools in the background (the turn uses the tools already cached; one refresh at a time). The app passes the servers' environment values (secrets) in JARVIS_MCP_ENV, JSON {server: {NAME: value}}, never in the settings file; the worker removes that variable from its environment at start, before any child process (MeloTTS) runs, hands it to the brain, and the brain adds each server's values to that server's environment only. Each server's outcome is printed without debug logging (category llm): MCP server <name>: <n> tool(s), or Warning: MCP server <name> failed: <reason>. The KMA key (JARVIS_KMA_API_KEY, app/src/admin.spec.md) leaves the environment at the same point; the weather tool keeps it and uses it only for the current weather inside Korea (geocoded or detected country KR): KMA's 초단기실황 (temperature, humidity, wind, precipitation) and 초단기예보 (the coming hour's sky). Any KMA failure falls back to open-meteo, then met.no; the key is in KMA's URLs, so a failure is logged by its kind only.
  • prepare tells the brain that user is saying a call whose partial transcript is text (the app sends it while the speaker is still talking, app/src/conversation.spec.md). The brain runs the intent judge's tool pick for it on its own thread, with the asker's dialogue memory and the tool catalogue the reply will have. A reply from that user whose text is about the same (letters and digits, 85% alike), within 30 s and with the dialogue memory unchanged, uses that pick instead of making its own, waiting up to 5 s for one still being made. Each prepared judgement is used at most once. One prepare per user runs at a time: while it runs only the newest text waits, a text about the same as the one last prepared is dropped, and a reply from that user drops the waiting text. Nothing is sent back; without a brain, or with one that cannot prepare, it is ignored. The asker is set in jarvis.reply.turn_state for the turn, so tools that keep personal data (meals) store and read it under that person only.
  • addressed and rank are the conversation's questions (app/src/conversation.spec.md), answered on their own threads, so they never wait behind a running reply. addressed: was text said to the assistant? The session's hotwords are given to the judge as the assistant's names, and people (in the voice channel, the speaker included) and callers (of them, who may call Jarvis), when the app sends them (positive integers; otherwise not known). The question follows the room: with 2 people most of what the speaker says counts as said to the assistant (questions, requests, complaints and "you" remarks unless clearly said to the other person; unclear is yes); with 3 or more, or not known, most talk is between people (unclear is no; not known asks exactly as before the counts existed); fewer callers than people adds "only N of the M can use the assistant". rank: the candidate ids most important first (ids the brain left out follow in their order, unknown ids are dropped). A judgement the brain could not make, or that raised, comes back failed: true (addressed: false, empty order). Both are one short non-thinking call to the chat model with the heard text fenced as data (voice_worker/judges.py, eval evals/test_voice_judges.py). Without a brain they answer error brain_unavailable with the id.
  • While a reply runs, the process's stdout and stderr are muted: the v1 engine and its tools print questions, answers, tool arguments and memory text, also from helper threads. The worker's own log (voice_worker/log.py) writes to the original stderr and keeps working.
  • Chat channel tool (src/jarvis/tools/builtin/chat_channels.spec.md): the app sends the registered channels, and the brain's tool asks the app to act. Reads and searches come from the history store.
    -> {"type": "chat_channels", "bot": "<bot user id>", "channels": [{"id": "...", "name": "...",
        "description": "...", "read": true, "send": true, "edit": true, "react": true,
        "delete": false, "pin": false}]}
    <- {"type": "chat_action", "id": "a1", "action": "send|edit|react|delete|pin|unpin",
        "channel": "...", "message_id": "...", "text": "...", "emoji": "...",
        "user": "<id>"}                                // who asked; the app records it (Activity)
    -> {"type": "chat_action_result", "id": "a1", "ok": true, "message_ids": ["..."]}
    -> {"type": "chat_action_result", "id": "a1", "ok": false, "error": "..."}
    
    Action ids are unique across connections and worker runs, and an answer whose id is not waiting on this connection is ignored, so a late answer meant for an earlier connection never completes a newer action. With no registered channels the tool is not offered. An action waits up to 15 s for its result (then it fails), and a closed session fails every waiting action. Each reply counts as one turn per person, so a delete is confirmed only in that person's next reply. An answer reaches the person when its speech ends with tts_done neither failed nor cancelled (speak: true), when the app reports it posted with reply_delivered ok: true (via: "chat"), or at reply_text for an unspoken voice answer. It never reaches them when the brain gave none, its speech was stopped, failed or had no TTS (tts_unavailable), the app reports ok: false, no report comes within 30 s, or the session ends first. A reply sent before the person's previous answer reached them cannot answer it, and an answer that never reached them asked them nothing: in both cases a turn number is skipped, so a delete asked in that previous answer lapses.
  • Memory settings, optional in hello as "memory": {...} and changeable with {"type": "memory_config", ...}: chat (true), others (true), off (false), since_days (null: whole history), recent_minutes (30), recent_records (20), matches (5), budget_tokens (4000).
  • Logs carry IDs, counts and timings only, never record, question or answer text.

Schedules

The brain's schedule tool (src/jarvis/tools/builtin/schedule.spec.md) asks the app, which keeps the schedules (app/src/admin.spec.md, "Schedules"):

<- {"type": "schedule_action", "id": "s1", "action": "add|list|delete", "user": "<id>",
    "via": "voice|chat", "channel": "<chat channel id>",       // channel: only for via chat, when given
    "kind": "remind|task", "text": "...", "at": 1762830000.0, "repeat": "none|daily|weekdays|weekly",
    "schedule": "<id>"}                                         // add: kind..repeat; delete: schedule
-> {"type": "schedule_action_result", "id": "s1", "ok": true, "item": {...}}    // add, delete
-> {"type": "schedule_action_result", "id": "s1", "ok": true, "items": [...]}   // list
-> {"type": "schedule_action_result", "id": "s1", "ok": false, "error": "..."}
-> {"type": "reply", "id": "t1", "user": "<id>", "text": "<the task>", "via": "task"}
<- {"type": "reply_text", "id": "t1", "text": "...", "failed": false}
  • Every voice and chat reply gets a schedule backend for its asker, the reply's via and, for chat, the channel the reply named. Ids and waiting are those of chat actions (unique across connections and runs, 15 s, a closed session fails what waits).
  • via: "task" runs a scheduled task the app fires: in the chat lane and one at a time with the person's other replies, with their history scope, but its own short-term memory, no chat or schedule backend and the read-only tools only (brain.TASK_TOOLS, no MCP tools). It is not a turn of the person's conversation (their turn count and delete confirmations stay), is never spoken (speak: true is bad_message), and its answer is not stored; the app delivers it. It waits for LLM switches like any reply.

Personal notes

The brain's personalNotes tool (src/jarvis/tools/builtin/personal_notes.spec.md) asks the app, which keeps each person's standing instructions and sends the asker's with every reply (app/src/admin.spec.md, "Personal notes"):

-> {"type": "reply", ..., "notes": ["나한테는 반말로 해"]}     // optional; at most 20, each at most 300 characters
<- {"type": "note_action", "id": "n1", "action": "add|list|delete", "user": "<id>", "via": "voice|chat",
    "channel": "...", "text": "...", "note": "<id>"}            // add: text; delete: note
-> {"type": "note_action_result", "id": "n1", "ok": true, "item": {...}}   // or "items", or ok false with "error"
  • notes that are not a list of at most 20 strings of at most 300 characters are bad_message. The brain puts them in the reply's prompt (person_notes); a scheduled task (via: "task") gets them too, but no notes tool.
  • Every voice and chat reply gets a notes backend for its asker; ids and waiting are those of chat and schedule actions.

Screen viewing (M7)

The brain's screenView tool (src/jarvis/tools/builtin/screen_view.spec.md) asks the app, which captures and decides who may see (app/src/admin.spec.md, "Screen viewing"):

<- {"type": "screen_action", "id": "s1", "action": "capture|list", "user": "<id>", "via": "voice|chat",
    "channel": "...", "target": "screen|monitor|window", "monitor": 2, "window": "chrome", "detail": "fast|normal|detailed"}
-> {"type": "screen_action_result", "id": "s1", "ok": true, "item": {...}, "images": ["<base64 JPEG>"]}  // or ok false with "error"
  • Every voice and chat reply gets a screen backend for its asker (not scheduled tasks); ids and waiting are those of chat and schedule actions. images that are not a list of strings are dropped. The pictures stay in that one reply (screen_view.spec.md).
  • Messages may be up to 4 MiB (pictures), the WebSocket's limit.

PC control (M7)

The brain's pcControl tool (src/jarvis/tools/builtin/pc_control.spec.md) asks the app, which acts through agent-desktop and decides who may (app/src/admin.spec.md, "PC control"):

<- {"type": "pc_action", "id": "p1", "action": "windows|read|click|type|keys|run|close", "user": "<id>",
    "via": "voice|chat", "channel": "...", "window": "...", "ref": "@s1:e8", "text": "...", "keys": "ctrl+s",
    "program": "notepad", "consent": true}
-> {"type": "pc_action_result", "id": "p1", "ok": true, "item": {...}}  // or ok false with "error", or "ask": true
  • Every voice and chat reply gets a PC backend for its asker (not scheduled tasks); ids and waiting are those of chat and schedule actions. The worker keeps each asker's requests awaiting consent with the turn they were asked in, numbered like the chat delete confirmations ("History and brain"), so only a message sent after the question counts as the next turn.

Background collection (M6)

With JARVIS_COLLECT_PATH set, the worker collects on its own thread, apart from the voice and the brain (app/src/admin.spec.md, "Background collection"). The file is {"dir": "<data>/collect", "keepDays": 30, "targets": [{"name", "kind", "value", "everyMinutes"}]}; the app writes only enabled targets. Nothing is collected without targets, but old day files are still deleted (below).

  • A target runs when everyMinutes passed since its last run (also across restarts; the last run is kept in <dir>/<name>/state.json), one target at a time, checked every 30 s. A failed run counts as a run, so a broken site is tried again only at its next interval.
  • Kinds: site keeps the page's title and text (at most 20,000 characters) when either differs from the last kept ones; rss (RSS or Atom) keeps each new item (title, link, time, summary of at most 2,000 characters, new by its id or link); search keeps each new web search result (title and link) for the words, from DuckDuckGo's lite page, which needs no key (its bot challenge is a failed run, not "no results").
  • Every request checks its address like the web search (web_search.spec.md): only http(s), and every address the name resolves to must be public (for IPv6 only global unicast 2000::/3, without 6to4); the connection goes only to the checked addresses in their order (TLS still checks the name), so a second lookup cannot lead into the local network. Every redirect is checked the same way. At most 1 MB and 15 s in all: the name lookup, connecting, TLS, the headers and the body of every redirect count, and a server that sends slowly, even byte by byte within the per-read limit, is cut off at 15 s. A response cut short is a failure. Characters outside ASCII (and spaces) in the path and words go as UTF-8 %HH; existing %HH and separators stay. The check logs nothing about the address. The collected text is web content: it goes only to <dir>/<name>/<YYYY-MM-DD>.jsonl (one JSON object per line: at, then title, url, text as the kind has them), never to a log. The collector itself never hands it to the brain; the brain reads it only through its collectedNews tool, fenced as untrusted web content (src/jarvis/tools/builtin/collected_news.spec.md), and the app announces new titles and links when asked to (app/src/admin.spec.md, "News notifications").
  • Day files older than keepDays are deleted in every folder of dir at each check, also of targets no longer listed.
  • Each run is a notice line (category voice_worker): the target's name and how many new items, or a Warning with the error's type. Never a URL, words or collected text.

LLM switches (M7)

The app restarts the LLM with another placement when a game starts or ends (app/src/admin.spec.md, "Performance stages"). It tells the worker first and waits for the brain to be idle:

-> {"type": "llm", "state": "switching", "id": 7, "notice": "잠깐만요, 게임 모드로 바꾸는 중이에요"}   // id, notice optional
<- {"type": "llm_idle", "id": 7}            // nothing that uses the LLM runs any more (id: the switch's, when it had one)
-> {"type": "llm", "state": "ready", "game": true}   // the LLM answers again; game optional (default false)
-> {"type": "llm", "state": "paused"}       // the LLM is stopped (a game runs)
  • Brain work that uses the LLM (reply, prepare, addressed, rank) counts as running from its start to its end. On switching new work waits; llm_idle goes to the connection that sent switching once nothing runs (at once when nothing does). On ready what waited goes on.
  • The first spoken reply that waits during one switch hears the notice under its own id (its answer follows in the same speech when ready); later ones wait without it. A notice ends with a newline, so it is spoken at once and never joined to the answer. A request waits at most 180 s, then fails without calling the brain (the LLM may still be away, or a measuring server stand in).
  • On paused brain work answers nothing at once (reply_text failed, judgements failed) until ready; what waited for a switch stops waiting the same way.
  • game on ready says the LLM runs in the game placement: from then on, while a reply is being written, partial passes wait (finals go on at once, "Scheduling"); when no answer is being written they go on, the newest audio per speaker. ready without it (or false) ends that. Without a game partials never wait (M0 gamemode_tune: with partials held the game placement answered 1.7 times faster, 64.9 against 37.5 tok/s with a partial pass every 2 s).
  • Any other state, a notice that is not a string, an id that is not an integer or a game that is not a boolean is bad_message.
  • The app sends its latest llm message again first on every connection, so a worker that missed one (a dropped connection, a restarted worker) catches up.

Recordings

For checking what was heard against what was transcribed, and what Jarvis said. Off unless the app sets JARVIS_RECORDINGS_DIR (setting voice.record, off by default; the app passes <data folder>/recordings at start). Then voice_worker/recordings.py keeps:

  • each finished utterance (every final, also one with no text) as a 16 kHz mono WAV of the clip as it was cut for transcription, before loudness levelling: <day>/<HHMMSS>-<ms>-said-<speaker>.wav;
  • each spoken answer as a mono WAV at the session's TTS rate, as far as the app played it: <day>/<HHMMSS>-<ms>-jarvis.wav, timed at its first sentence. The app sends tts_played (played: how many of its sentences started playing, cancelled: whether it was stopped, last_ms: when it was stopped while a sentence played, how long that sentence had played) when each request ends; the worker keeps the request's sentences until it is made and that report came, then writes the first played sentences, the last cut to last_ms (at the TTS rate, the rate it is played at), with cancelled. Nothing played (0), never reported (the session ends, an older app; at most 50 are held) or ids it does not know (the earcon): no file;
  • one JSON line per file in <day>/index.jsonl: {file, at, ms, kind: said|spoke, who, name, channel, text, id, cancelled} (the transcript, or the text sent to tts).

<day> is the local day. Files are written on a thread of their own, so the voice path never waits, and a failed write is a debug line (no text) that changes nothing else. Day folders older than 7 days are removed when the worker starts and with the first recording of each new day while it runs. Nothing is recorded without the variable.

Errors

{"type": "error", "code": "...", "message": "...", "id": "..."}. Codes: bad_message (unknown type, bad JSON, bad field), bad_frame (malformed binary frame), tts_unavailable, tts_failed, brain_unavailable. Errors never close the connection after the handshake.

Debug logging

debug_log(message, category) at: connection accepted or refused, settings changed, speaker left, utterance end (speaker, reason, length), each STT batch (kind, size, model time), each TTS sentence (length, time), cancel, and engine failures. The category names the service, so the app's log screen puts each line in its tab (app/src/admin.spec.md, "Logs"): stt (utterance ends, STT batches, the Whisper model), tts (sentences, cancels, MeloTTS), llm (the brain's replies, prepares and judgements), voice_worker for the rest (connections, sessions, channels).

Without debug logging too, each answer and each scheduled task logs one llm line once the LLM is done with it: the least free graphics memory read while it was written, the card's total, how many readings and how long it took (answer <id> card: least N MiB free of T MiB (k readings, ms)). The card is read with nvidia-smi (memory.free, which leaves out what the driver reserves) once a second while the answer is written, in a thread, so the answer never waits for it: a reading still running when the answer ends finishes on its own and the line follows it, so a short answer gets its line too. Without nvidia-smi nothing is read or logged. The line carries no text, so a slow answer can be matched to a full card (the engine log has the same answer's timings).