The 기능 screen's 기상청 날씨 card stores the KMA API hub key encrypted (kmaApiKey, masked in logs). On start the app passes it to the worker in JARVIS_KMA_API_KEY; the worker takes it out of its environment before any child process starts. Inside Korea getWeather reads 초단기실황 and the coming hour's sky from 초단기예보, and falls back to open-meteo, then met.no, on any KMA failure. The key is in KMA's URLs, so failures are logged by their kind only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
38 KiB
Voice worker spec (v2 M1)
The voice worker is the Python process that turns many Discord speakers' audio into text and the
assistant's reply text into speech. The Electron app's Discord voice core talks to it over one local
WebSocket. Design background lives in docs/v2/voice-and-llm.md and docs/v2/architecture.md.
Principles
- Local and private. The server binds to a loopback address only and refuses any other host. Every
connection must present the token the app generated; the token comes from the environment
(
JARVIS_VOICE_TOKEN), never from the command line, so it does not show in process listings. Logs carry timings, sizes and counts, never transcript or reply text. - One shared model. A single faster-whisper model serves every speaker. Speakers that need transcription at the same moment go through the model in one batch.
- Finals before partials. A finished utterance is always transcribed before any live caption.
- Language-agnostic. Nothing in the worker matches words. The STT language is a setting (
nullmeans detect per utterance); sentence splitting uses punctuation only. - Engines are pluggable. STT, VAD and TTS sit behind small interfaces so the logic is tested with fakes and the real models are tested separately.
Running
JARVIS_VOICE_TOKEN=<secret> python -m voice_worker --host 127.0.0.1 --port 8771 \
[--stt-model large-v3-turbo] [--stt-device cuda] [--stt-compute int8_float16] \
[--melo-python <path to the MeloTTS venv python>] [--melo-device cuda] [--melo-speed 1.5]
[--history <history.sqlite>] [--brain]
Run from src/ (or with src/ on PYTHONPATH) in a venv built from voice_worker/requirements.txt. MeloTTS needs its own Python 3.11 venv
(transformers 4.27), so it runs as a child process (voice_worker/melo_child.py) under
--melo-python. Without --melo-python the worker has no TTS and answers tts requests with an
error. Both models load and warm up once at start; hello_ok reports whether each is ready.
Protocol
Text frames are JSON objects with a type. Binary frames start with a one-byte kind.
Handshake
The first frame must be hello within 5 seconds, otherwise the worker closes with code 4400. A wrong
token closes with 4401.
-> {"type": "hello", "token": "...",
"audio": {"rate": 48000, "channels": 2}, // incoming PCM format: rate 16000|48000, channels 1|2
"tts": {"rate": 48000, "channels": 2}, // outgoing PCM format (any rate, channels 1|2)
"stt": { ...settings, see config... }} // optional
<- {"type": "hello_ok", "stt_ready": true, "tts_ready": true,
"audio": {...}, "tts": {...}, "stt": { ...effective settings... }}
STT
-> binary 0x01 | u8 n | speaker id (n bytes UTF-8) | PCM16LE samples in the hello "audio" format
-> {"type": "speaking_end", "speaker": "..."} // Discord speaking stopped for this speaker
-> {"type": "speaker_leave", "speaker": "..."} // the speaker left the voice channel
-> {"type": "config", ...any STT settings...} // takes effect immediately
<- {"type": "speech_start", "speaker": "...", "utt": 3}
<- {"type": "partial", "speaker": "...", "utt": 3, "text": "..."}
<- {"type": "final", "speaker": "...", "utt": 3, "text": "...", "end_reason": "silence",
"audio_ms": 2310, "wait_ms": 160, "stt_ms": 120}
speaker_leave ends an open utterance as speaking_end does (its final is still sent) and frees the
speaker's state. A session holds at most 32 speakers at once, so the app sends it whenever someone
leaves; a speaker who comes back starts again at utt 1.
utt counts utterances per speaker. final always closes an utterance that sent speech_start, even
when the text is empty (noise that Whisper rejected). wait_ms is from the moment the end was decided
to the moment the final was sent; stt_ms is the model time of the batch it went through.
STT settings (all optional in hello and config):
| Setting | Default | Meaning |
|---|---|---|
partials |
true |
Send live captions. false drops queued partials and stops new ones; finals continue. The app turns it off while the LLM answers during a game. |
partial_interval_ms |
400 | At most one partial per speaker per interval, only when audio grew. |
end_silence_ms |
300 | Trailing VAD silence that ends an utterance. Range 100-2000. |
final_beam |
2 | Beam size for finals (partials always use 1). Range 1-5. |
language |
"ko" |
Whisper language code, or null to detect per utterance. |
hotwords |
[] |
Words (wake words, names) passed to Whisper as a hint. The hint alone made Whisper write the wake word on laughter, so with hotwords every clip is also decoded without them and that plain decode decides (see Hallucination gate). |
Utterance rules
Each speaker's audio is mixed to mono, resampled to 16 kHz with a streaming filter, and run through its own streaming Silero VAD (512-sample windows, state kept between frames).
- Start: the first window with speech probability at or above 0.5. The 200 ms before it are kept.
- speech_start is sent once the utterance holds 250 ms of clearly voiced windows (probability at or above 0.5). An utterance that ends before that is dropped without any message and never reaches the model: a click, a cough, or noise.
- Speech continues while windows stay at or above 0.35 (hysteresis); a quieter window counts as silence. Windows between 0.35 and 0.5 keep an utterance open but are not evidence of a voice, so noise the VAD half-believes (fans, typing) never adds up to an utterance.
- End, with
end_reason:silence: trailing silence reachedend_silence_ms.speaking_end: the app sentspeaking_end. Discord stops sending packets when a user stops talking, so the worker cannot wait for silence that never arrives.idle: no audio arrived forend_silence_ms(a missedspeaking_end).max_length: the utterance reached 28 seconds (Whisper's window). The next audio starts a new utterance.
- The final clip keeps at most 200 ms of trailing silence.
- Loudness equalisation before transcription (partials and finals), so a quiet and a loud speaker reach the model alike: rumble below 60 Hz is removed, the level is measured as the RMS of the voiced windows only (silence, a click or a knock do not set the gain), the clip is brought to -20 dBFS with the gain kept between -20 and +40 dB, and peaks above 0.8 are soft-limited below 0.98 instead of clipping. Measured: speech from -60 dBFS to clipped shouting was transcribed equally well, and quiet speech under loud clicks or knocks better than with peak normalisation.
Scheduling
One thread owns the STT model and loops:
- If any finals are waiting, take all of them (up to 16) and transcribe in one batch with
final_beam. - Otherwise take up to 2 waiting partials and transcribe them in one batch with beam 1. A speaker has at most one waiting partial; a newer request replaces the older one and goes to the back of the line, so speakers take turns.
Whisper's encoder always runs on a full 30-second window, so its cost grows with every clip in a batch (measured on an RTX 5050: 0.2 s for one clip, 2.1 s for ten). A large partial batch would make a final wait seconds behind it; two clips keep that wait to about one clip's time. With many people talking, captions fall behind and only the newest audio per speaker is transcribed.
Batches are grouped by decode settings (beam, language, hotwords). A partial whose utterance has already ended, or that is older than the last partial sent, is dropped. In a game, partials wait while an answer is being written ("LLM switches"); finals never wait.
Hallucination gate
Whisper writes text for voice-like sounds without words (coughs, laughter, a crying baby) and, prompted with a hint, the hint itself. A transcript is kept only when all of these hold, else the text is empty:
- It has letters or digits (not only punctuation).
- Its mean log-probability per token is at least -0.7. This is the decoder's own per-token mean; the
avg_logprobused by faster-whisper divides by one token more, which lifted a three-token hallucination from -0.73 to -0.55 and let it through. - Its compression ratio (UTF-8 bytes over zlib bytes) is at most 2.4, which catches repetition loops.
- With hotwords: the clip is decoded once with them and once without, from the same encoder pass. The plain decode must pass the checks above, otherwise nothing was said. The hinted text is used only when it stays within half the plain text's letters of it and contains every hotword exactly as often as the plain one (ignoring case, spacing and punctuation, no fuzzy matching), so the hint can never add, drop or create a wake word: not from laughter, not after a real sentence, not from a one-letter wake word, and not by turning a similar word ("서비스", or the near miss "자비수") into it. Otherwise the plain text is used; matching near misses is left to call detection.
no_speech_prob is not used: large-v3-turbo reports 0.0 for silence and noise alike.
Measured (2026-11-11, RTX 5050) with 54 real noise recordings from ESC-50 (typing, mouse, breathing,
coughing, laughing, crying, vacuum, ...), each at its own level and at -25 dBFS: of the 108 noise-only
clips streamed through the worker with the wake word hint on, 103 never reached the model (the VAD
rule above) and the 5 that did (coughs, laughter, crying) all came out empty
(test_real_noise_recordings_produce_no_text with JARVIS_NOISE_DIR). Over 89 mixes of speech with that noise
(SNR 20, 10 and 0 dB, plus clean) the mean character error rate was 4.8%. Left over: one
"thank you" written for laughter just before a sentence (mean log-probability -0.70, right at the
line), two short words lost in loud noise, and one word repeated three times at SNR 0. A stricter
line (-0.6) would lose more real words than it saves.
TTS
-> {"type": "tts", "id": "r1", "text": "...", "more": false} // more=true: more text follows for r1
-> {"type": "tts_cancel", "id": "r1"}
-> {"type": "tts_speed", "speed": 1.8} // 0.5-2.0, from the next sentence on
-> {"type": "tts_played", "id": "r1", "played": 1, "cancelled": true, "last_ms": 1040} // how much the app played (Recordings)
<- binary 0x02 | u8 n | id (n bytes UTF-8) | u32 LE seq | PCM16LE in the hello "tts" format
<- {"type": "tts_done", "id": "r1", "chunks": 3, "cancelled": false, "failed": false}
Text is split into sentences on punctuation only: a run of newlines, CJK terminators (。!?…), or
ASCII terminators followed by whitespace, a closing quote or bracket, or the end of the text, so
decimals, versions and URLs stay whole. Fragments under 5 characters are merged into a neighbour.
With more=true the worker synthesises every complete sentence at once and holds the unfinished tail
until more text or more=false arrives (a short last fragment is then spoken on its own, because the
sentence before it has already gone out). One audio chunk is sent per sentence, in order, as soon as it
is ready. Requests are served first come, first served by one TTS engine. tts_cancel drops the
request's unsent sentences, and tts_done follows with cancelled: true. A sentence that fails to
synthesise is reported with error (code: "tts_failed") and the request ends with failed: true.
Speed starts at --melo-speed and tts_speed changes it for every sentence made after it (a speed
outside 0.5-2.0 or not a number is bad_message and the last good one stays). MeloTTS itself speaks
up to 1.25: its own speed shortens every sound, and past that it drops them (five Korean sentences,
three renders each, read back by Whisper large-v3: at 1.5 sentence-final "요" and first syllables went
missing, 4.6% character errors, 9.6% at 1.75, against 1.2% at 1.0). Faster speech is made at 1.25 and
shortened with WSOLA, a time stretch that keeps the pitch (30 ms half-overlapping Hann frames, each
shifted within 5 ms to where it best continues the one before): 2.6% at 1.5 and 1.9% at 1.75. The
stretch costs naturalness the more it does (UTMOS at 1.5: 3.56, against 3.13 when all of it is
stretched from 1.0), so it does only the part the model cannot; up to 1.25 nothing is stretched. About 15 ms per 4 s
of speech.
History and brain (M3)
The worker also hosts the brain (the existing jarvis reply engine) and the conversation history
store (src/jarvis/memory/history.spec.md). Start it with --history <file> to keep history and
--brain to answer (--brain needs --history; it loads the jarvis settings, whose
ollama_base_url points at Strata's /v1). Without them the messages below are accepted and
ignored, except reply, which answers error brain_unavailable.
The app tells the worker where Jarvis is and who is there. Speaker IDs are Discord user IDs, the same strings as in audio frames. Times are taken by the worker on arrival, except chat messages, which carry Discord's own time.
-> {"type": "voice_join", "channel": "<voice channel id>"} // Jarvis joined (or moved to) this channel
-> {"type": "voice_leave"} // Jarvis left the voice channel
-> {"type": "member_join", "user": "<id>", "name": "<display name>"} // also for everyone already there
-> {"type": "member_leave", "user": "<id>"}
-> {"type": "chat_message", "channel": "...", "message_id": "...", "author": "<id>", "name": "...",
"text": "...", "ts": 1762830000.5, "to": "<id>"} // to: only on Jarvis's own answer to someone
-> {"type": "chat_edit", "channel": "...", "message_id": "...", "text": "..."}
-> {"type": "chat_delete", "channel": "...", "message_id": "..."}
-> {"type": "reply", "id": "r1", "user": "<id>", "text": "...", "via": "voice", "speak": true}
// via chat: optional "channel" it was written in;
// optional "notes": the asker's standing instructions
<- {"type": "reply_text", "id": "r1", "text": "...", "failed": false}
-> {"type": "reply_delivered", "id": "c1", "ok": true} // a chat answer was posted (ok: false: it was not)
-> {"type": "prepare", "user": "<id>", "text": "..."} // a call still being said; nothing comes back
-> {"type": "addressed", "id": "q1", "user": "<id>", "text": "...", "people": 2, "callers": 1}
<- {"type": "addressed_result", "id": "q1", "addressed": true, "failed": false}
-> {"type": "rank", "id": "k1", "candidates": [{"id": "7", "user": "<id>", "text": "..."}]}
<- {"type": "rank_result", "id": "k1", "order": ["7"], "failed": false}
- At start the worker closes every presence interval a previous run left open (it may have been killed); the app then reports who is there.
voice_joincloses the presence of everyone in the previous channel and starts a new one;voice_leaveand the end of the session close it, because nothing said after that is heard.member_joinopens the user's presence in the current channel and remembers their name;member_leavecloses it. Members outside a voice channel are ignored.- Every non-empty
finalis stored as avoicerecord under the channel, the speaker's name (the user ID untilmember_joinnames them) and the time of the moment its end was decided, all fixed then: decoding takes a while, and a move, a rename or a late joiner meanwhile must not change where or when it was said, or who heard it. A final said outside a voice channel is not stored. - Chat messages are stored only as given; the app sends only registered channels.
replyasks the brain to answeruser'stext. The worker builds the history context for that user (their scope only) and runs the reply engine on a worker thread, one voice reply and one chat reply at a time: the two kinds have separate lanes, so a voice answer never waits behind another person's chat answer (app/src/conversation.spec.md, chat calls). One person's replies stay one at a time across both lanes, in the order asked, so their dialogue memory and delete confirmations ("the very next message") follow that order.reply_textcarries the answer (failed: truewith empty text when the brain gave none). A voice answer (via: "voice") is stored as areplyrecord to the user in the current channel; a chat answer is stored when the app reports the sent message aschat_messagewithto. Withspeak: truethe answer is also spoken through the TTS path under the same id (audio frames andtts_done), andtts_cancelwith that id stops it. Atts_cancelthat arrives while the brain is still answering means the answer is not spoken:reply_textstill comes, thentts_donewithcancelled: trueand no audio. Every reply, a task's too and one that failed, leaves a trace of how it was made (history.spec.md, "Traces"): the history lines and number of notes it was given, then what the engine wrote down. It is stored as soon as the request arrives (before it waits for the asker's previous answer) and again as each step is added, then ends with the answer, or with none when the reply fails, the session closes or the reply is cancelled; a trace that cannot be stored never stops the answer. At start the worker ends the traces a killed run left unfinished and removes those older than 30 days. Each user keeps their own short-term dialogue memory (tool carry-over) across replies. The brain runs the reply engine with v1's single-user memory off (legacy_memory=False: no warm profile, diary or graph), because several people share this assistant and only the asker's scoped history may reach the prompt. It answers in Korean by default (--reply-language, defaultKorean;''answers in the asker's language), whatever language the question was heard or written in, and in another language only when the asker explicitly asks for it (just that answer, or from then on, as they said). Tools get the ISO code--tool-language(defaultko). v1'stts_enginesetting (an English-only Piper voice by default) does not lock replies to English, because v2 speaks through the worker's TTS. Anoutput_languagelock in the jarvis settings still wins. Eval:evals/test_reply_language_default.py. STT (language"ko") and TTS (--melo-language KR) are Korean by default too. Two reply options keep the time to the first word short (Strata reads every prompt from the start, about 0.36 s per 2,048 tokens, and each LLM call before the answer adds its whole time): the short voice persona (compact_prompt=True: the same butler in a tenth of v1's words, without v1's single-user memory and on-screen browser rules, answers as short as possible, v1's tool guidance in short only on turns that offer a data tool), and the intent judge (intent_judge=True: the tool router judges whether the turn needs tools and which, and the planner runs only when it does and the pick is not a lone one-step tool;planner.spec.md, "Intent judge (v2)"). v1's screen-share browser (stream_browser) is off: it needs v1's Node bot (M6 brings the v2 one), so its tools are not offered.- Tools: the brain offers and runs no built-in tool in the jarvis settings'
disabled_tools(the app writes the tools of the features turned off on its 기능 screen,app/src/admin.spec.md, "Features and MCP servers"), and never v1's host toolslocalFiles(the PC owner's home folder: read, write, delete) andscreenshot(the PC's screen): in v2 anyone in the voice channel or a registered chat channel asks, so they stay off until M7 brings screen viewing with its own permissions. The jarvis settings'mcpsare connected on a thread of their own when the brain starts, so a slow server does not hold up the voice; their tools are offered once discovered. Discovery starts the servers, so it runs outside the tool cache's lock: a turn reading the cached tools never waits for a slow server, and a new conversation refreshes the MCP tools in the background (the turn uses the tools already cached; one refresh at a time). The app passes the servers' environment values (secrets) inJARVIS_MCP_ENV, JSON{server: {NAME: value}}, never in the settings file; the worker removes that variable from its environment at start, before any child process (MeloTTS) runs, hands it to the brain, and the brain adds each server's values to that server's environment only. Each server's outcome is printed without debug logging (categoryllm):MCP server <name>: <n> tool(s), orWarning: MCP server <name> failed: <reason>. The KMA key (JARVIS_KMA_API_KEY,app/src/admin.spec.md) leaves the environment at the same point; the weather tool keeps it and uses it only for the current weather inside Korea (geocoded or detected countryKR): KMA's 초단기실황 (temperature, humidity, wind, precipitation) and 초단기예보 (the coming hour's sky). Any KMA failure falls back to open-meteo, then met.no; the key is in KMA's URLs, so a failure is logged by its kind only. preparetells the brain thatuseris saying a call whose partial transcript istext(the app sends it while the speaker is still talking,app/src/conversation.spec.md). The brain runs the intent judge's tool pick for it on its own thread, with the asker's dialogue memory and the tool catalogue the reply will have. Areplyfrom that user whose text is about the same (letters and digits, 85% alike), within 30 s and with the dialogue memory unchanged, uses that pick instead of making its own, waiting up to 5 s for one still being made. Each prepared judgement is used at most once. Oneprepareper user runs at a time: while it runs only the newest text waits, a text about the same as the one last prepared is dropped, and areplyfrom that user drops the waiting text. Nothing is sent back; without a brain, or with one that cannot prepare, it is ignored. The asker is set injarvis.reply.turn_statefor the turn, so tools that keep personal data (meals) store and read it under that person only.addressedandrankare the conversation's questions (app/src/conversation.spec.md), answered on their own threads, so they never wait behind a running reply.addressed: wastextsaid to the assistant? The session'shotwordsare given to the judge as the assistant's names, andpeople(in the voice channel, the speaker included) andcallers(of them, who may call Jarvis), when the app sends them (positive integers; otherwise not known). The question follows the room: with 2 people most of what the speaker says counts as said to the assistant (questions, requests, complaints and "you" remarks unless clearly said to the other person; unclear is yes); with 3 or more, or not known, most talk is between people (unclear is no; not known asks exactly as before the counts existed); fewer callers than people adds "only N of the M can use the assistant".rank: the candidate ids most important first (ids the brain left out follow in their order, unknown ids are dropped). A judgement the brain could not make, or that raised, comes backfailed: true(addressed: false, emptyorder). Both are one short non-thinking call to the chat model with the heard text fenced as data (voice_worker/judges.py, evalevals/test_voice_judges.py). Without a brain they answererrorbrain_unavailablewith the id.- While a reply runs, the process's stdout and stderr are muted: the v1 engine and its tools print
questions, answers, tool arguments and memory text, also from helper threads. The worker's own log
(
voice_worker/log.py) writes to the original stderr and keeps working. - Chat channel tool (
src/jarvis/tools/builtin/chat_channels.spec.md): the app sends the registered channels, and the brain's tool asks the app to act. Reads and searches come from the history store.Action ids are unique across connections and worker runs, and an answer whose id is not waiting on this connection is ignored, so a late answer meant for an earlier connection never completes a newer action. With no registered channels the tool is not offered. An action waits up to 15 s for its result (then it fails), and a closed session fails every waiting action. Each reply counts as one turn per person, so a delete is confirmed only in that person's next reply. An answer reaches the person when its speech ends with-> {"type": "chat_channels", "bot": "<bot user id>", "channels": [{"id": "...", "name": "...", "description": "...", "read": true, "send": true, "edit": true, "react": true, "delete": false, "pin": false}]} <- {"type": "chat_action", "id": "a1", "action": "send|edit|react|delete|pin|unpin", "channel": "...", "message_id": "...", "text": "...", "emoji": "...", "user": "<id>"} // who asked; the app records it (Activity) -> {"type": "chat_action_result", "id": "a1", "ok": true, "message_ids": ["..."]} -> {"type": "chat_action_result", "id": "a1", "ok": false, "error": "..."}tts_doneneither failed nor cancelled (speak: true), when the app reports it posted withreply_deliveredok: true(via: "chat"), or atreply_textfor an unspoken voice answer. It never reaches them when the brain gave none, its speech was stopped, failed or had no TTS (tts_unavailable), the app reportsok: false, no report comes within 30 s, or the session ends first. A reply sent before the person's previous answer reached them cannot answer it, and an answer that never reached them asked them nothing: in both cases a turn number is skipped, so a delete asked in that previous answer lapses. - Memory settings, optional in
helloas"memory": {...}and changeable with{"type": "memory_config", ...}:chat(true),others(true),off(false),since_days(null: whole history),recent_minutes(30),recent_records(20),matches(5),budget_tokens(4000). - Logs carry IDs, counts and timings only, never record, question or answer text.
Schedules
The brain's schedule tool (src/jarvis/tools/builtin/schedule.spec.md) asks the app, which keeps
the schedules (app/src/admin.spec.md, "Schedules"):
<- {"type": "schedule_action", "id": "s1", "action": "add|list|delete", "user": "<id>",
"via": "voice|chat", "channel": "<chat channel id>", // channel: only for via chat, when given
"kind": "remind|task", "text": "...", "at": 1762830000.0, "repeat": "none|daily|weekdays|weekly",
"schedule": "<id>"} // add: kind..repeat; delete: schedule
-> {"type": "schedule_action_result", "id": "s1", "ok": true, "item": {...}} // add, delete
-> {"type": "schedule_action_result", "id": "s1", "ok": true, "items": [...]} // list
-> {"type": "schedule_action_result", "id": "s1", "ok": false, "error": "..."}
-> {"type": "reply", "id": "t1", "user": "<id>", "text": "<the task>", "via": "task"}
<- {"type": "reply_text", "id": "t1", "text": "...", "failed": false}
- Every voice and chat reply gets a schedule backend for its asker, the reply's
viaand, for chat, thechannelthereplynamed. Ids and waiting are those of chat actions (unique across connections and runs, 15 s, a closed session fails what waits). via: "task"runs a scheduled task the app fires: in the chat lane and one at a time with the person's other replies, with their history scope, but its own short-term memory, no chat or schedule backend and the read-only tools only (brain.TASK_TOOLS, no MCP tools). It is not a turn of the person's conversation (their turn count and delete confirmations stay), is never spoken (speak: trueisbad_message), and its answer is not stored; the app delivers it. It waits for LLM switches like any reply.
Personal notes
The brain's personalNotes tool (src/jarvis/tools/builtin/personal_notes.spec.md) asks the app,
which keeps each person's standing instructions and sends the asker's with every reply
(app/src/admin.spec.md, "Personal notes"):
-> {"type": "reply", ..., "notes": ["나한테는 반말로 해"]} // optional; at most 20, each at most 300 characters
<- {"type": "note_action", "id": "n1", "action": "add|list|delete", "user": "<id>", "via": "voice|chat",
"channel": "...", "text": "...", "note": "<id>"} // add: text; delete: note
-> {"type": "note_action_result", "id": "n1", "ok": true, "item": {...}} // or "items", or ok false with "error"
notesthat are not a list of at most 20 strings of at most 300 characters arebad_message. The brain puts them in the reply's prompt (person_notes); a scheduled task (via: "task") gets them too, but no notes tool.- Every voice and chat reply gets a notes backend for its asker; ids and waiting are those of chat and schedule actions.
Screen viewing (M7)
The brain's screenView tool (src/jarvis/tools/builtin/screen_view.spec.md) asks the app, which
captures and decides who may see (app/src/admin.spec.md, "Screen viewing"):
<- {"type": "screen_action", "id": "s1", "action": "capture|list", "user": "<id>", "via": "voice|chat",
"channel": "...", "target": "screen|monitor|window", "monitor": 2, "window": "chrome", "detail": "fast|normal|detailed"}
-> {"type": "screen_action_result", "id": "s1", "ok": true, "item": {...}, "images": ["<base64 JPEG>"]} // or ok false with "error"
- Every voice and chat reply gets a screen backend for its asker (not scheduled tasks); ids and
waiting are those of chat and schedule actions.
imagesthat are not a list of strings are dropped. The pictures stay in that one reply (screen_view.spec.md). - Messages may be up to 4 MiB (pictures), the WebSocket's limit.
PC control (M7)
The brain's pcControl tool (src/jarvis/tools/builtin/pc_control.spec.md) asks the app, which acts
through agent-desktop and decides who may (app/src/admin.spec.md, "PC control"):
<- {"type": "pc_action", "id": "p1", "action": "windows|read|click|type|keys|run|close", "user": "<id>",
"via": "voice|chat", "channel": "...", "window": "...", "ref": "@s1:e8", "text": "...", "keys": "ctrl+s",
"program": "notepad", "consent": true}
-> {"type": "pc_action_result", "id": "p1", "ok": true, "item": {...}} // or ok false with "error", or "ask": true
- Every voice and chat reply gets a PC backend for its asker (not scheduled tasks); ids and waiting are those of chat and schedule actions. The worker keeps each asker's requests awaiting consent with the turn they were asked in, numbered like the chat delete confirmations ("History and brain"), so only a message sent after the question counts as the next turn.
Background collection (M6)
With JARVIS_COLLECT_PATH set, the worker collects on its own thread, apart from the voice and the
brain (app/src/admin.spec.md, "Background collection"). The file is
{"dir": "<data>/collect", "keepDays": 30, "targets": [{"name", "kind", "value", "everyMinutes"}]};
the app writes only enabled targets. Nothing is collected without targets, but old day files are
still deleted (below).
- A target runs when
everyMinutespassed since its last run (also across restarts; the last run is kept in<dir>/<name>/state.json), one target at a time, checked every 30 s. A failed run counts as a run, so a broken site is tried again only at its next interval. - Kinds:
sitekeeps the page's title and text (at most 20,000 characters) when either differs from the last kept ones;rss(RSS or Atom) keeps each new item (title, link, time, summary of at most 2,000 characters, new by its id or link);searchkeeps each new web search result (title and link) for the words, from DuckDuckGo's lite page, which needs no key (its bot challenge is a failed run, not "no results"). - Every request checks its address like the web search (
web_search.spec.md): only http(s), and every address the name resolves to must be public (for IPv6 only global unicast2000::/3, without 6to4); the connection goes only to the checked addresses in their order (TLS still checks the name), so a second lookup cannot lead into the local network. Every redirect is checked the same way. At most 1 MB and 15 s in all: the name lookup, connecting, TLS, the headers and the body of every redirect count, and a server that sends slowly, even byte by byte within the per-read limit, is cut off at 15 s. A response cut short is a failure. Characters outside ASCII (and spaces) in the path and words go as UTF-8%HH; existing%HHand separators stay. The check logs nothing about the address. The collected text is web content: it goes only to<dir>/<name>/<YYYY-MM-DD>.jsonl(one JSON object per line:at, thentitle,url,textas the kind has them), never to a log. The collector itself never hands it to the brain; the brain reads it only through itscollectedNewstool, fenced as untrusted web content (src/jarvis/tools/builtin/collected_news.spec.md), and the app announces new titles and links when asked to (app/src/admin.spec.md, "News notifications"). - Day files older than
keepDaysare deleted in every folder ofdirat each check, also of targets no longer listed. - Each run is a
noticeline (categoryvoice_worker): the target's name and how many new items, or aWarningwith the error's type. Never a URL, words or collected text.
LLM switches (M7)
The app restarts the LLM with another placement when a game starts or ends (app/src/admin.spec.md, "Performance stages"). It tells the worker first and waits for the brain to be idle:
-> {"type": "llm", "state": "switching", "id": 7, "notice": "잠깐만요, 게임 모드로 바꾸는 중이에요"} // id, notice optional
<- {"type": "llm_idle", "id": 7} // nothing that uses the LLM runs any more (id: the switch's, when it had one)
-> {"type": "llm", "state": "ready", "game": true} // the LLM answers again; game optional (default false)
-> {"type": "llm", "state": "paused"} // the LLM is stopped (a game runs)
- Brain work that uses the LLM (
reply,prepare,addressed,rank) counts as running from its start to its end. Onswitchingnew work waits;llm_idlegoes to the connection that sentswitchingonce nothing runs (at once when nothing does). Onreadywhat waited goes on. - The first spoken
replythat waits during one switch hears the notice under its own id (its answer follows in the same speech when ready); later ones wait without it. A notice ends with a newline, so it is spoken at once and never joined to the answer. A request waits at most 180 s, then fails without calling the brain (the LLM may still be away, or a measuring server stand in). - On
pausedbrain work answers nothing at once (reply_textfailed, judgements failed) untilready; what waited for a switch stops waiting the same way. gameonreadysays the LLM runs in the game placement: from then on, while areplyis being written, partial passes wait (finals go on at once, "Scheduling"); when no answer is being written they go on, the newest audio per speaker.readywithout it (or false) ends that. Without a game partials never wait (M0gamemode_tune: with partials held the game placement answered 1.7 times faster, 64.9 against 37.5 tok/s with a partial pass every 2 s).- Any other
state, a notice that is not a string, an id that is not an integer or agamethat is not a boolean isbad_message. - The app sends its latest
llmmessage again first on every connection, so a worker that missed one (a dropped connection, a restarted worker) catches up.
Recordings
For checking what was heard against what was transcribed, and what Jarvis said. Off unless the app
sets JARVIS_RECORDINGS_DIR (setting voice.record, off by default; the app passes
<data folder>/recordings at start). Then voice_worker/recordings.py keeps:
- each finished utterance (every final, also one with no text) as a 16 kHz mono WAV of the clip as it
was cut for transcription, before loudness levelling:
<day>/<HHMMSS>-<ms>-said-<speaker>.wav; - each spoken answer as a mono WAV at the session's TTS rate, as far as the app played it:
<day>/<HHMMSS>-<ms>-jarvis.wav, timed at its first sentence. The app sendstts_played(played: how many of its sentences started playing,cancelled: whether it was stopped,last_ms: when it was stopped while a sentence played, how long that sentence had played) when each request ends; the worker keeps the request's sentences until it is made and that report came, then writes the firstplayedsentences, the last cut tolast_ms(at the TTS rate, the rate it is played at), withcancelled. Nothing played (0), never reported (the session ends, an older app; at most 50 are held) or ids it does not know (the earcon): no file; - one JSON line per file in
<day>/index.jsonl:{file, at, ms, kind: said|spoke, who, name, channel, text, id, cancelled}(the transcript, or the text sent totts).
<day> is the local day. Files are written on a thread of their own, so the voice path never waits,
and a failed write is a debug line (no text) that changes nothing else. Day folders older than 7 days
are removed when the worker starts and with the first recording of each new day while it runs.
Nothing is recorded without the variable.
Errors
{"type": "error", "code": "...", "message": "...", "id": "..."}. Codes: bad_message (unknown type,
bad JSON, bad field), bad_frame (malformed binary frame), tts_unavailable, tts_failed,
brain_unavailable. Errors
never close the connection after the handshake.
Debug logging
debug_log(message, category) at: connection accepted or refused, settings changed, speaker left, utterance end
(speaker, reason, length), each STT batch (kind, size, model time), each TTS sentence (length, time),
cancel, and engine failures. The category names the service, so the app's log screen puts each line
in its tab (app/src/admin.spec.md, "Logs"): stt (utterance ends, STT batches, the Whisper model),
tts (sentences, cancels, MeloTTS), llm (the brain's replies, prepares and judgements),
voice_worker for the rest (connections, sessions, channels).
Without debug logging too, each answer and each scheduled task logs one llm line once the LLM is
done with it: the least free graphics memory read while it was written, the card's total, how many
readings and how long it took (answer <id> card: least N MiB free of T MiB (k readings, ms)). The
card is read with nvidia-smi (memory.free, which leaves out what the driver reserves) once a
second while the answer is written, in a thread, so the answer never waits for it: a reading still
running when the answer ends finishes on its own and the line follows it, so a short answer gets its
line too. Without nvidia-smi nothing is read or logged. The line carries no text, so a
slow answer can be matched to a full card (the engine log has the same answer's timings).