Files
javis_bot/app/src/conversation.spec.md
javis-bot b88e20134f feat(app): a call still being said lets the brain get ready
A partial transcript that call detection already reads as a request is sent
to the worker as prepare, so the intent judge runs while the speaker talks.
The live round trip now times the earcon and the answer's own sound apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:25:15 +09:00

16 KiB
Raw Blame History

Conversation spec: call detection and response policies (v2 M4)

The part of the app that decides which finished utterances (and, later, chat messages) Jarvis answers and in what order, and when an answer that is being spoken stops. It sits between the Discord voice core (discord_core.spec.md), which reports what was heard and plays speech, and the voice worker's brain (src/voice_worker/voice_worker.spec.md, reply), which writes the answer. The user-facing design is docs/v2/conversation.md; this spec is the contract the code follows.

Principles

  • Plain types, tested without Discord. Call detection (app/src/calls.rs) is a pure function of the settings, the speaker and the text. The response policy (app/src/policy.rs) is a state machine driven by events with the time passed in, so every rule is tested with fake times.
  • Language-agnostic. Nothing matches a language's words in code. Wake words and stop words are settings; matching works on Unicode-normalised letters, so it behaves the same in any script.
  • No text in logs. Logs carry speaker IDs, decisions, reasons, counts and timings, never what was said.
  • Nothing is dropped silently. Every request the policy does not answer, and every answer it stops, is reported as missed with a reason (the dashboard's "놓친 요청" list, M5).

Call detection

Settings (settings.json, section call; wake words stay in voice.wakeWords):

Key Default Meaning
voice.wakeWords ["자비스"] Wake words for voice (also the STT hint).
call.chatWakeWords null Wake words for chat; null uses voice.wakeWords.
call.similarity 0.2 How far a heard word may be from a wake word, as a share of the wake word's normalised letters (0-0.5). 0 needs an exact match.
call.mode "wakeWord" wakeWord: answer only when called by a wake word. addressed: also answer what is judged to be said to Jarvis. everything: answer every utterance.
call.designated [] Discord user IDs of the designated people.
call.designatedOnly false Answer only designated people.
call.designatedNoWakeWord false Answer designated people without a wake word.
call.stopWords ["멈춰", "그만", "정지", "stop"] Said together with a wake word, they stop Jarvis in any policy.
call.earcon true Play the earcon (below) when Jarvis takes a voice call.
call.followUpMs 1500 After a wake word said alone, how long the speaker's next utterance may take to start and still be joined to it. 0-5000.

Normalisation: Unicode NFKD, lower case, and only letters and digits kept (spaces, punctuation and combining marks dropped). NFKD splits Hangul syllables into their letters, so "자비즈" is one letter away from "자비스" (of six) while "서비스" is two away.

A wake word is heard when some stretch of the normalised text is at most floor(similarity × letters of the wake word) edits (insert, delete, substitute) from it. With the default 0.2, "자비스" (six letters) allows one edit. A wake word anywhere in the utterance counts.

Decision for one utterance (judge(speaker, text)), in order:

  1. Empty text (after normalisation): ignore.
  2. designatedOnly and the speaker is not designated: ignore.
  3. A wake word is heard and the rest of the text, with that stretch removed, starts with a stop word and has at most 3 more letters ("멈춰", "멈춰 줘", "그만해"): stop.
  4. A wake word is heard: request, direct (called by name).
  5. designatedNoWakeWord and the speaker is designated: request, not direct.
  6. call.mode everything: request, not direct. addressed: ask (the brain judges whether it was said to Jarvis; a yes becomes a request, not direct). wakeWord: ignore.

The text sent to the brain is the utterance as heard; the brain treats the wake word as address.

A wake word said alone. People often pause after the name ("자비스, ... 지금 몇 시야?"), and a pause longer than the worker's end_silence_ms ends the utterance there. A voice utterance that is a wake word and nothing else (no letters or digits left once the wake word is removed) is therefore not a request yet: the conversation keeps it for that speaker and joins it to their next final, with a space between, and judges the joined text as one utterance (so "자비스" then "멈춰" is a stop). If the speaker's next utterance already started when the wake word's final came, or starts within call.followUpMs of it, the wake word waits for that utterance's final however long it takes; otherwise, after call.followUpMs, the wake word alone is the request. Other speakers' utterances never join it, and a stop (voice, slash command or leaving) drops it. So does a lost worker connection (WorkerLost): the final it waited for will not come, and the restarted worker's utterances are not continuations of it. Chat messages are not joined: a message is complete as sent.

Earcon. A short two-tone "띠링" (about 0.3 s, made by the app, app/src/earcon.rs) tells the speaker Jarvis heard the call, long before the answer is written: it is the first sound after a call. It plays right before each voice answer starts (the policy's Answer), and at once when a wake word said alone is kept waiting while no answer is running (the question that joins it then gets no second earcon). It is queued like speech, so it waits behind speech already queued and a stop drops it. A call the policy does not take (busy, replaced, stopped) gets none, nor does a wake word alone while an answer runs. Chat answers get none. call.earcon: false turns it off.

Response policy

Settings (section response):

Key Default Meaning
response.policy "first" first (처음 대화만), inOrder (들어온 순서대로), latest (마지막 대화만), important (제일 중요한 대화만).
response.queueMax 5 inOrder: most waiting requests; past it the oldest is dropped. 1-20.
response.collectMs 700 important: how long candidates are collected before choosing. 0-3000.
response.settleMs 800 Longest wait for an earlier-ended utterance that is still being transcribed. 0-3000.
response.priorities {} important: user ID → priority (higher first, default 0).

Terms:

  • A request has an id, the speaker, the text, whether it was direct, and ended_at: when the utterance ended (the final's arrival minus its wait_ms).
  • An answer runs from the moment the policy starts it until it is done (its speech ended, failed or was stopped). There is at most one answer at a time: Jarvis has one voice.
  • An utterance is open from its speech_start until its final. An open utterance whose speaker has stopped speaking (speaking_end sent) is ending: its final may still turn out to be a call.

Rules shared by every policy:

  • Ordering by end time. Requests are compared by ended_at, not by when their final arrived, so a long utterance is not pushed back by its transcription time.
  • Settling. When a request could start now but some other utterance is ending and stopped earlier than the request's ended_at (in important: any ending utterance, since it competes on score), the policy waits for that final, at most settleMs from the first waiting request's arrival. A final that is not a call ends the wait for it. latest does not settle: an earlier utterance would lose anyway.
  • Stop. A stop decision stops the current answer and drops every waiting request, in every policy. The stopped answer and the dropped requests are missed (stopped).
  • An answer that is already done is never started again.

Per policy:

Policy No answer running A new request while an answer runs
first Answer the request that ended first (after settling). Missed (busy).
inOrder Answer the earliest-ended waiting request. Queued by ended_at. Past queueMax, the earliest-ended waiting one is missed (queue_full).
latest Answer the latest-ended waiting request; the others are missed (replaced). The current answer stops (missed, interrupted) and the new request is answered. A request that ended before the current one is missed (replaced).
important Collect for collectMs from the first request, then answer the best; the other candidates are missed (lower_score). Kept as a candidate. If it beats the current answer, the current answer stops (missed, interrupted) and the best candidate is answered at once; otherwise, when the current answer ends, the best of the candidates collected meanwhile is answered and the rest are missed (lower_score).

important ranks by (1) priority, (2) direct, then (3) the brain's importance ranking, asked for only when two or more candidates tie on (1) and (2). While the ranking is pending nothing new starts; if it does not come within 1.5 s (a ranking that arrives later counts as late), or fails, the earliest-ended of the tied candidates wins. Candidates are kept in ended_at order, whenever they arrived.

Each ranking carries an id (Rank { id, candidates }, answered with ranked(id, order)). Only the pending ranking's answer is used: an answer to a ranking that was replaced, or dropped (its answer ended, or a stop), is ignored. A candidate that ties the ranked ones while a ranking is pending replaces it with a new ranking over all of them (and the running answer, when the ranking was about interrupting it), keeping the first ranking's deadline, so only candidates that were compared are missed (lower_score). A candidate that beats the ranked ones makes the ranking moot.

A new request that beats the running answer on (1) or (2) interrupts it at once. One that ties it on both is ranked against it (the running answer keeps playing meanwhile): if the ranking puts the new request first, the answer stops (missed, interrupted) and the new request is answered; if it puts the running answer first, or does not come in time, the answer goes on and the new request waits. The ranking is dropped when the running answer ends first.

Chat calls

Registered chat channels are answered in chat, by text, beside the voice conversation: a chat message never takes the voice turn and never goes through the response policy.

Key Default Meaning
chatChannels[].answer "wakeWord" wakeWord: answer messages that call a wake word. everything: answer every message.
  • The wake words are call.chatWakeWords (or voice.wakeWords when it is null), with the same similarity and the same designated-people switches as voice; call.mode is voice only.
  • Only messages of a registered channel with reading on reach the loop (the core's rule), and only a channel with sending on is answered: there an answer could not be posted, so none is asked for.
  • Messages from bots (other bots and Jarvis itself) are never answered, so two bots cannot keep answering each other.
  • Only a request is answered: a stop in chat ("자비스 멈춰") is not a question and gets no answer.
  • The brain answers through reply with via: "chat" and speak: false; the answer is posted in the same channel as a Discord reply to the message (so the history records it as an answer to that person), pinging nobody. A failed or empty answer posts nothing. An answer that arrives 5 minutes or more after it was asked is not posted. The worker is told whether each answer was posted (reply_delivered).
  • The worker writes voice and chat answers in separate lanes (voice_worker.spec.md), so a voice answer never waits behind another person's chat answer; one person's answers are written one at a time, in the order they asked. For the two to run at the same time the LLM must take two requests at once (docs/v2/conversation.md, "LLM 동시 요청 수"): until the app starts Strata itself (M5), Strata is started with --parallel 2 when some channel answers everything; with --parallel 1 the two lanes' LLM calls take turns at Strata.

The conversation loop

app/src/conversation.rs runs between the core's events and everything else that reads them (every event is passed on unchanged). It talks to the core through a small Voice interface (answer, cancel one request, stop all, earcon, prepare, ask "addressed", ask "rank"), so the glue is tested with a fake.

  • Chat and ReplyText: chat calls, above.
  • Partial (getting ready): a partial transcript that call detection already reads as a request (joined to the speaker's waiting wake word alone, as its final would be; not the wake word alone, not a stop) is sent to the brain as prepare(user, text), so the brain judges what the call needs while the speaker is still talking (voice_worker.spec.md, prepare). Not while an answer runs: the call may not be taken, and the brain is busy answering.
  • SpeechStart opens the speaker's utterance; SpeakingStopped marks it ending.
  • Final: a request gets the next id, ended_at = arrival minus the final's wait_ms, and goes through call detection. Ignore closes the utterance. Stop stops all speech, runs the policy's stop, voids every pending "addressed" question (closing its utterance) and closes the utterance. Request goes to the policy. Ask sends addressed to the brain; a yes becomes the request, a no (or a failed judgement) closes the utterance, and no answer within 5 s counts as no, also when the answer arrives after that but before the timer ran.
  • Policy actions: Answer → the earcon (unless it already played for this request), then answer(user, text) (the brain answers and the answer is spoken; when it cannot be queued the answer is done at once); StopAnswer → cancel of that answer's speech; Rank { id } → rank with id k<id> and the candidates' request ids; Missed → a debug log and the missed list (the last 100: speaker, reason, time and length, never the text).
  • SpeakEnd of an answer's speech is that answer's done. The end of a speech the loop already cancelled starts nothing.
  • Stopped (all speech stopped from elsewhere: the slash command, stop(), the session ending) and Left: the same stop. The core sends Stopped before the SpeakEnds of the speech it stopped, so a cancelled answer's end never starts the next waiting request.
  • A timer wakes the loop at the policy's next deadline or the earliest "addressed" timeout.

The brain's two judgements are src/voice_worker/judges.py (voice_worker.spec.md): one short call each to the chat model, the heard text fenced as data. Eval: evals/test_voice_judges.py.

Tests

cargo test covers call detection (exact and near wake words in Hangul and Latin, a different word two letters away, wake word anywhere, stop with and without a wake word, designated combinations, modes) and every policy rule above with fake times (settling, ordering by end time, busy, queue limit, interruption, ranking tie, ranking timeout, stop dropping the queue), and the conversation loop with a fake voice side (answering and queueing, a busy drop recorded without text, voice stop, addressed yes, no and timeout, the earcon before each answer, once for a wake word alone, none while busy or when off, partials that read as a call sent to get ready (joined to a waiting wake word, none while answering), interruption cancelling the right speech, a ranking deciding, end times from wait_ms, and chat: an answer posted as a reply once, per-channel switches, no answers to bots or to a stop, chat wake words and designated people, a failed answer posting nothing). tests/worker_protocol.rs checks an answer spoken under its id and both judgements against the real Python worker with a fake brain.