A partial transcript that call detection already reads as a request is sent to the worker as prepare, so the intent judge runs while the speaker talks. The live round trip now times the earcon and the answer's own sound apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
16 KiB
Conversation spec: call detection and response policies (v2 M4)
The part of the app that decides which finished utterances (and, later, chat messages) Jarvis answers
and in what order, and when an answer that is being spoken stops. It sits between the Discord voice
core (discord_core.spec.md), which reports what was heard and plays speech, and the voice worker's
brain (src/voice_worker/voice_worker.spec.md, reply), which writes the answer. The user-facing
design is docs/v2/conversation.md; this spec is the contract the code follows.
Principles
- Plain types, tested without Discord. Call detection (
app/src/calls.rs) is a pure function of the settings, the speaker and the text. The response policy (app/src/policy.rs) is a state machine driven by events with the time passed in, so every rule is tested with fake times. - Language-agnostic. Nothing matches a language's words in code. Wake words and stop words are settings; matching works on Unicode-normalised letters, so it behaves the same in any script.
- No text in logs. Logs carry speaker IDs, decisions, reasons, counts and timings, never what was said.
- Nothing is dropped silently. Every request the policy does not answer, and every answer it stops, is reported as missed with a reason (the dashboard's "놓친 요청" list, M5).
Call detection
Settings (settings.json, section call; wake words stay in voice.wakeWords):
| Key | Default | Meaning |
|---|---|---|
voice.wakeWords |
["자비스"] |
Wake words for voice (also the STT hint). |
call.chatWakeWords |
null |
Wake words for chat; null uses voice.wakeWords. |
call.similarity |
0.2 |
How far a heard word may be from a wake word, as a share of the wake word's normalised letters (0-0.5). 0 needs an exact match. |
call.mode |
"wakeWord" |
wakeWord: answer only when called by a wake word. addressed: also answer what is judged to be said to Jarvis. everything: answer every utterance. |
call.designated |
[] |
Discord user IDs of the designated people. |
call.designatedOnly |
false |
Answer only designated people. |
call.designatedNoWakeWord |
false |
Answer designated people without a wake word. |
call.stopWords |
["멈춰", "그만", "정지", "stop"] |
Said together with a wake word, they stop Jarvis in any policy. |
call.earcon |
true |
Play the earcon (below) when Jarvis takes a voice call. |
call.followUpMs |
1500 |
After a wake word said alone, how long the speaker's next utterance may take to start and still be joined to it. 0-5000. |
Normalisation: Unicode NFKD, lower case, and only letters and digits kept (spaces, punctuation and combining marks dropped). NFKD splits Hangul syllables into their letters, so "자비즈" is one letter away from "자비스" (of six) while "서비스" is two away.
A wake word is heard when some stretch of the normalised text is at most
floor(similarity × letters of the wake word) edits (insert, delete, substitute) from it. With the
default 0.2, "자비스" (six letters) allows one edit. A wake word anywhere in the utterance counts.
Decision for one utterance (judge(speaker, text)), in order:
- Empty text (after normalisation): ignore.
designatedOnlyand the speaker is not designated: ignore.- A wake word is heard and the rest of the text, with that stretch removed, starts with a stop word and has at most 3 more letters ("멈춰", "멈춰 줘", "그만해"): stop.
- A wake word is heard: request,
direct(called by name). designatedNoWakeWordand the speaker is designated: request, not direct.call.modeeverything: request, not direct.addressed: ask (the brain judges whether it was said to Jarvis; a yes becomes a request, not direct).wakeWord: ignore.
The text sent to the brain is the utterance as heard; the brain treats the wake word as address.
A wake word said alone. People often pause after the name ("자비스, ... 지금 몇 시야?"), and a
pause longer than the worker's end_silence_ms ends the utterance there. A voice utterance that is a
wake word and nothing else (no letters or digits left once the wake word is removed) is therefore
not a request yet: the conversation keeps it for that speaker and joins it to their next final, with
a space between, and judges the joined text as one utterance (so "자비스" then "멈춰" is a stop). If
the speaker's next utterance already started when the wake word's final came, or starts within
call.followUpMs of it, the wake word waits for that utterance's final however long it takes;
otherwise, after call.followUpMs, the wake word alone is the request. Other speakers' utterances
never join it, and a stop (voice, slash command or leaving) drops it. So does a lost worker
connection (WorkerLost): the final it waited for will not come, and the restarted worker's
utterances are not continuations of it. Chat messages are not joined:
a message is complete as sent.
Earcon. A short two-tone "띠링" (about 0.3 s, made by the app, app/src/earcon.rs) tells the
speaker Jarvis heard the call, long before the answer is written: it is the first sound after a call.
It plays right before each voice answer starts (the policy's Answer), and at once when a wake word
said alone is kept waiting while no answer is running (the question that joins it then gets no second
earcon). It is queued like speech, so it waits behind speech already queued and a stop drops it. A
call the policy does not take (busy, replaced, stopped) gets none, nor does a wake word alone while an
answer runs. Chat answers get none. call.earcon: false turns it off.
Response policy
Settings (section response):
| Key | Default | Meaning |
|---|---|---|
response.policy |
"first" |
first (처음 대화만), inOrder (들어온 순서대로), latest (마지막 대화만), important (제일 중요한 대화만). |
response.queueMax |
5 |
inOrder: most waiting requests; past it the oldest is dropped. 1-20. |
response.collectMs |
700 |
important: how long candidates are collected before choosing. 0-3000. |
response.settleMs |
800 |
Longest wait for an earlier-ended utterance that is still being transcribed. 0-3000. |
response.priorities |
{} |
important: user ID → priority (higher first, default 0). |
Terms:
- A request has an id, the speaker, the text, whether it was
direct, andended_at: when the utterance ended (the final's arrival minus itswait_ms). - An answer runs from the moment the policy starts it until it is done (its speech ended, failed or was stopped). There is at most one answer at a time: Jarvis has one voice.
- An utterance is open from its
speech_startuntil its final. An open utterance whose speaker has stopped speaking (speaking_endsent) is ending: its final may still turn out to be a call.
Rules shared by every policy:
- Ordering by end time. Requests are compared by
ended_at, not by when their final arrived, so a long utterance is not pushed back by its transcription time. - Settling. When a request could start now but some other utterance is ending and stopped earlier
than the request's
ended_at(inimportant: any ending utterance, since it competes on score), the policy waits for that final, at mostsettleMsfrom the first waiting request's arrival. A final that is not a call ends the wait for it.latestdoes not settle: an earlier utterance would lose anyway. - Stop. A stop decision stops the current answer and drops every waiting request, in every policy.
The stopped answer and the dropped requests are missed (
stopped). - An answer that is already done is never started again.
Per policy:
| Policy | No answer running | A new request while an answer runs |
|---|---|---|
first |
Answer the request that ended first (after settling). | Missed (busy). |
inOrder |
Answer the earliest-ended waiting request. | Queued by ended_at. Past queueMax, the earliest-ended waiting one is missed (queue_full). |
latest |
Answer the latest-ended waiting request; the others are missed (replaced). |
The current answer stops (missed, interrupted) and the new request is answered. A request that ended before the current one is missed (replaced). |
important |
Collect for collectMs from the first request, then answer the best; the other candidates are missed (lower_score). |
Kept as a candidate. If it beats the current answer, the current answer stops (missed, interrupted) and the best candidate is answered at once; otherwise, when the current answer ends, the best of the candidates collected meanwhile is answered and the rest are missed (lower_score). |
important ranks by (1) priority, (2) direct, then (3) the brain's importance ranking, asked for only
when two or more candidates tie on (1) and (2). While the ranking is pending nothing new starts; if it
does not come within 1.5 s (a ranking that arrives later counts as late), or fails, the earliest-ended
of the tied candidates wins. Candidates are kept in ended_at order, whenever they arrived.
Each ranking carries an id (Rank { id, candidates }, answered with ranked(id, order)). Only the
pending ranking's answer is used: an answer to a ranking that was replaced, or dropped (its answer ended,
or a stop), is ignored. A candidate that ties the ranked ones while a ranking is pending replaces it with
a new ranking over all of them (and the running answer, when the ranking was about interrupting it),
keeping the first ranking's deadline, so only candidates that were compared are missed (lower_score).
A candidate that beats the ranked ones makes the ranking moot.
A new request that beats the running answer on (1) or (2) interrupts it at once. One that ties it on
both is ranked against it (the running answer keeps playing meanwhile): if the ranking puts the new
request first, the answer stops (missed, interrupted) and the new request is answered; if it puts the
running answer first, or does not come in time, the answer goes on and the new request waits. The
ranking is dropped when the running answer ends first.
Chat calls
Registered chat channels are answered in chat, by text, beside the voice conversation: a chat message never takes the voice turn and never goes through the response policy.
| Key | Default | Meaning |
|---|---|---|
chatChannels[].answer |
"wakeWord" |
wakeWord: answer messages that call a wake word. everything: answer every message. |
- The wake words are
call.chatWakeWords(orvoice.wakeWordswhen it isnull), with the same similarity and the same designated-people switches as voice;call.modeis voice only. - Only messages of a registered channel with reading on reach the loop (the core's rule), and only a channel with sending on is answered: there an answer could not be posted, so none is asked for.
- Messages from bots (other bots and Jarvis itself) are never answered, so two bots cannot keep answering each other.
- Only a request is answered: a stop in chat ("자비스 멈춰") is not a question and gets no answer.
- The brain answers through
replywithvia: "chat"andspeak: false; the answer is posted in the same channel as a Discord reply to the message (so the history records it as an answer to that person), pinging nobody. A failed or empty answer posts nothing. An answer that arrives 5 minutes or more after it was asked is not posted. The worker is told whether each answer was posted (reply_delivered). - The worker writes voice and chat answers in separate lanes (
voice_worker.spec.md), so a voice answer never waits behind another person's chat answer; one person's answers are written one at a time, in the order they asked. For the two to run at the same time the LLM must take two requests at once (docs/v2/conversation.md, "LLM 동시 요청 수"): until the app starts Strata itself (M5), Strata is started with--parallel 2when some channel answers everything; with--parallel 1the two lanes' LLM calls take turns at Strata.
The conversation loop
app/src/conversation.rs runs between the core's events and everything else that reads them (every
event is passed on unchanged). It talks to the core through a small Voice interface (answer, cancel
one request, stop all, earcon, prepare, ask "addressed", ask "rank"), so the glue is tested with a fake.
ChatandReplyText: chat calls, above.Partial(getting ready): a partial transcript that call detection already reads as a request (joined to the speaker's waiting wake word alone, as its final would be; not the wake word alone, not a stop) is sent to the brain asprepare(user, text), so the brain judges what the call needs while the speaker is still talking (voice_worker.spec.md,prepare). Not while an answer runs: the call may not be taken, and the brain is busy answering.SpeechStartopens the speaker's utterance;SpeakingStoppedmarks it ending.Final: a request gets the next id,ended_at= arrival minus the final'swait_ms, and goes through call detection. Ignore closes the utterance. Stop stops all speech, runs the policy's stop, voids every pending "addressed" question (closing its utterance) and closes the utterance. Request goes to the policy. Ask sendsaddressedto the brain; a yes becomes the request, a no (or a failed judgement) closes the utterance, and no answer within 5 s counts as no, also when the answer arrives after that but before the timer ran.- Policy actions:
Answer→ the earcon (unless it already played for this request), thenanswer(user, text)(the brain answers and the answer is spoken; when it cannot be queued the answer is done at once);StopAnswer→cancelof that answer's speech;Rank { id }→rankwith idk<id>and the candidates' request ids;Missed→ a debug log and the missed list (the last 100: speaker, reason, time and length, never the text). SpeakEndof an answer's speech is that answer's done. The end of a speech the loop already cancelled starts nothing.Stopped(all speech stopped from elsewhere: the slash command,stop(), the session ending) andLeft: the same stop. The core sendsStoppedbefore theSpeakEnds of the speech it stopped, so a cancelled answer's end never starts the next waiting request.- A timer wakes the loop at the policy's next deadline or the earliest "addressed" timeout.
The brain's two judgements are src/voice_worker/judges.py (voice_worker.spec.md): one short call
each to the chat model, the heard text fenced as data. Eval: evals/test_voice_judges.py.
Tests
cargo test covers call detection (exact and near wake words in Hangul and Latin, a different word
two letters away, wake word anywhere, stop with and without a wake word, designated combinations,
modes) and every policy rule above with fake times (settling, ordering by end time, busy, queue limit,
interruption, ranking tie, ranking timeout, stop dropping the queue), and the conversation loop with a
fake voice side (answering and queueing, a busy drop recorded without text, voice stop, addressed yes,
no and timeout, the earcon before each answer, once for a wake word alone, none while busy or when
off, partials that read as a call sent to get ready (joined to a waiting wake word, none while
answering), interruption cancelling the right speech, a ranking deciding, end times from
wait_ms, and chat: an answer posted as a reply once, per-channel switches, no answers to bots or to
a stop, chat wake words and designated people, a failed answer posting nothing). tests/worker_protocol.rs checks an answer spoken under its id and both judgements against
the real Python worker with a fake brain.