21 KiB
Conversation spec: call detection and response policies (v2 M4)
The part of the app that decides which finished utterances (and, later, chat messages) Jarvis answers
and in what order, and when an answer that is being spoken stops. It sits between the Discord voice
core (discord_core.spec.md), which reports what was heard and plays speech, and the voice worker's
brain (src/voice_worker/voice_worker.spec.md, reply), which writes the answer. The user-facing
design is docs/v2/conversation.md; this spec is the contract the code follows.
Principles
- Plain types, tested without Discord. Call detection (
app/src/calls.rs) is a pure function of the settings, the speaker and the text. The response policy (app/src/policy.rs) is a state machine driven by events with the time passed in, so every rule is tested with fake times. - Language-agnostic. Nothing matches a language's words in code. Wake words and stop words are settings; matching works on Unicode-normalised letters, so it behaves the same in any script.
- No text in logs. Logs carry speaker IDs, decisions, reasons, counts and timings, never what was said.
- Nothing is dropped silently. Every request the policy does not answer, and every answer it stops, is reported as missed with a reason (the dashboard's "놓친 요청" list, M5).
Call detection
Settings (settings.json, section call; wake words stay in voice.wakeWords):
| Key | Default | Meaning |
|---|---|---|
voice.wakeWords |
["비서", "비서야", "비서님", "김비서", "김비서야", "김비서님"] |
Wake words for voice (also the STT hint). The default calls Jarvis like a secretary, by the word and its usual call forms (user request, 2026-11-19; before it was ["자비스"]). |
call.chatWakeWords |
null |
Wake words for chat; null uses voice.wakeWords. |
call.similarity |
0.2 |
How far a heard word may be from a wake word, as a share of the wake word's normalised letters (0-0.5). 0 needs an exact match. |
call.mode |
"wakeWord" |
wakeWord: answer only when called by a wake word. addressed: also answer what is judged to be said to Jarvis. everything: answer every utterance. |
call.designated |
[] |
Discord user IDs of the designated people. |
call.designatedOnly |
false |
Answer only designated people. |
call.designatedNoWakeWord |
false |
Answer designated people without a wake word. |
call.stopWords |
["멈춰", "그만", "정지", "stop"] |
Said together with a wake word, they stop Jarvis in any policy. |
call.earcon |
true |
Play the earcon (below) when Jarvis takes a voice call. |
call.followUpMs |
1500 |
After a wake word said alone, how long the speaker's next utterance may take to start and still be joined to it. 0-5000. |
call.continueMs |
5000 |
Going on (below): how long after a request's utterance ended the speaker's next utterance may start and still be part of it, while its answer is not heard yet. 0 turns joining off. 0-15000. |
Normalisation: Unicode NFKD, lower case, and only letters and digits kept (spaces, punctuation and combining marks dropped). NFKD splits Hangul syllables into their letters, so "자비즈" is one letter away from "자비스" (of six) while "서비스" is two away.
A wake word is heard when some stretch of the normalised text is at most
floor(similarity × letters of the wake word) edits (insert, delete, substitute) from it. With the
default 0.2, "자비스" (six letters) allows one edit. A wake word anywhere in the utterance counts.
The earliest heard wake word is the call; where several are heard starting there, the one with the
fewest edits, then the longest, so a call form ("비서야") is removed whole and leaves nothing over
("비서야 멈춰" is a stop), and a near match of a longer word does not eat into what follows.
Decision for one utterance (judge(speaker, text)), in order:
- Empty text (after normalisation): ignore.
designatedOnlyand the speaker is not designated: ignore.- A wake word is heard and the rest of the text, with that stretch removed, starts with a stop word and has at most 3 more letters ("멈춰", "멈춰 줘", "그만해"): stop.
- A wake word is heard: request,
direct(called by name). designatedNoWakeWordand the speaker is designated: request, not direct.call.modeeverything: request, not direct.addressed: when the speaker is the only person in Jarvis's voice channel (the people it hears, not Jarvis or ignored bots; the speaker always counts, also while the channel list lags), request, not direct: alone with Jarvis, whatever they say is said to it. Otherwise ask: the brain judges whether it was said to Jarvis, told how many people are in the channel and how many of them may call Jarvis (all, or withdesignatedOnlythe designated ones present), since with one other person most of what the speaker says is for Jarvis and with more most talk is between people (voice_worker.spec.md,addressed); a yes becomes a request, not direct.wakeWord: ignore.
The text sent to the brain is the utterance as heard; the brain treats the wake word as address.
A wake word said alone. People often pause after the name ("자비스, ... 지금 몇 시야?"), and a
pause longer than the worker's end_silence_ms ends the utterance there. A voice utterance that is a
wake word and nothing else (no letters or digits left once the wake word is removed) is therefore
not a request yet: the conversation keeps it for that speaker and joins it to their next final, with
a space between, and judges the joined text as one utterance (so "자비스" then "멈춰" is a stop). If
the speaker's next utterance already started when the wake word's final came, or starts within
call.followUpMs of it, the wake word waits for that utterance's final however long it takes;
otherwise, after call.followUpMs, the wake word alone is the request. Other speakers' utterances
never join it, and a stop (voice, slash command or leaving) drops it. So does a lost worker
connection (WorkerLost): the final it waited for will not come, and the restarted worker's
utterances are not continuations of it. Chat messages are not joined:
a message is complete as sent.
Going on. People say one thing in pieces, with pauses of a fraction of a second to several
seconds between them (live 2026-11-19: 0.1-4.7 s inside one thought), and every pause longer than
the worker's end_silence_ms ends an utterance. Each piece used to be answered on its own, one after
another, and the answers piled up (user request, 2026-11-19: what is said in pieces is one request).
So a speaker's voice request stays open until its answer is heard (the first audio of its speech
plays, SpeakStart), and while it is open their next final is joined to it when that utterance
started within call.continueMs of the request's end (start = the final's arrival minus its
wait_ms and audio_ms):
- The earlier request is taken back (the policy's
withdraw): a running answer is stopped (its speechcancelled, which also stops the brain,voice_worker.spec.md), a waiting one leaves the queue, a pending "addressed" question is void. It is not a missed request. - The joined text (the earlier text, a space, the new final) is judged and answered as one request, as if it had been said in one go; it ends when the new final ended. It may be joined again.
- The new final is joined whatever it is on its own (a piece needs no wake word of its own), except a stop (a stop stops, as always) and an empty final (noise the worker rejected: nothing is joined or asked again, the request goes on as it was).
- When the speaker's next utterance starts (
SpeechStart) withincall.continueMsof the request's end, or had already started when the request's final came (a pause shorter than the transcription), the request is held open for that utterance's final, also if its answer starts playing or even ends meanwhile; the joined request then stops that answer if it still plays. An empty final or a stop ends the hold, and nothing is asked again. - Other speakers' utterances never join it. An answer already heard, an answer that ended, a missed request, a stop, leaving and a lost worker end it; then what the speaker says is a new request.
- The joined request gets no second earcon when the first one's already played.
- While a request is open, its speaker's partials are sent to get ready as the joined text, also while its answer is being written (it is the request that will be answered).
Earcon. A short two-tone "띠링" (about 0.3 s, made by the app, app/src/earcon.rs) tells the
speaker Jarvis heard the call, long before the answer is written: it is the first sound after a call.
It plays right before each voice answer starts (the policy's Answer), and at once when a wake word
said alone is kept waiting while no answer is running (the question that joins it then gets no second
earcon). It is queued like speech, so it waits behind speech already queued and a stop drops it. A
call the policy does not take (busy, replaced, stopped) gets none, nor does a wake word alone while an
answer runs. Chat answers get none. call.earcon: false turns it off.
Response policy
Settings (section response):
| Key | Default | Meaning |
|---|---|---|
response.policy |
"latest" |
first (처음 대화만), inOrder (들어온 순서대로), latest (마지막 대화만), important (제일 중요한 대화만). |
response.queueMax |
5 |
inOrder: most waiting requests; past it the oldest is dropped. 1-20. |
response.collectMs |
700 |
important: how long candidates are collected before choosing. 0-3000. |
response.settleMs |
800 |
Longest wait for an earlier-ended utterance that is still being transcribed. 0-3000. |
response.priorities |
{} |
important: user ID → priority (higher first, default 0). |
response.interrupt |
true |
latest and important: a new request may stop the running answer (below). false: the running answer is always spoken to its end and the new request waits for it (latest: the newest waiting request is answered next and older waiting ones are missed, replaced, also a request whose final arrives after a newer one's; important: the best candidate collected meanwhile is answered next, no ranking against the running answer). A voice stop still stops it. first and inOrder never interrupt. |
Terms:
- A request has an id, the speaker, the text, whether it was
direct, andended_at: when the utterance ended (the final's arrival minus itswait_ms). - An answer runs from the moment the policy starts it until it is done (its speech ended, failed or was stopped). There is at most one answer at a time: Jarvis has one voice.
- An utterance is open from its
speech_startuntil its final. An open utterance whose speaker has stopped speaking (speaking_endsent) is ending: its final may still turn out to be a call.
Rules shared by every policy:
- Ordering by end time. Requests are compared by
ended_at, not by when their final arrived, so a long utterance is not pushed back by its transcription time. - Settling. When a request could start now but some other utterance is ending and stopped earlier
than the request's
ended_at(inimportant: any ending utterance, since it competes on score), the policy waits for that final, at mostsettleMsfrom the first waiting request's arrival. A final that is not a call ends the wait for it.latestdoes not settle: an earlier utterance would lose anyway. - Taking back. A request whose speaker went on (Going on) is withdrawn: stopped when it is the running answer, removed when it waits, dropped from a pending ranking. Not missed; the joined request that follows goes through the policy as any new request.
- Stop. A stop decision stops the current answer and drops every waiting request, in every policy.
The stopped answer and the dropped requests are missed (
stopped). - An answer that is already done is never started again.
Per policy (with response.interrupt on; off, no answer is stopped for a new request):
| Policy | No answer running | A new request while an answer runs |
|---|---|---|
first |
Answer the request that ended first (after settling). | Missed (busy). |
inOrder |
Answer the earliest-ended waiting request. | Queued by ended_at. Past queueMax, the earliest-ended waiting one is missed (queue_full). |
latest |
Answer the latest-ended waiting request; the others are missed (replaced). |
The current answer stops (missed, interrupted) and the new request is answered. A request that ended before the current one is missed (replaced). |
important |
Collect for collectMs from the first request, then answer the best; the other candidates are missed (lower_score). |
Kept as a candidate. If it beats the current answer, the current answer stops (missed, interrupted) and the best candidate is answered at once; otherwise, when the current answer ends, the best of the candidates collected meanwhile is answered and the rest are missed (lower_score). |
important ranks by (1) priority, (2) direct, then (3) the brain's importance ranking, asked for only
when two or more candidates tie on (1) and (2). While the ranking is pending nothing new starts; if it
does not come within 1.5 s (a ranking that arrives later counts as late), or fails, the earliest-ended
of the tied candidates wins. Candidates are kept in ended_at order, whenever they arrived.
Each ranking carries an id (Rank { id, candidates }, answered with ranked(id, order)). Only the
pending ranking's answer is used: an answer to a ranking that was replaced, or dropped (its answer ended,
or a stop), is ignored. A candidate that ties the ranked ones while a ranking is pending replaces it with
a new ranking over all of them (and the running answer, when the ranking was about interrupting it),
keeping the first ranking's deadline, so only candidates that were compared are missed (lower_score).
A candidate that beats the ranked ones makes the ranking moot.
A new request that beats the running answer on (1) or (2) interrupts it at once. One that ties it on
both is ranked against it (the running answer keeps playing meanwhile): if the ranking puts the new
request first, the answer stops (missed, interrupted) and the new request is answered; if it puts the
running answer first, or does not come in time, the answer goes on and the new request waits. The
ranking is dropped when the running answer ends first.
Chat calls
Registered chat channels are answered in chat, by text, beside the voice conversation: a chat message never takes the voice turn and never goes through the response policy.
| Key | Default | Meaning |
|---|---|---|
chatChannels[].answer |
"wakeWord" |
wakeWord: answer messages that call a wake word. everything: answer every message. |
- The wake words are
call.chatWakeWords(orvoice.wakeWordswhen it isnull), with the same similarity and the same designated-people switches as voice;call.modeis voice only. - Only messages of a registered channel with reading on reach the loop (the core's rule), and only a channel with sending on is answered: there an answer could not be posted, so none is asked for.
- Messages from bots (other bots and Jarvis itself) are never answered, so two bots cannot keep answering each other.
- Only a request is answered: a stop in chat ("자비스 멈춰") is not a question and gets no answer.
- The brain answers through
replywithvia: "chat"andspeak: false; the answer is posted in the same channel as a Discord reply to the message (so the history records it as an answer to that person), pinging nobody. A failed or empty answer posts nothing. An answer that arrives 5 minutes or more after it was asked is not posted. The worker is told whether each answer was posted (reply_delivered). - The worker writes voice and chat answers in separate lanes (
voice_worker.spec.md), so a voice answer never waits behind another person's chat answer; one person's answers are written one at a time, in the order they asked. The LLM takes one request at a time (docs/v2/conversation.md, "LLM 동시 요청 수";--parallel 1, also when a channel answers everything), so the two lanes' LLM calls take turns at Strata.
The conversation loop
app/src/conversation.rs runs between the core's events and everything else that reads them (every
event is passed on unchanged). It talks to the core through a small Voice interface (answer, cancel
one request, stop all, earcon, prepare, ask "addressed", ask "rank"), so the glue is tested with a fake.
ChatandReplyText: chat calls, above.Partial(getting ready): a partial of a speaker whose request is open (Going on) is sent as the joined text. Otherwise a partial transcript that call detection already reads as a request (joined to the speaker's waiting wake word alone, as its final would be; not the wake word alone, not a stop) is sent to the brain asprepare(user, text), so the brain judges what the call needs while the speaker is still talking (voice_worker.spec.md,prepare). Not while an answer runs: the call may not be taken, and the brain is busy answering.SpeechStartopens the speaker's utterance;SpeakingStoppedmarks it ending.Final: a request gets the next id,ended_at= arrival minus the final'swait_ms, and goes through call detection. Ignore closes the utterance. Stop stops all speech, runs the policy's stop, voids every pending "addressed" question (closing its utterance) and closes the utterance. Request goes to the policy. Ask sendsaddressedto the brain; a yes becomes the request, a no (or a failed judgement) closes the utterance, and no answer within 5 s counts as no, also when the answer arrives after that but before the timer ran.- Policy actions:
Answer→ the earcon (unless it already played for this request), thenanswer(user, text)(the brain answers and the answer is spoken; when it cannot be queued the answer is done at once);StopAnswer→cancelof that answer's speech;Rank { id }→rankwith idk<id>and the candidates' request ids;Missed→ a debug log and the missed list (the last 100: speaker, reason, time and length, never the text). SpeakStartof an answer's speech: the answer is heard, its request is no longer open (Going on), unless its speaker already started their next utterance (held).SpeakEndof an answer's speech is that answer's done. The end of a speech the loop already cancelled starts nothing.Stopped(all speech stopped from elsewhere: the slash command,stop(), the session ending) andLeft: the same stop. The core sendsStoppedbefore theSpeakEnds of the speech it stopped, so a cancelled answer's end never starts the next waiting request.- A timer wakes the loop at the policy's next deadline or the earliest "addressed" timeout.
The brain's two judgements are src/voice_worker/judges.py (voice_worker.spec.md): one short call
each to the chat model, the heard text fenced as data. Eval: evals/test_voice_judges.py.
Tests
cargo test covers call detection (exact and near wake words in Hangul and Latin, a different word
two letters away, wake word anywhere, the longest call form where several start (the default secretary call forms), stop with and without a wake word, designated combinations,
modes) and every policy rule above with fake times (settling, ordering by end time, busy, queue limit,
interruption and response.interrupt off, ranking tie, ranking timeout, stop dropping the queue), and the conversation loop with a
fake voice side (answering and queueing, a busy drop recorded without text, voice stop, addressed alone
with Jarvis answered without asking, the room counts sent with the question, addressed yes,
no and timeout, the earcon before each answer, once for a wake word alone, none while busy or when
off, partials that read as a call sent to get ready (joined to a waiting wake word, none while
answering someone else, the joined text while going on), going on (pieces joined into one answer in
every policy without a missed record, a piece without a wake word joined, other speakers never, a
new request once the answer is heard or the pause is longer than call.continueMs, a stop still
stopping, no second earcon, an utterance started before the answer played or before the first final
came still joined, also after a short answer ended, in every policy; an empty final or a stop
after a hold asking nothing again), interruption cancelling the right speech, a ranking deciding, end times from
wait_ms, and chat: an answer posted as a reply once, per-channel switches, no answers to bots or to
a stop, chat wake words and designated people, a failed answer posting nothing). tests/worker_protocol.rs checks an answer spoken under its id and both judgements against
the real Python worker with a fake brain.