# Conversation spec: call detection and response policies (v2 M4) The part of the app that decides which finished utterances (and, later, chat messages) Jarvis answers and in what order, and when an answer that is being spoken stops. It sits between the Discord voice core (`discord_core.spec.md`), which reports what was heard and plays speech, and the voice worker's brain (`src/voice_worker/voice_worker.spec.md`, `reply`), which writes the answer. The user-facing design is `docs/v2/conversation.md`; this spec is the contract the code follows. ## Principles - **Plain types, tested without Discord.** Call detection (`app/src/calls.rs`) is a pure function of the settings, the speaker and the text. The response policy (`app/src/policy.rs`) is a state machine driven by events with the time passed in, so every rule is tested with fake times. - **Language-agnostic.** Nothing matches a language's words in code. Wake words and stop words are settings; matching works on Unicode-normalised letters, so it behaves the same in any script. - **No text in logs.** Logs carry speaker IDs, decisions, reasons, counts and timings, never what was said. - **Nothing is dropped silently.** Every request the policy does not answer, and every answer it stops, is reported as missed with a reason (the dashboard's "놓친 요청" list, M5). ## Call detection Settings (`settings.json`, section `call`; wake words stay in `voice.wakeWords`): | Key | Default | Meaning | |---|---|---| | `voice.wakeWords` | `["자비스"]` | Wake words for voice (also the STT hint). | | `call.chatWakeWords` | `null` | Wake words for chat; `null` uses `voice.wakeWords`. | | `call.similarity` | `0.2` | How far a heard word may be from a wake word, as a share of the wake word's normalised letters (0-0.5). `0` needs an exact match. | | `call.mode` | `"wakeWord"` | `wakeWord`: answer only when called by a wake word. `addressed`: also answer what is judged to be said to Jarvis. `everything`: answer every utterance. | | `call.designated` | `[]` | Discord user IDs of the designated people. | | `call.designatedOnly` | `false` | Answer only designated people. | | `call.designatedNoWakeWord` | `false` | Answer designated people without a wake word. | | `call.stopWords` | `["멈춰", "그만", "정지", "stop"]` | Said together with a wake word, they stop Jarvis in any policy. | | `call.earcon` | `true` | Play the earcon (below) when Jarvis takes a voice call. | | `call.followUpMs` | `1500` | After a wake word said alone, how long the speaker's next utterance may take to start and still be joined to it. 0-5000. | Normalisation: Unicode NFKD, lower case, and only letters and digits kept (spaces, punctuation and combining marks dropped). NFKD splits Hangul syllables into their letters, so "자비즈" is one letter away from "자비스" (of six) while "서비스" is two away. A wake word is heard when some stretch of the normalised text is at most `floor(similarity × letters of the wake word)` edits (insert, delete, substitute) from it. With the default 0.2, "자비스" (six letters) allows one edit. A wake word anywhere in the utterance counts. Decision for one utterance (`judge(speaker, text)`), in order: 1. Empty text (after normalisation): ignore. 2. `designatedOnly` and the speaker is not designated: ignore. 3. A wake word is heard and the rest of the text, with that stretch removed, starts with a stop word and has at most 3 more letters ("멈춰", "멈춰 줘", "그만해"): **stop**. 4. A wake word is heard: **request**, `direct` (called by name). 5. `designatedNoWakeWord` and the speaker is designated: **request**, not direct. 6. `call.mode` `everything`: **request**, not direct. `addressed`: when the speaker is the only person in Jarvis's voice channel (the people it hears, not Jarvis or ignored bots; the speaker always counts, also while the channel list lags), **request**, not direct: alone with Jarvis, whatever they say is said to it. Otherwise **ask**: the brain judges whether it was said to Jarvis, told how many people are in the channel and how many of them may call Jarvis (all, or with `designatedOnly` the designated ones present), since with one other person most of what the speaker says is for Jarvis and with more most talk is between people (`voice_worker.spec.md`, `addressed`); a yes becomes a request, not direct. `wakeWord`: ignore. The text sent to the brain is the utterance as heard; the brain treats the wake word as address. **A wake word said alone.** People often pause after the name ("자비스, ... 지금 몇 시야?"), and a pause longer than the worker's `end_silence_ms` ends the utterance there. A voice utterance that is a wake word and nothing else (no letters or digits left once the wake word is removed) is therefore not a request yet: the conversation keeps it for that speaker and joins it to their next final, with a space between, and judges the joined text as one utterance (so "자비스" then "멈춰" is a stop). If the speaker's next utterance already started when the wake word's final came, or starts within `call.followUpMs` of it, the wake word waits for that utterance's final however long it takes; otherwise, after `call.followUpMs`, the wake word alone is the request. Other speakers' utterances never join it, and a stop (voice, slash command or leaving) drops it. So does a lost worker connection (`WorkerLost`): the final it waited for will not come, and the restarted worker's utterances are not continuations of it. Chat messages are not joined: a message is complete as sent. **Earcon.** A short two-tone "띠링" (about 0.3 s, made by the app, `app/src/earcon.rs`) tells the speaker Jarvis heard the call, long before the answer is written: it is the first sound after a call. It plays right before each voice answer starts (the policy's `Answer`), and at once when a wake word said alone is kept waiting while no answer is running (the question that joins it then gets no second earcon). It is queued like speech, so it waits behind speech already queued and a stop drops it. A call the policy does not take (busy, replaced, stopped) gets none, nor does a wake word alone while an answer runs. Chat answers get none. `call.earcon: false` turns it off. ## Response policy Settings (section `response`): | Key | Default | Meaning | |---|---|---| | `response.policy` | `"latest"` | `first` (처음 대화만), `inOrder` (들어온 순서대로), `latest` (마지막 대화만), `important` (제일 중요한 대화만). | | `response.queueMax` | `5` | `inOrder`: most waiting requests; past it the oldest is dropped. 1-20. | | `response.collectMs` | `700` | `important`: how long candidates are collected before choosing. 0-3000. | | `response.settleMs` | `800` | Longest wait for an earlier-ended utterance that is still being transcribed. 0-3000. | | `response.priorities` | `{}` | `important`: user ID → priority (higher first, default 0). | | `response.interrupt` | `true` | `latest` and `important`: a new request may stop the running answer (below). `false`: the running answer is always spoken to its end and the new request waits for it (`latest`: the newest waiting request is answered next and older waiting ones are missed, `replaced`, also a request whose final arrives after a newer one's; `important`: the best candidate collected meanwhile is answered next, no ranking against the running answer). A voice stop still stops it. `first` and `inOrder` never interrupt. | Terms: - A **request** has an id, the speaker, the text, whether it was `direct`, and `ended_at`: when the utterance ended (the final's arrival minus its `wait_ms`). - An **answer** runs from the moment the policy starts it until it is done (its speech ended, failed or was stopped). There is at most one answer at a time: Jarvis has one voice. - An utterance is **open** from its `speech_start` until its final. An open utterance whose speaker has stopped speaking (`speaking_end` sent) is **ending**: its final may still turn out to be a call. Rules shared by every policy: - **Ordering by end time.** Requests are compared by `ended_at`, not by when their final arrived, so a long utterance is not pushed back by its transcription time. - **Settling.** When a request could start now but some other utterance is ending and stopped earlier than the request's `ended_at` (in `important`: any ending utterance, since it competes on score), the policy waits for that final, at most `settleMs` from the first waiting request's arrival. A final that is not a call ends the wait for it. `latest` does not settle: an earlier utterance would lose anyway. - **Stop.** A stop decision stops the current answer and drops every waiting request, in every policy. The stopped answer and the dropped requests are missed (`stopped`). - An answer that is already done is never started again. Per policy (with `response.interrupt` on; off, no answer is stopped for a new request): | Policy | No answer running | A new request while an answer runs | |---|---|---| | `first` | Answer the request that ended first (after settling). | Missed (`busy`). | | `inOrder` | Answer the earliest-ended waiting request. | Queued by `ended_at`. Past `queueMax`, the earliest-ended waiting one is missed (`queue_full`). | | `latest` | Answer the latest-ended waiting request; the others are missed (`replaced`). | The current answer stops (missed, `interrupted`) and the new request is answered. A request that ended before the current one is missed (`replaced`). | | `important` | Collect for `collectMs` from the first request, then answer the best; the other candidates are missed (`lower_score`). | Kept as a candidate. If it beats the current answer, the current answer stops (missed, `interrupted`) and the best candidate is answered at once; otherwise, when the current answer ends, the best of the candidates collected meanwhile is answered and the rest are missed (`lower_score`). | `important` ranks by (1) priority, (2) `direct`, then (3) the brain's importance ranking, asked for only when two or more candidates tie on (1) and (2). While the ranking is pending nothing new starts; if it does not come within 1.5 s (a ranking that arrives later counts as late), or fails, the earliest-ended of the tied candidates wins. Candidates are kept in `ended_at` order, whenever they arrived. Each ranking carries an id (`Rank { id, candidates }`, answered with `ranked(id, order)`). Only the pending ranking's answer is used: an answer to a ranking that was replaced, or dropped (its answer ended, or a stop), is ignored. A candidate that ties the ranked ones while a ranking is pending replaces it with a new ranking over all of them (and the running answer, when the ranking was about interrupting it), keeping the first ranking's deadline, so only candidates that were compared are missed (`lower_score`). A candidate that beats the ranked ones makes the ranking moot. A new request that beats the running answer on (1) or (2) interrupts it at once. One that ties it on both is ranked against it (the running answer keeps playing meanwhile): if the ranking puts the new request first, the answer stops (missed, `interrupted`) and the new request is answered; if it puts the running answer first, or does not come in time, the answer goes on and the new request waits. The ranking is dropped when the running answer ends first. ## Chat calls Registered chat channels are answered in chat, by text, beside the voice conversation: a chat message never takes the voice turn and never goes through the response policy. | Key | Default | Meaning | |---|---|---| | `chatChannels[].answer` | `"wakeWord"` | `wakeWord`: answer messages that call a wake word. `everything`: answer every message. | - The wake words are `call.chatWakeWords` (or `voice.wakeWords` when it is `null`), with the same similarity and the same designated-people switches as voice; `call.mode` is voice only. - Only messages of a registered channel with reading on reach the loop (the core's rule), and only a channel with sending on is answered: there an answer could not be posted, so none is asked for. - Messages from bots (other bots and Jarvis itself) are never answered, so two bots cannot keep answering each other. - Only a request is answered: a stop in chat ("자비스 멈춰") is not a question and gets no answer. - The brain answers through `reply` with `via: "chat"` and `speak: false`; the answer is posted in the same channel as a Discord reply to the message (so the history records it as an answer to that person), pinging nobody. A failed or empty answer posts nothing. An answer that arrives 5 minutes or more after it was asked is not posted. The worker is told whether each answer was posted (`reply_delivered`). - The worker writes voice and chat answers in separate lanes (`voice_worker.spec.md`), so a voice answer never waits behind another person's chat answer; one person's answers are written one at a time, in the order they asked. The LLM takes one request at a time (`docs/v2/conversation.md`, "LLM 동시 요청 수"; `--parallel 1`, also when a channel answers everything), so the two lanes' LLM calls take turns at Strata. ## The conversation loop `app/src/conversation.rs` runs between the core's events and everything else that reads them (every event is passed on unchanged). It talks to the core through a small `Voice` interface (answer, cancel one request, stop all, earcon, prepare, ask "addressed", ask "rank"), so the glue is tested with a fake. - `Chat` and `ReplyText`: chat calls, above. - `Partial` (getting ready): a partial transcript that call detection already reads as a request (joined to the speaker's waiting wake word alone, as its final would be; not the wake word alone, not a stop) is sent to the brain as `prepare(user, text)`, so the brain judges what the call needs while the speaker is still talking (`voice_worker.spec.md`, `prepare`). Not while an answer runs: the call may not be taken, and the brain is busy answering. - `SpeechStart` opens the speaker's utterance; `SpeakingStopped` marks it ending. - `Final`: a request gets the next id, `ended_at` = arrival minus the final's `wait_ms`, and goes through call detection. Ignore closes the utterance. Stop stops all speech, runs the policy's stop, voids every pending "addressed" question (closing its utterance) and closes the utterance. Request goes to the policy. Ask sends `addressed` to the brain; a yes becomes the request, a no (or a failed judgement) closes the utterance, and no answer within 5 s counts as no, also when the answer arrives after that but before the timer ran. - Policy actions: `Answer` → the earcon (unless it already played for this request), then `answer(user, text)` (the brain answers and the answer is spoken; when it cannot be queued the answer is done at once); `StopAnswer` → `cancel` of that answer's speech; `Rank { id }` → `rank` with id `k` and the candidates' request ids; `Missed` → a debug log and the missed list (the last 100: speaker, reason, time and length, never the text). - `SpeakEnd` of an answer's speech is that answer's done. The end of a speech the loop already cancelled starts nothing. - `Stopped` (all speech stopped from elsewhere: the slash command, `stop()`, the session ending) and `Left`: the same stop. The core sends `Stopped` before the `SpeakEnd`s of the speech it stopped, so a cancelled answer's end never starts the next waiting request. - A timer wakes the loop at the policy's next deadline or the earliest "addressed" timeout. The brain's two judgements are `src/voice_worker/judges.py` (`voice_worker.spec.md`): one short call each to the chat model, the heard text fenced as data. Eval: `evals/test_voice_judges.py`. ## Tests `cargo test` covers call detection (exact and near wake words in Hangul and Latin, a different word two letters away, wake word anywhere, stop with and without a wake word, designated combinations, modes) and every policy rule above with fake times (settling, ordering by end time, busy, queue limit, interruption and `response.interrupt` off, ranking tie, ranking timeout, stop dropping the queue), and the conversation loop with a fake voice side (answering and queueing, a busy drop recorded without text, voice stop, addressed alone with Jarvis answered without asking, the room counts sent with the question, addressed yes, no and timeout, the earcon before each answer, once for a wake word alone, none while busy or when off, partials that read as a call sent to get ready (joined to a waiting wake word, none while answering), interruption cancelling the right speech, a ranking deciding, end times from `wait_ms`, and chat: an answer posted as a reply once, per-channel switches, no answers to bots or to a stop, chat wake words and designated people, a failed answer posting nothing). `tests/worker_protocol.rs` checks an answer spoken under its id and both judgements against the real Python worker with a fake brain.