feat(voice_worker): a short voice prompt and one judgement before each v2 answer

- compact_prompt: the v2 persona in a tenth of v1's words, answers as short as
  possible, the full tool guidance only on turns with a data tool. A greeting's
  reply prompt went from about 4,500 to 1,400 tokens on Strata.
- intent_judge: the tool router judges whether the turn needs tools; the
  planner runs only when it does. prepare_intent runs the pick ahead on a
  partial transcript (worker message prepare), and the reply reuses it.
- Without stream_browser the screen-share tools are not offered.
- The time context carries the 12-hour form (10:37 was read as pm).
- A <tool_call> block written as the reply is parsed as a call, not spoken.

Evals (Qwen3.6 on Strata, EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1):
221 passed, 4 failed (known failures); new evals/test_voice_reply_length.py
(v2 answers 234 chars against v1's 416) and test_reply_language_default.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-12 12:25:15 +09:00
parent 7623d6d002
commit 46e4e75d05
13 changed files with 721 additions and 44 deletions

View File

@@ -4,7 +4,9 @@ interface, so the worker logic is tested with a fake (voice_worker.spec.md, "His
from __future__ import annotations
import dataclasses
import difflib
import sys
import unicodedata
import threading
import time
from dataclasses import dataclass
@@ -28,6 +30,11 @@ class Brain(Protocol):
"""Blocking: candidate ids (``{"id", "user", "text"}``) most important first; None on failure.
Runs on its own thread, also while a reply runs."""
def prepare(self, user: str, text: str, chat: bool) -> None:
"""Blocking: get ready to answer ``text`` (a partial transcript of a call still being said),
so the reply to about the same text starts further along. ``chat``: chat channels are
registered. Runs on its own thread, also while a reply runs. Optional."""
@dataclass
class MemorySettings:
@@ -110,6 +117,32 @@ class Muted:
cls._depth -= 1
# A prepared intent is used for a reply whose text is at least this close to the text it was
# prepared for (letters and digits only): the last partial often lacks a final syllable or two.
PREPARED_SIMILARITY = 0.85
# ... and only this soon after it was prepared, and at most this long is waited for one in flight.
PREPARED_MAX_AGE_S = 30.0
PREPARED_WAIT_S = 5.0
def _letters(text: str) -> str:
return "".join(c for c in unicodedata.normalize("NFKD", text.lower()) if c.isalnum())
def similar(a: str, b: str) -> bool:
x, y = _letters(a), _letters(b)
return bool(x and y) and difflib.SequenceMatcher(None, x, y).ratio() >= PREPARED_SIMILARITY
@dataclass
class _Prepared:
text: str
turns: int # the asker's dialogue memory size it was made with
at: float
done: threading.Event = dataclasses.field(default_factory=threading.Event)
tools: Optional[list] = None # the intent judge's pick; None: none made
class JarvisBrain:
"""The existing jarvis reply engine with the user's history context.
@@ -125,13 +158,16 @@ class JarvisBrain:
# v1's tts_engine names the v1 desktop voice (English-only Piper by default), which makes the
# engine lock replies to English. v2 speaks through the worker's TTS and chat answers are
# not spoken; the brain answers in ``reply_language`` (an OUTPUT_LANGUAGE lock still wins).
self.cfg = dataclasses.replace(load_settings(), tts_engine="melo")
# stream_browser: v1's screen-share browser (controlBrowser, browseAndPlay, setBroadcast and
# browser-first web search) needs v1's Node bot; v2 has no on-screen browser until M6.
self.cfg = dataclasses.replace(load_settings(), tts_engine="melo", stream_browser=False)
self.db = Database(self.cfg.db_path, self.cfg.sqlite_vss_path)
# ``language``: the ISO code tools get (e.g. the Wikipedia host); ``reply_language``: replies
# come in it unless the asker explicitly asks for another language (None: the asker's).
self.language = language
self.reply_language = reply_language
self._memories: dict = {}
self._prepared: dict[str, _Prepared] = {}
self._lock = threading.Lock()
debug_log(f"brain ready: llm={self.cfg.ollama_base_url} model={self.cfg.ollama_chat_model}", "voice_worker")
@@ -145,11 +181,41 @@ class JarvisBrain:
self._memories[user] = mem
return mem
@staticmethod
def _turns(mem) -> int:
return len(mem.get_recent_messages()) if mem.has_recent_messages() else 0
def prepare(self, user: str, text: str, chat: bool) -> None:
from jarvis.reply.engine import prepare_intent
mem = self._memory(user)
p = _Prepared(text=text, turns=self._turns(mem), at=time.monotonic())
with self._lock:
self._prepared[user] = p
t0 = time.perf_counter()
try:
p.tools = Muted.run(prepare_intent, self.cfg, text, mem, chat)
finally:
p.done.set()
debug_log(f"brain prepare user={user} picked={p.tools is not None} "
f"ms={(time.perf_counter() - t0) * 1000:.0f}", "voice_worker")
def _prepared_tools(self, user: str, text: str) -> Optional[list]:
"""The pick prepared for about this text, waiting for one still being made; None if none fits."""
with self._lock:
p = self._prepared.pop(user, None)
if p is None or not similar(p.text, text) or time.monotonic() - p.at > PREPARED_MAX_AGE_S:
return None
if not p.done.wait(PREPARED_WAIT_S) or p.turns != self._turns(self._memory(user)):
return None
return p.tools
def reply(self, user: str, text: str, history: HistoryContext, chat=None) -> Optional[str]:
from jarvis.reply import turn_state
from jarvis.reply.engine import run_reply_engine
t0 = time.perf_counter()
prepared = self._prepared_tools(user, text)
# legacy_memory=False: v1's profile, diary and graph belong to one user; several people
# share this assistant, so only the asker's scoped history may reach the prompt.
def turn():
@@ -160,13 +226,14 @@ class JarvisBrain:
try:
return run_reply_engine(self.db, self.cfg, None, text, self._memory(user),
language=self.language, history=history, legacy_memory=False,
default_language=self.reply_language)
default_language=self.reply_language, compact_prompt=True,
intent_judge=True, prepared_tools=prepared)
finally:
turn_state.reset()
answer = Muted.run(turn)
debug_log(f"brain reply user={user} ok={bool(answer)} ms={(time.perf_counter() - t0) * 1000:.0f}",
"voice_worker")
debug_log(f"brain reply user={user} ok={bool(answer)} prepared={prepared is not None} "
f"ms={(time.perf_counter() - t0) * 1000:.0f}", "voice_worker")
return answer
def addressed(self, user: str, text: str, names: list[str]) -> Optional[bool]:

View File

@@ -176,6 +176,9 @@ class Session:
# Spoken replies whose speech was cancelled while the brain was still answering.
self.cancelled_replies: set[str] = set()
self.thinking_replies: set[str] = set()
# user -> the newest call text waiting for that user's running prepare to finish (or None).
self.preparing: dict[str, str | None] = {}
self.last_prepared: dict[str, str] = {} # user -> the text of their last prepare
# Answers sent but not yet known to have reached the person: id -> (user, turn, timer).
self.deliveries: dict[str, tuple[str, int, asyncio.TimerHandle | None]] = {}
self._handed_over: dict[str, bool] = {} # replies whose end is counted by _delivered
@@ -316,7 +319,8 @@ class Session:
"chat_delete": self._on_chat_delete, "reply": self._on_reply,
"addressed": self._on_addressed, "rank": self._on_rank,
"chat_channels": self._on_chat_channels, "chat_action_result": self._on_chat_action_result,
"memory_config": self._on_memory_config, "reply_delivered": self._on_reply_delivered}.get(kind)
"memory_config": self._on_memory_config, "reply_delivered": self._on_reply_delivered,
"prepare": self._on_prepare}.get(kind)
if handler is None:
self.error("bad_message", f"unknown type: {kind!r}")
return
@@ -507,11 +511,47 @@ class Session:
return
if speak:
self.thinking_replies.add(rid)
if user in self.preparing:
self.preparing[user] = None # the call was answered: no newer text to prepare
self.last_prepared.pop(user, None)
waiting = self.worker.replies_unanswered
early = waiting.get(user, 0) > 0 # sent before their previous answer: not a reply to it
waiting[user] = waiting.get(user, 0) + 1
asyncio.get_running_loop().create_task(self._reply(rid, user, text, via, speak, early))
def _on_prepare(self, msg: dict) -> None:
"""A call still being said: the brain gets ready for it. One prepare per user runs at a time;
while it runs only the newest text waits, and one about the same as the last one prepared
(or being prepared) is dropped. Nothing is sent back."""
from .brain import similar
got = self._strings(msg, "user", "text")
if got is None:
return
user, text = got
if self.worker.brain is None or not hasattr(self.worker.brain, "prepare"):
return
if user in self.last_prepared and similar(self.last_prepared[user], text):
return
if user in self.preparing:
self.preparing[user] = text
return
self.preparing[user] = None
asyncio.get_running_loop().create_task(self._prepare(user, text))
async def _prepare(self, user: str, text: str | None) -> None:
from .brain import similar
try:
while text is not None and not self.closed:
self.last_prepared[user] = text
try:
await asyncio.to_thread(self.worker.brain.prepare, user, text, bool(self.chat_channels))
except Exception as e: # the brain must never take the session down
debug_log(f"brain prepare failed: {type(e).__name__}", "voice_worker")
newer, self.preparing[user] = self.preparing[user], None
text = newer if newer is not None and not similar(newer, self.last_prepared.get(user, "")) else None
finally:
self.preparing.pop(user, None)
def _on_addressed(self, msg: dict) -> None:
got = self._strings(msg, "id", "user", "text")
if got is None:

View File

@@ -201,6 +201,7 @@ which carry Discord's own time.
-> {"type": "reply", "id": "r1", "user": "<id>", "text": "...", "via": "voice", "speak": true}
<- {"type": "reply_text", "id": "r1", "text": "...", "failed": false}
-> {"type": "reply_delivered", "id": "c1", "ok": true} // a chat answer was posted (ok: false: it was not)
-> {"type": "prepare", "user": "<id>", "text": "..."} // a call still being said; nothing comes back
-> {"type": "addressed", "id": "q1", "user": "<id>", "text": "..."}
<- {"type": "addressed_result", "id": "q1", "addressed": true, "failed": false}
-> {"type": "rank", "id": "k1", "candidates": [{"id": "7", "user": "<id>", "text": "..."}]}
@@ -241,6 +242,23 @@ which carry Discord's own time.
default) does not lock replies to English, because v2 speaks through the worker's TTS. An
`output_language` lock in the jarvis settings still wins. Eval: `evals/test_reply_language_default.py`.
STT (`language` `"ko"`) and TTS (`--melo-language KR`) are Korean by default too.
Two reply options keep the time to the first word short (Strata reads every prompt from the
start, about 0.36 s per 2,048 tokens, and each LLM call before the answer adds its whole time):
the short voice persona (`compact_prompt=True`: the same butler in a tenth of v1's words, without
v1's single-user memory and on-screen browser rules, answers as short as possible, the full tool
guidance only on turns that offer a data tool), and the intent judge (`intent_judge=True`: the tool
router judges whether the turn needs tools and which, and the planner runs only when it does;
`planner.spec.md`, "Intent judge (v2)"). v1's screen-share browser (`stream_browser`) is off: it
needs v1's Node bot (M6 brings the v2 one), so its tools are not offered.
- `prepare` tells the brain that `user` is saying a call whose partial transcript is `text` (the app
sends it while the speaker is still talking, `app/src/conversation.spec.md`). The brain runs the
intent judge's tool pick for it on its own thread, with the asker's dialogue memory and the tool
catalogue the reply will have. A `reply` from that user whose text is about the same (letters and
digits, 85% alike), within 30 s and with the dialogue memory unchanged, uses that pick instead of
making its own, waiting up to 5 s for one still being made. Each prepared judgement is used at most once.
One `prepare` per user runs at a time: while it runs only the newest text waits, a text about the
same as the one last prepared is dropped, and a `reply` from that user drops the waiting text.
Nothing is sent back; without a brain, or with one that cannot prepare, it is ignored.
The asker is set in `jarvis.reply.turn_state` for the turn, so tools that keep personal data
(meals) store and read it under that person only.
- `addressed` and `rank` are the conversation's questions (`app/src/conversation.spec.md`), answered