feat(voice_worker): a short voice prompt and one judgement before each v2 answer
- compact_prompt: the v2 persona in a tenth of v1's words, answers as short as possible, the full tool guidance only on turns with a data tool. A greeting's reply prompt went from about 4,500 to 1,400 tokens on Strata. - intent_judge: the tool router judges whether the turn needs tools; the planner runs only when it does. prepare_intent runs the pick ahead on a partial transcript (worker message prepare), and the reply reuses it. - Without stream_browser the screen-share tools are not offered. - The time context carries the 12-hour form (10:37 was read as pm). - A <tool_call> block written as the reply is parsed as a call, not spoken. Evals (Qwen3.6 on Strata, EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1): 221 passed, 4 failed (known failures); new evals/test_voice_reply_length.py (v2 answers 234 chars against v1's 416) and test_reply_language_default.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -4,7 +4,9 @@ interface, so the worker logic is tested with a fake (voice_worker.spec.md, "His
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import difflib
|
||||
import sys
|
||||
import unicodedata
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
@@ -28,6 +30,11 @@ class Brain(Protocol):
|
||||
"""Blocking: candidate ids (``{"id", "user", "text"}``) most important first; None on failure.
|
||||
Runs on its own thread, also while a reply runs."""
|
||||
|
||||
def prepare(self, user: str, text: str, chat: bool) -> None:
|
||||
"""Blocking: get ready to answer ``text`` (a partial transcript of a call still being said),
|
||||
so the reply to about the same text starts further along. ``chat``: chat channels are
|
||||
registered. Runs on its own thread, also while a reply runs. Optional."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class MemorySettings:
|
||||
@@ -110,6 +117,32 @@ class Muted:
|
||||
cls._depth -= 1
|
||||
|
||||
|
||||
# A prepared intent is used for a reply whose text is at least this close to the text it was
|
||||
# prepared for (letters and digits only): the last partial often lacks a final syllable or two.
|
||||
PREPARED_SIMILARITY = 0.85
|
||||
# ... and only this soon after it was prepared, and at most this long is waited for one in flight.
|
||||
PREPARED_MAX_AGE_S = 30.0
|
||||
PREPARED_WAIT_S = 5.0
|
||||
|
||||
|
||||
def _letters(text: str) -> str:
|
||||
return "".join(c for c in unicodedata.normalize("NFKD", text.lower()) if c.isalnum())
|
||||
|
||||
|
||||
def similar(a: str, b: str) -> bool:
|
||||
x, y = _letters(a), _letters(b)
|
||||
return bool(x and y) and difflib.SequenceMatcher(None, x, y).ratio() >= PREPARED_SIMILARITY
|
||||
|
||||
|
||||
@dataclass
|
||||
class _Prepared:
|
||||
text: str
|
||||
turns: int # the asker's dialogue memory size it was made with
|
||||
at: float
|
||||
done: threading.Event = dataclasses.field(default_factory=threading.Event)
|
||||
tools: Optional[list] = None # the intent judge's pick; None: none made
|
||||
|
||||
|
||||
class JarvisBrain:
|
||||
"""The existing jarvis reply engine with the user's history context.
|
||||
|
||||
@@ -125,13 +158,16 @@ class JarvisBrain:
|
||||
# v1's tts_engine names the v1 desktop voice (English-only Piper by default), which makes the
|
||||
# engine lock replies to English. v2 speaks through the worker's TTS and chat answers are
|
||||
# not spoken; the brain answers in ``reply_language`` (an OUTPUT_LANGUAGE lock still wins).
|
||||
self.cfg = dataclasses.replace(load_settings(), tts_engine="melo")
|
||||
# stream_browser: v1's screen-share browser (controlBrowser, browseAndPlay, setBroadcast and
|
||||
# browser-first web search) needs v1's Node bot; v2 has no on-screen browser until M6.
|
||||
self.cfg = dataclasses.replace(load_settings(), tts_engine="melo", stream_browser=False)
|
||||
self.db = Database(self.cfg.db_path, self.cfg.sqlite_vss_path)
|
||||
# ``language``: the ISO code tools get (e.g. the Wikipedia host); ``reply_language``: replies
|
||||
# come in it unless the asker explicitly asks for another language (None: the asker's).
|
||||
self.language = language
|
||||
self.reply_language = reply_language
|
||||
self._memories: dict = {}
|
||||
self._prepared: dict[str, _Prepared] = {}
|
||||
self._lock = threading.Lock()
|
||||
debug_log(f"brain ready: llm={self.cfg.ollama_base_url} model={self.cfg.ollama_chat_model}", "voice_worker")
|
||||
|
||||
@@ -145,11 +181,41 @@ class JarvisBrain:
|
||||
self._memories[user] = mem
|
||||
return mem
|
||||
|
||||
@staticmethod
|
||||
def _turns(mem) -> int:
|
||||
return len(mem.get_recent_messages()) if mem.has_recent_messages() else 0
|
||||
|
||||
def prepare(self, user: str, text: str, chat: bool) -> None:
|
||||
from jarvis.reply.engine import prepare_intent
|
||||
|
||||
mem = self._memory(user)
|
||||
p = _Prepared(text=text, turns=self._turns(mem), at=time.monotonic())
|
||||
with self._lock:
|
||||
self._prepared[user] = p
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
p.tools = Muted.run(prepare_intent, self.cfg, text, mem, chat)
|
||||
finally:
|
||||
p.done.set()
|
||||
debug_log(f"brain prepare user={user} picked={p.tools is not None} "
|
||||
f"ms={(time.perf_counter() - t0) * 1000:.0f}", "voice_worker")
|
||||
|
||||
def _prepared_tools(self, user: str, text: str) -> Optional[list]:
|
||||
"""The pick prepared for about this text, waiting for one still being made; None if none fits."""
|
||||
with self._lock:
|
||||
p = self._prepared.pop(user, None)
|
||||
if p is None or not similar(p.text, text) or time.monotonic() - p.at > PREPARED_MAX_AGE_S:
|
||||
return None
|
||||
if not p.done.wait(PREPARED_WAIT_S) or p.turns != self._turns(self._memory(user)):
|
||||
return None
|
||||
return p.tools
|
||||
|
||||
def reply(self, user: str, text: str, history: HistoryContext, chat=None) -> Optional[str]:
|
||||
from jarvis.reply import turn_state
|
||||
from jarvis.reply.engine import run_reply_engine
|
||||
|
||||
t0 = time.perf_counter()
|
||||
prepared = self._prepared_tools(user, text)
|
||||
# legacy_memory=False: v1's profile, diary and graph belong to one user; several people
|
||||
# share this assistant, so only the asker's scoped history may reach the prompt.
|
||||
def turn():
|
||||
@@ -160,13 +226,14 @@ class JarvisBrain:
|
||||
try:
|
||||
return run_reply_engine(self.db, self.cfg, None, text, self._memory(user),
|
||||
language=self.language, history=history, legacy_memory=False,
|
||||
default_language=self.reply_language)
|
||||
default_language=self.reply_language, compact_prompt=True,
|
||||
intent_judge=True, prepared_tools=prepared)
|
||||
finally:
|
||||
turn_state.reset()
|
||||
|
||||
answer = Muted.run(turn)
|
||||
debug_log(f"brain reply user={user} ok={bool(answer)} ms={(time.perf_counter() - t0) * 1000:.0f}",
|
||||
"voice_worker")
|
||||
debug_log(f"brain reply user={user} ok={bool(answer)} prepared={prepared is not None} "
|
||||
f"ms={(time.perf_counter() - t0) * 1000:.0f}", "voice_worker")
|
||||
return answer
|
||||
|
||||
def addressed(self, user: str, text: str, names: list[str]) -> Optional[bool]:
|
||||
|
||||
@@ -176,6 +176,9 @@ class Session:
|
||||
# Spoken replies whose speech was cancelled while the brain was still answering.
|
||||
self.cancelled_replies: set[str] = set()
|
||||
self.thinking_replies: set[str] = set()
|
||||
# user -> the newest call text waiting for that user's running prepare to finish (or None).
|
||||
self.preparing: dict[str, str | None] = {}
|
||||
self.last_prepared: dict[str, str] = {} # user -> the text of their last prepare
|
||||
# Answers sent but not yet known to have reached the person: id -> (user, turn, timer).
|
||||
self.deliveries: dict[str, tuple[str, int, asyncio.TimerHandle | None]] = {}
|
||||
self._handed_over: dict[str, bool] = {} # replies whose end is counted by _delivered
|
||||
@@ -316,7 +319,8 @@ class Session:
|
||||
"chat_delete": self._on_chat_delete, "reply": self._on_reply,
|
||||
"addressed": self._on_addressed, "rank": self._on_rank,
|
||||
"chat_channels": self._on_chat_channels, "chat_action_result": self._on_chat_action_result,
|
||||
"memory_config": self._on_memory_config, "reply_delivered": self._on_reply_delivered}.get(kind)
|
||||
"memory_config": self._on_memory_config, "reply_delivered": self._on_reply_delivered,
|
||||
"prepare": self._on_prepare}.get(kind)
|
||||
if handler is None:
|
||||
self.error("bad_message", f"unknown type: {kind!r}")
|
||||
return
|
||||
@@ -507,11 +511,47 @@ class Session:
|
||||
return
|
||||
if speak:
|
||||
self.thinking_replies.add(rid)
|
||||
if user in self.preparing:
|
||||
self.preparing[user] = None # the call was answered: no newer text to prepare
|
||||
self.last_prepared.pop(user, None)
|
||||
waiting = self.worker.replies_unanswered
|
||||
early = waiting.get(user, 0) > 0 # sent before their previous answer: not a reply to it
|
||||
waiting[user] = waiting.get(user, 0) + 1
|
||||
asyncio.get_running_loop().create_task(self._reply(rid, user, text, via, speak, early))
|
||||
|
||||
def _on_prepare(self, msg: dict) -> None:
|
||||
"""A call still being said: the brain gets ready for it. One prepare per user runs at a time;
|
||||
while it runs only the newest text waits, and one about the same as the last one prepared
|
||||
(or being prepared) is dropped. Nothing is sent back."""
|
||||
from .brain import similar
|
||||
got = self._strings(msg, "user", "text")
|
||||
if got is None:
|
||||
return
|
||||
user, text = got
|
||||
if self.worker.brain is None or not hasattr(self.worker.brain, "prepare"):
|
||||
return
|
||||
if user in self.last_prepared and similar(self.last_prepared[user], text):
|
||||
return
|
||||
if user in self.preparing:
|
||||
self.preparing[user] = text
|
||||
return
|
||||
self.preparing[user] = None
|
||||
asyncio.get_running_loop().create_task(self._prepare(user, text))
|
||||
|
||||
async def _prepare(self, user: str, text: str | None) -> None:
|
||||
from .brain import similar
|
||||
try:
|
||||
while text is not None and not self.closed:
|
||||
self.last_prepared[user] = text
|
||||
try:
|
||||
await asyncio.to_thread(self.worker.brain.prepare, user, text, bool(self.chat_channels))
|
||||
except Exception as e: # the brain must never take the session down
|
||||
debug_log(f"brain prepare failed: {type(e).__name__}", "voice_worker")
|
||||
newer, self.preparing[user] = self.preparing[user], None
|
||||
text = newer if newer is not None and not similar(newer, self.last_prepared.get(user, "")) else None
|
||||
finally:
|
||||
self.preparing.pop(user, None)
|
||||
|
||||
def _on_addressed(self, msg: dict) -> None:
|
||||
got = self._strings(msg, "id", "user", "text")
|
||||
if got is None:
|
||||
|
||||
@@ -201,6 +201,7 @@ which carry Discord's own time.
|
||||
-> {"type": "reply", "id": "r1", "user": "<id>", "text": "...", "via": "voice", "speak": true}
|
||||
<- {"type": "reply_text", "id": "r1", "text": "...", "failed": false}
|
||||
-> {"type": "reply_delivered", "id": "c1", "ok": true} // a chat answer was posted (ok: false: it was not)
|
||||
-> {"type": "prepare", "user": "<id>", "text": "..."} // a call still being said; nothing comes back
|
||||
-> {"type": "addressed", "id": "q1", "user": "<id>", "text": "..."}
|
||||
<- {"type": "addressed_result", "id": "q1", "addressed": true, "failed": false}
|
||||
-> {"type": "rank", "id": "k1", "candidates": [{"id": "7", "user": "<id>", "text": "..."}]}
|
||||
@@ -241,6 +242,23 @@ which carry Discord's own time.
|
||||
default) does not lock replies to English, because v2 speaks through the worker's TTS. An
|
||||
`output_language` lock in the jarvis settings still wins. Eval: `evals/test_reply_language_default.py`.
|
||||
STT (`language` `"ko"`) and TTS (`--melo-language KR`) are Korean by default too.
|
||||
Two reply options keep the time to the first word short (Strata reads every prompt from the
|
||||
start, about 0.36 s per 2,048 tokens, and each LLM call before the answer adds its whole time):
|
||||
the short voice persona (`compact_prompt=True`: the same butler in a tenth of v1's words, without
|
||||
v1's single-user memory and on-screen browser rules, answers as short as possible, the full tool
|
||||
guidance only on turns that offer a data tool), and the intent judge (`intent_judge=True`: the tool
|
||||
router judges whether the turn needs tools and which, and the planner runs only when it does;
|
||||
`planner.spec.md`, "Intent judge (v2)"). v1's screen-share browser (`stream_browser`) is off: it
|
||||
needs v1's Node bot (M6 brings the v2 one), so its tools are not offered.
|
||||
- `prepare` tells the brain that `user` is saying a call whose partial transcript is `text` (the app
|
||||
sends it while the speaker is still talking, `app/src/conversation.spec.md`). The brain runs the
|
||||
intent judge's tool pick for it on its own thread, with the asker's dialogue memory and the tool
|
||||
catalogue the reply will have. A `reply` from that user whose text is about the same (letters and
|
||||
digits, 85% alike), within 30 s and with the dialogue memory unchanged, uses that pick instead of
|
||||
making its own, waiting up to 5 s for one still being made. Each prepared judgement is used at most once.
|
||||
One `prepare` per user runs at a time: while it runs only the newest text waits, a text about the
|
||||
same as the one last prepared is dropped, and a `reply` from that user drops the waiting text.
|
||||
Nothing is sent back; without a brain, or with one that cannot prepare, it is ignored.
|
||||
The asker is set in `jarvis.reply.turn_state` for the turn, so tools that keep personal data
|
||||
(meals) store and read it under that person only.
|
||||
- `addressed` and `rank` are the conversation's questions (`app/src/conversation.spec.md`), answered
|
||||
|
||||
Reference in New Issue
Block a user