feat(voice_worker): a short voice prompt and one judgement before each v2 answer
- compact_prompt: the v2 persona in a tenth of v1's words, answers as short as possible, the full tool guidance only on turns with a data tool. A greeting's reply prompt went from about 4,500 to 1,400 tokens on Strata. - intent_judge: the tool router judges whether the turn needs tools; the planner runs only when it does. prepare_intent runs the pick ahead on a partial transcript (worker message prepare), and the reply reuses it. - Without stream_browser the screen-share tools are not offered. - The time context carries the 12-hour form (10:37 was read as pm). - A <tool_call> block written as the reply is parsed as a call, not spoken. Evals (Qwen3.6 on Strata, EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1): 221 passed, 4 failed (known failures); new evals/test_voice_reply_length.py (v2 answers 234 chars against v1's 416) and test_reply_language_default.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -15,6 +15,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
|
||||
- Redacted user query
|
||||
- Recent dialogue (last 5 minutes), including in-loop tool-call + tool-role messages from prior replies within the active conversation (tool carryover, `DialogueMemory.record_tool_turn` / `get_recent_turns_with_tools` in [src/jarvis/memory/conversation.py](src/jarvis/memory/conversation.py); per-prompt cap via `cfg.tool_carryover_max_turns` / `tool_carryover_per_entry_chars`; storage cap `_tool_turns_max_storage = 16`; cleared on `stop` signal AND on new-conversation entry; UNTRUSTED WEB EXTRACT fence markers preserved on truncation; both `content` and `tool_calls[*].function.arguments` scrubbed on write)
|
||||
- Unified system prompt from [src/jarvis/system_prompt.py](src/jarvis/system_prompt.py) + ASR note + tool-protocol guidance. Reply language is resolved by `reply_language_directive(lang, cfg.tts_engine)` where `lang = _resolve_output_language()` — the single source of truth that prefers the settings-web UI value (config JSON `output_language`) over the compose `OUTPUT_LANGUAGE` env, so changing the language in the settings page takes effect. An explicit lock wins (forces "reply only in `<language>`", also forbidding other scripts so small models stop leaking trailing CJK/Hanja); else a Piper/Chatterbox TTS forces English (English-only voices); else (multilingual TTS, no lock) the assistant replies in the user's own language. The v2 voice worker's brain (`src/voice_worker/brain.py`) runs with `tts_engine="melo"`, so v1's English-only Piper default never locks v2 voice or chat answers to English, and passes `default_language` (worker `--reply-language`, default `Korean`): with no lock, `default_language_directive()` makes replies come in that language unless the user explicitly asks for another one, and `build_system_prompt()` rewrites the persona's "in the user's language" clause to "in the reply language" (eval `evals/test_reply_language_default.py`). The directive is inserted near the FRONT of the guidance list so a small model gives it primacy, and the SAME resolved `lang` feeds `build_system_prompt()`, which rewrites the persona's "in the user's language" clause to the locked language so the persona cannot contradict the directive (previously the persona read the raw env while the directive read the config value, so a settings-UI change was honoured by one and ignored by the other). Gated in `_build_initial_system_message()` at [engine.py](src/jarvis/reply/engine.py).
|
||||
- **v2 compact prompt** (`compact_prompt=True`, the v2 brain): `build_voice_persona()` replaces the v1 persona (about 1.3k characters instead of 7.4k: no single-user long-term memory or on-screen browser rules, answers as short as possible), and of the size-specific components only the ASR note stays; the full tool guidance (`tool_guidance` + `tool_constraints`) is added only when a tool other than `stop`, `toolSearchTool` and `chatChannels` is allowed, else the one-line `COMPACT_ACTION_RULE`. Measured on Strata (Qwen3.6, 2026-11-12): a greeting's reply prompt went from about 4,500 to 1,350 tokens and its prompt read from 1.07 s to 0.36 s (Strata reads in 2,048-token steps). The v2 brain also turns `stream_browser` off. The `[Context: ...]` time carries the 12-hour form beside the 24-hour one (`10:37 (10:37 AM)`): with the short prompt Qwen3.6 read 10:37 as 오후 (pm) 4 of 4 times.
|
||||
- **Tool honesty rules**: `TOOL_HONESTY_RULES` inside the persona, plus `tool_honesty_reminder(allowed_tools)` appended after every context block (just before the closing reply-language lock) naming the turn's tools. Added after M0 (2026-11-10) caught models answering past an available tool.
|
||||
- **Operator instructions** (two sources, both framed "Additional instructions from the operator:" and appended near the end of the guidance list): the settings-UI `llm_instructions` config field, and every `*.md` file in `AGENTS_DIR` (default `/app/agents`, bind-mounted from `./agents`). The file-based set is read once per turn by `load_agent_instructions()` in [src/jarvis/system_prompt.py](src/jarvis/system_prompt.py) and concatenated in filename order, so dropping/editing a `.md` applies on the next reply with no rebuild/restart; fail-open to `""` when the folder is absent/empty/unreadable.
|
||||
- **Warm profile block** (query-agnostic User + Directives excerpt from the knowledge graph, composed by `build_warm_profile()` / `format_warm_profile_block()` in [src/jarvis/memory/graph_ops.py](src/jarvis/memory/graph_ops.py) at Step 3.5 of `reply()`; no LLM call, pure SQLite read; injected unconditionally so personalisation is the default; result cached in `DialogueMemory._hot_cache` under `DialogueMemory.WARM_PROFILE_CACHE_KEY` for the lifetime of the active conversation. Invalidated on `stop`, on new-conversation entry, AND on User/Directives graph mutations via the listener registered in [src/jarvis/daemon.py](src/jarvis/daemon.py) against `register_graph_mutation_listener` in [src/jarvis/memory/graph.py](src/jarvis/memory/graph.py); World-branch writes are ignored)
|
||||
@@ -95,7 +96,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
|
||||
## 7. Tool Router (pre-loop tool selection)
|
||||
|
||||
- **File**: [src/jarvis/tools/selection.py](src/jarvis/tools/selection.py) — `select_tools_with_llm()` (~line 331).
|
||||
- **Trigger**: once per reply, **at the very front of the flow before the planner (#12)**. Always runs — the router is the authoritative tool picker, and its narrowed catalogue is what the planner sees. When the planner later references tools, those names are unioned into the router's allow-list but never replace it; small models tend to default to `webSearch` where a dedicated tool like `getWeather` should win, and the router is tuned for that classification. `tool_selection_strategy == "llm"` is the default; other strategies (`all`, `keyword`, `embedding`) also run here.
|
||||
- **Trigger**: once per reply, **at the very front of the flow before the planner (#12)**. Always runs — the router is the authoritative tool picker, and its narrowed catalogue is what the planner sees. When the planner later references tools, those names are unioned into the router's allow-list but never replace it; small models tend to default to `webSearch` where a dedicated tool like `getWeather` should win, and the router is tuned for that classification. `tool_selection_strategy == "llm"` is the default; other strategies (`all`, `keyword`, `embedding`) also run here. **v2 brain** (`run_reply_engine(intent_judge=True)`): the router is the turn's judgement; the planner (#12) runs only when it picked a tool other than `stop`, `toolSearchTool` and `chatChannels`. `prepare_intent()` runs the same pick ahead on a partial transcript while the user is still talking, and the reply uses it (`prepared_tools`) without a router call of its own (planner.spec.md, "Intent judge (v2)"). Without `stream_browser` the screen-share tools (`controlBrowser`, `browseAndPlay`, `setBroadcast`) are not in the router's or the chat model's catalogue.
|
||||
- **Model / gating**: `resolve_tool_router_model(cfg)` chain — `tool_router_model → intent_judge_model → ollama_chat_model`.
|
||||
- **Inputs**: user query, tool catalogue (builtin + MCP with descriptions), optional narrow-down hint.
|
||||
- **System prompt**: inline (~lines 260-315). Teaches pick up-to-5 tools or `none`.
|
||||
@@ -156,7 +157,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
|
||||
## 12. Task-list Planner (pre-flight decomposition, gates the whole turn)
|
||||
|
||||
- **File**: [src/jarvis/reply/planner.py](src/jarvis/reply/planner.py) — `plan_query()`.
|
||||
- **Trigger**: once per reply, **after the tool router and before memory search**. Skipped when `cfg.planner_enabled = False`, when the query is shorter than `MIN_QUERY_CHARS` (4), or when no model / base URL is available.
|
||||
- **Trigger**: once per reply, **after the tool router and before memory search**. **v2 brain** (`intent_judge=True`): runs only on turns where the router picked a tool (not for `stop`, `toolSearchTool` or `chatChannels` alone), so a turn without tools has one LLM call before its answer. Skipped when `cfg.planner_enabled = False`, when the query is shorter than `MIN_QUERY_CHARS` (4), or when no model / base URL is available.
|
||||
- **Model / gating**: resolution chain `planner_model (override) → ollama_chat_model`. The planner tracks the chat model so upgrading the chat model (via setup wizard or config) automatically upgrades plan quality.
|
||||
- **Inputs**: user query, dialogue context, the live time/location line (`LIVE CONTEXT:`, so a location-dependent ask plans `searchMemory topic='where the user lives'` only when the location is unknown or disabled), **router-narrowed** tool catalogue (names + one-line descriptions + argument names, built by `planner_tool_catalog()`) — not the full 30+ list. When the carry-over guard from #7 fires, the previous turn's failed tool name is unioned into this catalogue before the planner sees it, so the planner can plan a re-call without `toolSearchTool` round-tripping. **No** memory context — the planner decides *whether* memory is needed.
|
||||
- **System prompt**: `_PROMPT_TEMPLATE` in `planner.py`. Teaches the `searchMemory topic='...'` directive for prior-conversation lookups, short imperative tool steps, angle-bracket entity placeholders, final synthesis step, same-language output, no numbering.
|
||||
@@ -204,7 +205,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
|
||||
| 4 | Memory digest | 0-N | auto by size | SMALL (uses chat model) |
|
||||
| 5 | Tool-result digest | 0-N | auto by size | SMALL (uses chat model) |
|
||||
| 6 | Max-turn digest | 0-1 | No | SMALL |
|
||||
| 7 | Tool router | 1 | always runs; planner picks unioned in | SMALL |
|
||||
| 7 | Tool router | 0-1 | always runs (v2: unless prepared on a partial); planner picks unioned in | SMALL |
|
||||
| 8 | Tool searcher | 0-3 | model-initiated | SMALL (reuses #7) |
|
||||
| 9 | Summariser | ~1/session | No (background) | LARGE |
|
||||
| 10 | Graph extraction | ~1/session | No (background) | LARGE |
|
||||
|
||||
@@ -39,6 +39,30 @@ from helpers import MockConfig, JUDGE_MODEL, is_judge_llm_available
|
||||
# Shared Markers
|
||||
# =============================================================================
|
||||
|
||||
# The v2 brain's reply options, to compare them with v1's on the same cases:
|
||||
# EVAL_COMPACT_PROMPT=1 runs every eval with run_reply_engine(compact_prompt=True),
|
||||
# EVAL_INTENT_JUDGE=1 with intent_judge=True (both together are the v2 brain's setup).
|
||||
@pytest.fixture(autouse=True)
|
||||
def _v2_reply_options(monkeypatch):
|
||||
on = lambda name: os.environ.get(name, "").strip() in ("1", "true", "yes")
|
||||
options = {k: True for k, env in (("compact_prompt", "EVAL_COMPACT_PROMPT"),
|
||||
("intent_judge", "EVAL_INTENT_JUDGE")) if on(env)}
|
||||
if not options:
|
||||
return
|
||||
import functools
|
||||
import jarvis.reply.engine as engine
|
||||
|
||||
original = engine.run_reply_engine
|
||||
|
||||
@functools.wraps(original)
|
||||
def v2(*args, **kwargs):
|
||||
for k, v in options.items():
|
||||
kwargs.setdefault(k, v)
|
||||
return original(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(engine, "run_reply_engine", v2)
|
||||
|
||||
|
||||
_JUDGE_LLM_AVAILABLE = is_judge_llm_available()
|
||||
requires_judge_llm = pytest.mark.skipif(
|
||||
not _JUDGE_LLM_AVAILABLE,
|
||||
|
||||
68
evals/test_voice_reply_length.py
Normal file
68
evals/test_voice_reply_length.py
Normal file
@@ -0,0 +1,68 @@
|
||||
"""
|
||||
Eval: the v2 voice prompt keeps answers short (user request 2026-11-12: "답변은 최대한 짧게").
|
||||
|
||||
The v2 brain runs the reply engine with ``compact_prompt=True`` (system_prompt.build_voice_persona)
|
||||
and its intent judge. Every answer here must be short to speak: at most 70 characters (about four
|
||||
seconds of speech) and at most two sentences (a joke may need a question and its punchline),
|
||||
measured by script. The comparison runs the same questions with v1's prompt:
|
||||
the v2 answers must be shorter in total, so the prompt (not the model) is what makes them short.
|
||||
|
||||
Run: EVAL_JUDGE_MODEL=... EVAL_JUDGE_BASE_URL=... python -m pytest evals/test_voice_reply_length.py -v
|
||||
"""
|
||||
|
||||
import re
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from conftest import requires_judge_llm
|
||||
from helpers import ToolCallCapture, assert_not_fallback_reply, create_mock_tool_run
|
||||
|
||||
QUESTIONS = [
|
||||
"자비스, 하늘은 왜 파래?",
|
||||
"자비스, 고마워",
|
||||
"자비스, 오늘 하루 어땠어?",
|
||||
"자비스, 피곤하다",
|
||||
"자비스, 물은 몇 도에서 끓어?",
|
||||
"자비스, 대한민국 수도가 어디야?",
|
||||
"자비스, 재밌는 얘기 하나 해 줘",
|
||||
]
|
||||
MAX_CHARS = 70
|
||||
# A sentence end: . ! ? (or their full-width forms) followed by more text.
|
||||
_SENTENCE_BREAK = re.compile(r"[.!?。!?](?=\s*\S)")
|
||||
|
||||
|
||||
def _reply(mock_config, eval_db, eval_dialogue_memory, query, v2, monkeypatch):
|
||||
from helpers import JUDGE_BASE_URL, JUDGE_MODEL
|
||||
from jarvis.memory.conversation import DialogueMemory
|
||||
from jarvis.reply.engine import run_reply_engine
|
||||
|
||||
monkeypatch.delenv("OUTPUT_LANGUAGE", raising=False)
|
||||
mock_config.ollama_base_url = JUDGE_BASE_URL
|
||||
mock_config.ollama_chat_model = JUDGE_MODEL
|
||||
mock_config.tts_engine = "melo"
|
||||
mock_config.stream_browser = False
|
||||
capture = ToolCallCapture()
|
||||
with patch("jarvis.reply.engine.run_tool_with_retries", side_effect=create_mock_tool_run(capture, {})), \
|
||||
patch("jarvis.reply.engine._extra_config", lambda key, default="": default):
|
||||
return run_reply_engine(db=eval_db, cfg=mock_config, tts=None, text=query,
|
||||
dialogue_memory=DialogueMemory(), legacy_memory=False,
|
||||
default_language="Korean", compact_prompt=v2, intent_judge=v2) or ""
|
||||
|
||||
|
||||
@pytest.mark.eval
|
||||
@requires_judge_llm
|
||||
class TestVoiceReplyLength:
|
||||
def test_v2_answers_are_short_and_shorter_than_v1(self, mock_config, eval_db,
|
||||
eval_dialogue_memory, monkeypatch):
|
||||
v2 = [_reply(mock_config, eval_db, eval_dialogue_memory, q, True, monkeypatch) for q in QUESTIONS]
|
||||
v1 = [_reply(mock_config, eval_db, eval_dialogue_memory, q, False, monkeypatch) for q in QUESTIONS]
|
||||
for q, a, b in zip(QUESTIONS, v2, v1):
|
||||
print(f"\n {q}\n v2 ({len(a)}): {a}\n v1 ({len(b)}): {b}")
|
||||
too_long = [(q, a) for q, a in zip(QUESTIONS, v2)
|
||||
if len(a) > MAX_CHARS or len(_SENTENCE_BREAK.findall(a.strip())) > 1]
|
||||
for q, a in zip(QUESTIONS, v2):
|
||||
assert_not_fallback_reply(a, context=q)
|
||||
print(f"\n total chars v2={sum(map(len, v2))} v1={sum(map(len, v1))}")
|
||||
assert not too_long, f"not short: {too_long}"
|
||||
assert sum(map(len, v2)) < sum(map(len, v1))
|
||||
@@ -11,6 +11,8 @@ from typing import Optional, TYPE_CHECKING
|
||||
from ..utils.redact import redact
|
||||
from ..system_prompt import (
|
||||
build_system_prompt,
|
||||
build_voice_persona,
|
||||
COMPACT_ACTION_RULE,
|
||||
load_agent_instructions,
|
||||
reply_language_directive,
|
||||
tool_honesty_reminder,
|
||||
@@ -321,6 +323,26 @@ def _extract_text_tool_call(content_field: str, known_names: set):
|
||||
return None, None, None
|
||||
content_field = content_field
|
||||
|
||||
# Form: `<tool_call>` block as text — Qwen's XML (`<function=name><parameter=key>value</parameter>`)
|
||||
# or Hermes JSON (`{"name": ..., "arguments": {...}}`). Field run 2026-11-12 (Qwen3.6 on Strata):
|
||||
# asked for a timer with no timer tool offered, the model wrote this block as its reply, which
|
||||
# TTS would read aloud. Any name is returned: one not on offer gets the "not available" answer.
|
||||
tc_block = re.search(r"<tool_call>\s*(.+?)\s*(?:</tool_call>|$)", content_field, re.DOTALL)
|
||||
if tc_block:
|
||||
body = tc_block.group(1)
|
||||
fn = re.match(r"<function=([^>\s]+)>(.*?)(?:</function>|$)", body, re.DOTALL)
|
||||
if fn:
|
||||
args = {k: v.strip() for k, v in re.findall(r"<parameter=([^>\s]+)>(.*?)</parameter>", fn.group(2), re.DOTALL)}
|
||||
return fn.group(1).strip(), args, f"call_{uuid.uuid4().hex[:8]}"
|
||||
try:
|
||||
data = json.loads(body)
|
||||
name = str(data.get("name", "")).strip() if isinstance(data, dict) else ""
|
||||
if name:
|
||||
args = data.get("arguments", {})
|
||||
return name, (args if isinstance(args, dict) else {}), f"call_{uuid.uuid4().hex[:8]}"
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Form: markdown fence
|
||||
fence_match = re.search(
|
||||
r"```tool_call\s*\n({.+?})\s*\n```",
|
||||
@@ -977,12 +999,95 @@ def _history_block(history: Any) -> str:
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def _dialogue_context(recent_messages: list) -> str:
|
||||
"""The planner's DIALOGUE CONTEXT: the last few user/assistant lines, shortened."""
|
||||
lines: list[str] = []
|
||||
for m in (recent_messages or [])[-6:]:
|
||||
role = m.get("role", "")
|
||||
content = (m.get("content") or "").strip().replace("\n", " ")
|
||||
if role in ("user", "assistant") and content:
|
||||
lines.append(f"{role}: {content[:200]}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _recent_messages(cfg, dialogue_memory) -> list:
|
||||
if not (dialogue_memory and dialogue_memory.has_recent_messages()):
|
||||
return []
|
||||
if hasattr(dialogue_memory, "get_recent_turns_with_tools"):
|
||||
return dialogue_memory.get_recent_turns_with_tools(
|
||||
max_tool_turns=getattr(cfg, "tool_carryover_max_turns", 2),
|
||||
per_entry_chars=getattr(cfg, "tool_carryover_per_entry_chars", 1200),
|
||||
)
|
||||
return dialogue_memory.get_recent_messages()
|
||||
|
||||
|
||||
# The on-screen browser tools work only in screen-share mode (cfg.stream_browser); without it they
|
||||
# only refuse, and offered anyway the router picked them for unrelated asks (field run 2026-11-12,
|
||||
# v2: "피곤하다" routed to browseAndPlay).
|
||||
_SCREEN_SHARE_TOOLS = ("controlBrowser", "browseAndPlay", "setBroadcast")
|
||||
|
||||
|
||||
def _offered_builtins(cfg, chat_available: bool) -> dict:
|
||||
"""The built-in tools this turn can offer: chatChannels only with a chat backend
|
||||
(chat_channels.spec.md), the screen-share tools only in screen-share mode."""
|
||||
stream = getattr(cfg, "stream_browser", True)
|
||||
return {k: v for k, v in BUILTIN_TOOLS.items()
|
||||
if (k != "chatChannels" or chat_available) and (stream or k not in _SCREEN_SHARE_TOOLS)}
|
||||
|
||||
|
||||
def _route_tools(cfg, redacted: str, builtin_tools: dict, mcp_tools: dict, recent_messages: list):
|
||||
"""The tool router's pick for this query (``select_tools`` with the engine's settings)."""
|
||||
try:
|
||||
strategy = ToolSelectionStrategy(getattr(cfg, "tool_selection_strategy", "llm"))
|
||||
except ValueError:
|
||||
strategy = ToolSelectionStrategy.LLM
|
||||
return select_tools(
|
||||
query=redacted,
|
||||
builtin_tools=builtin_tools,
|
||||
mcp_tools=mcp_tools,
|
||||
strategy=strategy,
|
||||
llm_base_url=cfg.ollama_base_url,
|
||||
llm_model=resolve_tool_router_model(cfg),
|
||||
llm_timeout_sec=float(getattr(cfg, "llm_tools_timeout_sec", 8.0)),
|
||||
embed_model=getattr(cfg, "ollama_embed_model", "nomic-embed-text"),
|
||||
embed_timeout_sec=float(getattr(cfg, "llm_embed_timeout_sec", 10.0)),
|
||||
context_hint=_build_enrichment_context_hint(cfg, recent_messages),
|
||||
)
|
||||
|
||||
|
||||
def prepare_intent(cfg, text: str, dialogue_memory, chat_available: bool) -> Optional[list]:
|
||||
"""Run the intent judge (the tool router) for ``text`` ahead of its reply (v2: on a partial
|
||||
transcript while the user still speaks), with the inputs ``run_reply_engine(intent_judge=True)``
|
||||
would give it, so its pick can be passed back as ``prepared_tools``. None when the router gave
|
||||
up (its fall-open to the whole catalogue is not a pick). ``chat_available``: the reply will
|
||||
have a chat backend (the chatChannels tool is in the catalogue)."""
|
||||
mcp_tools: dict = {}
|
||||
if getattr(cfg, "mcps", {}):
|
||||
try:
|
||||
from ..tools.registry import get_cached_mcp_tools
|
||||
mcp_tools = get_cached_mcp_tools()
|
||||
except Exception:
|
||||
mcp_tools = {}
|
||||
builtin = _offered_builtins(cfg, chat_available)
|
||||
try:
|
||||
routed = _route_tools(cfg, redact(text), builtin, mcp_tools, _recent_messages(cfg, dialogue_memory))
|
||||
except Exception as exc: # pragma: no cover — defensive
|
||||
debug_log(f"intent judge (prepare) failed (non-fatal): {exc}", "planning")
|
||||
return None
|
||||
if routed is None or set(routed) >= set(builtin) | set(mcp_tools):
|
||||
return None
|
||||
return list(routed)
|
||||
|
||||
|
||||
def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
||||
text: str, dialogue_memory: "DialogueMemory",
|
||||
language: Optional[str] = None,
|
||||
history: Optional[Any] = None,
|
||||
legacy_memory: bool = True,
|
||||
default_language: Optional[str] = None) -> Optional[str]:
|
||||
default_language: Optional[str] = None,
|
||||
compact_prompt: bool = False,
|
||||
intent_judge: bool = False,
|
||||
prepared_tools: Optional[list] = None) -> Optional[str]:
|
||||
"""
|
||||
Main entry point for reply generation.
|
||||
|
||||
@@ -1008,6 +1113,15 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
||||
default_language: v2 only. The language to reply in unless the user
|
||||
explicitly asks for another one (e.g. "Korean"). An
|
||||
``output_language`` lock still wins. None: the user's language.
|
||||
compact_prompt: v2 only. The short voice persona and, on turns where
|
||||
no data tool was routed, one action rule instead of the full tool
|
||||
guidance (prompt length is reply latency on Strata).
|
||||
intent_judge: v2 only. One judgement before the reply: the tool
|
||||
router decides which tools the turn needs and the planner does
|
||||
not run (planner.spec.md, "Intent judge (v2)").
|
||||
prepared_tools: v2 only, with ``intent_judge``. The router's tools
|
||||
``prepare_intent`` already chose for this text (while the user was
|
||||
still speaking); the router is not run again.
|
||||
|
||||
Returns:
|
||||
Generated reply text or None
|
||||
@@ -1095,19 +1209,12 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
||||
# (chat_channels.spec.md); otherwise it is not in the catalogue at all.
|
||||
from .turn_state import get_chat as _get_chat
|
||||
_chat_backend = _get_chat()
|
||||
_builtin_tools = {k: v for k, v in BUILTIN_TOOLS.items()
|
||||
if k != "chatChannels" or _chat_backend is not None}
|
||||
_builtin_tools = _offered_builtins(cfg, _chat_backend is not None)
|
||||
_all_builtin_names = list(_builtin_tools.keys())
|
||||
_all_mcp_names = list(mcp_tools.keys())
|
||||
_full_catalog_names = _all_builtin_names + _all_mcp_names
|
||||
|
||||
_dialogue_lines: list[str] = []
|
||||
for _m in (recent_messages or [])[-6:]:
|
||||
_role = _m.get("role", "")
|
||||
_content = (_m.get("content") or "").strip().replace("\n", " ")
|
||||
if _role in ("user", "assistant") and _content:
|
||||
_dialogue_lines.append(f"{_role}: {_content[:200]}")
|
||||
_dialogue_ctx = "\n".join(_dialogue_lines)
|
||||
_dialogue_ctx = _dialogue_context(recent_messages)
|
||||
|
||||
# Step 2a: Tool routing FIRST.
|
||||
#
|
||||
@@ -1138,22 +1245,15 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
||||
dialogue_memory.hot_cache_get(_router_cache_key)
|
||||
if dialogue_memory and hasattr(dialogue_memory, "hot_cache_get") else None
|
||||
)
|
||||
if isinstance(_cached_routed, list):
|
||||
if intent_judge and prepared_tools is not None:
|
||||
# Intent judge (v2): the router's pick made while the user was still speaking.
|
||||
routed_tools = list(prepared_tools)
|
||||
debug_log(f"intent judge: prepared pick used ({len(routed_tools)} tool(s))", "planning")
|
||||
elif isinstance(_cached_routed, list):
|
||||
routed_tools = list(_cached_routed)
|
||||
debug_log("tool router served from hot-window cache", "planning")
|
||||
else:
|
||||
routed_tools = select_tools(
|
||||
query=redacted,
|
||||
builtin_tools=_builtin_tools,
|
||||
mcp_tools=mcp_tools,
|
||||
strategy=strategy,
|
||||
llm_base_url=cfg.ollama_base_url,
|
||||
llm_model=resolve_tool_router_model(cfg),
|
||||
llm_timeout_sec=float(getattr(cfg, "llm_tools_timeout_sec", 8.0)),
|
||||
embed_model=getattr(cfg, "ollama_embed_model", "nomic-embed-text"),
|
||||
embed_timeout_sec=float(getattr(cfg, "llm_embed_timeout_sec", 10.0)),
|
||||
context_hint=context_hint,
|
||||
)
|
||||
routed_tools = _route_tools(cfg, redacted, _builtin_tools, mcp_tools, recent_messages)
|
||||
# Don't cache the router's "fall open to all tools" fallback. That
|
||||
# path fires when the LLM router times out, returns empty, or emits
|
||||
# a response no token of which matches a known tool name — i.e. the
|
||||
@@ -1217,6 +1317,9 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
||||
"deleteMeal", "localFiles", "controlBrowser", "browseAndPlay", "screenshot",
|
||||
}
|
||||
_router_wants_data = any(t in routed_tools for t in _DATA_TOOLS)
|
||||
# Intent judge (v2): the planner only runs on turns that need a tool (the chat tool aside,
|
||||
# which is offered on every turn below); a turn without one goes straight to its answer.
|
||||
_turn_needs_tools = any(t not in ("chatChannels", "stop", "toolSearchTool") for t in routed_tools)
|
||||
|
||||
# In screen-share mode, always offer setBroadcast. "Turn the broadcast
|
||||
# on/off" is language-agnostic intent the embedding/keyword router won't
|
||||
@@ -1261,13 +1364,16 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
||||
|
||||
action_plan: list[str] = []
|
||||
try:
|
||||
action_plan = plan_query(
|
||||
cfg=cfg,
|
||||
query=redacted,
|
||||
dialogue_context=_dialogue_ctx,
|
||||
tools=_planner_tool_catalog,
|
||||
live_context=_live_time_location_string(cfg),
|
||||
)
|
||||
# Intent judge (v2): the router decides whether the turn needs tools; only then does the
|
||||
# planner order the steps (planner.spec.md, "Intent judge (v2)").
|
||||
if not intent_judge or _turn_needs_tools:
|
||||
action_plan = plan_query(
|
||||
cfg=cfg,
|
||||
query=redacted,
|
||||
dialogue_context=_dialogue_ctx,
|
||||
tools=_planner_tool_catalog,
|
||||
live_context=_live_time_location_string(cfg),
|
||||
)
|
||||
except Exception as _plan_exc: # pragma: no cover — defensive
|
||||
debug_log(f"planner step failed (non-fatal): {_plan_exc}", "planning")
|
||||
action_plan = []
|
||||
@@ -1717,7 +1823,8 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
||||
# _resolve_output_language). Building the persona from the raw env var while
|
||||
# the directive used the config value made the two contradict each other.
|
||||
_output_language = _resolve_output_language()
|
||||
_persona_prompt = build_system_prompt(_assistant_name, _output_language, default_language)
|
||||
_persona_prompt = (build_voice_persona if compact_prompt else build_system_prompt)(
|
||||
_assistant_name, _output_language, default_language)
|
||||
# File-based operator instructions: every *.md in AGENTS_DIR (default
|
||||
# /app/agents, bind-mounted from ./agents). Read once per turn so edits in
|
||||
# the folder apply on the next reply without a restart; fail-open to "".
|
||||
@@ -1726,8 +1833,18 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
||||
def _build_initial_system_message() -> str:
|
||||
guidance = [_persona_prompt.strip()]
|
||||
|
||||
# Add model-size-appropriate prompt components
|
||||
guidance.extend(prompts.to_list())
|
||||
if compact_prompt:
|
||||
# The persona already covers voice style, inference and tool use; the tool guidance
|
||||
# (named entities, auto-derived and self-contained arguments) matters only when a
|
||||
# data tool is on offer. The sentinels and the always-offered chat tool are not.
|
||||
guidance.append(prompts.asr_note)
|
||||
if any(t not in ("stop", "toolSearchTool", "chatChannels") for t in allowed_tools):
|
||||
guidance.extend(c for c in (prompts.tool_guidance, prompts.tool_constraints) if c)
|
||||
else:
|
||||
guidance.append(COMPACT_ACTION_RULE)
|
||||
else:
|
||||
# Add model-size-appropriate prompt components
|
||||
guidance.extend(prompts.to_list())
|
||||
|
||||
# Reply-language policy: an explicit OUTPUT_LANGUAGE lock wins, else
|
||||
# Piper/Chatterbox TTS forces English (English-only voices), else the
|
||||
|
||||
@@ -204,6 +204,30 @@ The engine consumes the plan in two phases.
|
||||
the synthetic assistant `tool_calls` message, and no progress nudge is
|
||||
appended.
|
||||
|
||||
### Intent judge (v2)
|
||||
|
||||
The v2 brain (`src/voice_worker/voice_worker.spec.md`, "History and brain") judges each turn once
|
||||
before the answer: `run_reply_engine(intent_judge=True)` runs the tool router as usual (the harness's
|
||||
judgement of whether the turn needs a tool and which ones) and runs the planner **only when the
|
||||
router picked a tool** other than the sentinels (`stop`, `toolSearchTool`) and the always-offered
|
||||
`chatChannels`. A turn without a tool (greetings, small talk, facts the model knows) goes straight
|
||||
to its answer with one LLM call before it; a tool turn keeps the planner's ordered steps, which
|
||||
multi-step asks need ("who directed Possessor, and what else did they make").
|
||||
|
||||
Measured on Strata (Qwen3.6, 2026-11-12): the router took 0.42 s and the planner 0.49-0.94 s, one
|
||||
after the other, before every answer; each call costs at least one full prompt read. Tried first
|
||||
and dropped: the planner alone over the whole catalogue (one call) planned "reply with the
|
||||
weather" without a tool step and lost the weather tool (20 eval failures against 4), and the router
|
||||
alone without a planner failed the chained Possessor research 3 of 3 times. With this rule the
|
||||
whole eval suite passes as with v1's flow (`EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1`).
|
||||
|
||||
`prepare_intent(cfg, text, dialogue_memory, chat_available)` runs the same router pick ahead of
|
||||
time, with the inputs the reply would give it (the catalogue with `chatChannels` when a chat backend
|
||||
will be there, the context hint from the asker's dialogue memory); None when the router gave up (its
|
||||
fall-open to the whole catalogue is not a pick). The pick is passed back as
|
||||
`run_reply_engine(prepared_tools=...)` and the router is not run again. The brain calls it on a
|
||||
partial transcript while the user is still talking.
|
||||
|
||||
### resolve_next_tool_call
|
||||
|
||||
- **Fast path**: if the step text is fully concrete (tool name in the
|
||||
|
||||
@@ -157,6 +157,57 @@ _SYSTEM_PROMPT_TEMPLATE: str = (
|
||||
)
|
||||
|
||||
|
||||
# The v2 voice persona (voice_worker.spec.md, "History and brain"): the same butler in a tenth of
|
||||
# the words. v1's persona spends most of its length on the single-user long-term memory (profile,
|
||||
# diary, graph) and on the on-screen browser, neither of which the v2 brain has; Strata reads every
|
||||
# prompt from the start, so each dropped token is latency saved on every reply. The user asked for
|
||||
# answers as short as possible (2026-11-12): the aside is optional and only when nearly free.
|
||||
_VOICE_PERSONA_TEMPLATE: str = (
|
||||
"You are {name}, the voice assistant in a group voice chat, with the manner of a composed, "
|
||||
"polite butler and a dry sense of humour. "
|
||||
"Keep every reply as short as it can be while still answering well: exactly one short "
|
||||
"sentence that gives the answer first, with no second sentence even for a remark or advice "
|
||||
"(join them into the one sentence or drop one). No preamble, no repeating the question, no "
|
||||
"follow-up offers, no lists. Meet small talk in kind, warmly and briefly. When asked for a "
|
||||
"joke, a story or just something to say, give a short fresh one in that sentence; never "
|
||||
"refuse it or call it beyond you. "
|
||||
"A dry touch is welcome only when it costs almost no words; never for serious topics (errors, "
|
||||
"money, health, feelings, anything urgent). "
|
||||
"Never be mean, condescending or sycophantic ('great question'), never call the user 'sir' or "
|
||||
"'madam', and never answer with a bare greeting: engage with what was actually said. "
|
||||
"Your reply is spoken aloud by text-to-speech: plain words only, no markdown, emoji or code. "
|
||||
"{tool_rules} "
|
||||
"The [Context: ...] line at the top holds the real current time and location: answer time and "
|
||||
"date questions from it, phrased naturally in the user's language; never say you lack a clock. "
|
||||
"When 'Information shared in previous conversations' is present it is your memory of earlier "
|
||||
"conversations: what was said there is not a live fact and needs no tool, so answer questions "
|
||||
"about it from that section directly, and never say you cannot check or remember it."
|
||||
)
|
||||
|
||||
|
||||
def build_voice_persona(
|
||||
assistant_name: str = "Jarvis", output_language: Optional[str] = None,
|
||||
default_language: Optional[str] = None,
|
||||
) -> str:
|
||||
"""The short v2 voice persona; language handling as in ``build_system_prompt``."""
|
||||
name = (assistant_name or "Jarvis").strip() or "Jarvis"
|
||||
prompt = _VOICE_PERSONA_TEMPLATE.format(name=name, tool_rules=TOOL_HONESTY_RULES)
|
||||
lang = (output_language or "").strip()
|
||||
if lang:
|
||||
prompt = prompt.replace("in the user's language", f"in {lang}")
|
||||
elif (default_language or "").strip():
|
||||
prompt = prompt.replace("in the user's language", "in the reply language")
|
||||
return prompt
|
||||
|
||||
|
||||
# Compact prompts, turns where no data tool was routed: what the tool guidance says about acting
|
||||
# instead of refusing, in one line (the full guidance is only for turns with data tools).
|
||||
COMPACT_ACTION_RULE: str = (
|
||||
"If the user asks for an action or for live information that your tools here do not cover, "
|
||||
"call toolSearchTool with a short description of it before saying you cannot."
|
||||
)
|
||||
|
||||
|
||||
def tool_honesty_reminder(tool_names) -> str:
|
||||
"""End-of-prompt restatement of the tool honesty rules for this turn.
|
||||
|
||||
|
||||
@@ -14,8 +14,10 @@ except ImportError: # pragma: no cover - Python < 3.9
|
||||
ZoneInfoNotFoundError = Exception # type: ignore[assignment,misc]
|
||||
|
||||
|
||||
_UTC_FORMAT = "%A, %B %d, %Y at %H:%M UTC"
|
||||
_LOCAL_FORMAT = "%A, %B %d, %Y at %H:%M %Z"
|
||||
# The 12-hour form beside the 24-hour one: field run 2026-11-12 (Qwen3.6, Korean) read 10:37 as
|
||||
# 오후 (pm) 4 times of 4 with the short v2 prompt, and as 오전 (am) every time with it given.
|
||||
_UTC_FORMAT = "%A, %B %d, %Y at %H:%M (%I:%M %p) UTC"
|
||||
_LOCAL_FORMAT = "%A, %B %d, %Y at %H:%M (%I:%M %p) %Z"
|
||||
|
||||
|
||||
def format_time_context(
|
||||
|
||||
@@ -4,7 +4,9 @@ interface, so the worker logic is tested with a fake (voice_worker.spec.md, "His
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import difflib
|
||||
import sys
|
||||
import unicodedata
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
@@ -28,6 +30,11 @@ class Brain(Protocol):
|
||||
"""Blocking: candidate ids (``{"id", "user", "text"}``) most important first; None on failure.
|
||||
Runs on its own thread, also while a reply runs."""
|
||||
|
||||
def prepare(self, user: str, text: str, chat: bool) -> None:
|
||||
"""Blocking: get ready to answer ``text`` (a partial transcript of a call still being said),
|
||||
so the reply to about the same text starts further along. ``chat``: chat channels are
|
||||
registered. Runs on its own thread, also while a reply runs. Optional."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class MemorySettings:
|
||||
@@ -110,6 +117,32 @@ class Muted:
|
||||
cls._depth -= 1
|
||||
|
||||
|
||||
# A prepared intent is used for a reply whose text is at least this close to the text it was
|
||||
# prepared for (letters and digits only): the last partial often lacks a final syllable or two.
|
||||
PREPARED_SIMILARITY = 0.85
|
||||
# ... and only this soon after it was prepared, and at most this long is waited for one in flight.
|
||||
PREPARED_MAX_AGE_S = 30.0
|
||||
PREPARED_WAIT_S = 5.0
|
||||
|
||||
|
||||
def _letters(text: str) -> str:
|
||||
return "".join(c for c in unicodedata.normalize("NFKD", text.lower()) if c.isalnum())
|
||||
|
||||
|
||||
def similar(a: str, b: str) -> bool:
|
||||
x, y = _letters(a), _letters(b)
|
||||
return bool(x and y) and difflib.SequenceMatcher(None, x, y).ratio() >= PREPARED_SIMILARITY
|
||||
|
||||
|
||||
@dataclass
|
||||
class _Prepared:
|
||||
text: str
|
||||
turns: int # the asker's dialogue memory size it was made with
|
||||
at: float
|
||||
done: threading.Event = dataclasses.field(default_factory=threading.Event)
|
||||
tools: Optional[list] = None # the intent judge's pick; None: none made
|
||||
|
||||
|
||||
class JarvisBrain:
|
||||
"""The existing jarvis reply engine with the user's history context.
|
||||
|
||||
@@ -125,13 +158,16 @@ class JarvisBrain:
|
||||
# v1's tts_engine names the v1 desktop voice (English-only Piper by default), which makes the
|
||||
# engine lock replies to English. v2 speaks through the worker's TTS and chat answers are
|
||||
# not spoken; the brain answers in ``reply_language`` (an OUTPUT_LANGUAGE lock still wins).
|
||||
self.cfg = dataclasses.replace(load_settings(), tts_engine="melo")
|
||||
# stream_browser: v1's screen-share browser (controlBrowser, browseAndPlay, setBroadcast and
|
||||
# browser-first web search) needs v1's Node bot; v2 has no on-screen browser until M6.
|
||||
self.cfg = dataclasses.replace(load_settings(), tts_engine="melo", stream_browser=False)
|
||||
self.db = Database(self.cfg.db_path, self.cfg.sqlite_vss_path)
|
||||
# ``language``: the ISO code tools get (e.g. the Wikipedia host); ``reply_language``: replies
|
||||
# come in it unless the asker explicitly asks for another language (None: the asker's).
|
||||
self.language = language
|
||||
self.reply_language = reply_language
|
||||
self._memories: dict = {}
|
||||
self._prepared: dict[str, _Prepared] = {}
|
||||
self._lock = threading.Lock()
|
||||
debug_log(f"brain ready: llm={self.cfg.ollama_base_url} model={self.cfg.ollama_chat_model}", "voice_worker")
|
||||
|
||||
@@ -145,11 +181,41 @@ class JarvisBrain:
|
||||
self._memories[user] = mem
|
||||
return mem
|
||||
|
||||
@staticmethod
|
||||
def _turns(mem) -> int:
|
||||
return len(mem.get_recent_messages()) if mem.has_recent_messages() else 0
|
||||
|
||||
def prepare(self, user: str, text: str, chat: bool) -> None:
|
||||
from jarvis.reply.engine import prepare_intent
|
||||
|
||||
mem = self._memory(user)
|
||||
p = _Prepared(text=text, turns=self._turns(mem), at=time.monotonic())
|
||||
with self._lock:
|
||||
self._prepared[user] = p
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
p.tools = Muted.run(prepare_intent, self.cfg, text, mem, chat)
|
||||
finally:
|
||||
p.done.set()
|
||||
debug_log(f"brain prepare user={user} picked={p.tools is not None} "
|
||||
f"ms={(time.perf_counter() - t0) * 1000:.0f}", "voice_worker")
|
||||
|
||||
def _prepared_tools(self, user: str, text: str) -> Optional[list]:
|
||||
"""The pick prepared for about this text, waiting for one still being made; None if none fits."""
|
||||
with self._lock:
|
||||
p = self._prepared.pop(user, None)
|
||||
if p is None or not similar(p.text, text) or time.monotonic() - p.at > PREPARED_MAX_AGE_S:
|
||||
return None
|
||||
if not p.done.wait(PREPARED_WAIT_S) or p.turns != self._turns(self._memory(user)):
|
||||
return None
|
||||
return p.tools
|
||||
|
||||
def reply(self, user: str, text: str, history: HistoryContext, chat=None) -> Optional[str]:
|
||||
from jarvis.reply import turn_state
|
||||
from jarvis.reply.engine import run_reply_engine
|
||||
|
||||
t0 = time.perf_counter()
|
||||
prepared = self._prepared_tools(user, text)
|
||||
# legacy_memory=False: v1's profile, diary and graph belong to one user; several people
|
||||
# share this assistant, so only the asker's scoped history may reach the prompt.
|
||||
def turn():
|
||||
@@ -160,13 +226,14 @@ class JarvisBrain:
|
||||
try:
|
||||
return run_reply_engine(self.db, self.cfg, None, text, self._memory(user),
|
||||
language=self.language, history=history, legacy_memory=False,
|
||||
default_language=self.reply_language)
|
||||
default_language=self.reply_language, compact_prompt=True,
|
||||
intent_judge=True, prepared_tools=prepared)
|
||||
finally:
|
||||
turn_state.reset()
|
||||
|
||||
answer = Muted.run(turn)
|
||||
debug_log(f"brain reply user={user} ok={bool(answer)} ms={(time.perf_counter() - t0) * 1000:.0f}",
|
||||
"voice_worker")
|
||||
debug_log(f"brain reply user={user} ok={bool(answer)} prepared={prepared is not None} "
|
||||
f"ms={(time.perf_counter() - t0) * 1000:.0f}", "voice_worker")
|
||||
return answer
|
||||
|
||||
def addressed(self, user: str, text: str, names: list[str]) -> Optional[bool]:
|
||||
|
||||
@@ -176,6 +176,9 @@ class Session:
|
||||
# Spoken replies whose speech was cancelled while the brain was still answering.
|
||||
self.cancelled_replies: set[str] = set()
|
||||
self.thinking_replies: set[str] = set()
|
||||
# user -> the newest call text waiting for that user's running prepare to finish (or None).
|
||||
self.preparing: dict[str, str | None] = {}
|
||||
self.last_prepared: dict[str, str] = {} # user -> the text of their last prepare
|
||||
# Answers sent but not yet known to have reached the person: id -> (user, turn, timer).
|
||||
self.deliveries: dict[str, tuple[str, int, asyncio.TimerHandle | None]] = {}
|
||||
self._handed_over: dict[str, bool] = {} # replies whose end is counted by _delivered
|
||||
@@ -316,7 +319,8 @@ class Session:
|
||||
"chat_delete": self._on_chat_delete, "reply": self._on_reply,
|
||||
"addressed": self._on_addressed, "rank": self._on_rank,
|
||||
"chat_channels": self._on_chat_channels, "chat_action_result": self._on_chat_action_result,
|
||||
"memory_config": self._on_memory_config, "reply_delivered": self._on_reply_delivered}.get(kind)
|
||||
"memory_config": self._on_memory_config, "reply_delivered": self._on_reply_delivered,
|
||||
"prepare": self._on_prepare}.get(kind)
|
||||
if handler is None:
|
||||
self.error("bad_message", f"unknown type: {kind!r}")
|
||||
return
|
||||
@@ -507,11 +511,47 @@ class Session:
|
||||
return
|
||||
if speak:
|
||||
self.thinking_replies.add(rid)
|
||||
if user in self.preparing:
|
||||
self.preparing[user] = None # the call was answered: no newer text to prepare
|
||||
self.last_prepared.pop(user, None)
|
||||
waiting = self.worker.replies_unanswered
|
||||
early = waiting.get(user, 0) > 0 # sent before their previous answer: not a reply to it
|
||||
waiting[user] = waiting.get(user, 0) + 1
|
||||
asyncio.get_running_loop().create_task(self._reply(rid, user, text, via, speak, early))
|
||||
|
||||
def _on_prepare(self, msg: dict) -> None:
|
||||
"""A call still being said: the brain gets ready for it. One prepare per user runs at a time;
|
||||
while it runs only the newest text waits, and one about the same as the last one prepared
|
||||
(or being prepared) is dropped. Nothing is sent back."""
|
||||
from .brain import similar
|
||||
got = self._strings(msg, "user", "text")
|
||||
if got is None:
|
||||
return
|
||||
user, text = got
|
||||
if self.worker.brain is None or not hasattr(self.worker.brain, "prepare"):
|
||||
return
|
||||
if user in self.last_prepared and similar(self.last_prepared[user], text):
|
||||
return
|
||||
if user in self.preparing:
|
||||
self.preparing[user] = text
|
||||
return
|
||||
self.preparing[user] = None
|
||||
asyncio.get_running_loop().create_task(self._prepare(user, text))
|
||||
|
||||
async def _prepare(self, user: str, text: str | None) -> None:
|
||||
from .brain import similar
|
||||
try:
|
||||
while text is not None and not self.closed:
|
||||
self.last_prepared[user] = text
|
||||
try:
|
||||
await asyncio.to_thread(self.worker.brain.prepare, user, text, bool(self.chat_channels))
|
||||
except Exception as e: # the brain must never take the session down
|
||||
debug_log(f"brain prepare failed: {type(e).__name__}", "voice_worker")
|
||||
newer, self.preparing[user] = self.preparing[user], None
|
||||
text = newer if newer is not None and not similar(newer, self.last_prepared.get(user, "")) else None
|
||||
finally:
|
||||
self.preparing.pop(user, None)
|
||||
|
||||
def _on_addressed(self, msg: dict) -> None:
|
||||
got = self._strings(msg, "id", "user", "text")
|
||||
if got is None:
|
||||
|
||||
@@ -201,6 +201,7 @@ which carry Discord's own time.
|
||||
-> {"type": "reply", "id": "r1", "user": "<id>", "text": "...", "via": "voice", "speak": true}
|
||||
<- {"type": "reply_text", "id": "r1", "text": "...", "failed": false}
|
||||
-> {"type": "reply_delivered", "id": "c1", "ok": true} // a chat answer was posted (ok: false: it was not)
|
||||
-> {"type": "prepare", "user": "<id>", "text": "..."} // a call still being said; nothing comes back
|
||||
-> {"type": "addressed", "id": "q1", "user": "<id>", "text": "..."}
|
||||
<- {"type": "addressed_result", "id": "q1", "addressed": true, "failed": false}
|
||||
-> {"type": "rank", "id": "k1", "candidates": [{"id": "7", "user": "<id>", "text": "..."}]}
|
||||
@@ -241,6 +242,23 @@ which carry Discord's own time.
|
||||
default) does not lock replies to English, because v2 speaks through the worker's TTS. An
|
||||
`output_language` lock in the jarvis settings still wins. Eval: `evals/test_reply_language_default.py`.
|
||||
STT (`language` `"ko"`) and TTS (`--melo-language KR`) are Korean by default too.
|
||||
Two reply options keep the time to the first word short (Strata reads every prompt from the
|
||||
start, about 0.36 s per 2,048 tokens, and each LLM call before the answer adds its whole time):
|
||||
the short voice persona (`compact_prompt=True`: the same butler in a tenth of v1's words, without
|
||||
v1's single-user memory and on-screen browser rules, answers as short as possible, the full tool
|
||||
guidance only on turns that offer a data tool), and the intent judge (`intent_judge=True`: the tool
|
||||
router judges whether the turn needs tools and which, and the planner runs only when it does;
|
||||
`planner.spec.md`, "Intent judge (v2)"). v1's screen-share browser (`stream_browser`) is off: it
|
||||
needs v1's Node bot (M6 brings the v2 one), so its tools are not offered.
|
||||
- `prepare` tells the brain that `user` is saying a call whose partial transcript is `text` (the app
|
||||
sends it while the speaker is still talking, `app/src/conversation.spec.md`). The brain runs the
|
||||
intent judge's tool pick for it on its own thread, with the asker's dialogue memory and the tool
|
||||
catalogue the reply will have. A `reply` from that user whose text is about the same (letters and
|
||||
digits, 85% alike), within 30 s and with the dialogue memory unchanged, uses that pick instead of
|
||||
making its own, waiting up to 5 s for one still being made. Each prepared judgement is used at most once.
|
||||
One `prepare` per user runs at a time: while it runs only the newest text waits, a text about the
|
||||
same as the one last prepared is dropped, and a `reply` from that user drops the waiting text.
|
||||
Nothing is sent back; without a brain, or with one that cannot prepare, it is ignored.
|
||||
The asker is set in `jarvis.reply.turn_state` for the turn, so tools that keep personal data
|
||||
(meals) store and read it under that person only.
|
||||
- `addressed` and `rank` are the conversation's questions (`app/src/conversation.spec.md`), answered
|
||||
|
||||
108
tests/test_engine_compact_prompt.py
Normal file
108
tests/test_engine_compact_prompt.py
Normal file
@@ -0,0 +1,108 @@
|
||||
"""The v2 brain's compact prompt (voice_worker.spec.md, "History and brain").
|
||||
|
||||
Strata reads every prompt from the start, so prompt length is reply latency. Behaviour: the compact
|
||||
prompt is far shorter than v1's for the same turn, still carries the tool honesty rules and the
|
||||
reply-language default, and brings the full tool guidance only on turns that offer a data tool.
|
||||
"""
|
||||
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from src.jarvis.memory.conversation import DialogueMemory
|
||||
from src.jarvis.memory.history import HistoryContext
|
||||
from src.jarvis.reply.engine import run_reply_engine
|
||||
from src.jarvis.reply.prompts.model_variants import TOOL_CONSTRAINTS_LARGE
|
||||
from src.jarvis.system_prompt import COMPACT_ACTION_RULE, TOOL_HONESTY_RULES, default_language_directive
|
||||
|
||||
from tests.test_ollama_num_predict import _mock_cfg
|
||||
|
||||
|
||||
def _system_prompt(compact, routed):
|
||||
cfg = _mock_cfg(512)
|
||||
cfg.stream_browser = False
|
||||
cfg.tool_selection_strategy = "llm"
|
||||
with patch("src.jarvis.reply.engine.plan_query", return_value=["Reply to the user."]), \
|
||||
patch("src.jarvis.reply.engine.select_tools", return_value=list(routed)), \
|
||||
patch("src.jarvis.reply.engine._extra_config", lambda key, default="": default), \
|
||||
patch("src.jarvis.reply.engine.extract_text_from_response", return_value="네."), \
|
||||
patch("src.jarvis.reply.engine.chat_with_messages") as mock_chat:
|
||||
mock_chat.return_value = {"message": {"content": "네."}}
|
||||
run_reply_engine(db=Mock(), cfg=cfg, tts=None, text="안녕?", dialogue_memory=DialogueMemory(),
|
||||
history=HistoryContext(), legacy_memory=False, default_language="Korean",
|
||||
compact_prompt=compact)
|
||||
messages = mock_chat.call_args.kwargs.get("messages") or mock_chat.call_args.args[2]
|
||||
return messages[0]["content"]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@pytest.mark.parametrize("routed", [[], ["getWeather"]])
|
||||
def test_the_compact_prompt_is_far_shorter_and_keeps_the_rules(routed):
|
||||
full, compact = _system_prompt(False, routed), _system_prompt(True, routed)
|
||||
assert len(compact) < len(full) * 0.6, (len(compact), len(full))
|
||||
for prompt in (full, compact):
|
||||
assert TOOL_HONESTY_RULES in prompt
|
||||
assert default_language_directive("Korean") in prompt
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_the_full_tool_guidance_comes_only_with_a_data_tool():
|
||||
chatty = _system_prompt(True, [])
|
||||
assert COMPACT_ACTION_RULE in chatty and TOOL_CONSTRAINTS_LARGE not in chatty
|
||||
weather = _system_prompt(True, ["getWeather"])
|
||||
assert TOOL_CONSTRAINTS_LARGE in weather and COMPACT_ACTION_RULE not in weather
|
||||
|
||||
|
||||
def _intent_turn(prepared, routed=("getWeather",)):
|
||||
cfg = _mock_cfg(512)
|
||||
cfg.stream_browser = False
|
||||
cfg.tool_selection_strategy = "llm"
|
||||
with patch("src.jarvis.reply.engine.plan_query", return_value=["Reply to the user."]) as planner, \
|
||||
patch("src.jarvis.reply.engine.select_tools", return_value=list(routed)) as router, \
|
||||
patch("src.jarvis.reply.engine._extra_config", lambda key, default="": default), \
|
||||
patch("src.jarvis.reply.engine.extract_text_from_response", return_value="네."), \
|
||||
patch("src.jarvis.reply.engine.chat_with_messages") as mock_chat:
|
||||
mock_chat.return_value = {"message": {"content": "네."}}
|
||||
run_reply_engine(db=Mock(), cfg=cfg, tts=None, text="날씨 어때?", dialogue_memory=DialogueMemory(),
|
||||
history=HistoryContext(), legacy_memory=False, intent_judge=True,
|
||||
prepared_tools=prepared)
|
||||
tools = [t["function"]["name"] for t in (mock_chat.call_args.kwargs.get("tools") or [])]
|
||||
return planner.call_count, router.call_count, tools
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_the_intent_judge_plans_only_turns_that_need_a_tool():
|
||||
# No tool needed: one router call, then the answer.
|
||||
assert _intent_turn(None, routed=())[:2] == (0, 1)
|
||||
# A tool turn: the planner orders the steps.
|
||||
planner, router, tools = _intent_turn(None)
|
||||
assert (planner, router) == (1, 1) and "getWeather" in tools
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_prepared_pick_replaces_the_router_call():
|
||||
planner, router, tools = _intent_turn([])
|
||||
assert (planner, router) == (0, 0)
|
||||
planner, router, tools = _intent_turn(["webSearch"])
|
||||
assert (planner, router) == (1, 0) and "webSearch" in tools and "getWeather" not in tools
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@pytest.mark.parametrize("stream", [False, True])
|
||||
def test_screen_share_tools_are_offered_only_in_screen_share_mode(stream):
|
||||
cfg = _mock_cfg(512)
|
||||
cfg.stream_browser = stream
|
||||
cfg.tool_selection_strategy = "llm"
|
||||
with patch("src.jarvis.reply.engine.plan_query", return_value=[]), \
|
||||
patch("src.jarvis.reply.engine.select_tools", return_value=[]) as router, \
|
||||
patch("src.jarvis.reply.engine._extra_config", lambda key, default="": default), \
|
||||
patch("src.jarvis.reply.engine._maybe_deterministic_site_search", return_value=None), \
|
||||
patch("src.jarvis.reply.engine.extract_text_from_response", return_value="네."), \
|
||||
patch("src.jarvis.reply.engine.chat_with_messages") as mock_chat:
|
||||
mock_chat.return_value = {"message": {"content": "네."}}
|
||||
run_reply_engine(db=Mock(), cfg=cfg, tts=None, text="피곤하다", dialogue_memory=DialogueMemory(),
|
||||
history=HistoryContext(), legacy_memory=False)
|
||||
offered = set(router.call_args.kwargs["builtin_tools"])
|
||||
screen = {"controlBrowser", "browseAndPlay", "setBroadcast"}
|
||||
assert (screen <= offered) if stream else not (screen & offered)
|
||||
assert "getWeather" in offered
|
||||
@@ -297,3 +297,21 @@ class TestTextToolCallGuidancePrompt:
|
||||
"an example would reinforce the exact behaviour we want to "
|
||||
"stop."
|
||||
)
|
||||
|
||||
|
||||
class TestToolCallBlockAsText:
|
||||
"""`<tool_call>` blocks written as the reply (Qwen XML or Hermes JSON) are tool calls, not
|
||||
speech, also for a tool that is not on offer (the engine then says it is not available)."""
|
||||
|
||||
def test_qwen_xml(self):
|
||||
content = ("<tool_call>\n<function=setTimer>\n<parameter=duration>\n5 minutes\n</parameter>\n"
|
||||
"</function>\n</tool_call>")
|
||||
name, args, call_id = _extract(content)
|
||||
assert name == "setTimer" and args == {"duration": "5 minutes"} and call_id
|
||||
|
||||
def test_hermes_json(self):
|
||||
name, args, _ = _extract('<tool_call>\n{"name": "webSearch", "arguments": {"search_query": "x"}}\n</tool_call>')
|
||||
assert name == "webSearch" and args == {"search_query": "x"}
|
||||
|
||||
def test_prose_mentioning_tags_is_not_a_call(self):
|
||||
assert _extract("I would call <tool_call> if I could.")[0] is None
|
||||
|
||||
@@ -372,9 +372,11 @@ def test_the_brain_answers_in_korean_by_default_even_with_a_v1_english_voice(tmp
|
||||
seen = {}
|
||||
|
||||
def fake_engine(db, cfg, tts, text, mem, language=None, history=None, legacy_memory=True,
|
||||
default_language=None):
|
||||
default_language=None, compact_prompt=False, **kw):
|
||||
seen["directive"] = reply_language_directive(None, cfg.tts_engine, default_language)
|
||||
seen["language"] = language
|
||||
seen["compact"] = compact_prompt
|
||||
seen["stream_browser"] = cfg.stream_browser
|
||||
return "ok"
|
||||
|
||||
import jarvis.reply.engine as engine
|
||||
@@ -382,6 +384,8 @@ def test_the_brain_answers_in_korean_by_default_even_with_a_v1_english_voice(tmp
|
||||
brain_mod.JarvisBrain().reply("100", "what time is it?", HistoryContext())
|
||||
assert seen["directive"] == default_language_directive("Korean")
|
||||
assert seen["language"] == "ko"
|
||||
# The short voice prompt, and no v1 screen-share browser (it needs v1's Node bot).
|
||||
assert seen["compact"] is True and seen["stream_browser"] is False
|
||||
# Without a default the brain answers in the asker's language.
|
||||
brain_mod.JarvisBrain(language=None, reply_language=None).reply("100", "q", HistoryContext())
|
||||
assert seen["directive"] is None and seen["language"] is None
|
||||
@@ -966,3 +970,138 @@ def test_a_chat_question_the_app_never_reports_counts_as_not_posted(store):
|
||||
actions, results = run_delete_then_yes(store, dict(via="chat", speak=False), no_report,
|
||||
delivery_timeout_s=0.1)
|
||||
assert actions == [] and results == [("지워", True), ("응, 지워", False)]
|
||||
|
||||
|
||||
class PreparingBrain(FakeBrain):
|
||||
"""Each prepare waits until the test lets it finish."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
import threading
|
||||
self.prepared = []
|
||||
self.release = threading.Event()
|
||||
|
||||
def prepare(self, user, text, chat):
|
||||
self.prepared.append((user, text, chat))
|
||||
self.release.wait(5)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_prepare_runs_one_at_a_time_per_user_and_only_the_newest_different_text_waits(store):
|
||||
brain = PreparingBrain()
|
||||
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws, stt={"partials": False})
|
||||
await send(ws, type="prepare", user="100", text="자비스 오늘")
|
||||
await asyncio.sleep(0.2)
|
||||
await send(ws, type="prepare", user="100", text="자비스 오늘 서울")
|
||||
await send(ws, type="prepare", user="100", text="자비스 오늘 서울 날씨 어때")
|
||||
await send(ws, type="prepare", user="200", text="자비스 안녕")
|
||||
await asyncio.sleep(0.2)
|
||||
assert sorted(brain.prepared) == [("100", "자비스 오늘", False), ("200", "자비스 안녕", False)]
|
||||
brain.release.set()
|
||||
await asyncio.sleep(0.3)
|
||||
# About the same as the one just prepared: not prepared again.
|
||||
await send(ws, type="prepare", user="100", text="자비스 오늘 서울 날씨 어때?")
|
||||
await asyncio.sleep(0.3)
|
||||
await send(ws, type="reply", id="r1", user="100", text="x", via="voice", speak=False)
|
||||
return await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text")
|
||||
|
||||
seen = asyncio.run(with_worker(body, store=store, brain=brain))
|
||||
assert [t for u, t, _ in brain.prepared if u == "100"] == ["자비스 오늘", "자비스 오늘 서울 날씨 어때"]
|
||||
assert not [m for m in seen if isinstance(m, dict) and m.get("type") == "error"], "nothing is sent back"
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_brain_without_prepare_ignores_it(store):
|
||||
async def body(url):
|
||||
async with connect(url) as ws:
|
||||
await hello(ws, stt={"partials": False})
|
||||
await send(ws, type="prepare", user="100", text="자비스 안녕")
|
||||
await send(ws, type="reply", id="r1", user="100", text="hi", via="voice", speak=False)
|
||||
return await recv_until(ws, lambda m: isinstance(m, dict) and m.get("type") == "reply_text")
|
||||
|
||||
seen = asyncio.run(with_worker(body, store=store, brain=FakeBrain()))
|
||||
assert not [m for m in seen if isinstance(m, dict) and m.get("type") == "error"]
|
||||
|
||||
|
||||
def _jarvis_brain(tmp_path, monkeypatch, plans):
|
||||
"""A real JarvisBrain with the engine's judge and reply replaced: ``plans`` maps text to a pick."""
|
||||
pytest.importorskip("mcp") # the real brain needs the jarvis dependencies (--brain)
|
||||
from voice_worker import brain as brain_mod
|
||||
import jarvis.reply.engine as engine
|
||||
|
||||
cfg = tmp_path / "config.json"
|
||||
cfg.write_text(json.dumps({"db_path": str(tmp_path / "jarvis.db")}))
|
||||
monkeypatch.setenv("JARVIS_CONFIG_PATH", str(cfg))
|
||||
seen = {"judged": [], "prepared": []}
|
||||
|
||||
def fake_prepare(cfg, text, mem, chat_available):
|
||||
seen["judged"].append((text, chat_available))
|
||||
return plans.get(text)
|
||||
|
||||
def fake_engine(db, cfg, tts, text, mem, prepared_tools=None, intent_judge=False, **kw):
|
||||
seen["prepared"].append(prepared_tools)
|
||||
seen["intent_judge"] = intent_judge
|
||||
mem.add_message("user", text)
|
||||
mem.add_message("assistant", "ok")
|
||||
return "ok"
|
||||
|
||||
monkeypatch.setattr(engine, "prepare_intent", fake_prepare)
|
||||
monkeypatch.setattr(engine, "run_reply_engine", fake_engine)
|
||||
return brain_mod.JarvisBrain(), seen
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_reply_uses_the_pick_prepared_for_about_the_same_text(tmp_path, monkeypatch):
|
||||
from jarvis.memory.history import HistoryContext
|
||||
|
||||
plan = ["getWeather"]
|
||||
b, seen = _jarvis_brain(tmp_path, monkeypatch, {"자비스 오늘 서울 날씨 어": plan})
|
||||
b.prepare("100", "자비스 오늘 서울 날씨 어", True)
|
||||
assert seen["judged"] == [("자비스 오늘 서울 날씨 어", True)]
|
||||
b.reply("100", "자비스, 오늘 서울 날씨 어때?", HistoryContext())
|
||||
assert seen["prepared"] == [plan] and seen["intent_judge"] is True
|
||||
# Used once: the next reply judges for itself.
|
||||
b.reply("100", "자비스, 오늘 서울 날씨 어때?", HistoryContext())
|
||||
assert seen["prepared"][-1] is None
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_prepared_pick_is_not_used_for_other_words_another_person_or_a_changed_dialogue(tmp_path, monkeypatch):
|
||||
from jarvis.memory.history import HistoryContext
|
||||
|
||||
plan = ["webSearch"]
|
||||
b, seen = _jarvis_brain(tmp_path, monkeypatch, {"자비스 오늘 서울 날씨 어때": plan})
|
||||
b.prepare("100", "자비스 오늘 서울 날씨 어때", False)
|
||||
b.reply("100", "자비스, 내일 회의 몇 시지?", HistoryContext())
|
||||
b.prepare("100", "자비스 오늘 서울 날씨 어때", False)
|
||||
b.reply("200", "자비스 오늘 서울 날씨 어때", HistoryContext())
|
||||
assert seen["prepared"] == [None, None]
|
||||
b.prepare("100", "자비스 오늘 서울 날씨 어때", False)
|
||||
b._memory("100").add_message("user", "something said meanwhile")
|
||||
b.reply("100", "자비스 오늘 서울 날씨 어때", HistoryContext())
|
||||
assert seen["prepared"][-1] is None
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_reply_waits_for_a_pick_still_being_prepared_for_its_text(tmp_path, monkeypatch):
|
||||
import threading
|
||||
import time as _time
|
||||
from jarvis.memory.history import HistoryContext
|
||||
|
||||
plan = [] # "no tools" is a pick too
|
||||
b, seen = _jarvis_brain(tmp_path, monkeypatch, {})
|
||||
import jarvis.reply.engine as engine
|
||||
|
||||
def slow_prepare(cfg, text, mem, chat_available):
|
||||
_time.sleep(0.4)
|
||||
return plan
|
||||
monkeypatch.setattr(engine, "prepare_intent", slow_prepare)
|
||||
t = threading.Thread(target=b.prepare, args=("100", "자비스 날씨 어때", False))
|
||||
t.start()
|
||||
_time.sleep(0.1)
|
||||
b.reply("100", "자비스 날씨 어때?", HistoryContext())
|
||||
t.join()
|
||||
assert seen["prepared"] == [plan]
|
||||
|
||||
Reference in New Issue
Block a user