perf(reply): v2 tool turns get the tool guidance in short, and a shorter reply-language default
Strata reads every reply prompt from the start in 2,048-token steps. The v2 compact prompt carried v1's full tool guidance and constraints (about 900 Qwen3.6 tokens) on every turn with a data tool, and the reply-language default (about 120 tokens) twice. COMPACT_TOOL_GUIDANCE keeps the same rules (act instead of refusing, facts only from the result, look up named things, auto-derived and self-contained arguments) in about a quarter of the tokens, without English examples; the reply-language default says the same in half. v1's prompts are unchanged. Measured on Strata (Qwen3.6, 20 lines of recent conversation): a weather turn's prompt 3,436 -> 2,722 tokens, a search turn's second call 4,096 -> 3,383. Evals (compact prompt + intent judge, Qwen3.6): 24/24 before, 30/30 twice after (weather location, follow-up arguments, tool-skip confabulation, search language, reply language, Possessor fidelity, reply length, multi-turn context). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -15,7 +15,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
|
||||
- Redacted user query
|
||||
- Recent dialogue (last 5 minutes), including in-loop tool-call + tool-role messages from prior replies within the active conversation (tool carryover, `DialogueMemory.record_tool_turn` / `get_recent_turns_with_tools` in [src/jarvis/memory/conversation.py](src/jarvis/memory/conversation.py); per-prompt cap via `cfg.tool_carryover_max_turns` / `tool_carryover_per_entry_chars`; storage cap `_tool_turns_max_storage = 16`; cleared on `stop` signal AND on new-conversation entry; UNTRUSTED WEB EXTRACT fence markers preserved on truncation; both `content` and `tool_calls[*].function.arguments` scrubbed on write)
|
||||
- Unified system prompt from [src/jarvis/system_prompt.py](src/jarvis/system_prompt.py) + ASR note + tool-protocol guidance. Reply language is resolved by `reply_language_directive(lang, cfg.tts_engine)` where `lang = _resolve_output_language()` — the single source of truth that prefers the settings-web UI value (config JSON `output_language`) over the compose `OUTPUT_LANGUAGE` env, so changing the language in the settings page takes effect. An explicit lock wins (forces "reply only in `<language>`", also forbidding other scripts so small models stop leaking trailing CJK/Hanja); else a Piper/Chatterbox TTS forces English (English-only voices); else (multilingual TTS, no lock) the assistant replies in the user's own language. The v2 voice worker's brain (`src/voice_worker/brain.py`) runs with `tts_engine="melo"`, so v1's English-only Piper default never locks v2 voice or chat answers to English, and passes `default_language` (worker `--reply-language`, default `Korean`): with no lock, `default_language_directive()` makes replies come in that language unless the user explicitly asks for another one, and `build_system_prompt()` rewrites the persona's "in the user's language" clause to "in the reply language" (eval `evals/test_reply_language_default.py`). The directive is inserted near the FRONT of the guidance list so a small model gives it primacy, and the SAME resolved `lang` feeds `build_system_prompt()`, which rewrites the persona's "in the user's language" clause to the locked language so the persona cannot contradict the directive (previously the persona read the raw env while the directive read the config value, so a settings-UI change was honoured by one and ignored by the other). Gated in `_build_initial_system_message()` at [engine.py](src/jarvis/reply/engine.py).
|
||||
- **v2 compact prompt** (`compact_prompt=True`, the v2 brain): `build_voice_persona()` replaces the v1 persona (about 1.3k characters instead of 7.4k: no single-user long-term memory or on-screen browser rules, answers as short as possible), and of the size-specific components only the ASR note stays; the full tool guidance (`tool_guidance` + `tool_constraints`) is added only when a tool other than `stop`, `toolSearchTool` and `chatChannels` is allowed, else the one-line `COMPACT_ACTION_RULE`. Measured on Strata (Qwen3.6, 2026-11-12): a greeting's reply prompt went from about 4,500 to 1,350 tokens and its prompt read from 1.07 s to 0.36 s (Strata reads in 2,048-token steps). The v2 brain also turns `stream_browser` off. The `[Context: ...]` time carries the 12-hour form beside the 24-hour one (`10:37 (10:37 AM)`): with the short prompt Qwen3.6 read 10:37 as 오후 (pm) 4 of 4 times.
|
||||
- **v2 compact prompt** (`compact_prompt=True`, the v2 brain): `build_voice_persona()` replaces the v1 persona (about 1.3k characters instead of 7.4k: no single-user long-term memory or on-screen browser rules, answers as short as possible), and of the size-specific components only the ASR note stays; the tool guidance is added only when a tool other than `stop`, `toolSearchTool` and `chatChannels` is allowed, as `COMPACT_TOOL_GUIDANCE` (v1's `tool_guidance` + `tool_constraints` rules in about a quarter of the tokens), else the one-line `COMPACT_ACTION_RULE`; the reply-language default (`default_language_directive`, at the front and the end) is short too. Measured on Strata (Qwen3.6, 2026-11-16, four lines of recent conversation): a weather turn's reply prompt went from 3,075 to 2,360 tokens and a time question's from 2,115 to 2,031, inside one 2,048-token step (Strata reads about 0.44 s up to 2,048 tokens, 0.86 s up to 4,096, 1.25 s up to 6,144). Measured on Strata (Qwen3.6, 2026-11-12): a greeting's reply prompt went from about 4,500 to 1,350 tokens and its prompt read from 1.07 s to 0.36 s (Strata reads in 2,048-token steps). The v2 brain also turns `stream_browser` off. The `[Context: ...]` time carries the 12-hour form beside the 24-hour one (`10:37 (10:37 AM)`): with the short prompt Qwen3.6 read 10:37 as 오후 (pm) 4 of 4 times.
|
||||
- **Tool honesty rules**: `TOOL_HONESTY_RULES` inside the persona, plus `tool_honesty_reminder(allowed_tools)` appended after every context block (just before the closing reply-language lock) naming the turn's tools. Added after M0 (2026-11-10) caught models answering past an available tool.
|
||||
- **Operator instructions** (two sources, both framed "Additional instructions from the operator:" and appended near the end of the guidance list): the settings-UI `llm_instructions` config field, and every `*.md` file in `AGENTS_DIR` (default `/app/agents`, bind-mounted from `./agents`). The file-based set is read once per turn by `load_agent_instructions()` in [src/jarvis/system_prompt.py](src/jarvis/system_prompt.py) and concatenated in filename order, so dropping/editing a `.md` applies on the next reply with no rebuild/restart; fail-open to `""` when the folder is absent/empty/unreadable.
|
||||
- **Warm profile block** (query-agnostic User + Directives excerpt from the knowledge graph, composed by `build_warm_profile()` / `format_warm_profile_block()` in [src/jarvis/memory/graph_ops.py](src/jarvis/memory/graph_ops.py) at Step 3.5 of `reply()`; no LLM call, pure SQLite read; injected unconditionally so personalisation is the default; result cached in `DialogueMemory._hot_cache` under `DialogueMemory.WARM_PROFILE_CACHE_KEY` for the lifetime of the active conversation. Invalidated on `stop`, on new-conversation entry, AND on User/Directives graph mutations via the listener registered in [src/jarvis/daemon.py](src/jarvis/daemon.py) against `register_graph_mutation_listener` in [src/jarvis/memory/graph.py](src/jarvis/memory/graph.py); World-branch writes are ignored)
|
||||
|
||||
@@ -13,6 +13,7 @@ from ..system_prompt import (
|
||||
build_system_prompt,
|
||||
build_voice_persona,
|
||||
COMPACT_ACTION_RULE,
|
||||
COMPACT_TOOL_GUIDANCE,
|
||||
load_agent_instructions,
|
||||
reply_language_directive,
|
||||
tool_honesty_reminder,
|
||||
@@ -1941,9 +1942,10 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
||||
# The persona already covers voice style, inference and tool use; the tool guidance
|
||||
# (named entities, auto-derived and self-contained arguments) matters only when a
|
||||
# data tool is on offer. The sentinels and the always-offered chat tool are not.
|
||||
# It is v1's guidance in short (COMPACT_TOOL_GUIDANCE): every token is read each reply.
|
||||
guidance.append(prompts.asr_note)
|
||||
if any(t not in ("stop", "toolSearchTool", "chatChannels") for t in allowed_tools):
|
||||
guidance.extend(c for c in (prompts.tool_guidance, prompts.tool_constraints) if c)
|
||||
guidance.append(COMPACT_TOOL_GUIDANCE)
|
||||
else:
|
||||
guidance.append(COMPACT_ACTION_RULE)
|
||||
else:
|
||||
|
||||
@@ -209,6 +209,27 @@ COMPACT_ACTION_RULE: str = (
|
||||
)
|
||||
|
||||
|
||||
# Compact prompts, turns with a data tool: v1's tool guidance and constraints (about 900 Qwen3.6
|
||||
# tokens) in a quarter of the words. Strata reads prompts in 2,048-token steps of about 0.4 s each,
|
||||
# so these tokens decide whether a tool turn with recent conversation stays inside one step fewer.
|
||||
# The same rules as v1's (acting instead of refusing, facts only from the result, looking up named
|
||||
# things, auto-derived and self-contained arguments), without English examples.
|
||||
COMPACT_TOOL_GUIDANCE: str = (
|
||||
"Using tools: when a tool here covers the request, call it straight away, without announcing it "
|
||||
"or asking permission; when none fits an action or a live fact, call toolSearchTool with a short "
|
||||
"description before saying you cannot. Answer from tool results, never about them: every "
|
||||
"specific fact in the reply (names, dates, numbers, places, cast, plot) must come from the tool "
|
||||
"result or the user's own words, not from your prior knowledge; if the result lacks it, say so. "
|
||||
"When a search result has a 'Content from top result' section, give its facts, not the links. "
|
||||
"A specific named thing (a film, song, game, product, person, place or event) you cannot state "
|
||||
"verified facts about is looked up before you answer. When a tool fills an argument itself (for "
|
||||
"example the current location), call it with what you have instead of asking; a value the user "
|
||||
"already gave in the conversation is passed. Free-text arguments such as search queries must "
|
||||
"stand on their own, because the tool does not see earlier turns: resolve pronouns and "
|
||||
"references into a short keyword phrase."
|
||||
)
|
||||
|
||||
|
||||
def tool_honesty_reminder(tool_names) -> str:
|
||||
"""End-of-prompt restatement of the tool honesty rules for this turn.
|
||||
|
||||
@@ -307,14 +328,12 @@ def default_language_directive(language: Optional[str]) -> Optional[str]:
|
||||
lang = (language or "").strip()
|
||||
if not lang:
|
||||
return None
|
||||
# Short on purpose: the reply engine puts it at the front and the end of every v2 prompt.
|
||||
return (
|
||||
f"REPLY LANGUAGE: write your reply in {lang} by default, whatever language the user's "
|
||||
f"message or any reference text is in. The one exception overrides this default: when "
|
||||
f"the user explicitly asks, in any language, for the answer in another language (for "
|
||||
f"example 'answer in English'), reply in that language, for just this reply or from now "
|
||||
f"on in this conversation, as they asked. Do not mix in other scripts: no Chinese/Hanja characters, "
|
||||
f"Japanese kana or Cyrillic letters inside a {lang} reply. (Numerals and unavoidable "
|
||||
f"proper-noun brand names are fine.)"
|
||||
f"REPLY LANGUAGE: reply in {lang}, whatever language the message or any reference text is "
|
||||
f"in, unless the user explicitly asks, in any language, for another language: then use that "
|
||||
f"one, for this reply or from then on, as they asked. No Chinese/Hanja, Japanese kana or "
|
||||
f"Cyrillic inside a {lang} reply; numerals and brand names are fine."
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -267,8 +267,8 @@ which carry Discord's own time.
|
||||
Two reply options keep the time to the first word short (Strata reads every prompt from the
|
||||
start, about 0.36 s per 2,048 tokens, and each LLM call before the answer adds its whole time):
|
||||
the short voice persona (`compact_prompt=True`: the same butler in a tenth of v1's words, without
|
||||
v1's single-user memory and on-screen browser rules, answers as short as possible, the full tool
|
||||
guidance only on turns that offer a data tool), and the intent judge (`intent_judge=True`: the tool
|
||||
v1's single-user memory and on-screen browser rules, answers as short as possible, v1's tool
|
||||
guidance in short only on turns that offer a data tool), and the intent judge (`intent_judge=True`: the tool
|
||||
router judges whether the turn needs tools and which, and the planner runs only when it does
|
||||
and the pick is not a lone one-step tool;
|
||||
`planner.spec.md`, "Intent judge (v2)"). v1's screen-share browser (`stream_browser`) is off: it
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Strata reads every prompt from the start, so prompt length is reply latency. Behaviour: the compact
|
||||
prompt is far shorter than v1's for the same turn, still carries the tool honesty rules and the
|
||||
reply-language default, and brings the full tool guidance only on turns that offer a data tool.
|
||||
reply-language default, and brings the (short) tool guidance only on turns that offer a data tool.
|
||||
"""
|
||||
|
||||
from unittest.mock import Mock, patch
|
||||
@@ -13,7 +13,9 @@ from src.jarvis.memory.conversation import DialogueMemory
|
||||
from src.jarvis.memory.history import HistoryContext
|
||||
from src.jarvis.reply.engine import run_reply_engine
|
||||
from src.jarvis.reply.prompts.model_variants import TOOL_CONSTRAINTS_LARGE
|
||||
from src.jarvis.system_prompt import COMPACT_ACTION_RULE, TOOL_HONESTY_RULES, default_language_directive
|
||||
from src.jarvis.system_prompt import (
|
||||
COMPACT_ACTION_RULE, COMPACT_TOOL_GUIDANCE, TOOL_HONESTY_RULES, default_language_directive,
|
||||
)
|
||||
|
||||
from tests.test_ollama_num_predict import _mock_cfg
|
||||
|
||||
@@ -46,11 +48,20 @@ def test_the_compact_prompt_is_far_shorter_and_keeps_the_rules(routed):
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_the_full_tool_guidance_comes_only_with_a_data_tool():
|
||||
def test_the_tool_guidance_comes_only_with_a_data_tool():
|
||||
chatty = _system_prompt(True, [])
|
||||
assert COMPACT_ACTION_RULE in chatty and TOOL_CONSTRAINTS_LARGE not in chatty
|
||||
assert COMPACT_ACTION_RULE in chatty and COMPACT_TOOL_GUIDANCE not in chatty
|
||||
weather = _system_prompt(True, ["getWeather"])
|
||||
assert TOOL_CONSTRAINTS_LARGE in weather and COMPACT_ACTION_RULE not in weather
|
||||
assert COMPACT_TOOL_GUIDANCE in weather and COMPACT_ACTION_RULE not in weather
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_a_data_tool_turn_gets_the_short_tool_guidance_not_v1s():
|
||||
# v1 keeps its full guidance; the compact prompt says the same rules in a fraction of it.
|
||||
assert TOOL_CONSTRAINTS_LARGE in _system_prompt(False, ["getWeather"])
|
||||
weather = _system_prompt(True, ["getWeather"])
|
||||
assert TOOL_CONSTRAINTS_LARGE not in weather
|
||||
assert len(COMPACT_TOOL_GUIDANCE) < len(TOOL_CONSTRAINTS_LARGE) / 2
|
||||
|
||||
|
||||
def _intent_turn(prepared, routed=("getWeather",), first=None):
|
||||
|
||||
Reference in New Issue
Block a user