feat(brain): make OUTPUT_LANGUAGE lock robust on small models
Harden the reply-language lock so qwen2.5:3b reliably stays in the locked language instead of leaking the query language back in: - reply_language_directive(): single resolver with clear precedence — explicit OUTPUT_LANGUAGE lock wins over the Piper/Chatterbox English-only fallback (this deployment's actual TTS is Korean MeloTTS, so the legacy English lock was both wrong and contradicting the Korean lock). - Stronger, override-explicit directive wording, inserted near the FRONT of the system prompt so a small model gives it primacy over the persona. - build_system_prompt(output_language=...): rewrite the persona's "in the user's language" clause to the locked language so the persona stops fighting the lock. - docs/llm_contexts.md: document the resolver, precedence, and placement. Live-verified on the running brain (qwen2.5:3b): Korean voice-style input and a cold English query both return fully Korean replies with no CJK/Hanja leak. Tests cover unset/set/agnostic/whitespace + precedence + persona rewrite.
This commit is contained in:
javis-bot
@@ -12,7 +12,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
|
||||
- **Inputs**:
|
||||
- Redacted user query
|
||||
- Recent dialogue (last 5 minutes), including in-loop tool-call + tool-role messages from prior replies within the active conversation (tool carryover, `DialogueMemory.record_tool_turn` / `get_recent_turns_with_tools` in [src/jarvis/memory/conversation.py](src/jarvis/memory/conversation.py); per-prompt cap via `cfg.tool_carryover_max_turns` / `tool_carryover_per_entry_chars`; storage cap `_tool_turns_max_storage = 16`; cleared on `stop` signal AND on new-conversation entry; UNTRUSTED WEB EXTRACT fence markers preserved on truncation; both `content` and `tool_calls[*].function.arguments` scrubbed on write)
|
||||
- Unified system prompt from [src/jarvis/system_prompt.py](src/jarvis/system_prompt.py) + ASR note + tool-protocol guidance. When the `OUTPUT_LANGUAGE` env var is set, `output_language_directive()` appends a "respond only in `<language>`" instruction (also forbids other scripts, suppressing trailing CJK leakage on small models); empty (default) keeps the multilingual "reply in the user's language" behaviour. Gated in `_build_initial_system_message()` at [engine.py](src/jarvis/reply/engine.py) alongside the TTS English-only lock.
|
||||
- Unified system prompt from [src/jarvis/system_prompt.py](src/jarvis/system_prompt.py) + ASR note + tool-protocol guidance. Reply language is resolved by `reply_language_directive(OUTPUT_LANGUAGE, cfg.tts_engine)`: an explicit `OUTPUT_LANGUAGE` env lock wins (forces "reply only in `<language>`", also forbidding other scripts so small models stop leaking trailing CJK/Hanja); else a Piper/Chatterbox TTS forces English (English-only voices); else (multilingual TTS, no lock) the assistant replies in the user's own language. The directive is inserted near the FRONT of the guidance list so a small model gives it primacy, and when the lock is set `build_system_prompt()` also rewrites the persona's "in the user's language" clause to the locked language so the persona does not contradict the lock. Gated in `_build_initial_system_message()` at [engine.py](src/jarvis/reply/engine.py).
|
||||
- **Warm profile block** (query-agnostic User + Directives excerpt from the knowledge graph, composed by `build_warm_profile()` / `format_warm_profile_block()` in [src/jarvis/memory/graph_ops.py](src/jarvis/memory/graph_ops.py) at Step 3.5 of `reply()`; no LLM call, pure SQLite read; injected unconditionally so personalisation is the default; result cached in `DialogueMemory._hot_cache` under `DialogueMemory.WARM_PROFILE_CACHE_KEY` for the lifetime of the active conversation. Invalidated on `stop`, on new-conversation entry, AND on User/Directives graph mutations via the listener registered in [src/jarvis/daemon.py](src/jarvis/daemon.py) against `register_graph_mutation_listener` in [src/jarvis/memory/graph.py](src/jarvis/memory/graph.py); World-branch writes are ignored)
|
||||
- Digested memory enrichment (optional, see #4)
|
||||
- Time + location context (re-injected each turn)
|
||||
|
||||
@@ -9,7 +9,7 @@ import os
|
||||
from typing import Optional, TYPE_CHECKING
|
||||
|
||||
from ..utils.redact import redact
|
||||
from ..system_prompt import build_system_prompt, output_language_directive
|
||||
from ..system_prompt import build_system_prompt, reply_language_directive
|
||||
from ..tools.registry import run_tool_with_retries, generate_tools_description, generate_tools_json_schema, BUILTIN_TOOLS
|
||||
from ..tools.builtin.stop import STOP_SIGNAL
|
||||
from ..debug import debug_log
|
||||
@@ -1425,7 +1425,7 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
||||
action_plan = strip_memory_directives(action_plan)
|
||||
|
||||
_assistant_name = str(getattr(cfg, "wake_word", "jarvis") or "jarvis").strip().capitalize()
|
||||
_persona_prompt = build_system_prompt(_assistant_name)
|
||||
_persona_prompt = build_system_prompt(_assistant_name, os.environ.get("OUTPUT_LANGUAGE"))
|
||||
|
||||
def _build_initial_system_message() -> str:
|
||||
guidance = [_persona_prompt.strip()]
|
||||
@@ -1433,22 +1433,19 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
||||
# Add model-size-appropriate prompt components
|
||||
guidance.extend(prompts.to_list())
|
||||
|
||||
# Both current TTS engines (Piper, Chatterbox) only support English.
|
||||
# Responding in another language would produce garbled audio.
|
||||
# Remove this constraint when a multilingual TTS engine is added.
|
||||
tts_engine = getattr(cfg, 'tts_engine', 'piper')
|
||||
if tts_engine in ('piper', 'chatterbox'):
|
||||
guidance.append(
|
||||
"Always respond in English regardless of the language the user speaks in."
|
||||
)
|
||||
|
||||
# Deployment-level output-language lock (OUTPUT_LANGUAGE). When set,
|
||||
# force every reply into that single language and forbid stray
|
||||
# characters from other scripts. Empty (default) keeps the
|
||||
# multilingual behaviour of replying in the user's own language.
|
||||
_lang_directive = output_language_directive(os.environ.get("OUTPUT_LANGUAGE"))
|
||||
# Reply-language policy: an explicit OUTPUT_LANGUAGE lock wins, else
|
||||
# Piper/Chatterbox TTS forces English (English-only voices), else the
|
||||
# assistant replies in the user's own language. See
|
||||
# reply_language_directive() for the precedence rationale.
|
||||
# Placed at the FRONT (after the persona header) so a small model gives
|
||||
# it primacy over the persona's "use the user's language" lines — a tail
|
||||
# instruction loses to those when the query itself is in another language.
|
||||
_lang_directive = reply_language_directive(
|
||||
os.environ.get("OUTPUT_LANGUAGE"),
|
||||
getattr(cfg, "tts_engine", "piper"),
|
||||
)
|
||||
if _lang_directive:
|
||||
guidance.append(_lang_directive)
|
||||
guidance.insert(1, _lang_directive)
|
||||
|
||||
if warm_profile_block:
|
||||
# Pre-query, query-agnostic user context. Lives OUTSIDE the
|
||||
|
||||
@@ -81,14 +81,26 @@ _SYSTEM_PROMPT_TEMPLATE: str = (
|
||||
)
|
||||
|
||||
|
||||
def build_system_prompt(assistant_name: str = "Jarvis") -> str:
|
||||
def build_system_prompt(
|
||||
assistant_name: str = "Jarvis", output_language: Optional[str] = None
|
||||
) -> str:
|
||||
"""Render the persona prompt with the configured assistant name.
|
||||
|
||||
The name comes from the user's wake word (capitalised); defaults to
|
||||
"Jarvis" when no config is available (tests, eval harnesses).
|
||||
|
||||
When ``output_language`` is set (a single-language deployment), the
|
||||
persona's "reply in the user's language" clause is rewritten to that
|
||||
language so the persona does not contradict the OUTPUT_LANGUAGE lock — a
|
||||
small model otherwise honours the persona's instruction to mirror the
|
||||
query language and leaks the other language back in.
|
||||
"""
|
||||
name = (assistant_name or "Jarvis").strip() or "Jarvis"
|
||||
return _SYSTEM_PROMPT_TEMPLATE.format(name=name)
|
||||
prompt = _SYSTEM_PROMPT_TEMPLATE.format(name=name)
|
||||
lang = (output_language or "").strip()
|
||||
if lang:
|
||||
prompt = prompt.replace("in the user's language", f"in {lang}")
|
||||
return prompt
|
||||
|
||||
|
||||
def output_language_directive(language: Optional[str]) -> Optional[str]:
|
||||
@@ -108,7 +120,41 @@ def output_language_directive(language: Optional[str]) -> Optional[str]:
|
||||
if not lang:
|
||||
return None
|
||||
return (
|
||||
f"Always respond only in {lang}, regardless of the language the user "
|
||||
f"writes in. Do not mix in words, characters, or punctuation from any "
|
||||
f"other language or script."
|
||||
f"CRITICAL OUTPUT RULE: write your ENTIRE reply only in {lang}. Even if "
|
||||
f"the user writes in English or any other language, you must still reply "
|
||||
f"only in {lang}. This rule overrides every other instruction about "
|
||||
f"matching or using the user's language. Never mix in words, characters, "
|
||||
f"or punctuation from any other language or script."
|
||||
)
|
||||
|
||||
|
||||
# TTS engines that can only synthesise English. Replying in another language
|
||||
# with one of these produces garbled audio, so those deployments force English.
|
||||
_TTS_ENGLISH_ONLY = frozenset({"piper", "chatterbox"})
|
||||
|
||||
# Kept verbatim for backward compatibility with anything asserting on the wording.
|
||||
ENGLISH_ONLY_DIRECTIVE = (
|
||||
"Always respond in English regardless of the language the user speaks in."
|
||||
)
|
||||
|
||||
|
||||
def reply_language_directive(
|
||||
output_language: Optional[str], tts_engine: Optional[str]
|
||||
) -> Optional[str]:
|
||||
"""Resolve the reply-language instruction for the chat loop, or None.
|
||||
|
||||
Precedence:
|
||||
1. An explicit ``output_language`` lock wins — the deployment serves a
|
||||
single language and owns a TTS voice that can speak it (e.g. Korean
|
||||
MeloTTS). This intentionally overrides the English-only fallback.
|
||||
2. Otherwise, a Piper/Chatterbox TTS engine can only synthesise English,
|
||||
so force English to avoid garbled audio.
|
||||
3. Otherwise (multilingual TTS, no lock) → None: the assistant replies in
|
||||
the user's own language.
|
||||
"""
|
||||
forced = output_language_directive(output_language)
|
||||
if forced:
|
||||
return forced
|
||||
if (tts_engine or "piper").strip().lower() in _TTS_ENGLISH_ONLY:
|
||||
return ENGLISH_ONLY_DIRECTIVE
|
||||
return None
|
||||
|
||||
@@ -5,7 +5,12 @@ wake word to e.g. "Friday" produces a butler named Friday, not one still
|
||||
hardcoded to Jarvis.
|
||||
"""
|
||||
|
||||
from jarvis.system_prompt import build_system_prompt, output_language_directive
|
||||
from jarvis.system_prompt import (
|
||||
build_system_prompt,
|
||||
output_language_directive,
|
||||
reply_language_directive,
|
||||
ENGLISH_ONLY_DIRECTIVE,
|
||||
)
|
||||
|
||||
|
||||
class TestBuildSystemPrompt:
|
||||
@@ -27,6 +32,17 @@ class TestBuildSystemPrompt:
|
||||
assert "named Jarvis" in build_system_prompt(" ")
|
||||
assert "named Jarvis" in build_system_prompt(None) # type: ignore[arg-type]
|
||||
|
||||
def test_default_keeps_user_language_clause(self):
|
||||
# Without a lock, the persona still mirrors the user's language.
|
||||
assert "in the user's language" in build_system_prompt("Jarvis")
|
||||
|
||||
def test_language_lock_rewrites_user_language_clause(self):
|
||||
# With a lock, the contradicting "user's language" clause is rewritten
|
||||
# so the persona does not fight the OUTPUT_LANGUAGE directive.
|
||||
prompt = build_system_prompt("Jarvis", "Korean")
|
||||
assert "in the user's language" not in prompt
|
||||
assert "in Korean" in prompt
|
||||
|
||||
|
||||
class TestOutputLanguageDirective:
|
||||
"""A deployment may lock replies to a single language via OUTPUT_LANGUAGE.
|
||||
@@ -58,3 +74,37 @@ class TestOutputLanguageDirective:
|
||||
assert directive is not None
|
||||
assert "Korean" in directive
|
||||
assert " Korean" not in directive
|
||||
|
||||
|
||||
class TestReplyLanguageDirective:
|
||||
"""Precedence: explicit OUTPUT_LANGUAGE lock > English-only TTS > free.
|
||||
|
||||
The lock must override the Piper/Chatterbox English fallback, because a
|
||||
deployment that sets OUTPUT_LANGUAGE (e.g. Korean) also runs a TTS voice
|
||||
that can speak it. Without this, the English lock and the Korean lock
|
||||
contradict each other and the model reverts to English.
|
||||
"""
|
||||
|
||||
def test_lock_overrides_english_only_tts(self):
|
||||
directive = reply_language_directive("Korean", "piper")
|
||||
assert directive is not None
|
||||
assert "Korean" in directive
|
||||
assert directive != ENGLISH_ONLY_DIRECTIVE
|
||||
|
||||
def test_english_only_tts_forces_english_without_lock(self):
|
||||
assert reply_language_directive(None, "piper") == ENGLISH_ONLY_DIRECTIVE
|
||||
assert reply_language_directive("", "chatterbox") == ENGLISH_ONLY_DIRECTIVE
|
||||
|
||||
def test_no_lock_multilingual_tts_is_free(self):
|
||||
# A non-English-only engine (e.g. melo) with no lock → reply in the
|
||||
# user's own language, so no directive.
|
||||
assert reply_language_directive(None, "melo") is None
|
||||
|
||||
def test_unknown_tts_defaults_to_english_only(self):
|
||||
# Preserves the original getattr(cfg, 'tts_engine', 'piper') default:
|
||||
# an unknown/missing engine is treated conservatively as English-only.
|
||||
assert reply_language_directive(None, None) == ENGLISH_ONLY_DIRECTIVE
|
||||
|
||||
def test_lock_wins_even_with_multilingual_tts(self):
|
||||
directive = reply_language_directive("Korean", "melo")
|
||||
assert directive is not None and "Korean" in directive
|
||||
|
||||
Reference in New Issue
Block a user