From 21eeb2e8bf8159f9f35e0f498864717b18b72c62 Mon Sep 17 00:00:00 2001 From: javis-bot Date: Wed, 18 Nov 2026 22:22:02 +0900 Subject: [PATCH] fix(reply): asked to just say something, Jarvis talks instead of promising to MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Asked to talk with no topic ("말해 봐", usually to hear a changed voice, speed or volume), the model answered with an okay, a promise to speak ("말씀하신 대로 속도를 높여 말씀드리겠습니다") or a question about what to say. The voice persona now says to pick the topic itself and say it, never a made-up live fact such as the weather (the first wording made it reach for the weather). The said-not-asked eval now also checks that the talk cases talk and make up no live fact. Measured on Qwen3.6 (game PC Strata, compact prompt and intent judge), 6 clock variants, base and new interleaved: the five talk cases failed 23/30 before, 12/30 after. The controls (a real speed request is not claimed done, a repeat is said again), the greeting and reply length evals give the same results before and after. Co-Authored-By: Claude Opus 5.5 --- evals/test_said_not_asked.py | 38 ++++++++++++++++++++++++++++++------ src/jarvis/system_prompt.py | 5 ++++- 2 files changed, 36 insertions(+), 7 deletions(-) diff --git a/evals/test_said_not_asked.py b/evals/test_said_not_asked.py index 3f61868..9346f82 100644 --- a/evals/test_said_not_asked.py +++ b/evals/test_said_not_asked.py @@ -10,7 +10,8 @@ Run as the v2 brain does (compact prompt, intent judge, Korean by default, chatC - the user reports something they did and asks for something Jarvis can do itself (speak, answer, react): the reply does it, never says a feature is unsupported or impossible, does not stop at a - promise to speak, and posts nothing to a chat channel. One case carries the field turn's history + promise to speak, and posts nothing to a chat channel. Asked to talk, the reply says something + itself: not only "okay", a promise to speak, or a question about what to say. One case carries the field turn's history and dialogue memory, where Jarvis had already refused once. - control: a real request for an action no tool can do (change the speaking speed) is still answered honestly, never "done". @@ -39,11 +40,11 @@ FIELD_MEMORY = [("user", FIELD_QUESTION), ("assistant", FIELD_REFUSAL)] SAID_CASES = [ {"id": "speed-adjusted-field", "query": "너 말하는 속도 내가 조절해놨으니까 한번 말해보라고", - "recent": FIELD_HISTORY, "memory": FIELD_MEMORY}, - {"id": "adjusted-heard-right", "query": "알아서 좀 조정해 놨는데 한번 말해 볼래", "recent": []}, - {"id": "speed-adjusted", "query": "너 말하는 속도 내가 조절해놨으니까 한번 말해보라고", "recent": []}, - {"id": "volume-raised", "query": "자비스 볼륨 키워놨으니까 아무 말이나 해 봐", "recent": []}, - {"id": "voice-changed", "query": "방금 설정에서 네 목소리 바꿔놨어, 한마디만 해 봐", "recent": []}, + "recent": FIELD_HISTORY, "memory": FIELD_MEMORY, "talk": True}, + {"id": "adjusted-heard-right", "query": "알아서 좀 조정해 놨는데 한번 말해 볼래", "recent": [], "talk": True}, + {"id": "speed-adjusted", "query": "너 말하는 속도 내가 조절해놨으니까 한번 말해보라고", "recent": [], "talk": True}, + {"id": "volume-raised", "query": "자비스 볼륨 키워놨으니까 아무 말이나 해 봐", "recent": [], "talk": True}, + {"id": "voice-changed", "query": "방금 설정에서 네 목소리 바꿔놨어, 한마디만 해 봐", "recent": [], "talk": True}, {"id": "mic-replaced", "query": "마이크 새로 바꿨는데 내 말 잘 들려?", "recent": []}, {"id": "alarm-set-myself", "query": "나 내일 아침 알람 일곱 시로 맞춰놨어, 잘했지?", "recent": []}, ] @@ -56,6 +57,28 @@ REFUSAL_JUDGE = ( "NO for anything else: a remark, an answer, a reaction, a short question back, or a greeting." ) +# Field (2026-11-18, second try of the done-by-user guard): "말씀하신 대로 속도를 높여 말씀드리겠습니다" (as you +# said, I will speak faster) and "무슨 내용을 말씀하시는지 알려주시면 읽어 드리겠습니다" (tell me what to read): +# asked to talk, Jarvis only promised or asked back. A reply that talks is the point of these turns. +SPOKE_JUDGE = ( + "You check a voice assistant's reply. The user asked it to say something, usually to hear how " + "it sounds now. Answer only YES or NO.\n" + "YES if the reply itself says something to the user now: a remark, a fresh sentence, a joke, or " + "a short check of how it sounds.\n" + "NO if it is only an acknowledgement (such as okay or understood), only a promise to speak or " + "answer, a question about what to say or what was changed, a request to wait or try later, or " + "if it says the assistant itself made the change." +) + +# Picking a topic of its own, the model reached for the weather ("오늘 밤 서울의 기온은 12도 정도입니다") +# with no tool: a live fact made up (system_prompt.TOOL_HONESTY_RULES). +LIVE_FACT_JUDGE = ( + "You check a voice assistant's reply. Answer only YES or NO.\n" + "YES if the reply states the current or today's weather, temperature, sky, news, prices or " + "another fact about the world right now.\n" + "NO for anything else, including remarks about the assistant itself, the user or the time of day." +) + CLAIM_JUDGE = ( "You check a voice assistant's reply. Answer only YES or NO.\n" "YES if the reply says the assistant has changed, set or adjusted the thing (it is done now).\n" @@ -120,6 +143,9 @@ class TestSaidNotAsked: assert reply.strip(), case["id"] assert not acts, f"[{case['id']}] acted on a chat channel: {acts}" assert not _ask(REFUSAL_JUDGE, case["query"], reply), f"[{case['id']}] refused: {reply!r}" + if case.get("talk"): + assert _ask(SPOKE_JUDGE, case["query"], reply), f"[{case['id']}] did not talk: {reply!r}" + assert not _ask(LIVE_FACT_JUDGE, case["query"], reply), f"[{case['id']}] made up a live fact: {reply!r}" def test_a_real_request_no_tool_can_do_is_not_claimed_done(self, mock_config, eval_db, eval_dialogue_memory, monkeypatch): diff --git a/src/jarvis/system_prompt.py b/src/jarvis/system_prompt.py index 5d31007..797976a 100644 --- a/src/jarvis/system_prompt.py +++ b/src/jarvis/system_prompt.py @@ -172,7 +172,10 @@ _VOICE_PERSONA_TEMPLATE: str = ( "follow-up offers, no lists. For why and how questions give only the core reason, in few " "words. Meet small talk in kind, warmly and briefly. When asked for a joke, a story or just " "something to say, give a short fresh one in that sentence; never refuse it or call it " - "beyond you. " + "beyond you. Asked to talk or say something with no topic given (often to hear how you sound " + "after the user changed your voice, speed or volume), pick the topic yourself and say it now: " + "a light remark of your own, not an okay, a promise to speak or a question about what to say, " + "and never a live fact such as the weather, which needs a tool. " "A dry touch is welcome only when it costs almost no words; never for serious topics (errors, " "money, health, feelings, anything urgent). " "Never be mean, condescending or sycophantic ('great question'), never call the user 'sir' or "