fix(reply): asked to just say something, Jarvis talks instead of promising to
Asked to talk with no topic ("말해 봐", usually to hear a changed voice,
speed or volume), the model answered with an okay, a promise to speak
("말씀하신 대로 속도를 높여 말씀드리겠습니다") or a question about what to
say. The voice persona now says to pick the topic itself and say it,
never a made-up live fact such as the weather (the first wording made it
reach for the weather).
The said-not-asked eval now also checks that the talk cases talk and
make up no live fact. Measured on Qwen3.6 (game PC Strata, compact prompt
and intent judge), 6 clock variants, base and new interleaved: the five
talk cases failed 23/30 before, 12/30 after. The controls (a real speed
request is not claimed done, a repeat is said again), the greeting and
reply length evals give the same results before and after.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -10,7 +10,8 @@ Run as the v2 brain does (compact prompt, intent judge, Korean by default, chatC
|
||||
|
||||
- the user reports something they did and asks for something Jarvis can do itself (speak, answer,
|
||||
react): the reply does it, never says a feature is unsupported or impossible, does not stop at a
|
||||
promise to speak, and posts nothing to a chat channel. One case carries the field turn's history
|
||||
promise to speak, and posts nothing to a chat channel. Asked to talk, the reply says something
|
||||
itself: not only "okay", a promise to speak, or a question about what to say. One case carries the field turn's history
|
||||
and dialogue memory, where Jarvis had already refused once.
|
||||
- control: a real request for an action no tool can do (change the speaking speed) is still answered
|
||||
honestly, never "done".
|
||||
@@ -39,11 +40,11 @@ FIELD_MEMORY = [("user", FIELD_QUESTION), ("assistant", FIELD_REFUSAL)]
|
||||
|
||||
SAID_CASES = [
|
||||
{"id": "speed-adjusted-field", "query": "너 말하는 속도 내가 조절해놨으니까 한번 말해보라고",
|
||||
"recent": FIELD_HISTORY, "memory": FIELD_MEMORY},
|
||||
{"id": "adjusted-heard-right", "query": "알아서 좀 조정해 놨는데 한번 말해 볼래", "recent": []},
|
||||
{"id": "speed-adjusted", "query": "너 말하는 속도 내가 조절해놨으니까 한번 말해보라고", "recent": []},
|
||||
{"id": "volume-raised", "query": "자비스 볼륨 키워놨으니까 아무 말이나 해 봐", "recent": []},
|
||||
{"id": "voice-changed", "query": "방금 설정에서 네 목소리 바꿔놨어, 한마디만 해 봐", "recent": []},
|
||||
"recent": FIELD_HISTORY, "memory": FIELD_MEMORY, "talk": True},
|
||||
{"id": "adjusted-heard-right", "query": "알아서 좀 조정해 놨는데 한번 말해 볼래", "recent": [], "talk": True},
|
||||
{"id": "speed-adjusted", "query": "너 말하는 속도 내가 조절해놨으니까 한번 말해보라고", "recent": [], "talk": True},
|
||||
{"id": "volume-raised", "query": "자비스 볼륨 키워놨으니까 아무 말이나 해 봐", "recent": [], "talk": True},
|
||||
{"id": "voice-changed", "query": "방금 설정에서 네 목소리 바꿔놨어, 한마디만 해 봐", "recent": [], "talk": True},
|
||||
{"id": "mic-replaced", "query": "마이크 새로 바꿨는데 내 말 잘 들려?", "recent": []},
|
||||
{"id": "alarm-set-myself", "query": "나 내일 아침 알람 일곱 시로 맞춰놨어, 잘했지?", "recent": []},
|
||||
]
|
||||
@@ -56,6 +57,28 @@ REFUSAL_JUDGE = (
|
||||
"NO for anything else: a remark, an answer, a reaction, a short question back, or a greeting."
|
||||
)
|
||||
|
||||
# Field (2026-11-18, second try of the done-by-user guard): "말씀하신 대로 속도를 높여 말씀드리겠습니다" (as you
|
||||
# said, I will speak faster) and "무슨 내용을 말씀하시는지 알려주시면 읽어 드리겠습니다" (tell me what to read):
|
||||
# asked to talk, Jarvis only promised or asked back. A reply that talks is the point of these turns.
|
||||
SPOKE_JUDGE = (
|
||||
"You check a voice assistant's reply. The user asked it to say something, usually to hear how "
|
||||
"it sounds now. Answer only YES or NO.\n"
|
||||
"YES if the reply itself says something to the user now: a remark, a fresh sentence, a joke, or "
|
||||
"a short check of how it sounds.\n"
|
||||
"NO if it is only an acknowledgement (such as okay or understood), only a promise to speak or "
|
||||
"answer, a question about what to say or what was changed, a request to wait or try later, or "
|
||||
"if it says the assistant itself made the change."
|
||||
)
|
||||
|
||||
# Picking a topic of its own, the model reached for the weather ("오늘 밤 서울의 기온은 12도 정도입니다")
|
||||
# with no tool: a live fact made up (system_prompt.TOOL_HONESTY_RULES).
|
||||
LIVE_FACT_JUDGE = (
|
||||
"You check a voice assistant's reply. Answer only YES or NO.\n"
|
||||
"YES if the reply states the current or today's weather, temperature, sky, news, prices or "
|
||||
"another fact about the world right now.\n"
|
||||
"NO for anything else, including remarks about the assistant itself, the user or the time of day."
|
||||
)
|
||||
|
||||
CLAIM_JUDGE = (
|
||||
"You check a voice assistant's reply. Answer only YES or NO.\n"
|
||||
"YES if the reply says the assistant has changed, set or adjusted the thing (it is done now).\n"
|
||||
@@ -120,6 +143,9 @@ class TestSaidNotAsked:
|
||||
assert reply.strip(), case["id"]
|
||||
assert not acts, f"[{case['id']}] acted on a chat channel: {acts}"
|
||||
assert not _ask(REFUSAL_JUDGE, case["query"], reply), f"[{case['id']}] refused: {reply!r}"
|
||||
if case.get("talk"):
|
||||
assert _ask(SPOKE_JUDGE, case["query"], reply), f"[{case['id']}] did not talk: {reply!r}"
|
||||
assert not _ask(LIVE_FACT_JUDGE, case["query"], reply), f"[{case['id']}] made up a live fact: {reply!r}"
|
||||
|
||||
def test_a_real_request_no_tool_can_do_is_not_claimed_done(self, mock_config, eval_db,
|
||||
eval_dialogue_memory, monkeypatch):
|
||||
|
||||
@@ -172,7 +172,10 @@ _VOICE_PERSONA_TEMPLATE: str = (
|
||||
"follow-up offers, no lists. For why and how questions give only the core reason, in few "
|
||||
"words. Meet small talk in kind, warmly and briefly. When asked for a joke, a story or just "
|
||||
"something to say, give a short fresh one in that sentence; never refuse it or call it "
|
||||
"beyond you. "
|
||||
"beyond you. Asked to talk or say something with no topic given (often to hear how you sound "
|
||||
"after the user changed your voice, speed or volume), pick the topic yourself and say it now: "
|
||||
"a light remark of your own, not an okay, a promise to speak or a question about what to say, "
|
||||
"and never a live fact such as the weather, which needs a tool. "
|
||||
"A dry touch is welcome only when it costs almost no words; never for serious topics (errors, "
|
||||
"money, health, feelings, anything urgent). "
|
||||
"Never be mean, condescending or sycophantic ('great question'), never call the user 'sir' or "
|
||||
|
||||
Reference in New Issue
Block a user