fix(planner): plan in the query's own writing system only
Live 2026-11-18 '자비 사과랑 배 중에 뭐가 더 달아' was searched as '사과와 배 중哪个가 더 단가요'. The rule and the user message's closing line now say to use only the query's letters. Eval test_search_in_users_language.py, new case fruit (qwen3.6-35b-a3b, game PC): Han in the search 7/7 before; rule only 3/3, closing line only 3/3, both 0/6; whole file 12/12. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -164,7 +164,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
|
||||
- **Trigger**: once per reply, **after the tool router and before memory search**. **v2 brain** (`intent_judge=True`): runs only on turns where the router picked a tool (not for the sentinels `stop` or `toolSearchTool` alone; a router-picked `chatChannels` counts), so a turn without tools has one LLM call before its answer. Also skipped when the router's only pick is a one-step tool (`_ONE_STEP_TOOLS` in `engine.py`: `getWeather`, `logMeal`, `fetchMeals`, `screenView`) and the chat model calls tools natively; the chat model then gets the tool directly, and an answer it gives without the tool is dropped and followed by one nudge to call it; a second answer without the tool runs the planner after all (one extra call, only then) and the skipped-tool guard applies (planner.spec.md, "Intent judge (v2)"). Skipped when `cfg.planner_enabled = False`, when the query is shorter than `MIN_QUERY_CHARS` (4), or when no model / base URL is available.
|
||||
- **Model / gating**: resolution chain `planner_model (override) → ollama_chat_model`. The planner tracks the chat model so upgrading the chat model (via setup wizard or config) automatically upgrades plan quality.
|
||||
- **Inputs**: user query, dialogue context, the live time/location line (`LIVE CONTEXT:`, so a location-dependent ask plans `searchMemory topic='where the user lives'` only when the location is unknown or disabled), **router-narrowed** tool catalogue (names + one-line descriptions + argument names, built by `planner_tool_catalog()`) — not the full 30+ list. When the carry-over guard from #7 fires, the previous turn's failed tool name is unioned into this catalogue before the planner sees it, so the planner can plan a re-call without `toolSearchTool` round-tripping. **No** memory context — the planner decides *whether* memory is needed.
|
||||
- **System prompt**: `_PROMPT_TEMPLATE` in `planner.py`. Teaches the `searchMemory topic='...'` directive for prior-conversation lookups, short imperative tool steps, angle-bracket entity placeholders, final synthesis step, same-language output (the words of search and play arguments too, repeated in the user message's closing line), no numbering.
|
||||
- **System prompt**: `_PROMPT_TEMPLATE` in `planner.py`. Teaches the `searchMemory topic='...'` directive for prior-conversation lookups, short imperative tool steps, angle-bracket entity placeholders, final synthesis step, same-language output in the query's own writing system (the words of search and play arguments too, repeated in the user message's closing line), no numbering.
|
||||
- **Output**: list of plan steps (max `MAX_STEPS` = 5). Gates memory enrichment (#3 / #4) and augments the tool router (#7 — planner's picks are unioned in, not replacing). Single-step `["Reply to the user."]` plans are the planner's positive "no memory, no tools" signal. An empty list is fail-open — the engine reverts to running #3 unconditionally. Consumed further by the engine to build the `ACTION PLAN:` system-message block and drive the direct-exec loop (#13) for small models.
|
||||
- **Limits**: `planner_timeout_sec` (6s). Fail-open → `[]`.
|
||||
|
||||
|
||||
@@ -22,6 +22,8 @@ CASES = [
|
||||
("netflix", "요즘 볼 만한 넷플릭스 드라마 세 개만 추천해 줘"),
|
||||
("price", "엔비디아 RTX 5050 그래픽카드 가격 검색해 줘"),
|
||||
("restaurants", "이번 주말에 갈 만한 맛집 좀 찾아 줘"),
|
||||
# Live 2026-11-18: the search was "사과와 배 중哪个가 더 단가요" (another script mixed in).
|
||||
("fruit", "자비 사과랑 배 중에 뭐가 더 달아 검색해줘"),
|
||||
]
|
||||
|
||||
|
||||
|
||||
@@ -235,7 +235,9 @@ _PROMPT_TEMPLATE = (
|
||||
"12. Write the steps in the same language the user wrote the query in. "
|
||||
"The words of every search or play argument stay in the user's "
|
||||
"language too, never translated into English, because a search in the "
|
||||
"user's language finds what people where they are get.\n"
|
||||
"user's language finds what people where they are get. Use only the "
|
||||
"writing system of the query; never mix in characters of another "
|
||||
"script.\n"
|
||||
)
|
||||
|
||||
|
||||
@@ -283,7 +285,7 @@ def _build_user_message(
|
||||
parts.append("DIALOGUE CONTEXT: (empty)")
|
||||
parts.append(f"USER QUERY: {query.strip()}")
|
||||
parts.append("\nEmit the plan now, one step per line, no numbering, written in the language of "
|
||||
"the USER QUERY (the words of searches too).")
|
||||
"the USER QUERY (the words of searches too), in its own letters only.")
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
|
||||
@@ -116,6 +116,13 @@ The planner prompt instructs the model to emit:
|
||||
(qwen3:8b, 3 runs of 3 Korean requests): Korean searches 6/9 before
|
||||
("요즘 볼 만한 넷플릭스 드라마" searched "best netflix dramas 2026" 3/3), 8/9
|
||||
after (that one 2/3).
|
||||
- Only the writing system of the query, no characters of another script,
|
||||
in the rule and in the closing line ("in its own letters only"). Live
|
||||
2026-11-18 "자비 사과랑 배 중에 뭐가 더 달아" was searched as
|
||||
"사과와 배 중哪个가 더 단가요". Same eval, case `fruit` (qwen3.6-35b-a3b):
|
||||
Han in the search 7/7 before; with only the rule 3/3, with only the
|
||||
closing line 3/3, with both 0/6, the other cases unchanged (12/12).
|
||||
`webSearch` also drops such letters itself (web_search.spec.md).
|
||||
|
||||
### Parsing and hygiene
|
||||
|
||||
|
||||
@@ -17,8 +17,8 @@ Netflix dramas searched in English and recommended American shows; eval
|
||||
Letters of a writing system the user did not use are dropped from the query before any provider
|
||||
runs (`_drop_foreign_script`, by Unicode script name against `context.redacted_text`; Latin is always
|
||||
kept for brand names and model numbers). Live 2026-11-18 a Hangul question was searched as
|
||||
"사과와 배 중哪个가 더 단가요". The same eval checks that no other script is mixed in; the planner
|
||||
alone did not reproduce it in 24 runs, so no prompt rule was added for it.
|
||||
"사과와 배 중哪个가 더 단가요". The planner is also told to use only the query's writing system
|
||||
(planner.spec.md); `evals/test_search_in_users_language.py` checks that no other script is mixed in.
|
||||
|
||||
### Real-time info routing (`STREAM_BROWSER`)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user