perf(reply): v2 turns with one one-step tool skip the planner

When the router's only pick is a tool that does its job in one call
(getWeather, logMeal, fetchMeals, screenView) and the chat model calls
tools natively, the planner no longer runs: on Strata it averaged 2.5 s
per tool turn (2026-11-16 16:45-16:58 engine log, 9 calls) before the
answer could start. Search, browsers, schedule, notes, chat channels and
picks of two or more tools still plan. The rule reads the router's pick,
not the words, so it is language-agnostic.

Without a plan the skipped-tool guard has nothing to run, so these turns
get one nudge instead: an answer given before any tool ran is dropped and
the model is asked to call the tool (or ask for a missing detail).

Evals (qwen3:8b, EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1, 3 runs of
tool_skip_confabulation, weather_autoderive_location, screen_view_tool,
followup_supplies_missing_tool_arg, tool_router_implicit): 71/84 before,
72/84 after, same known failures; planner calls 14 -> 3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-16 19:00:36 +09:00
parent 6a9b66bdcd
commit a6932d9b16
7 changed files with 204 additions and 13 deletions

View File

@@ -161,7 +161,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
## 12. Task-list Planner (pre-flight decomposition, gates the whole turn)
- **File**: [src/jarvis/reply/planner.py](src/jarvis/reply/planner.py) — `plan_query()`.
- **Trigger**: once per reply, **after the tool router and before memory search**. **v2 brain** (`intent_judge=True`): runs only on turns where the router picked a tool (not for the sentinels `stop` or `toolSearchTool` alone; a router-picked `chatChannels` counts), so a turn without tools has one LLM call before its answer. Skipped when `cfg.planner_enabled = False`, when the query is shorter than `MIN_QUERY_CHARS` (4), or when no model / base URL is available.
- **Trigger**: once per reply, **after the tool router and before memory search**. **v2 brain** (`intent_judge=True`): runs only on turns where the router picked a tool (not for the sentinels `stop` or `toolSearchTool` alone; a router-picked `chatChannels` counts), so a turn without tools has one LLM call before its answer. Also skipped when the router's only pick is a one-step tool (`_ONE_STEP_TOOLS` in `engine.py`: `getWeather`, `logMeal`, `fetchMeals`, `screenView`) and the chat model calls tools natively; the chat model then gets the tool directly, and an answer it gives without the tool is dropped and followed by one nudge to call it (planner.spec.md, "Intent judge (v2)"). Skipped when `cfg.planner_enabled = False`, when the query is shorter than `MIN_QUERY_CHARS` (4), or when no model / base URL is available.
- **Model / gating**: resolution chain `planner_model (override) → ollama_chat_model`. The planner tracks the chat model so upgrading the chat model (via setup wizard or config) automatically upgrades plan quality.
- **Inputs**: user query, dialogue context, the live time/location line (`LIVE CONTEXT:`, so a location-dependent ask plans `searchMemory topic='where the user lives'` only when the location is unknown or disabled), **router-narrowed** tool catalogue (names + one-line descriptions + argument names, built by `planner_tool_catalog()`) — not the full 30+ list. When the carry-over guard from #7 fires, the previous turn's failed tool name is unioned into this catalogue before the planner sees it, so the planner can plan a re-call without `toolSearchTool` round-tripping. **No** memory context — the planner decides *whether* memory is needed.
- **System prompt**: `_PROMPT_TEMPLATE` in `planner.py`. Teaches the `searchMemory topic='...'` directive for prior-conversation lookups, short imperative tool steps, angle-bracket entity placeholders, final synthesis step, same-language output (the words of search and play arguments too, repeated in the user message's closing line), no numbering.
@@ -215,7 +215,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
| 10 | Graph extraction | ~1/session | No (background) | LARGE |
| 11 | Graph best-child | 0-N | No (background) | SMALL (via router chain) |
| 11b | Graph node merge | 0-N (per node, batched) | No (background) | SMALL (via router chain) |
| 12 | Planner (plan_query) | 1 | yes (planner_enabled) | LARGE/SMALL (tracks chat model) |
| 12 | Planner (plan_query) | 0-1 (v2: tool turns except one-step tools) | yes (planner_enabled) | LARGE/SMALL (tracks chat model) |
| 13 | Plan step resolver | 0-N (SMALL only) | auto by size + plan | SMALL (via router chain) |
| 14 | Tool-specific | per-tool | n/a | LARGE |
| 15 | Voice call judgements | 0 (per utterance / tie, outside the reply) | call mode / policy | LARGE (chat model) |

View File

@@ -1095,6 +1095,23 @@ def _attach_images(message: dict, result) -> None:
message["images"] = list(images)
def _starts_with_text_tools(cfg) -> bool:
"""Whether the reply starts in text-tool mode: a small model outside the native tool-call
families (see the loop's comment where it is used)."""
model_l = (cfg.ollama_chat_model or "").lower()
native_capable = any(k in model_l for k in (
"qwen2.5", "qwen2", "qwen3", "llama3.1", "llama3.2", "llama3.3",
"mistral", "hermes", "command-r", "firefunction",
))
return detect_model_size(cfg.ollama_chat_model) == ModelSize.SMALL and not native_capable
# Tools that do their job in one call (planner.spec.md, "Intent judge (v2)"): a v2 turn whose only
# router pick is one of them needs no plan. Tools a request may use several times or chain (search,
# browsers, schedule and notes lists before a change, chat channels) are not here.
_ONE_STEP_TOOLS = frozenset({"getWeather", "logMeal", "fetchMeals", "screenView"})
def _offered_builtins(cfg, chat_available: bool, schedule_available: bool = False,
notes_available: bool = False, screen_available: bool = False) -> dict:
"""The built-in tools this turn can offer: the enabled ones (``enabled_builtin_tools``),
@@ -1400,6 +1417,14 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
# aside; the chat tool counts here, its every-turn offer is added below); a turn without one
# goes straight to its answer.
_turn_needs_tools = any(t not in ("stop", "toolSearchTool") for t in routed_tools)
# Intent judge (v2), native tool calls: a lone one-step tool goes to the chat model without a
# plan; the nudge in the loop below stands in for the plan-based skipped-tool guard.
_picked_tools = [t for t in routed_tools if t not in ("stop", "toolSearchTool")]
_one_step_tool = (
_picked_tools[0]
if intent_judge and len(_picked_tools) == 1 and _picked_tools[0] in _ONE_STEP_TOOLS
and not _starts_with_text_tools(cfg) else None
)
# In screen-share mode, always offer setBroadcast. "Turn the broadcast
# on/off" is language-agnostic intent the embedding/keyword router won't
@@ -1447,7 +1472,9 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
try:
# Intent judge (v2): the router decides whether the turn needs tools; only then does the
# planner order the steps (planner.spec.md, "Intent judge (v2)").
if not intent_judge or _turn_needs_tools:
if _one_step_tool:
debug_log(f"intent judge: one-step tool {_one_step_tool}, no plan", "planning")
elif not intent_judge or _turn_needs_tools:
action_plan = plan_query(
cfg=cfg,
query=redacted,
@@ -1866,12 +1893,7 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
# call, so getWeather/webSearch/controlBrowser never run. Use native for the
# tool-capable families (native still auto-falls-back to text on HTTP 400);
# only genuinely non-tool small models (e.g. gemma) default to text.
_model_l = (cfg.ollama_chat_model or "").lower()
_native_capable = any(k in _model_l for k in (
"qwen2.5", "qwen2", "qwen3", "llama3.1", "llama3.2", "llama3.3",
"mistral", "hermes", "command-r", "firefunction",
))
use_text_tools = (model_size == ModelSize.SMALL) and not _native_capable
use_text_tools = _starts_with_text_tools(cfg)
prompts = get_system_prompts(model_size)
debug_log(f"Model size detected: {model_size.value} for {cfg.ollama_chat_model} (use_text_tools={use_text_tools})", "planning")
@@ -2281,6 +2303,9 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
# synthesise from the real result. Fires at most once per reply; a
# non-empty list means it fired. See planner.spec.md.
_forced_plan_steps: list[str] = []
# One-step turns have no plan: an answer given without their tool is dropped and the model is
# asked once to call it (planner.spec.md, "Intent judge (v2)").
_one_step_nudged = False
while turn < max_turns:
turn += 1
@@ -3024,6 +3049,18 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
)
print(" 🧭 Planned tool skipped by the model, running it directly", flush=True)
continue
if (
_one_step_tool and _one_step_tool in allowed_tools and not use_text_tools
and not _one_step_nudged and not malformed_fallback
and sum(1 for m in messages if m.get("tool_name")) == _plan_steps_baseline
):
messages.pop()
messages.append({"role": "user", "content": (
f"[Before answering, call {_one_step_tool}: this request needs its result. "
"If a detail it needs is missing, ask the user for it instead.]")})
_one_step_nudged = True
debug_log(f"one-step turn: answered without {_one_step_tool}; asking for the call", "planning")
continue
reply = candidate_reply
last_candidate_reply = candidate_reply

View File

@@ -222,6 +222,26 @@ needs the planner to name the channel); its every-turn offer, added after the ro
to its answer with one LLM call before it; a tool turn keeps the planner's ordered steps, which
multi-step asks need ("who directed Possessor, and what else did they make").
**One-step turns.** When the router's only pick (sentinels aside) is a tool that does its job in one
call, `_ONE_STEP_TOOLS` in `engine.py` (`getWeather`, `logMeal`, `fetchMeals`, `screenView`), and the
chat model calls tools natively, the planner does not run: its steps would only repeat the one call
the chat model makes itself. The rule reads the router's pick, never the words, so it is language-
agnostic. Tools a request may use several times or chain stay planned: search and page fetches,
the browsers, `schedule` and `personalNotes` (a list before a change), `chatChannels`. Two or more
picked tools are planned as before. Text-tool mode (small non-native models) always plans, since its
direct execution runs the plan.
Without a plan the skipped-tool guard above has nothing to run, so these turns get a nudge instead:
when the chat model answers in words before any tool ran in this reply, the answer is dropped (no
later turn sees it) and a note asks the model to call that tool first, or to ask the user for a
missing detail. The nudge is sent at most once per reply; a second answer without the tool (a
question back to the user, say) is delivered.
Measured on Strata (Qwen3.6, 2026-11-16 16:45-16:58, the engine log): the planner averaged 2.5 s
(9 calls), one after the other before the answer on every tool turn, while the router runs during
speech (`prepare_intent`). A planner call that does not run also leaves the engine's second session
to the chat model, whose prompt read is the largest cost.
Measured on Strata (Qwen3.6, 2026-11-12): the router took 0.42 s and the planner 0.49-0.94 s, one
after the other, before every answer; each call costs at least one full prompt read. Tried first
and dropped: the planner alone over the whole catalogue (one call) planned "reply with the

View File

@@ -106,7 +106,7 @@ Design principles enforced by the engine:
- When detected, the engine falls back to the standard "I had trouble understanding that request" error reply (model-size-aware). The malformed content is never shown to the user.
Skipped-tool guard (native tool calling):
- When a native-tool model answers in prose before any tool has run in this reply, and the plan names a callable tool as a concrete call, the engine discards that answer and direct-executes the plan's tool steps, then the model answers from the results. Only literal, schema-valid steps run (no LLM repair). When the planned call cannot run, the held-back answer is still discarded: the planned call is recorded as a failed tool result and the model answers from that (an honest "could not check or do it"). See `planner.spec.md` ("Skipped-tool guard").
- When a native-tool model answers in prose before any tool has run in this reply, and the plan names a callable tool as a concrete call, the engine discards that answer and direct-executes the plan's tool steps, then the model answers from the results. Only literal, schema-valid steps run (no LLM repair). When the planned call cannot run, the held-back answer is still discarded: the planned call is recorded as a failed tool result and the model answers from that (an honest "could not check or do it"). See `planner.spec.md` ("Skipped-tool guard"). v2 one-step turns (no plan) get one nudge to call the tool instead (`planner.spec.md`, "Intent judge (v2)").
Task-list planner (all model sizes, strongest impact on small models):
- The planner runs at the **front** of the reply flow (see step 3 above), not after tool selection. By the time the agentic loop starts, the plan already exists, the memory block has either been run or skipped based on the plan's `searchMemory` directive, and the tool allow-list has been derived from the tool names the plan referenced. See `planner.spec.md` for the prompt contract and fail-open semantics.

View File

@@ -269,7 +269,8 @@ which carry Discord's own time.
the short voice persona (`compact_prompt=True`: the same butler in a tenth of v1's words, without
v1's single-user memory and on-screen browser rules, answers as short as possible, the full tool
guidance only on turns that offer a data tool), and the intent judge (`intent_judge=True`: the tool
router judges whether the turn needs tools and which, and the planner runs only when it does;
router judges whether the turn needs tools and which, and the planner runs only when it does
and the pick is not a lone one-step tool;
`planner.spec.md`, "Intent judge (v2)"). v1's screen-share browser (`stream_browser`) is off: it
needs v1's Node bot (M6 brings the v2 one), so its tools are not offered.
- Tools: the brain offers and runs no built-in tool in the jarvis settings' `disabled_tools` (the

View File

@@ -74,9 +74,12 @@ def _intent_turn(prepared, routed=("getWeather",)):
def test_the_intent_judge_plans_only_turns_that_need_a_tool():
# No tool needed: one router call, then the answer.
assert _intent_turn(None, routed=())[:2] == (0, 1)
# A tool turn: the planner orders the steps.
# A tool turn that may take several steps: the planner orders them.
planner, router, tools = _intent_turn(None, routed=("webSearch",))
assert (planner, router) == (1, 1) and "webSearch" in tools
# A lone one-step tool needs no plan (tests/test_engine_one_step_turn.py).
planner, router, tools = _intent_turn(None)
assert (planner, router) == (1, 1) and "getWeather" in tools
assert (planner, router) == (0, 1) and "getWeather" in tools
@pytest.mark.unit

View File

@@ -0,0 +1,130 @@
"""One-step tool turns in the v2 brain skip the planner (planner.spec.md, "Intent judge (v2)").
When the router's only pick is a tool that does its job in one call (the weather, a meal log, a
look at the screen), the planner's ordered steps add nothing but a full LLM call before the answer.
The engine then gives the chat model the tool directly. Its skipped-tool guard, which reads the
plan, is replaced for these turns by one nudge: an answer given without the tool is dropped and the
model is asked once to call it.
"""
from __future__ import annotations
from unittest.mock import patch
import pytest
NATIVE_MODEL = "gpt-oss:20b"
FABRICATED = "서울은 지금 24도에 맑습니다."
GROUNDED = "서울은 지금 13도이고 맑습니다."
def _content(text: str) -> dict:
return {"message": {"role": "assistant", "content": text}}
def _native_call(name: str, args: dict) -> dict:
return {"message": {"role": "assistant", "content": "", "tool_calls": [
{"id": "call_1", "type": "function", "function": {"name": name, "arguments": args}}]}}
def _run(mock_config, db, dialogue_memory, *, routed, chat_replies, plan=("Reply to the user.",)):
from jarvis.memory.history import HistoryContext
from jarvis.reply import engine as engine_mod
from jarvis.tools.types import ToolExecutionResult
mock_config.ollama_chat_model = NATIVE_MODEL
mock_config.evaluator_enabled = False
mock_config.stream_browser = False
seen: list[list[dict]] = []
replies = iter(chat_replies)
def fake_chat(*args, **kwargs):
seen.append([dict(m) for m in (kwargs.get("messages") or args[2])])
return next(replies)
invoked: list[str] = []
def fake_tool(db, cfg, tool_name, tool_args, **kwargs):
invoked.append(tool_name)
return ToolExecutionResult(success=True, reply_text="Seoul: 13C, clear sky.", error_message=None)
with patch.object(engine_mod, "run_tool_with_retries", side_effect=fake_tool), \
patch.object(engine_mod, "chat_with_messages", side_effect=fake_chat), \
patch.object(engine_mod, "select_tools", return_value=list(routed)), \
patch.object(engine_mod, "plan_query", return_value=list(plan)) as planner:
reply = engine_mod.run_reply_engine(
db=db, cfg=mock_config, tts=None, text="서울 지금 우산 필요해?",
dialogue_memory=dialogue_memory, history=HistoryContext(), legacy_memory=False,
intent_judge=True)
return reply, invoked, seen, planner.call_count
@pytest.mark.unit
@pytest.mark.parametrize("routed", [["getWeather", "stop"], ["logMeal", "stop"], ["screenView"]])
def test_a_lone_one_step_tool_needs_no_plan(mock_config, db, dialogue_memory, routed):
_, _, _, planned = _run(mock_config, db, dialogue_memory, routed=routed,
chat_replies=[_content(GROUNDED)] * 2)
assert planned == 0
@pytest.mark.unit
@pytest.mark.parametrize("routed", [
["webSearch", "stop"], # searches chain (who directed it, what else they made)
["schedule", "stop"], # list, then cancel the one asked about
["getWeather", "webSearch", "stop"], # two tools
], ids=["search", "schedule", "two tools"])
def test_turns_that_may_take_several_steps_keep_the_plan(mock_config, db, dialogue_memory, routed):
_, _, _, planned = _run(mock_config, db, dialogue_memory, routed=routed,
chat_replies=[_content(GROUNDED)])
assert planned == 1
@pytest.mark.unit
def test_a_tool_less_answer_on_a_one_step_turn_is_dropped_and_the_tool_called(mock_config, db, dialogue_memory):
reply, invoked, seen, _ = _run(
mock_config, db, dialogue_memory, routed=["getWeather", "stop"],
chat_replies=[_content(FABRICATED), _native_call("getWeather", {"location": "Seoul"}),
_content(GROUNDED)])
assert invoked == ["getWeather"]
assert reply == GROUNDED
# The made-up answer never reaches a later turn, and the retry names the tool.
assert not any(FABRICATED in (m.get("content") or "") for m in seen[1])
assert "getWeather" in (seen[1][-1].get("content") or "")
@pytest.mark.unit
def test_the_tool_is_asked_for_only_once(mock_config, db, dialogue_memory):
# A second tool-less answer (for example a question back to the user) is delivered.
question = "어느 도시 날씨를 알려 드릴까요?"
reply, invoked, seen, _ = _run(
mock_config, db, dialogue_memory, routed=["getWeather", "stop"],
chat_replies=[_content(question), _content(question)])
assert invoked == []
assert reply == question
assert len(seen) == 2
@pytest.mark.unit
def test_a_model_that_calls_the_tool_is_not_asked_again(mock_config, db, dialogue_memory):
reply, invoked, seen, _ = _run(
mock_config, db, dialogue_memory, routed=["getWeather", "stop"],
chat_replies=[_native_call("getWeather", {"location": "Seoul"}), _content(GROUNDED)])
assert invoked == ["getWeather"]
assert reply == GROUNDED
assert len(seen) == 2
@pytest.mark.unit
def test_v1_turns_still_plan_a_one_step_tool(mock_config, db, dialogue_memory):
from jarvis.reply import engine as engine_mod
mock_config.ollama_chat_model = NATIVE_MODEL
mock_config.evaluator_enabled = False
with patch.object(engine_mod, "chat_with_messages", return_value=_content(GROUNDED)), \
patch.object(engine_mod, "select_tools", return_value=["getWeather", "stop"]), \
patch.object(engine_mod, "extract_search_params_for_memory", return_value={"keywords": []}), \
patch.object(engine_mod, "plan_query", return_value=["Reply to the user."]) as planner:
engine_mod.run_reply_engine(db=db, cfg=mock_config, tts=None, text="서울 지금 우산 필요해?",
dialogue_memory=dialogue_memory)
assert planner.call_count == 1