fix(reply): a one-step turn whose model skips the tool twice is planned and guarded

After the one nudge, a second answer without the tool no longer goes out
as it is: the turn takes the planned path (the planner runs then, one
extra call only on this failure) and the skipped-tool guard runs the
planned tool, so a made-up "24 degrees" answer cannot reach the user. A
plan that needs no tool (a question back to the user) still lets the
answer go out.

Evals (qwen3:8b, EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1, 3 runs):
the five tool-skip/weather/screen/follow-up/router files 72/84 (71/84 on
0.1.20, same known failures); Possessor director-then-filmography 0/3,
also 0/3 on 0.1.20 (qwen3:8b stops after one search; webSearch turns are
still planned).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-16 19:09:11 +09:00
parent a6932d9b16
commit 7da5bca783
6 changed files with 67 additions and 25 deletions

View File

@@ -3027,6 +3027,36 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
candidate_reply = content
malformed_fallback = False
_one_step_skipped = bool(
_one_step_tool and _one_step_tool in allowed_tools and not use_text_tools
and not _forced_plan_steps and not malformed_fallback
and sum(1 for m in messages if m.get("tool_name")) == _plan_steps_baseline
)
if _one_step_skipped and not _one_step_nudged:
messages.pop()
messages.append({"role": "user", "content": (
f"[Before answering, call {_one_step_tool}: this request needs its result. "
"If a detail it needs is missing, ask the user for it instead.]")})
_one_step_nudged = True
debug_log(f"one-step turn: answered without {_one_step_tool}; asking for the call", "planning")
continue
if _one_step_skipped and not action_plan:
# Asked once and still no tool: this turn takes the planned path after all, so the
# skipped-tool guard below runs the tool the plan names (or the plan says no tool is
# needed, a question back to the user say, and the answer goes out).
try:
action_plan = plan_query(
cfg=cfg, query=redacted, dialogue_context=_dialogue_ctx,
tools=_planner_tool_catalog, live_context=_live_time_location_string(cfg),
)
except Exception as _plan_exc: # pragma: no cover — defensive
debug_log(f"planner step failed (non-fatal): {_plan_exc}", "planning")
action_plan = []
if action_plan:
_turn_state.trace("plan", steps=list(action_plan))
debug_log(f"one-step turn: still no {_one_step_tool} after the nudge; planned "
f"{len(action_plan)} step(s)", "planning")
_guard_steps = [] if (use_text_tools or _forced_plan_steps or malformed_fallback) else (
named_tool_steps(
action_plan,
@@ -3049,18 +3079,6 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
)
print(" 🧭 Planned tool skipped by the model, running it directly", flush=True)
continue
if (
_one_step_tool and _one_step_tool in allowed_tools and not use_text_tools
and not _one_step_nudged and not malformed_fallback
and sum(1 for m in messages if m.get("tool_name")) == _plan_steps_baseline
):
messages.pop()
messages.append({"role": "user", "content": (
f"[Before answering, call {_one_step_tool}: this request needs its result. "
"If a detail it needs is missing, ask the user for it instead.]")})
_one_step_nudged = True
debug_log(f"one-step turn: answered without {_one_step_tool}; asking for the call", "planning")
continue
reply = candidate_reply
last_candidate_reply = candidate_reply

View File

@@ -234,8 +234,11 @@ direct execution runs the plan.
Without a plan the skipped-tool guard above has nothing to run, so these turns get a nudge instead:
when the chat model answers in words before any tool ran in this reply, the answer is dropped (no
later turn sees it) and a note asks the model to call that tool first, or to ask the user for a
missing detail. The nudge is sent at most once per reply; a second answer without the tool (a
question back to the user, say) is delivered.
missing detail. The nudge is sent at most once per reply. When the model answers without the tool
again, that turn falls back to the planned path: the planner runs then (one extra planner call, only
on this failure), and the skipped-tool guard above treats the second answer as on any planned turn.
A plan naming the tool runs it and the made-up answer never goes out; a plan that needs no tool (a
question back to the user, say) lets the answer go out, the planner judging that as before.
Measured on Strata (Qwen3.6, 2026-11-16 16:45-16:58, the engine log): the planner averaged 2.5 s
(9 calls), one after the other before the answer on every tool turn, while the router runs during

View File

@@ -106,7 +106,7 @@ Design principles enforced by the engine:
- When detected, the engine falls back to the standard "I had trouble understanding that request" error reply (model-size-aware). The malformed content is never shown to the user.
Skipped-tool guard (native tool calling):
- When a native-tool model answers in prose before any tool has run in this reply, and the plan names a callable tool as a concrete call, the engine discards that answer and direct-executes the plan's tool steps, then the model answers from the results. Only literal, schema-valid steps run (no LLM repair). When the planned call cannot run, the held-back answer is still discarded: the planned call is recorded as a failed tool result and the model answers from that (an honest "could not check or do it"). See `planner.spec.md` ("Skipped-tool guard"). v2 one-step turns (no plan) get one nudge to call the tool instead (`planner.spec.md`, "Intent judge (v2)").
- When a native-tool model answers in prose before any tool has run in this reply, and the plan names a callable tool as a concrete call, the engine discards that answer and direct-executes the plan's tool steps, then the model answers from the results. Only literal, schema-valid steps run (no LLM repair). When the planned call cannot run, the held-back answer is still discarded: the planned call is recorded as a failed tool result and the model answers from that (an honest "could not check or do it"). See `planner.spec.md` ("Skipped-tool guard"). v2 one-step turns (no plan) get one nudge to call the tool instead, and when the model skips it again the turn is planned and guarded as above (`planner.spec.md`, "Intent judge (v2)").
Task-list planner (all model sizes, strongest impact on small models):
- The planner runs at the **front** of the reply flow (see step 3 above), not after tool selection. By the time the agentic loop starts, the plan already exists, the memory block has either been run or skipped based on the plan's `searchMemory` directive, and the tool allow-list has been derived from the tool names the plan referenced. See `planner.spec.md` for the prompt contract and fail-open semantics.