fix(reply): a one-step turn whose model skips the tool twice is planned and guarded
After the one nudge, a second answer without the tool no longer goes out as it is: the turn takes the planned path (the planner runs then, one extra call only on this failure) and the skipped-tool guard runs the planned tool, so a made-up "24 degrees" answer cannot reach the user. A plan that needs no tool (a question back to the user) still lets the answer go out. Evals (qwen3:8b, EVAL_COMPACT_PROMPT=1 EVAL_INTENT_JUDGE=1, 3 runs): the five tool-skip/weather/screen/follow-up/router files 72/84 (71/84 on 0.1.20, same known failures); Possessor director-then-filmography 0/3, also 0/3 on 0.1.20 (qwen3:8b stops after one search; webSearch turns are still planned). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -3027,6 +3027,36 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
||||
candidate_reply = content
|
||||
malformed_fallback = False
|
||||
|
||||
_one_step_skipped = bool(
|
||||
_one_step_tool and _one_step_tool in allowed_tools and not use_text_tools
|
||||
and not _forced_plan_steps and not malformed_fallback
|
||||
and sum(1 for m in messages if m.get("tool_name")) == _plan_steps_baseline
|
||||
)
|
||||
if _one_step_skipped and not _one_step_nudged:
|
||||
messages.pop()
|
||||
messages.append({"role": "user", "content": (
|
||||
f"[Before answering, call {_one_step_tool}: this request needs its result. "
|
||||
"If a detail it needs is missing, ask the user for it instead.]")})
|
||||
_one_step_nudged = True
|
||||
debug_log(f"one-step turn: answered without {_one_step_tool}; asking for the call", "planning")
|
||||
continue
|
||||
if _one_step_skipped and not action_plan:
|
||||
# Asked once and still no tool: this turn takes the planned path after all, so the
|
||||
# skipped-tool guard below runs the tool the plan names (or the plan says no tool is
|
||||
# needed, a question back to the user say, and the answer goes out).
|
||||
try:
|
||||
action_plan = plan_query(
|
||||
cfg=cfg, query=redacted, dialogue_context=_dialogue_ctx,
|
||||
tools=_planner_tool_catalog, live_context=_live_time_location_string(cfg),
|
||||
)
|
||||
except Exception as _plan_exc: # pragma: no cover — defensive
|
||||
debug_log(f"planner step failed (non-fatal): {_plan_exc}", "planning")
|
||||
action_plan = []
|
||||
if action_plan:
|
||||
_turn_state.trace("plan", steps=list(action_plan))
|
||||
debug_log(f"one-step turn: still no {_one_step_tool} after the nudge; planned "
|
||||
f"{len(action_plan)} step(s)", "planning")
|
||||
|
||||
_guard_steps = [] if (use_text_tools or _forced_plan_steps or malformed_fallback) else (
|
||||
named_tool_steps(
|
||||
action_plan,
|
||||
@@ -3049,18 +3079,6 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
||||
)
|
||||
print(" 🧭 Planned tool skipped by the model, running it directly", flush=True)
|
||||
continue
|
||||
if (
|
||||
_one_step_tool and _one_step_tool in allowed_tools and not use_text_tools
|
||||
and not _one_step_nudged and not malformed_fallback
|
||||
and sum(1 for m in messages if m.get("tool_name")) == _plan_steps_baseline
|
||||
):
|
||||
messages.pop()
|
||||
messages.append({"role": "user", "content": (
|
||||
f"[Before answering, call {_one_step_tool}: this request needs its result. "
|
||||
"If a detail it needs is missing, ask the user for it instead.]")})
|
||||
_one_step_nudged = True
|
||||
debug_log(f"one-step turn: answered without {_one_step_tool}; asking for the call", "planning")
|
||||
continue
|
||||
|
||||
reply = candidate_reply
|
||||
last_candidate_reply = candidate_reply
|
||||
|
||||
@@ -234,8 +234,11 @@ direct execution runs the plan.
|
||||
Without a plan the skipped-tool guard above has nothing to run, so these turns get a nudge instead:
|
||||
when the chat model answers in words before any tool ran in this reply, the answer is dropped (no
|
||||
later turn sees it) and a note asks the model to call that tool first, or to ask the user for a
|
||||
missing detail. The nudge is sent at most once per reply; a second answer without the tool (a
|
||||
question back to the user, say) is delivered.
|
||||
missing detail. The nudge is sent at most once per reply. When the model answers without the tool
|
||||
again, that turn falls back to the planned path: the planner runs then (one extra planner call, only
|
||||
on this failure), and the skipped-tool guard above treats the second answer as on any planned turn.
|
||||
A plan naming the tool runs it and the made-up answer never goes out; a plan that needs no tool (a
|
||||
question back to the user, say) lets the answer go out, the planner judging that as before.
|
||||
|
||||
Measured on Strata (Qwen3.6, 2026-11-16 16:45-16:58, the engine log): the planner averaged 2.5 s
|
||||
(9 calls), one after the other before the answer on every tool turn, while the router runs during
|
||||
|
||||
@@ -106,7 +106,7 @@ Design principles enforced by the engine:
|
||||
- When detected, the engine falls back to the standard "I had trouble understanding that request" error reply (model-size-aware). The malformed content is never shown to the user.
|
||||
|
||||
Skipped-tool guard (native tool calling):
|
||||
- When a native-tool model answers in prose before any tool has run in this reply, and the plan names a callable tool as a concrete call, the engine discards that answer and direct-executes the plan's tool steps, then the model answers from the results. Only literal, schema-valid steps run (no LLM repair). When the planned call cannot run, the held-back answer is still discarded: the planned call is recorded as a failed tool result and the model answers from that (an honest "could not check or do it"). See `planner.spec.md` ("Skipped-tool guard"). v2 one-step turns (no plan) get one nudge to call the tool instead (`planner.spec.md`, "Intent judge (v2)").
|
||||
- When a native-tool model answers in prose before any tool has run in this reply, and the plan names a callable tool as a concrete call, the engine discards that answer and direct-executes the plan's tool steps, then the model answers from the results. Only literal, schema-valid steps run (no LLM repair). When the planned call cannot run, the held-back answer is still discarded: the planned call is recorded as a failed tool result and the model answers from that (an honest "could not check or do it"). See `planner.spec.md` ("Skipped-tool guard"). v2 one-step turns (no plan) get one nudge to call the tool instead, and when the model skips it again the turn is planned and guarded as above (`planner.spec.md`, "Intent judge (v2)").
|
||||
|
||||
Task-list planner (all model sizes, strongest impact on small models):
|
||||
- The planner runs at the **front** of the reply flow (see step 3 above), not after tool selection. By the time the agentic loop starts, the plan already exists, the memory block has either been run or skipped based on the plan's `searchMemory` directive, and the tool allow-list has been derived from the tool names the plan referenced. See `planner.spec.md` for the prompt contract and fail-open semantics.
|
||||
|
||||
Reference in New Issue
Block a user