fix(reply): never deliver the tool-less answer when the planned call cannot run

When the skipped-tool guard could not run the planned call (not a literal
call, or arguments the tool does not take) it delivered the answer it had
held back, which was given without the tool and could be a made-up value.
The planned call is now recorded as a failed tool result and the model
answers from that, so the user hears that it could not check or do it.

M0 kit: a case passes only when the final answer uses the tool result
(13 degrees, not a made-up 24) or, for the no-tool request, says it
cannot do it; the no-tool request now counts. The kit mirrors the
engine's failed-result path, and a failing model moves on to the next
candidate on these stricter checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-10 15:19:30 +09:00
parent 85a562c171
commit 8cb7e57cda
7 changed files with 176 additions and 60 deletions

View File

@@ -2261,14 +2261,37 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
continue
if _forced_plan_steps and _tool_results_so_far == 0:
# The guard could not run the planned tool after all
# (unresolvable step or arguments that do not fit). Fail
# open: deliver the answer it held back rather than ask
# the model again on the same messages.
debug_log("skipped-tool guard: planned tool not runnable, keeping the model answer", "planning")
messages.append(_guard_held_message)
reply = _guard_held_reply
last_candidate_reply = _guard_held_reply
break
# (not a literal call, or arguments the tool does not
# take). The held-back answer was given without the tool,
# so it may be a guess: never deliver it. Record the
# planned call as a failed tool result and let the model
# answer from that, which under the tool honesty rules
# means saying it could not check or do it.
_fail_step = _plan_tool_steps[0]
_fail_name = _fail_step.split()[0]
_fail_id = f"call_plan_{uuid.uuid4().hex[:8]}"
debug_log(
f"skipped-tool guard: planned call not runnable ({_fail_step[:80]!r}); "
"asking the model to answer from a failed tool result",
"planning",
)
messages.append({
"role": "assistant",
"content": "",
"tool_calls": [{"id": _fail_id, "type": "function",
"function": {"name": _fail_name, "arguments": {}}}],
})
messages.append({
"role": "tool",
"tool_call_id": _fail_id,
"tool_name": _fail_name,
"content": (
f"Error: the planned call `{_fail_step}` could not be run, so nothing "
"was fetched or done. Tell the user plainly that you could not check "
"or do it right now; do not guess a value or claim it was done."
),
"tool_failed": True,
})
# Update the system message with fresh context (time/location) before each LLM call
# Note: We update the first system message rather than appending a new one because
@@ -2757,8 +2780,7 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
# The model answered without calling the tool the plan chose, so
# the answer can only be a guess. Drop it (the synthesis turn must
# not anchor on it) and run the planned steps directly.
_guard_held_message = messages.pop()
_guard_held_reply = candidate_reply
messages.pop()
_forced_plan_steps = _guard_steps
debug_log(
f"skipped-tool guard: model answered without the planned tool; "

View File

@@ -183,10 +183,13 @@ The engine consumes the plan in two phases.
against the tool's input schema. An LLM resolver could "repair" a wrong
step into a valid but unwanted call (a timer request planned as
`setBroadcast duration=5 minutes` was turned into starting the broadcast).
- Fail-open: if the first planned step cannot be run as written, the
engine delivers the answer it held back, without a second chat call. The
guard fires at most once per reply and never in text-tool mode (that mode
already direct-executes every plan).
- If the first planned step cannot be run as written, the held-back answer
is still never delivered (it was given without the tool, so it may be a
guess). The engine records the planned call as a failed tool result
("could not be run, nothing was fetched or done") and the chat model
answers from that, which under the tool honesty rules means saying it
could not check or do it. The guard fires at most once per reply and never
in text-tool mode (that mode already direct-executes every plan).
- In native mode the direct-exec result is a `role=tool` message answering
the synthetic assistant `tool_calls` message, and no progress nudge is
appended.

View File

@@ -98,7 +98,7 @@ Design principles enforced by the engine:
- When detected, the engine falls back to the standard "I had trouble understanding that request" error reply (model-size-aware). The malformed content is never shown to the user.
Skipped-tool guard (native tool calling):
- When a native-tool model answers in prose before any tool has run in this reply, and the plan names a callable tool as a concrete call, the engine discards that answer and direct-executes the plan's tool steps, then the model answers from the results. Only literal, schema-valid steps run (no LLM repair); otherwise the held-back answer is delivered unchanged. See `planner.spec.md` ("Skipped-tool guard").
- When a native-tool model answers in prose before any tool has run in this reply, and the plan names a callable tool as a concrete call, the engine discards that answer and direct-executes the plan's tool steps, then the model answers from the results. Only literal, schema-valid steps run (no LLM repair). When the planned call cannot run, the held-back answer is still discarded: the planned call is recorded as a failed tool result and the model answers from that (an honest "could not check or do it"). See `planner.spec.md` ("Skipped-tool guard").
Task-list planner (all model sizes, strongest impact on small models):
- The planner runs at the **front** of the reply flow (see step 3 above), not after tool selection. By the time the agentic loop starts, the plan already exists, the memory block has either been run or skipped based on the plan's `searchMemory` directive, and the tool allow-list has been derived from the tool names the plan referenced. See `planner.spec.md` for the prompt contract and fail-open semantics.