fix(reply): never deliver the tool-less answer when the planned call cannot run
When the skipped-tool guard could not run the planned call (not a literal call, or arguments the tool does not take) it delivered the answer it had held back, which was given without the tool and could be a made-up value. The planned call is now recorded as a failed tool result and the model answers from that, so the user hears that it could not check or do it. M0 kit: a case passes only when the final answer uses the tool result (13 degrees, not a made-up 24) or, for the no-tool request, says it cannot do it; the no-tool request now counts. The kit mirrors the engine's failed-result path, and a failing model moves on to the next candidate on these stricter checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -2261,14 +2261,37 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
||||
continue
|
||||
if _forced_plan_steps and _tool_results_so_far == 0:
|
||||
# The guard could not run the planned tool after all
|
||||
# (unresolvable step or arguments that do not fit). Fail
|
||||
# open: deliver the answer it held back rather than ask
|
||||
# the model again on the same messages.
|
||||
debug_log("skipped-tool guard: planned tool not runnable, keeping the model answer", "planning")
|
||||
messages.append(_guard_held_message)
|
||||
reply = _guard_held_reply
|
||||
last_candidate_reply = _guard_held_reply
|
||||
break
|
||||
# (not a literal call, or arguments the tool does not
|
||||
# take). The held-back answer was given without the tool,
|
||||
# so it may be a guess: never deliver it. Record the
|
||||
# planned call as a failed tool result and let the model
|
||||
# answer from that, which under the tool honesty rules
|
||||
# means saying it could not check or do it.
|
||||
_fail_step = _plan_tool_steps[0]
|
||||
_fail_name = _fail_step.split()[0]
|
||||
_fail_id = f"call_plan_{uuid.uuid4().hex[:8]}"
|
||||
debug_log(
|
||||
f"skipped-tool guard: planned call not runnable ({_fail_step[:80]!r}); "
|
||||
"asking the model to answer from a failed tool result",
|
||||
"planning",
|
||||
)
|
||||
messages.append({
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [{"id": _fail_id, "type": "function",
|
||||
"function": {"name": _fail_name, "arguments": {}}}],
|
||||
})
|
||||
messages.append({
|
||||
"role": "tool",
|
||||
"tool_call_id": _fail_id,
|
||||
"tool_name": _fail_name,
|
||||
"content": (
|
||||
f"Error: the planned call `{_fail_step}` could not be run, so nothing "
|
||||
"was fetched or done. Tell the user plainly that you could not check "
|
||||
"or do it right now; do not guess a value or claim it was done."
|
||||
),
|
||||
"tool_failed": True,
|
||||
})
|
||||
|
||||
# Update the system message with fresh context (time/location) before each LLM call
|
||||
# Note: We update the first system message rather than appending a new one because
|
||||
@@ -2757,8 +2780,7 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
|
||||
# The model answered without calling the tool the plan chose, so
|
||||
# the answer can only be a guess. Drop it (the synthesis turn must
|
||||
# not anchor on it) and run the planned steps directly.
|
||||
_guard_held_message = messages.pop()
|
||||
_guard_held_reply = candidate_reply
|
||||
messages.pop()
|
||||
_forced_plan_steps = _guard_steps
|
||||
debug_log(
|
||||
f"skipped-tool guard: model answered without the planned tool; "
|
||||
|
||||
@@ -183,10 +183,13 @@ The engine consumes the plan in two phases.
|
||||
against the tool's input schema. An LLM resolver could "repair" a wrong
|
||||
step into a valid but unwanted call (a timer request planned as
|
||||
`setBroadcast duration=5 minutes` was turned into starting the broadcast).
|
||||
- Fail-open: if the first planned step cannot be run as written, the
|
||||
engine delivers the answer it held back, without a second chat call. The
|
||||
guard fires at most once per reply and never in text-tool mode (that mode
|
||||
already direct-executes every plan).
|
||||
- If the first planned step cannot be run as written, the held-back answer
|
||||
is still never delivered (it was given without the tool, so it may be a
|
||||
guess). The engine records the planned call as a failed tool result
|
||||
("could not be run, nothing was fetched or done") and the chat model
|
||||
answers from that, which under the tool honesty rules means saying it
|
||||
could not check or do it. The guard fires at most once per reply and never
|
||||
in text-tool mode (that mode already direct-executes every plan).
|
||||
- In native mode the direct-exec result is a `role=tool` message answering
|
||||
the synthetic assistant `tool_calls` message, and no progress nudge is
|
||||
appended.
|
||||
|
||||
@@ -98,7 +98,7 @@ Design principles enforced by the engine:
|
||||
- When detected, the engine falls back to the standard "I had trouble understanding that request" error reply (model-size-aware). The malformed content is never shown to the user.
|
||||
|
||||
Skipped-tool guard (native tool calling):
|
||||
- When a native-tool model answers in prose before any tool has run in this reply, and the plan names a callable tool as a concrete call, the engine discards that answer and direct-executes the plan's tool steps, then the model answers from the results. Only literal, schema-valid steps run (no LLM repair); otherwise the held-back answer is delivered unchanged. See `planner.spec.md` ("Skipped-tool guard").
|
||||
- When a native-tool model answers in prose before any tool has run in this reply, and the plan names a callable tool as a concrete call, the engine discards that answer and direct-executes the plan's tool steps, then the model answers from the results. Only literal, schema-valid steps run (no LLM repair). When the planned call cannot run, the held-back answer is still discarded: the planned call is recorded as a failed tool result and the model answers from that (an honest "could not check or do it"). See `planner.spec.md` ("Skipped-tool guard").
|
||||
|
||||
Task-list planner (all model sizes, strongest impact on small models):
|
||||
- The planner runs at the **front** of the reply flow (see step 3 above), not after tool selection. By the time the agentic loop starts, the plan already exists, the memory block has either been run or skipped based on the plan's `searchMemory` directive, and the tool allow-list has been derived from the tool names the plan referenced. See `planner.spec.md` for the prompt contract and fail-open semantics.
|
||||
|
||||
Reference in New Issue
Block a user