The tools stage decides whether to keep a model or measure the next one,
so a made-up answer must never pass. Each case now has a judge for the
spoken answer: numbers must match exactly with their unit (15 or 50
minutes fail a 5-minute timer), weather answers must agree with the rain
in the result and give no other temperature, and a refusal fails when it
also says the thing was done. A new case asks for a timer with no timer
tool offered, the shape Jarvis has today. Unit tests cover the reviewer's
counterexamples and that a lying model makes the stage move on to the
next candidate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the skipped-tool guard could not run the planned call (not a literal
call, or arguments the tool does not take) it delivered the answer it had
held back, which was given without the tool and could be a made-up value.
The planned call is now recorded as a failed tool result and the model
answers from that, so the user hears that it could not check or do it.
M0 kit: a case passes only when the final answer uses the tool result
(13 degrees, not a made-up 24) or, for the no-tool request, says it
cannot do it; the no-tool request now counts. The kit mirrors the
engine's failed-result path, and a failing model moves on to the next
candidate on these stricter checks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User rule: if the tool-calling fix does not work, change the model. The
tools_fast / tools_recommended stages now measure the next, less
compressed model of the same family (fast: UD-Q6_K_XL then Q8_0;
recommended: IQ3_XXS then IQ3_S) when the default still gets a scored
case wrong, stop at the first one that passes every case, and record it
as tools_verdict.chosen. Candidates without enough free disk are skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
M0 on Strata caught both v2 models skipping a tool they had been given:
the recommended model made up "24 degrees, sunny" instead of calling the
weather tool, and the fast model said "timer set" without calling the
timer tool (3 of 3 each). qwen2.5:7b does the same through the engine.
- Prompt: TOOL_HONESTY_RULES in the persona (live facts only from a tool
result this turn, actions only claimed after a tool reports success, no
answer text next to a tool call), restated at the end of the system
message by tool_honesty_reminder() with the turn's real tool names.
- Engine: skipped-tool guard. When a native-tool model answers before any
tool has run and the plan names a callable tool as a concrete call, the
answer is dropped and the plan steps are direct-executed, then the model
answers from the results. Only literal, schema-valid steps run (regex
fast path, no LLM repair); otherwise the held-back answer is delivered.
- Planner: the AVAILABLE TOOLS lines now carry argument names
(planner_tool_catalog), so concrete steps use the tool's real keys.
- Eval evals/test_tool_skip_confabulation.py; on qwen2.5:7b, 3 runs each,
available-tool cases went from 0/12 to 9/12 calling the tool.
- M0 kit: tools_fast / tools_recommended stages (run_m0_tools.bat) rerun
tool calling before and after the fix on the real models, loading the
production system_prompt.py and planner.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user wants the bot window on no real screen at all. Add a vdisplay
stage that drives the Parsec Virtual Display Adapter already on the PC
(no install, the display dies with the script), moves the clock window
onto it as a normal and as a tool window, and asks about real monitors,
Alt+Tab, Discord's Screens tab and whether the cursor can wander there.
Record the M0 defaults the user confirmed in docs/v2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Strata 0.1.59 gives Qwen3.8-Flash-Next Q2_0 the DFlash2 drafter alone
(--draft-fallback none), but DFlash2 needs a native pack and the canonical
Q2_0 pack is not one, so the engine stops at start ("DFlash2 is off here:
it needs a native pack ... the start ends here"). Ask for the MTP drafter
instead; Strata falls back to no drafter when the MTP layer is missing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first gaming-PC run failed the recommended model with only Strata's
one-line reason ("no CUDA-capable device"), which is the picture
reader's side message, not the cause. The engine log stayed on the PC.
A launch that does not become ready now copies the newest
*.engine.log (scrubbed, last 200 KB) into logs/, which the results zip
already packs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The context test re-estimated characters per token from the server's
prompt_tokens after every request. A server that reports a tiny count
made each next prompt grow by orders of magnitude, so the kit tests
reached 8 GB and were killed by the host memory guard. Clamp the
estimate to 0.5-4 characters per token.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A Strata install or start that never reached a server is saved as
"failed" (not "partial"), so the next run retries the stage.
- A launch that exits non-zero is not ready, even if something answers
on the port afterwards.
- Before the cold start, the stage stops if another program already
listens on port 8080 and tells the user to close it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One double-click kit (scripts/m0/run_m0.bat) that measures the gaming PC
and packs the results into a single zip:
- PC info (GPU, driver, RAM, disks, monitors)
- Windows checks with the user: bot browser window hidden from Alt+Tab
but listed in Discord's share picker and still drawing while covered;
Discord pop-out stream window identification and occlusion; installed
games, anti-cheat services and the borderless game detector
- MeloTTS, CosyVoice2 and Fun-CosyVoice3 Korean synthesis speed
- faster-whisper large-v3-turbo: final/partial latency, CER, wake-word
hint, batched decoding for several speakers
- Strata LLM (fast and recommended presets): TTFT by context length,
decode speed, tool calls, picture reading per quality tier, Korean
answers, parallel requests, VRAM give-back and restart/switch timings
uv and Python, per-stage venvs, models and Strata all live inside the
work folder. Results are scrubbed of user/computer names and home paths.
Verified on Linux GPU (LLM client against Ollama, STT, MeloTTS,
CosyVoice2/3) and on the Windows 11 CI VM (window checks, game scan,
MeloTTS/CosyVoice2/STT installs, Strata install and plan).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>