Ten eval files pinned their reply engine to http://localhost:11434 and the
judge only spoke Ollama's /api/tags and /api/chat, so pointing the suite at
Strata (OpenAI-compatible /v1 only) skipped or broke most evals.
- MockConfig and every eval config read EVAL_JUDGE_BASE_URL.
- The judge availability check reads /v1/models and the judge call goes
through jarvis.llm when the URL ends in /v1; the Ollama path is unchanged.
- run_evals.sh probes /models for a /v1 URL.
The intent-judge and listener evals stay pinned to gemma4:e2b on local Ollama.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the game PC the game went behind another window right after Jarvis
started and stayed there for three windows, drawing at its background rate
(about 58 fps). The kit only checked the window in front at the end of a
window, so the window in which the user clicked back into the game counted
and showed yielding as a 47 % drop in 1 % low.
- FrontWatch checks every second from the start of a window to its end and
spoils the window if the game was behind anything at any check.
- Before a window the kit asks the user, with a beep, to go back to the
game and waits up to two minutes.
- What was in front is kept only as a category (a Jarvis process, a console
window, Explorer, another app), never an app name.
Results: with the game in front, a busy Jarvis cost 0.2 % of the average
fps and 3.3 % of the 1 % low with no frames over 50 ms, so by the
pre-registered rule nothing is added and yielding is not adopted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Frame rate, 1 % low and hitches come from msBetweenDisplayChange of the
frames that reached the screen; frames never shown are counted apart and
the presented rate is kept as presented_fps. A game presenting 60 fps
while the screen changes every 167 ms now shows as the stutter it is.
- A window whose capture failed, a busy window where the LLM finished no
answer, and every busy window when the STT load failed are not counted.
- A yielding window counts only when every Jarvis process got both its
CPU and its GPU priority lowered.
- The "game hardly drawing" floor applies only with Jarvis off or idle, so
a game Jarvis slows that far still counts as harm.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 95 % GPU cap is gone: it did not lower utilisation, slowed the LLM and
a YouTube video did not stutter without it. What it was meant to protect
is now measured directly in a game. The new on-demand game_frames stage
records the game's frame times with PresentMon (pinned by hash, game
process only, CSV deleted after reading) with Jarvis off, idle and busy in
the game placement, and busy while Jarvis yields: its processes at
below-normal CPU and GPU scheduling priority, read back to confirm, then
restored. Busy rows run A B B A. The design docs record the candidate and
the pre-registered rule for adopting it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With DevTools on (port or pipe), Chrome tells pages a program drives it, and
YouTube then shows "문제가 발생했습니다" after about a minute of video, which
spoilt every window on the gaming PC. Reproduced on Linux: the port alone,
with no command sent, breaks playback at ~40 s; without it, or with
--disable-blink-features=AutomationControlled, it plays on. The browser is now
started with that switch, and one that still reports webdriver is refused
before Jarvis is started.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When loading the page again did not bring a player back, keep_room still
moved the missing player back to the start, the page script threw and the
stage stopped at the first window. Now a failed reload (or a refused
navigation) is recorded as "the video did not play again", only a playing
video is moved back (and the script checks the player exists), so the
window is measured as it is, not counted, and tried again up to the limit
before the stage goes on. The reload waits RELOAD_S for the video.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the gaming PC the player errored about 38 s into the first window with
Jarvis off; the kit had no way to notice, so every later window measured a
paused, empty player, and after the user reloaded the page the injected
counters were gone and the snapshot script threw, ending the stage early.
- The snapshot reports a lost page (counters gone, or the script could not
run while the page loaded) and the player's own error screen (first line
only, bracketed IDs removed) instead of throwing.
- Before each window the kit presses play on a stopped player and loads the
page again when that is not enough, or when the page was lost or errored.
- A window the player spoilt (error, reload, pause, ad, seek, buffering, no
frames) is measured again, up to 3 times in all, with the earlier attempts
and the fixes kept in the row; a lower quality or frame rate is not, since
a busy GPU can cause it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A window counts only when the player kept the asked quality and height
the whole time and played the source's 60 fps; a player stuck below
1080p is a stage error, not a note.
- Pauses, seeks and buffering are counted inside the page during the
window, so a pause that ended before the last snapshot no longer
passes as GPU stutter.
- A capped window whose cap found nothing to pause, ran no pause cycle,
or could not pause or resume a process is an error and does not count
for the cap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User decision (2026-11-11): the 95 % GPU cap is off by default because
gamemode_tune showed it did not lower nvidia-smi's utilisation and cost
6 to 10 % of the LLM's output. The worry behind it was a video stuttering
next to Jarvis, so the new on-demand stage video_drops measures that:
- A public Blender upload plays at 1080p60 in a throwaway Chrome/Edge
profile (removed afterwards), driven over DevTools with a small
standard-library WebSocket client, so no package is added to the
already built venvs.
- Each window counts the video's dropped frames (the player's own
counter) and the gaps over 50/100 ms between shown frames
(requestVideoFrameCallback); windows with an ad, a pause, the end or a
new video are kept but not counted.
- Rows: video alone, Jarvis idle (normal placement with the STT room and
1 GB for other programs), Jarvis busy (LLM answering, STT transcribing
ten speakers back to back) without and with the cap in A B B A order,
then the video alone again. The cap pauses the engine and the STT
worker only, never the browser.
- run_m0_video.bat runs just this stage.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resume that failed was only counted, and the process stayed suspended
after the cap ended. DutyCap now keeps what it still owes per process,
does not stack another suspend on it, retries the resume every period
and at the end, treats a raising call as a failure, and reports any
process it could not resume as left_paused (an error for the row).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User decision (2026-11-10, room to watch a video next to Jarvis): the
normal placement leaves 1 GB of VRAM for programs opened later, and
Jarvis's GPU work is capped at 95 %.
- gamemode_tune's normal rows take OTHER_ROOM_MIB off the expert budget.
- gpucap.DutyCap pauses the Strata engine (strata.exe --serve under the
kit) and the STT worker with their children together for 10 ms of
every 200 ms (NtSuspendProcess), and always resumes them.
- Every row samples nvidia-smi utilization.gpu; capped rows have an
uncapped twin (normal with and without the STT, game 2 GB with 2 s
partials) so the run shows the cost and what the GPU really gets back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User decision (2026-11-10): without a game the STT keeps every partial
pass while the LLM answers. gamemode_tune now measures the normal
placement with the STT room and back-to-back STT only, and paces the
2 GB game placement (back to back as the same-run baseline, every 2 s,
finals only). Docs say the slowdown applies in game mode only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gamemode_stt run showed Strata's automatic placement filling the card
next to the STT (15.9 of 16.3 GB, first token 1.6 -> 7.5 s) and the STT's
back-to-back partial passes cutting LLM output to a third. The new
gamemode_tune stage measures both fixes the user chose:
- stt_room: one start without experts (STT loaded) gives the LLM's base
VRAM; --expert-vram gets what is left after the STT's ten-speaker
growth (from this PC's stt stage) and a 768 MiB margin.
- stt_pace: a ten-speaker partial pass every 2 s, or none, with finals
every 0.5 s and finals first; records final wait and STT GPU share.
run_m0_gamemode_tune.bat runs it unattended.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker that did not stop in time returned its earlier "loaded" note, was
killed, and the placement still counted as ok with no transcription times.
finish() now reports a timeout, and a worker that timed out, died, never
started or measured no ten-speaker or single-clip timing is an error, so
the stage is partial and the next run retries it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gamemode now also starts the 2 GB expert placement with a 16K context and
drops the --kv q4_0 and --mtp off rows the Strata 0.1.59 qwen36 engine
ignores; the normal placement answers one request at a time.
gamemode_stt loads the STT worker (stt.py mode=load) before the LLM picks its
placement and keeps it transcribing ten speakers at once while the LLM is
measured, so the VRAM left is the real game share. The game placements run
with the STT on the GPU and on the CPU. gamemode_live repeats the game rows
while the user runs a real game; it only runs when asked for
(run_m0_game_live.bat) and is skipped when nobody is at the PC.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first game PC run showed that a 2 GB expert budget still left a 16 GB
card only 4.3 GB free: the weights, the 32K int8 conversation memory, the
MTP drafter and buffers take about 8.7 GB on their own. The game placements
now also shrink the memory format (q4_0), the context (16K, 8K) and the
drafter, with half the CPU workers (no speed cost measured). The engine log
is appended across starts, so each placement now reads only its own lines;
when no placement leaves the reserve the result names the most free one.
The gamemode runner skips the desktop stage, which is already measured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gamemode stage was "ok" when only the normal placement started; a
placement that does not start, or a measurement error inside one, now marks
the stage partial. The design no longer raises the expert budget above the
computed one: it shrinks the game reserve to what the smallest placement
leaves, shows that, and pauses the LLM during games when nothing is left.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a gamemode stage that restarts Qwen3.6-35B-A3B with a smaller routed
expert VRAM budget (6, 4, 2 GiB, and 2 GiB with half the CPU workers), then
the normal placement again, and records switch time, free VRAM, output speed,
first spoken sentence and 8K-context first token for each. It names the
fastest placement that leaves the game its 8 GB reserve. Strata 0.1.59's
qwen36 engine cannot shrink VRAM while running, so each placement is a restart.
The desktop stage now matches running apps by the process image name Windows
lists (Discord.exe) and closes the calculator it launched by image name when
close-app cannot find a UWP app.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
check_mcp timed five CLI and five MCP list-windows calls but ignored their
answers, so a server returning isError/ok=false still scored ok. Read the
envelope of every call, record the error code (never the message), and
fail the check on any failed call.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gaming-PC rerun scored the fast model's honest refusals ('...할 수 있는
도구는 없습니다', '...준비되어 있지 않습니다') as failures, so the kit
downloaded two more fast-model candidates for nothing. Rescored with the
fixed judge, UD-Q4_K_XL, UD-Q6_K_XL and Q8_0 all pass 27/27; the
recommended model's Q2_0/IQ3_XXS/IQ3_S still fail 3/27 by calling an
unrelated tool when no tool fits. Record both results in the v2 docs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a 'desktop' stage to the M0 kit that installs the pinned agent-desktop
1.2.0 Windows build (SHA-256 checked) and checks, on the gaming PC:
calculator buttons by automation id (12+3=15), set-value and read-back on
a test page in a throwaway browser profile, Discord/Explorer read time,
and a warm MCP server against one CLI process per call. Results keep only
counts, timings, error codes and the calculator digits.
Record agent-desktop (UI Automation) as the planned M7 control path in the
v2 design docs, with raw Windows input only for screens without
accessibility.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The temperature and minute checks dropped the sign and the decimal part,
so "13.9도" and "-13도" passed as 13 degrees. Numbers are now read whole
("-", "영하" and decimals included) and must equal the tool result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tools stage decides whether to keep a model or measure the next one,
so a made-up answer must never pass. Each case now has a judge for the
spoken answer: numbers must match exactly with their unit (15 or 50
minutes fail a 5-minute timer), weather answers must agree with the rain
in the result and give no other temperature, and a refusal fails when it
also says the thing was done. A new case asks for a timer with no timer
tool offered, the shape Jarvis has today. Unit tests cover the reviewer's
counterexamples and that a lying model makes the stage move on to the
next candidate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the skipped-tool guard could not run the planned call (not a literal
call, or arguments the tool does not take) it delivered the answer it had
held back, which was given without the tool and could be a made-up value.
The planned call is now recorded as a failed tool result and the model
answers from that, so the user hears that it could not check or do it.
M0 kit: a case passes only when the final answer uses the tool result
(13 degrees, not a made-up 24) or, for the no-tool request, says it
cannot do it; the no-tool request now counts. The kit mirrors the
engine's failed-result path, and a failing model moves on to the next
candidate on these stricter checks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User rule: if the tool-calling fix does not work, change the model. The
tools_fast / tools_recommended stages now measure the next, less
compressed model of the same family (fast: UD-Q6_K_XL then Q8_0;
recommended: IQ3_XXS then IQ3_S) when the default still gets a scored
case wrong, stop at the first one that passes every case, and record it
as tools_verdict.chosen. Candidates without enough free disk are skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
M0 on Strata caught both v2 models skipping a tool they had been given:
the recommended model made up "24 degrees, sunny" instead of calling the
weather tool, and the fast model said "timer set" without calling the
timer tool (3 of 3 each). qwen2.5:7b does the same through the engine.
- Prompt: TOOL_HONESTY_RULES in the persona (live facts only from a tool
result this turn, actions only claimed after a tool reports success, no
answer text next to a tool call), restated at the end of the system
message by tool_honesty_reminder() with the turn's real tool names.
- Engine: skipped-tool guard. When a native-tool model answers before any
tool has run and the plan names a callable tool as a concrete call, the
answer is dropped and the plan steps are direct-executed, then the model
answers from the results. Only literal, schema-valid steps run (regex
fast path, no LLM repair); otherwise the held-back answer is delivered.
- Planner: the AVAILABLE TOOLS lines now carry argument names
(planner_tool_catalog), so concrete steps use the tool's real keys.
- Eval evals/test_tool_skip_confabulation.py; on qwen2.5:7b, 3 runs each,
available-tool cases went from 0/12 to 9/12 calling the tool.
- M0 kit: tools_fast / tools_recommended stages (run_m0_tools.bat) rerun
tool calling before and after the fix on the real models, loading the
production system_prompt.py and planner.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user wants the bot window on no real screen at all. Add a vdisplay
stage that drives the Parsec Virtual Display Adapter already on the PC
(no install, the display dies with the script), moves the clock window
onto it as a normal and as a tool window, and asks about real monitors,
Alt+Tab, Discord's Screens tab and whether the cursor can wander there.
Record the M0 defaults the user confirmed in docs/v2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Strata 0.1.59 gives Qwen3.8-Flash-Next Q2_0 the DFlash2 drafter alone
(--draft-fallback none), but DFlash2 needs a native pack and the canonical
Q2_0 pack is not one, so the engine stops at start ("DFlash2 is off here:
it needs a native pack ... the start ends here"). Ask for the MTP drafter
instead; Strata falls back to no drafter when the MTP layer is missing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first gaming-PC run failed the recommended model with only Strata's
one-line reason ("no CUDA-capable device"), which is the picture
reader's side message, not the cause. The engine log stayed on the PC.
A launch that does not become ready now copies the newest
*.engine.log (scrubbed, last 200 KB) into logs/, which the results zip
already packs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The context test re-estimated characters per token from the server's
prompt_tokens after every request. A server that reports a tiny count
made each next prompt grow by orders of magnitude, so the kit tests
reached 8 GB and were killed by the host memory guard. Clamp the
estimate to 0.5-4 characters per token.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A Strata install or start that never reached a server is saved as
"failed" (not "partial"), so the next run retries the stage.
- A launch that exits non-zero is not ready, even if something answers
on the port afterwards.
- Before the cold start, the stage stops if another program already
listens on port 8080 and tells the user to close it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One double-click kit (scripts/m0/run_m0.bat) that measures the gaming PC
and packs the results into a single zip:
- PC info (GPU, driver, RAM, disks, monitors)
- Windows checks with the user: bot browser window hidden from Alt+Tab
but listed in Discord's share picker and still drawing while covered;
Discord pop-out stream window identification and occlusion; installed
games, anti-cheat services and the borderless game detector
- MeloTTS, CosyVoice2 and Fun-CosyVoice3 Korean synthesis speed
- faster-whisper large-v3-turbo: final/partial latency, CER, wake-word
hint, batched decoding for several speakers
- Strata LLM (fast and recommended presets): TTFT by context length,
decode speed, tool calls, picture reading per quality tier, Korean
answers, parallel requests, VRAM give-back and restart/switch timings
uv and Python, per-stage venvs, models and Strata all live inside the
work folder. Results are scrubbed of user/computer names and home paths.
Verified on Linux GPU (LLM client against Ollama, STT, MeloTTS,
CosyVoice2/3) and on the Windows 11 CI VM (window checks, game scan,
MeloTTS/CosyVoice2/STT installs, Strata install and plan).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>