live_round_trip.rs runs the app's services with the real worker and brain,
plays recorded questions from a second bot and times the final, the
answer text, playback start and the first audible frame after the speech
ends; a second worker session streams nine more voices for the load run.
Roadmap: first M4 step 4 numbers (median 3.5 s alone, 4.4 s under load).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pause after the name ("자비스, ... 지금 몇 시야?") is longer than the
worker's 300 ms end silence, so the wake word ended its own utterance:
Jarvis answered the bare name and ignored the question that followed (no
wake word). Found on real Discord. A voice utterance that is only a wake
word is now joined to the same speaker's next final when that utterance
is running or starts within call.followUpMs (default 1500); otherwise the
wake word alone is the request after that time. Stops drop it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
v1's tts_engine default (English-only Piper) made the reply engine force
English, so Korean questions got English answers. The v2 brain speaks
through the worker's MeloTTS and chat answers are not spoken, so it runs
with tts_engine=melo; an output_language lock still wins.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A person's reply stopped counting as "unanswered" as soon as the worker sent
reply_text, so a delete question that could not be spoken (no TTS, a failed
or stopped speech) or that the app could not post still let the next
"응, 지워" confirm the delete.
- Worker: an answer reaches the person on a tts_done neither failed nor
cancelled (spoken answers), on the app's reply_delivered ok (chat
answers), or at reply_text (unspoken voice answers). tts_unavailable, a
failed or cancelled tts_done, reply_delivered ok=false, no report within
30 s and the session ending all mean it never did: a turn number is
skipped, so the delete asked in it lapses. A reply sent before the
previous answer reached them is still not their next turn.
- App: a chat answer reports reply_delivered with whether Discord took the
post; a late, failed or empty answer reports it was not posted.
- Tests: no TTS, a failed speech, a chat post the app reports failed, and
no report at all do not delete; a spoken and a posted question do.
Conversation and worker client tests for the report.
- Specs: voice_worker.spec.md (reply_delivered, when an answer reaches the
person), discord_core.spec.md, conversation.spec.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A delete is confirmed only in the person's very next turn, but turn numbers
were given when a reply started. A "응, 지워" sent while the delete question
was still being written waited for the person's lock and then counted as the
next turn, so the delete went through.
- The worker counts each person's unanswered replies. A reply that arrived
before their previous answer was given skips a turn number, so a delete
asked in that answer lapses.
- After an answer the person never got (failed, or stopped before it was
spoken) a turn number is skipped as well: it asked them nothing.
- Tests with the real chatChannels tool and an agreeing consent check: a yes
sent before the answer does not delete, a yes after a failed answer does
not delete, a yes after the answer deletes.
- Specs: voice_worker.spec.md (turn numbering), chat_channels.spec.md (the
next turn is a message sent after the question was received).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The voice and chat lanes let a person's voice reply start while their chat
reply was still running. The chat delete consent counts "the very next
message" by turn number, so a message that started before the delete was
asked could pass as the confirmation, and the person's dialogue memory
was updated out of order.
- A reply takes its asker's lock, then its lane's: one person's replies
stay one at a time in the order asked, different people's voice and
chat answers still overlap, and a reply waiting for its asker holds no
lane. Test: a person's voice question starts only after their chat
answer.
- Specs and roadmap: the lanes are per person; the roadmap no longer says
voice answers wait behind chat answers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The voice worker writes voice and chat replies in separate lanes (one
of each at a time), so a chat answer being written never holds up a
voice answer. Tests: a voice reply finishes while a chat reply is still
running; replies of one kind stay one at a time.
- A chat answer arriving 5 minutes or more after it was asked is not
posted, checked when it arrives.
- Specs: the lanes, and that Strata needs --parallel 2 for both lanes'
LLM calls to run at once until the app starts Strata itself (M5).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- chatChannels[].answer: wakeWord (default) or everything. Chat uses
call.chatWakeWords (else the voice wake words) with the shared
designated-people switches.
- The brain's answer (reply via chat, not spoken) is posted as a Discord
reply to the message, pinging nobody; failed or empty answers post
nothing. Bots, channels with sending off and stops are not answered.
- Chat runs beside the voice conversation and never takes its turn.
- Core: send_chat_reply, chat_answer and the ReplyText event; the worker
client's reply takes via.
- Specs (conversation, discord core) and the roadmap updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A voice stop also voids pending "addressed" questions, so a yes that
arrives afterwards starts nothing.
- The core sends Stopped before the SpeakEnds of what stop() or the end
of a session cancelled; the conversation drops its queue on it, so the
slash command's stop no longer starts the next waiting request.
- An addressed answer after its 5 s timeout counts as no even when it is
handled before the timer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- conversation.rs: the loop between the core's events and the rest.
Finals go through call detection and the response policy; answers are
asked of the brain and spoken from the app's queue, an interruption
cancels only that answer, a voice stop silences everything, and missed
requests are kept without text (last 100). Tested with a fake voice side.
- Core: answer(user, text) adopts a playback request the worker speaks
under the same id, cancel(id) stops one request, ask_addressed and
ask_rank pass the conversation's questions; new SpeakingStopped,
Addressed and Ranked events.
- Worker client: reply, addressed and rank messages and their results; a
reply with no answer, refused or not sent ends its speech as failed.
- Voice worker: addressed and rank messages on their own threads, beside
a running reply; a spoken reply cancelled while the brain is answering
is not spoken. judges.py holds the two short LLM judgements with the
heard text fenced as data; eval evals/test_voice_judges.py.
- Specs (conversation, discord core, voice worker), docs/llm_contexts.md
and the roadmap updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Action::Rank and ranked() carry a ranking id; answers to a replaced or
dropped ranking are ignored.
- A candidate tying the ranked ones while a ranking is pending replaces
it with a new ranking over all of them (plus the running answer when
the ranking is about interrupting it), keeping the first deadline, so
only compared candidates are missed as lower_score. A better one makes
the ranking moot.
- next_deadline returns the collect timer while collecting, and the
settle deadline only while an ending utterance holds the choice up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A new request that ties the running answer on priority and wake word
is ranked against it by the brain; a win interrupts, a loss or a
timeout lets the answer go on. The ranking is dropped when the answer
ends first.
- A ranking that arrives after its deadline counts as a timeout.
- Candidates collected while an answer runs keep end-time order, so the
timeout fallback picks the earliest-ended one.
- A spent collect timer is cleared, so next_deadline reports the settle
deadline instead of a time already passed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- calls.rs: wake words anywhere in the utterance with near matches by
edit distance on NFKD-normalised letters (Hangul splits into its
letters), call modes (wakeWord, addressed, everything), designated
people with both switches, and a voice stop (wake word + stop word).
- policy.rs: first, inOrder, latest and important as an event-driven
state machine with the time passed in: ordering by end time, waiting
for an earlier-ended utterance still being transcribed, queue limit,
interruption, importance ranking only on a tie with a timeout, stop
dropping the queue, and a reason for every missed request.
- settings: new call and response sections, validated with the key.
- conversation.spec.md documents the contract; registry and roadmap
updated. Wiring into the Discord core and the brain comes next.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Drop the engine's deterministic Korean weather shortcut. It bypassed the
chat model and the tool runner, so a location the user had stated could
never be used. The tool already returns one clean Korean sentence.
- getWeather and the tool-argument prompts pass the place the user has
said they are (conversation or memory) and leave the argument empty
otherwise, so the detected location is not copied into it.
- getWeather retries a "district, city, country" name with its leading
parts dropped when the geocoder finds nothing.
- The planner gets the live time/location line and plans
searchMemory topic='where the user lives' only when the location is
unknown or disabled.
- The per-reply repeat guard counts only this reply's tool messages, so
calls carried over from the previous reply no longer block a retry the
user asks for.
- Memory extractor: the context block sits after the examples and the
no-location example uses a different query, so a location already in
context is not asked for again. New counterpart eval checks that it is
still asked for when the context has none.
- Tool retry eval fails every call in turn 1, so a model that retries by
itself inside turn 1 still exercises the user's "try again".
- Specs (planner, reply), docs/llm_contexts.md and the roadmap updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ten eval files pinned their reply engine to http://localhost:11434 and the
judge only spoke Ollama's /api/tags and /api/chat, so pointing the suite at
Strata (OpenAI-compatible /v1 only) skipped or broke most evals.
- MockConfig and every eval config read EVAL_JUDGE_BASE_URL.
- The judge availability check reads /v1/models and the judge call goes
through jarvis.llm when the URL ends in /v1; the Ollama path is unchanged.
- run_evals.sh probes /models for a /v1 URL.
The intent-judge and listener evals stay pinned to gemma4:e2b on local Ollama.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Weather: the tool deliberately returns one Korean sentence about now
(c522e1b), so the tests check the place, condition, rounded temperature and
feels-like, and that no forecast, Fahrenheit or degree sign is left for the
model to echo.
- Enrichment cache: the conversational fast-path skips enrichment when no
data tool is routed, so the test routes one to reach the extractor.
- Diary import: mock PyQt6, requests and psutil only when they are not
installed. The MagicMock left in sys.modules for an installed PyQt6 broke
every later GUI test file in the same run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The consent classifier is now told which message and channel are pending, so a yes for another
message ("yes, delete message 22" while 21 is pending) is a no and cancels the request. A reply
longer than 600 characters is no longer cut and judged by its start (which could drop a later
"don't"); it counts as no without a model call.
Eval evals/test_chat_delete_consent.py gains the different-message and long-reply cases:
17/17 on qwen2.5:7b, three runs. The previous prompt said YES 3/3 to
"응, 일정 채널에 있는 거 지워" while a message in 잡담 was pending.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- chatChannels delete with confirm=true now runs only in the person's very
next turn and only when their own message of that turn agrees, checked by a
small fail-closed YES/NO call on the router model chain (the message fenced
as data). A refusal, question, other topic or error cancels the request;
an unconfirmed request lapses after one turn. The reply model's confirm
flag alone can no longer delete.
- chat_action ids carry a per-run nonce and the session, so a late answer
meant for an earlier connection (or worker run) never completes a newer
action.
- Tests: agree, refuse, other topic, lapse, classifier failure, late answer
across reconnects. Eval test_chat_delete_consent: 12 of 12 on qwen2.5:7b,
3 runs. Spec and llm_contexts updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Tool (chat_channels.spec.md): list, read, search, send, edit, react,
delete, pin, unpin. Offered only when the v2 brain gives the turn a chat
backend, and then on every turn. Channel by ID, name, part of a name or a
name inside the text; a message ID finds its channel; planner forms of the
action ("send_message") work. Read and search results are fenced as
untrusted data. delete only reports until called again with confirm=true
in a later turn of the same person.
- Worker: chat_channels from the app, ChatBridge reads/searches the history
store and sends chat_action, waiting for chat_action_result (15 s timeout,
failed on session close); per-person reply turns and pending deletes.
- App: channel switches edit/react (on) and delete/pin (off); chat_channels
sent on worker ready and after login; chat_action runs send, edit (own
messages only), react, delete, pin, unpin and answers with ids or the error.
- Engine: chatChannels left out of the catalogue without a backend; the
"router returned everything" check now tolerates that.
- live_discord: edit own, refuse editing the speaker's, react, pin (Discord
refuses in a voice channel's chat, reported as an error), delete, refuse an
unregistered channel; passed in 109.4 s.
- Eval (qwen2.5:7b, 3 runs): 10 of 12; never obeyed the "delete everything"
message and never deleted in the same turn.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- meals.user_id (v1 databases gain it on open, DEFAULT '' keeps their meals
the single user's); insert_meal, get_meals_between and delete_meal take
the person, and delete only removes that person's meal.
- turn_state.set_user / ToolContext.user_id carry the asker; logMeal,
fetchMeals and deleteMeal use it. The v2 brain sets it for each reply.
- Tests: two people never see or delete each other's meals, no user is v1's
single user, and a v1 database opens with its meals kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A final is stored under the channel, speaker name and time fixed when its
end was decided. Decoding takes a while; a move, rename or late joiner
meanwhile used to change where and when it was said, and who heard it.
- At start the worker closes presence a killed run left open
(HistoryStore.close_open_presence), so a person who is gone no longer
hears new speech after a restart.
- run_reply_engine(legacy_memory=False) skips v1's single-user memory (warm
profile, extractor, diary, graph, digest); the v2 brain uses it so another
user's profile or diary never reaches the prompt.
- While a brain reply runs, stdout and stderr are muted (also helper threads
such as the web fetch pool); the worker logs through voice_worker/log.py to
the original stderr.
- Live check: Ollama's /v1 fixes the context at 2,048 tokens and cut the
prompt in earlier runs; with an 8k tag the chat-record question was answered
4 of 4 and nothing reached the output. Noted in llm_contexts and the eval.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Worker (voice_worker.spec.md "History and brain"):
- --history <file> keeps the conversation history store; --brain hosts the
jarvis reply engine (JarvisBrain, settings point ollama_base_url at /v1).
- voice_join / voice_leave / member_join / member_leave keep presence; the
session end closes it. Non-empty finals are stored under the speaker's name
in the current voice channel.
- chat_message / chat_edit / chat_delete keep registered chat channels.
- reply builds the asker's scoped history context, runs the brain on a thread
(one at a time), answers reply_text, stores a voice answer, and with
speak=true says it under the same id. memory settings in hello and
memory_config. brain_unavailable without --brain.
App (discord_core.spec.md "History reporting"):
- WorkerClient sends the new messages; the core reports the voice channel
and its heard members after a join, a move and every worker reconnect,
member joins and leaves, voice_leave when a session ends, and every message
of a readable registered channel (its own too, with `to` on replies), edits
and deletes.
- live_discord runs the worker with --history and checks the store at the
end: 8 finals under the speaker's name, the edited chat message only,
Jarvis's own messages, 5 presence intervals, none left open (passed, 107.8 s).
Eval: history recall gains a chat-record case and fails a reply that reads a
record stamp out (field run with qwen2.5:7b did once in four).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Search scored only the newest 5,000 matching records, so an old record
behind enough newer chatter was never found. Both passes (word rarity counts
and the score) now run in SQLite over every record in scope and only the
best rows come back.
- context() excluded the recent window from search and then cut that window
for the budget, so a relevant recent record could vanish. Search now covers
the recent window, matches keep their half of the budget, and a matched
record is removed from the recent block so it appears once.
- Matching used SQLite's ASCII-only lower() against Python's lowercasing.
Stored text gets a folded column (NFKC + casefold) and query words are
folded the same way, so case is ignored in every script.
- Prompt label "Records related to this question" (matches may now be recent);
eval re-run 3x on qwen2.5:7b, all pass. Spec, memory.md and roadmap updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- src/jarvis/memory/history.py keeps voice transcripts, Jarvis's answers,
registered chat channel messages (with edits and deletes) and voice channel
presence in one SQLite file. Every read is scoped to the asker: own records
always, chat channels and others' words (only while the asker was in the
same voice channel) behind settings, plus an optional since and off.
- Search is language-agnostic: Unicode words, prefixes keeping 60% of the
word, rarer words weigh more, newer wins ties. 100k records search in
24-61 ms here. context() returns the recent window and the matches cut to a
token budget, matches at most half so chatter cannot push out the record
the question is about.
- run_reply_engine(history=...) puts that context into the system prompt
under "Information shared in previous conversations", fenced as data.
evals/test_history_context_recall.py: with the block qwen2.5:7b answers the
stored day in Korean and English (3 of 3 runs), without it the day is unknown.
- Spec history.spec.md, registry entry in CLAUDE.md, reply.spec.md,
docs/llm_contexts.md, docs/v2/memory.md and roadmap updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A base URL that ends in /v1 now selects the OpenAI-compatible API: chat goes
to /chat/completions and embeddings to /embeddings. The conversion lives in
src/jarvis/llm.py only, so the reply engine and every other caller keep their
Ollama-shaped messages and responses.
- Requests drop private engine keys, send tool-call arguments as JSON strings,
map num_predict to max_tokens and the thinking switch to
chat_template_kwargs.enable_thinking, and leave out num_ctx and keep_alive.
- Responses come back with message.content, tool calls with dict arguments and
an id, and message.thinking from reasoning_content.
- Streaming, the warm-up (one-token chat completion), the intent judge and the
dictation clean-up use the same path. A server without an embedding model
fails open to the text paths.
- Specs and docs/llm_contexts.md describe the transport; roadmap records M3
step 1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Same-server moves point the existing songbird connection at the new channel and replace its
handlers; removing and rejoining at once let the late gateway "left" update cancel the new join.
- A driver disconnect without a reason (our own leave or move) is ignored, and a disconnect only ends
the session its handler belongs to, so an old connection's late event cannot end a newer one.
- Live test: move to a second voice channel and back; the session survives both, only the two moves
end one, and the speaker is still heard afterwards.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Live test: the speaker leaves and rejoins three times; each time its sentence is heard exactly
under the new SSRC (songbird issue #310 reports receive stalls after group changes; not reproduced).
- decisions.md: Rust over Electron, with the side-by-side results, the songbird risk and the fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user asked for a faster, real app instead of Electron. The app is now a Rust crate (serenity +
songbird); the GUI in M5 will be a Tauri 2 shell on the system WebView2.
- Same behaviour as the Electron core: one voice channel, per-speaker PCM to the voice worker,
speaking_end and speaker_leave, sentence playback with stop and cancel, transcription notice on
join, slash command join/leave/stop (create-or-update), registered chat channels with live events,
history and sends that never ping, worker process with a fresh token and restart backoff.
- songbird 0.6.1 garbles about a third of received frames under DAVE (its decoder keeps the bytes
DAVE decryption leaves at the end of the payload; a 20 ms stepped test tone showed repeats, skips
and reordering, while sending was exact). songbird now only decrypts; the core reorders packets in
a small window (Opus FEC for the last lost frame) and decodes them with libopus itself.
- Live test on real Discord with two bots: five sentences heard (CER 0/0/0/5/12.5%), 0.71-0.90 s
from end of speech to final, speech read back at 3.4%, sound gone 0.08 s after stop. The release
binary uses about 16 MB of memory; a Windows build (MinGW) logs in under Wine.
- Spec moved to app/src/discord_core.spec.md; architecture, decisions, roadmap and pc-features updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- app/ (Electron 45 + TypeScript): the core joins one voice channel, decodes each speaker's Opus to
48 kHz stereo and streams it to the voice worker, sends speaking_end and speaker_leave, and emits
speech_start/partial/final with the speaker's display name.
- Sentence playback from the worker's TTS, first come first served, with stop that silences the clip
at once and cancels unfinished requests; slash command join/leave/stop (create-or-update by name).
- Transcription notice posted on every join; other bots ignored by default; at most maxSpeakers
people transcribed at once.
- Registered chat channels: live message events, history reads, sends split at 2,000 characters with
mentions disabled; Message Content intent checked before login.
- The app starts the Python voice worker with a fresh token in the environment, restarts it with
backoff, and the worker client reconnects and fails in-flight TTS on a lost connection.
- Spec in app/src/core/discord_core.spec.md; Vitest unit tests, a protocol test against the real
Python worker server with fake engines, and an opt-in live test with two bots on real Discord.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- New speaker_leave message: ends the speaker's open utterance (its final is still sent) and drops
their state, so a long session never hits the 32-speaker limit as people come and go.
- Spec and test cover more than 32 speakers in one session, each leaving mid-utterance.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hotword occurrences were counted with fuzzy matching, which never counted a one-letter wake word
(so a hinted "봇 ..." was accepted over a plain decode without it) and counted the real word
"서비스" as "자비스" (so the hint could turn it into the wake word). Occurrences are now counted
exactly (case, spacing and punctuation ignored); near misses are left as heard for call detection.
The real-noise eval now plays each recording as recorded and at -25 dBFS and reports how many clips
reached the model.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Loudness equalisation replaces peak normalisation: rumble below 60 Hz removed, level measured on
the VAD-voiced windows only, brought to -20 dBFS (gain -20..+40 dB), soft limiter below 0.98.
- Only clearly voiced windows (VAD >= 0.5) count towards the 250 ms needed before an utterance
reaches the model, so noise the VAD half-believes never calls Whisper.
- Hallucination gate: mean token log-probability >= -0.7 (the old len/(len+1) correction let short
hallucinations through), compression ratio <= 2.4; no_speech_prob dropped (always 0.0 on turbo).
- With hotwords, the plain decode of the same encoder pass decides; the hint may fix a wake word's
spelling but cannot add or drop one (it wrote the wake word on laughter and after real sentences).
- Opt-in real-model evals: voice-like babble with the hint, speech over babble, quiet and clipped
speakers, and a JARVIS_NOISE_DIR folder of real noise recordings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- One shared faster-whisper model for every speaker, batched; finals go before partials and
partial batches are capped at two so a final never waits behind a ten-speaker pass.
- Per-speaker 48k stereo to 16k mono conversion, streaming Silero VAD, utterance ends on VAD
silence, Discord speaking_end, missing audio, or the 28 s Whisper window.
- MeloTTS in its own venv through a child process; sentences stream out one chunk each, with
incremental text, cancel and failure reporting.
- Loopback-only server with a token from the environment; logs carry no transcript text.
- Spec in src/voice_worker/voice_worker.spec.md; unit tests with fake engines and an opt-in
real-GPU test that streams ten speakers at once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the game PC the game went behind another window right after Jarvis
started and stayed there for three windows, drawing at its background rate
(about 58 fps). The kit only checked the window in front at the end of a
window, so the window in which the user clicked back into the game counted
and showed yielding as a 47 % drop in 1 % low.
- FrontWatch checks every second from the start of a window to its end and
spoils the window if the game was behind anything at any check.
- Before a window the kit asks the user, with a beep, to go back to the
game and waits up to two minutes.
- What was in front is kept only as a category (a Jarvis process, a console
window, Explorer, another app), never an app name.
Results: with the game in front, a busy Jarvis cost 0.2 % of the average
fps and 3.3 % of the 1 % low with no frames over 50 ms, so by the
pre-registered rule nothing is added and yielding is not adopted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Frame rate, 1 % low and hitches come from msBetweenDisplayChange of the
frames that reached the screen; frames never shown are counted apart and
the presented rate is kept as presented_fps. A game presenting 60 fps
while the screen changes every 167 ms now shows as the stutter it is.
- A window whose capture failed, a busy window where the LLM finished no
answer, and every busy window when the STT load failed are not counted.
- A yielding window counts only when every Jarvis process got both its
CPU and its GPU priority lowered.
- The "game hardly drawing" floor applies only with Jarvis off or idle, so
a game Jarvis slows that far still counts as harm.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 95 % GPU cap is gone: it did not lower utilisation, slowed the LLM and
a YouTube video did not stutter without it. What it was meant to protect
is now measured directly in a game. The new on-demand game_frames stage
records the game's frame times with PresentMon (pinned by hash, game
process only, CSV deleted after reading) with Jarvis off, idle and busy in
the game placement, and busy while Jarvis yields: its processes at
below-normal CPU and GPU scheduling priority, read back to confirm, then
restored. Busy rows run A B B A. The design docs record the candidate and
the pre-registered rule for adopting it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
YouTube 1080p60 kept 3-4 dropped frames per window with Jarvis off, idle
and busy (GPU up to 98%), with no gaps over 50 ms and no buffering. The
95% cap did not reduce drops (7 vs 7), so the setting is removed as the
spec already planned.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With DevTools on (port or pipe), Chrome tells pages a program drives it, and
YouTube then shows "문제가 발생했습니다" after about a minute of video, which
spoilt every window on the gaming PC. Reproduced on Linux: the port alone,
with no command sent, breaks playback at ~40 s; without it, or with
--disable-blink-features=AutomationControlled, it plays on. The browser is now
started with that switch, and one that still reports webdriver is refused
before Jarvis is started.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When loading the page again did not bring a player back, keep_room still
moved the missing player back to the start, the page script threw and the
stage stopped at the first window. Now a failed reload (or a refused
navigation) is recorded as "the video did not play again", only a playing
video is moved back (and the script checks the player exists), so the
window is measured as it is, not counted, and tried again up to the limit
before the stage goes on. The reload waits RELOAD_S for the video.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the gaming PC the player errored about 38 s into the first window with
Jarvis off; the kit had no way to notice, so every later window measured a
paused, empty player, and after the user reloaded the page the injected
counters were gone and the snapshot script threw, ending the stage early.
- The snapshot reports a lost page (counters gone, or the script could not
run while the page loaded) and the player's own error screen (first line
only, bracketed IDs removed) instead of throwing.
- Before each window the kit presses play on a stopped player and loads the
page again when that is not enough, or when the page was lost or errored.
- A window the player spoilt (error, reload, pause, ad, seek, buffering, no
frames) is measured again, up to 3 times in all, with the earlier attempts
and the fixes kept in the row; a lower quality or frame rate is not, since
a busy GPU can cause it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A window counts only when the player kept the asked quality and height
the whole time and played the source's 60 fps; a player stuck below
1080p is a stage error, not a note.
- Pauses, seeks and buffering are counted inside the page during the
window, so a pause that ended before the last snapshot no longer
passes as GPU stutter.
- A capped window whose cap found nothing to pause, ran no pause cycle,
or could not pause or resume a process is an error and does not count
for the cap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User decision (2026-11-11): the GPU share cap stays a setting but its
default is 100 % (off), since gamemode_tune showed it did not lower the
utilisation and only slowed the LLM. Whether the setting stays is decided
by the new M0 video_drops stage (a YouTube video's dropped frames with
Jarvis off, idle and busy, busy without and with the 95 % cap).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User decision (2026-11-11): the 95 % GPU cap is off by default because
gamemode_tune showed it did not lower nvidia-smi's utilisation and cost
6 to 10 % of the LLM's output. The worry behind it was a video stuttering
next to Jarvis, so the new on-demand stage video_drops measures that:
- A public Blender upload plays at 1080p60 in a throwaway Chrome/Edge
profile (removed afterwards), driven over DevTools with a small
standard-library WebSocket client, so no package is added to the
already built venvs.
- Each window counts the video's dropped frames (the player's own
counter) and the gaps over 50/100 ms between shown frames
(requestVideoFrameCallback); windows with an ad, a pause, the end or a
new video are kept but not counted.
- Rows: video alone, Jarvis idle (normal placement with the STT room and
1 GB for other programs), Jarvis busy (LLM answering, STT transcribing
ten speakers back to back) without and with the cap in A B B A order,
then the video alone again. The cap pauses the engine and the STT
worker only, never the browser.
- run_m0_video.bat runs just this stage.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 64.9 tok/s game-placement run was compared with the 64.4 tok/s
normal-placement run, which uses different expert, context and image
settings. Compare it only with the 2 s pace of the same placement, and
state that the final transcription in the kit is one fixed sample at
beam 1, so the real beam 2-3 / long-utterance final latency with paused
partials is still unmeasured (to be measured in M1).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Normal placement with the STT room and 1 GB for other programs:
expert VRAM 1.5 GB, 1.8 GB left at the STT peak, context TTFT back
to 1.9 s (was 7.5 s), 64.4 tok/s alone.
- Game mode pauses STT partials while the LLM answers: 64.9 tok/s vs
37.5 tok/s at a 2 s pace, final wait 0.001 s vs up to 1.2 s.
- The 95 % GPU cap paused for 5.2 % but nvidia-smi utilisation did not
drop (96->98 %, 40->39 %) and decode was 6-10 % slower; whether to
keep it is pending a user decision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A resume that failed was only counted, and the process stayed suspended
after the cap ended. DutyCap now keeps what it still owes per process,
does not stack another suspend on it, retries the resume every period
and at the end, treats a raising call as a failure, and reports any
process it could not resume as left_paused (an error for the row).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>