Commit Graph

30 Commits

Author SHA1 Message Date
javis-bot
f25c26cf53 refactor(app): rewrite the M2 Discord voice core in Rust
The user asked for a faster, real app instead of Electron. The app is now a Rust crate (serenity +
songbird); the GUI in M5 will be a Tauri 2 shell on the system WebView2.

- Same behaviour as the Electron core: one voice channel, per-speaker PCM to the voice worker,
  speaking_end and speaker_leave, sentence playback with stop and cancel, transcription notice on
  join, slash command join/leave/stop (create-or-update), registered chat channels with live events,
  history and sends that never ping, worker process with a fresh token and restart backoff.
- songbird 0.6.1 garbles about a third of received frames under DAVE (its decoder keeps the bytes
  DAVE decryption leaves at the end of the payload; a 20 ms stepped test tone showed repeats, skips
  and reordering, while sending was exact). songbird now only decrypts; the core reorders packets in
  a small window (Opus FEC for the last lost frame) and decodes them with libopus itself.
- Live test on real Discord with two bots: five sentences heard (CER 0/0/0/5/12.5%), 0.71-0.90 s
  from end of speech to final, speech read back at 3.4%, sound gone 0.08 s after stop. The release
  binary uses about 16 MB of memory; a Windows build (MinGW) logs in under Wine.
- Spec moved to app/src/discord_core.spec.md; architecture, decisions, roadmap and pc-features updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:57:53 +09:00
javis-bot
b4926d5e19 docs(v2): note that the game went behind the kit's console because the user clicked it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:36:11 +09:00
javis-bot
8f2470af2a fix(m0): keep the game in front for the whole frames window and record the game_frames results
On the game PC the game went behind another window right after Jarvis
started and stayed there for three windows, drawing at its background rate
(about 58 fps). The kit only checked the window in front at the end of a
window, so the window in which the user clicked back into the game counted
and showed yielding as a 47 % drop in 1 % low.

- FrontWatch checks every second from the start of a window to its end and
  spoils the window if the game was behind anything at any check.
- Before a window the kit asks the user, with a beep, to go back to the
  game and waits up to two minutes.
- What was in front is kept only as a category (a Jarvis process, a console
  window, Explorer, another app), never an app name.

Results: with the game in front, a busy Jarvis cost 0.2 % of the average
fps and 3.3 % of the 1 % low with no frames over 50 ms, so by the
pre-registered rule nothing is added and yielding is not adopted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:35:38 +09:00
javis-bot
cec52165b2 fix(m0): judge game frames by what the screen showed and drop spoilt windows
- Frame rate, 1 % low and hitches come from msBetweenDisplayChange of the
  frames that reached the screen; frames never shown are counted apart and
  the presented rate is kept as presented_fps. A game presenting 60 fps
  while the screen changes every 167 ms now shows as the stutter it is.
- A window whose capture failed, a busy window where the LLM finished no
  answer, and every busy window when the STT load failed are not counted.
- A yielding window counts only when every Jarvis process got both its
  CPU and its GPU priority lowered.
- The "game hardly drawing" floor applies only with Jarvis off or idle, so
  a game Jarvis slows that far still counts as harm.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:12:15 +09:00
javis-bot
dff18c5c7e feat(m0): measure the game's own frame times and try yielding by priority instead of a GPU cap
The 95 % GPU cap is gone: it did not lower utilisation, slowed the LLM and
a YouTube video did not stutter without it. What it was meant to protect
is now measured directly in a game. The new on-demand game_frames stage
records the game's frame times with PresentMon (pinned by hash, game
process only, CSV deleted after reading) with Jarvis off, idle and busy in
the game placement, and busy while Jarvis yields: its processes at
below-normal CPU and GPU scheduling priority, read back to confirm, then
restored. Busy rows run A B B A. The design docs record the candidate and
the pre-registered rule for adopting it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 09:05:55 +09:00
javis-bot
726fca3377 docs(v2): record M0 video_drops results and drop the GPU usage cap setting
YouTube 1080p60 kept 3-4 dropped frames per window with Jarvis off, idle
and busy (GPU up to 98%), with no gaps over 50 ms and no buffering. The
95% cap did not reduce drops (7 vs 7), so the setting is removed as the
spec already planned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:44:01 +09:00
javis-bot
a6908c386e docs(v2): turn the GPU cap off by default and measure video stutter instead
User decision (2026-11-11): the GPU share cap stays a setting but its
default is 100 % (off), since gamemode_tune showed it did not lower the
utilisation and only slowed the LLM. Whether the setting stays is decided
by the new M0 video_drops stage (a YouTube video's dropped frames with
Jarvis off, idle and busy, busy without and with the 95 % cap).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 00:55:26 +09:00
javis-bot
a4f77c3261 docs(v2): compare the paused-partials result only with the 2 s pace
The 64.9 tok/s game-placement run was compared with the 64.4 tok/s
normal-placement run, which uses different expert, context and image
settings. Compare it only with the 2 s pace of the same placement, and
state that the final transcription in the kit is one fixed sample at
beam 1, so the real beam 2-3 / long-utterance final latency with paused
partials is still unmeasured (to be measured in M1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 00:42:40 +09:00
javis-bot
aa34f707b5 docs(v2): record the gamemode_tune results
- Normal placement with the STT room and 1 GB for other programs:
  expert VRAM 1.5 GB, 1.8 GB left at the STT peak, context TTFT back
  to 1.9 s (was 7.5 s), 64.4 tok/s alone.
- Game mode pauses STT partials while the LLM answers: 64.9 tok/s vs
  37.5 tok/s at a 2 s pace, final wait 0.001 s vs up to 1.2 s.
- The 95 % GPU cap paused for 5.2 % but nvidia-smi utilisation did not
  drop (96->98 %, 40->39 %) and decode was 6-10 % slower; whether to
  keep it is pending a user decision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 00:40:43 +09:00
javis-bot
d0938c10f9 docs(v2): record the 1 GB room for other programs and the 95 % GPU cap
User decision (2026-11-10): the normal placement also subtracts a room
for other programs (default 1 GB, a setting), and the service manager
pauses the LLM engine and the voice worker together for 10 ms of every
200 ms (95 %, a setting, 100 % is off) in every stage. Both are to be
measured with M0 gamemode_tune.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 23:52:57 +09:00
javis-bot
3f0ed4343b fix(m0): pace STT partials only in the game placement
User decision (2026-11-10): without a game the STT keeps every partial
pass while the LLM answers. gamemode_tune now measures the normal
placement with the STT room and back-to-back STT only, and paces the
2 GB game placement (back to back as the same-run baseline, every 2 s,
finals only). Docs say the slowdown applies in game mode only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:37:16 +09:00
javis-bot
b3c26417d1 docs(v2): record the 2 GB game minimum, the STT room and paced partials
User decisions (2026-11-10): the game-mode minimum on a 16 GB card is the
2 GB expert placement, the normal placement leaves the STT its share by
formula instead of Strata's automatic placement, and STT partials slow
down while the LLM answers with finals handled at once. Also notes the
user's report of occasional, minor frame drops during the live run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:35:23 +09:00
javis-bot
a2e4f064d2 docs(v2): mark the game-mode latency sums and causes as estimates
The voice-start figures add medians from different runs and leave out
the TTS, the VRAM-full cause of the 7.5 s first token is a hypothesis,
and the 2 GB placement is 2.1x faster with the STT on the GPU but 1.5x
with it on the CPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:14:25 +09:00
javis-bot
7da86262de docs(v2): record the LLM next to the STT and in a real game
gamemode (third run), gamemode_stt and gamemode_live finished on the
game PC. The STT on the GPU takes 2.7 GB at its 10-speaker peak, as
estimated, and leaves the game 3.6 GB (experts 0 GB) or 2.6 GB (2 GB).
The STT stays on the GPU in games: on the CPU one utterance takes
3.0-3.8 s. In a real game the 2 GB expert placement is twice as fast as
0 GB. The GPU contention between the STT and the LLM, the in-game floor
placement and an STT margin for the normal placement wait for the user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 21:12:19 +09:00
javis-bot
ca5bb520b9 docs(v2): tie LLM concurrency to the chat channels that answer every message
The user meant the chat channel's "answer every message", not the voice
policy "in order": the LLM takes two or more requests at once only while a
chat channel answers every message, otherwise one whatever the voice policy.
The roadmap and the open measurements point at the new gamemode_stt and
gamemode_live stages, which also decide whether the STT moves to the CPU
during a game.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 19:12:53 +09:00
javis-bot
d0ffa7f55e docs(v2): budget STT at its 10-speaker peak in the game VRAM estimate
The M0 stt stage measured 1300 MiB for the model and another 1440 MiB while
transcribing ten speakers at once. The game share now subtracts the 2.7 GB
peak: about 3.1 GB on the 16 GB game PC (not 4.6 GB), and the 2 GB expert
placement returns at a reserve of about 2.2 GB. Both are marked as sums of
separate LLM and STT runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 18:44:29 +09:00
javis-bot
83304255f8 docs(v2): tie LLM concurrency to the response policy and record the game placement result
The user chose one request at a time by default (97 tok/s on a 16 GB card
against 70-75 with two) and two or more only under the "in order" policy,
which answers every request. The second gamemode run showed a 16 GB card
cannot free 8 GB for a game: the smallest placement still leaves 6.3 GB.
The Strata 0.1.59 qwen36 engine source confirms why --kv and --mtp off made
no difference: it takes no --kv flag and always loads the MTP block. The
game placement is now one request, CPU vision, 8 workers, 16K context and a
computed expert share, falling back to the minimum placement on this PC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 18:42:08 +09:00
javis-bot
165142be9a fix(m0): measure smaller game placements and read only the latest engine start
The first game PC run showed that a 2 GB expert budget still left a 16 GB
card only 4.3 GB free: the weights, the 32K int8 conversation memory, the
MTP drafter and buffers take about 8.7 GB on their own. The game placements
now also shrink the memory format (q4_0), the context (16K, 8K) and the
drafter, with half the CPU workers (no speed cost measured). The engine log
is appended across starts, so each placement now reads only its own lines;
when no placement leaves the reserve the result names the most free one.
The gamemode runner skips the desktop stage, which is already measured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 18:23:28 +09:00
javis-bot
8dfab3ab6e fix(m0,docs): fail gamemode on unstarted placements and never overrun the game reserve
The gamemode stage was "ok" when only the normal placement started; a
placement that does not start, or a measurement error inside one, now marks
the stage partial. The design no longer raises the expert budget above the
computed one: it shrinks the game reserve to what the smallest placement
leaves, shows that, and pauses the LLM during games when nothing is left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 17:43:26 +09:00
javis-bot
8124451402 docs(v2): use one model and tune it automatically for games and slow PCs
Drop the recommended model (Qwen3.8-Flash-Next) and the experimental one:
Qwen3.6-35B-A3B passed every tool call and answers first. Replace game-mode
actions (a)(b)(c) with automatic tuning: a normal and a game placement of the
same model, computed per PC from VRAM, a first-run self-check that lowers
defaults on slow PCs, and latency watching that lowers restart-free settings
first. Record the agent-desktop results and choose the per-call CLI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 17:39:42 +09:00
javis-bot
bf4138c9ee feat(m0): measure agent-desktop as the v2 PC-control path
Add a 'desktop' stage to the M0 kit that installs the pinned agent-desktop
1.2.0 Windows build (SHA-256 checked) and checks, on the gaming PC:
calculator buttons by automation id (12+3=15), set-value and read-back on
a test page in a throwaway browser profile, Discord/Explorer read time,
and a warm MCP server against one CLI process per call. Results keep only
counts, timings, error codes and the calculator digits.

Record agent-desktop (UI Automation) as the planned M7 control path in the
v2 design docs, with raw Windows input only for screens without
accessibility.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 17:13:33 +09:00
javis-bot
73fa93f346 docs(v2): record the recommended model rerun and drop the multi-sharer confirm
Keep the fast model as the default: the recommended model decodes faster
(131 vs 82 tok/s) but starts every voice answer, long-context reply and
image answer later. Note its working VRAM shrink for game mode and its
fabricated weather answer for the M3 evals.

A named share question now captures the popout straight away when that
person is sharing, even with several sharers (user decision).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 14:14:28 +09:00
javis-bot
01037944a0 docs(v2): confirm the selected stream before capturing a named share
Only capture a named person's share when they are the sole streamer or
the user has confirmed the selection in the popout. A timeout no longer
captures whatever is showing, a non-streaming target gets an honest
answer, and clicking the popout after confirmation asks again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 13:51:18 +09:00
javis-bot
bb0defd384 docs(v2): plan around Discord's single share popout window
Discord allows only one popped-out share window and picks the stream
inside it, so there is never a choice between windows. Use the voice
channel's streaming state instead: one streamer means capture directly,
two or more means asking the user to pick that person in the popout
when the requested person changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 13:50:10 +09:00
javis-bot
cbe1785e08 docs(v2): record the M0 tool, popout and game results and rerun the recommended model
The first run left three results out of the plan: the fast model skipped
the timer tool and claimed it was set, the popped-out share window carries
the voice channel name rather than the streamer's, and a covered popout
keeps drawing. The recommended model failed to start (DFlash2 needs a
native pack), so M0 stays open for that one stage with the fixed kit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 13:47:16 +09:00
javis-bot
26fd6e4a1f docs(v2): bundle Virtual Display Driver and close M0 without the rerun
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 13:42:21 +09:00
javis-bot
810c62ab29 docs(v2): ship our own virtual display driver, Parsec only for M0
The user wants the app to work on any Windows PC from its own installer,
so the Parsec driver is a test aid for M0, not a product dependency.
Scope the target to x64 Windows 10/11 outside S mode (ARM64 needs WHQL)
and re-verify the Discord share, cursor and unplug behaviour with the
driver that ships.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 13:34:00 +09:00
javis-bot
e0b172e796 feat(m0): check a virtual monitor for the bot browser window
The user wants the bot window on no real screen at all. Add a vdisplay
stage that drives the Parsec Virtual Display Adapter already on the PC
(no install, the display dies with the script), moves the clock window
onto it as a normal and as a tool window, and asks about real monitors,
Alt+Tab, Discord's Screens tab and whether the cursor can wander there.

Record the M0 defaults the user confirmed in docs/v2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 13:29:58 +09:00
javis-bot
4f90701112 docs(v2): record answers on channel sync, memory scope, popout and legacy branch
- Chat channel name/description edits also rename the real Discord
  channel and topic by default (toggle kept)
- Memory: separate toggles for registered chat channels and other
  speakers in the same voice channel, both on by default
- Screen-share popout is requested by voice, never automated
- v1 head bb60c4d preserved as the remote legacy branch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 13:25:28 +09:00
javis-bot
4a15384bc6 docs: add v2 plan README and docs/v2, move v1 README to docs/v1
Replace the root README with the v2 plan (game PC fully local, Windows
management app) and split the plan into docs/v2/*.md: architecture,
voice-and-llm, conversation, memory, desktop-app, pc-features, privacy,
roadmap, decisions. The v1 README moves to docs/v1/README.md with its
relative links fixed; .env.example points to the new location.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 13:10:47 +09:00