gamemode (third run), gamemode_stt and gamemode_live finished on the
game PC. The STT on the GPU takes 2.7 GB at its 10-speaker peak, as
estimated, and leaves the game 3.6 GB (experts 0 GB) or 2.6 GB (2 GB).
The STT stays on the GPU in games: on the CPU one utterance takes
3.0-3.8 s. In a real game the 2 GB expert placement is twice as fast as
0 GB. The GPU contention between the STT and the LLM, the in-game floor
placement and an STT margin for the normal placement wait for the user.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user meant the chat channel's "answer every message", not the voice
policy "in order": the LLM takes two or more requests at once only while a
chat channel answers every message, otherwise one whatever the voice policy.
The roadmap and the open measurements point at the new gamemode_stt and
gamemode_live stages, which also decide whether the STT moves to the CPU
during a game.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The M0 stt stage measured 1300 MiB for the model and another 1440 MiB while
transcribing ten speakers at once. The game share now subtracts the 2.7 GB
peak: about 3.1 GB on the 16 GB game PC (not 4.6 GB), and the 2 GB expert
placement returns at a reserve of about 2.2 GB. Both are marked as sums of
separate LLM and STT runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user chose one request at a time by default (97 tok/s on a 16 GB card
against 70-75 with two) and two or more only under the "in order" policy,
which answers every request. The second gamemode run showed a 16 GB card
cannot free 8 GB for a game: the smallest placement still leaves 6.3 GB.
The Strata 0.1.59 qwen36 engine source confirms why --kv and --mtp off made
no difference: it takes no --kv flag and always loads the MTP block. The
game placement is now one request, CPU vision, 8 workers, 16K context and a
computed expert share, falling back to the minimum placement on this PC.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first game PC run showed that a 2 GB expert budget still left a 16 GB
card only 4.3 GB free: the weights, the 32K int8 conversation memory, the
MTP drafter and buffers take about 8.7 GB on their own. The game placements
now also shrink the memory format (q4_0), the context (16K, 8K) and the
drafter, with half the CPU workers (no speed cost measured). The engine log
is appended across starts, so each placement now reads only its own lines;
when no placement leaves the reserve the result names the most free one.
The gamemode runner skips the desktop stage, which is already measured.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gamemode stage was "ok" when only the normal placement started; a
placement that does not start, or a measurement error inside one, now marks
the stage partial. The design no longer raises the expert budget above the
computed one: it shrinks the game reserve to what the smallest placement
leaves, shows that, and pauses the LLM during games when nothing is left.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop the recommended model (Qwen3.8-Flash-Next) and the experimental one:
Qwen3.6-35B-A3B passed every tool call and answers first. Replace game-mode
actions (a)(b)(c) with automatic tuning: a normal and a game placement of the
same model, computed per PC from VRAM, a first-run self-check that lowers
defaults on slow PCs, and latency watching that lowers restart-free settings
first. Record the agent-desktop results and choose the per-call CLI.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gaming-PC rerun scored the fast model's honest refusals ('...할 수 있는
도구는 없습니다', '...준비되어 있지 않습니다') as failures, so the kit
downloaded two more fast-model candidates for nothing. Rescored with the
fixed judge, UD-Q4_K_XL, UD-Q6_K_XL and Q8_0 all pass 27/27; the
recommended model's Q2_0/IQ3_XXS/IQ3_S still fail 3/27 by calling an
unrelated tool when no tool fits. Record both results in the v2 docs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a 'desktop' stage to the M0 kit that installs the pinned agent-desktop
1.2.0 Windows build (SHA-256 checked) and checks, on the gaming PC:
calculator buttons by automation id (12+3=15), set-value and read-back on
a test page in a throwaway browser profile, Discord/Explorer read time,
and a warm MCP server against one CLI process per call. Results keep only
counts, timings, error codes and the calculator digits.
Record agent-desktop (UI Automation) as the planned M7 control path in the
v2 design docs, with raw Windows input only for screens without
accessibility.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
User rule: if the tool-calling fix does not work, change the model. The
tools_fast / tools_recommended stages now measure the next, less
compressed model of the same family (fast: UD-Q6_K_XL then Q8_0;
recommended: IQ3_XXS then IQ3_S) when the default still gets a scored
case wrong, stop at the first one that passes every case, and record it
as tools_verdict.chosen. Candidates without enough free disk are skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
M0 on Strata caught both v2 models skipping a tool they had been given:
the recommended model made up "24 degrees, sunny" instead of calling the
weather tool, and the fast model said "timer set" without calling the
timer tool (3 of 3 each). qwen2.5:7b does the same through the engine.
- Prompt: TOOL_HONESTY_RULES in the persona (live facts only from a tool
result this turn, actions only claimed after a tool reports success, no
answer text next to a tool call), restated at the end of the system
message by tool_honesty_reminder() with the turn's real tool names.
- Engine: skipped-tool guard. When a native-tool model answers before any
tool has run and the plan names a callable tool as a concrete call, the
answer is dropped and the plan steps are direct-executed, then the model
answers from the results. Only literal, schema-valid steps run (regex
fast path, no LLM repair); otherwise the held-back answer is delivered.
- Planner: the AVAILABLE TOOLS lines now carry argument names
(planner_tool_catalog), so concrete steps use the tool's real keys.
- Eval evals/test_tool_skip_confabulation.py; on qwen2.5:7b, 3 runs each,
available-tool cases went from 0/12 to 9/12 calling the tool.
- M0 kit: tools_fast / tools_recommended stages (run_m0_tools.bat) rerun
tool calling before and after the fix on the real models, loading the
production system_prompt.py and planner.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the fast model as the default: the recommended model decodes faster
(131 vs 82 tok/s) but starts every voice answer, long-context reply and
image answer later. Note its working VRAM shrink for game mode and its
fabricated weather answer for the M3 evals.
A named share question now captures the popout straight away when that
person is sharing, even with several sharers (user decision).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only capture a named person's share when they are the sole streamer or
the user has confirmed the selection in the popout. A timeout no longer
captures whatever is showing, a non-streaming target gets an honest
answer, and clicking the popout after confirmation asks again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Discord allows only one popped-out share window and picks the stream
inside it, so there is never a choice between windows. Use the voice
channel's streaming state instead: one streamer means capture directly,
two or more means asking the user to pick that person in the popout
when the requested person changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first run left three results out of the plan: the fast model skipped
the timer tool and claimed it was set, the popped-out share window carries
the voice channel name rather than the streamer's, and a covered popout
keeps drawing. The recommended model failed to start (DFlash2 needs a
native pack), so M0 stays open for that one stage with the fixed kit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user wants the app to work on any Windows PC from its own installer,
so the Parsec driver is a test aid for M0, not a product dependency.
Scope the target to x64 Windows 10/11 outside S mode (ARM64 needs WHQL)
and re-verify the Discord share, cursor and unplug behaviour with the
driver that ships.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user wants the bot window on no real screen at all. Add a vdisplay
stage that drives the Parsec Virtual Display Adapter already on the PC
(no install, the display dies with the script), moves the clock window
onto it as a normal and as a tool window, and asks about real monitors,
Alt+Tab, Discord's Screens tab and whether the cursor can wander there.
Record the M0 defaults the user confirmed in docs/v2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One double-click kit (scripts/m0/run_m0.bat) that measures the gaming PC
and packs the results into a single zip:
- PC info (GPU, driver, RAM, disks, monitors)
- Windows checks with the user: bot browser window hidden from Alt+Tab
but listed in Discord's share picker and still drawing while covered;
Discord pop-out stream window identification and occlusion; installed
games, anti-cheat services and the borderless game detector
- MeloTTS, CosyVoice2 and Fun-CosyVoice3 Korean synthesis speed
- faster-whisper large-v3-turbo: final/partial latency, CER, wake-word
hint, batched decoding for several speakers
- Strata LLM (fast and recommended presets): TTFT by context length,
decode speed, tool calls, picture reading per quality tier, Korean
answers, parallel requests, VRAM give-back and restart/switch timings
uv and Python, per-stage venvs, models and Strata all live inside the
work folder. Results are scrubbed of user/computer names and home paths.
Verified on Linux GPU (LLM client against Ollama, STT, MeloTTS,
CosyVoice2/3) and on the Windows 11 CI VM (window checks, game scan,
MeloTTS/CosyVoice2/STT installs, Strata install and plan).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Chat channel name/description edits also rename the real Discord
channel and topic by default (toggle kept)
- Memory: separate toggles for registered chat channels and other
speakers in the same voice channel, both on by default
- Screen-share popout is requested by voice, never automated
- v1 head bb60c4d preserved as the remote legacy branch
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the root README with the v2 plan (game PC fully local, Windows
management app) and split the plan into docs/v2/*.md: architecture,
voice-and-llm, conversation, memory, desktop-app, pc-features, privacy,
roadmap, decisions. The v1 README moves to docs/v1/README.md with its
relative links fixed; .env.example points to the new location.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Switch the default TTS engine from online Edge TTS to offline GPU
MeloTTS (privacy-first: reply text never leaves the host). Adds a
supervisord melo-worker program that runs in full/bot/brain roles and
synthesises Korean on the GPU (MELO_DEVICE=cuda, MELO_SPEED=1.5), a
baked /opt/melo py3.11 venv layer in the Dockerfile, and compose/env
defaults. Edge TTS stays available as an opt-in online fallback via
TTS_ENGINE=edge.
Verified on the RTX 5050 (Blackwell sm_120): melo-worker logs
"ready (lang=KR speed=1.5 device=cuda)", /health returns ok, and a
/synth round-trip returns a 177KB WAV.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Run the GPU-heavy compute (Whisper STT, Ollama LLM, TTS) on a dedicated
brain host and the bot + browser on a separate no-GPU app host, talking
over the LAN.
- supervisord: new `brain` role (bridge only) and `app` role (bot +
desktop/Chrome, no local bridge); `full`/`browser`/`bot` unchanged.
- compose: make BRIDGE_URL overridable (was hardcoded to loopback, which
prevented the bot from reaching a remote bridge).
- docs: DEPLOY.md "GPU split" layout with per-host .env blocks; document
the new roles in run-if-role.sh and .env.example.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
For the split where the LLM runs on a separate LAN host (e.g. a GPU box at
192.168.10.9) and the app stack runs elsewhere (192.168.10.5), the brain must
reach Ollama over the network. The javis service hardcoded
OLLAMA_BASE_URL=http://ollama:11434, ignoring any .env value, so the app could
only use the in-stack Ollama. Make it ${OLLAMA_BASE_URL:-http://ollama:11434}:
default unchanged (all-in-one), overridable to point at an external Ollama.
Document the external-Ollama setup in .env.example and DEPLOY.md, refresh the
browser-host IP examples (.9→.5), and de-hardcode the .9 example in novnc.ts.
Replies stayed long because the prompt stack gave conflicting length signals:
the persona said "one sentence (two at the very most)" AND told the model to
"state the answer in a sentence, then add a dry observation" (a 2nd sentence),
while voice_style said "two to three sentences maximum". The model followed the
longest. Make all three sources agree on exactly one sentence: the persona's
aside must now fold into the same sentence as a trailing clause (never a 2nd
sentence), voice_style caps at one sentence, and agents/llm.md says 한 문장.
Shorter replies also cut Edge-TTS latency, since synth time scales with text
length. Specs (prompts.spec.md) and docs/llm_contexts.md updated; deterministic
prompt-contract tests added.
Follow-up to the OLLAMA_INTENT_MODEL split: document that the Docker stack runs
intent judging / tool routing / extraction on a small qwen2.5:3b (pulled by
ollama-init) kept separate from the big chat answer model, and that setting
OLLAMA_INTENT_MODEL = OLLAMA_CHAT_MODEL folds them back onto one resident model.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Spoken (TTS) replies are 1-2 sentences, so an unbounded num_predict only
exposes the worst case where the chat model rambles or loops. Add an
ollama_num_predict config (default 512, 0 disables) wired into the reply
loop's chat call on both the native- and text-tool paths. The 512-token
headroom stays well above this app's short tool-call JSON, so capping never
truncates a tool call. This keeps the user's quality model instead of
downgrading it. Configurable in the container via OLLAMA_NUM_PREDICT.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drop Markdown files into an agents/ folder and their contents are appended to
the main reply LLM's system prompt, so an operator can extend the assistant's
rules/tone without code changes. Files are concatenated in filename order
(use 00-, 10- prefixes to control ordering) and re-read once per turn, so edits
apply on the next reply with no rebuild/restart. Fail-open: a missing, empty,
or unreadable folder yields no instructions and never breaks a reply.
- load_agent_instructions() in system_prompt.py (AGENTS_DIR env, default
/app/agents); reads *.md only, skips blanks, ignores non-dir paths
- engine.py appends it alongside the existing settings-UI llm_instructions,
under the same "Additional instructions from the operator:" framing
- docker-compose.yml bind-mounts ./agents:/app/agents:ro and sets AGENTS_DIR
- agents/example.md.sample starter template (.sample is not loaded)
- tests cover ordering, md-only filtering, blank-skip, env/arg resolution,
and fail-open paths
- README, .env.example, docs/llm_contexts.md updated
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Windows users following the docs hit "The system cannot find the file
specified" because COMPOSE_FILE's separator is OS-specific (':' collides
with the C: drive letter). Fix every Windows example to use ';', add an
explicit OS-separator warning in .env.example, README, DEPLOY.md and the
gpu-windows compose comment, and point users at the explicit `-f` form as
a separator-agnostic alternative.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Google now rejects personal Google accounts on the Gemini CLI OAuth login
("This client is no longer supported for Gemini Code Assist for individuals").
The setup docs previously sent every user down "Sign in with Google" with no
warning. Note the block, recommend GEMINI_AUTH=apikey for personal accounts,
and clarify that real-time search fail-opens to DDG/Brave/Wikipedia regardless.
Docs only; no runtime default change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
OAuth cannot be done interactively in the headless container, so the login
must be seeded into the mounted ~/.gemini. Three problems are fixed:
- Mount fragility on the Windows Docker Desktop target: the creds mount
defaulted to ${HOME}/.config/javis/gemini, but ${HOME} is often unset when
compose runs outside a WSL shell, silently mounting the wrong dir. Default is
now the project-local ./docker/gemini-oauth (cross-platform), GEMINI_OAUTH_DIR
still overrides.
- No visibility: when oauth is selected but no login is seeded, the path
silently degraded to DDG/Brave. Added gemini_oauth_ready() + a one-time debug
hint and a startup entrypoint warning (skipped on the browser role, fail-open).
- Seeding guidance: oauth_creds.json is the essential credential (refresh token;
GOOGLE_GENAI_USE_GCA=true forces OAuth), which is what the readiness check and
warning verify; docs recommend copying the whole ~/.gemini for convenience.
Adds docker/gemini-oauth/ seed dir (.gitkeep) with the login files gitignored,
GEMINI_OAUTH_DIR in .env.example, and updates DEPLOY.md, stream_browser_modes.md
and llm_contexts.md. Covered by 3 new tests (10 passed total).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The persona prompt was built from the raw OUTPUT_LANGUAGE env while the
reply-language directive read the settings-web UI value (config JSON), so
changing the language in the settings page was honoured by the directive
but ignored by the persona, leaving them contradicting each other.
Add _resolve_output_language() as the single source of truth (config wins
over env) and feed the same resolved value to both build_system_prompt()
and reply_language_directive(). Update docs/llm_contexts.md to match.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Base compose is GPU-agnostic; GPU is added by a per-OS override selected via
COMPOSE_FILE in .env (docker-compose.gpu-linux.yml for Ubuntu/CDI,
docker-compose.gpu-windows.yml for Windows 11 Docker Desktop). Adds .env.example
split-deployment section + docs/DEPLOY.md covering all-in-one and browser+bot
layouts on both OSes.
Harden the reply-language lock so qwen2.5:3b reliably stays in the locked
language instead of leaking the query language back in:
- reply_language_directive(): single resolver with clear precedence —
explicit OUTPUT_LANGUAGE lock wins over the Piper/Chatterbox English-only
fallback (this deployment's actual TTS is Korean MeloTTS, so the legacy
English lock was both wrong and contradicting the Korean lock).
- Stronger, override-explicit directive wording, inserted near the FRONT of
the system prompt so a small model gives it primacy over the persona.
- build_system_prompt(output_language=...): rewrite the persona's "in the
user's language" clause to the locked language so the persona stops
fighting the lock.
- docs/llm_contexts.md: document the resolver, precedence, and placement.
Live-verified on the running brain (qwen2.5:3b): Korean voice-style input
and a cold English query both return fully Korean replies with no CJK/Hanja
leak. Tests cover unset/set/agnostic/whitespace + precedence + persona rewrite.
Add an optional OUTPUT_LANGUAGE env var that forces every reply into a
single language. When set, output_language_directive() injects a "respond
only in <language>" instruction (also forbidding other scripts) into the
chat loop's system prompt, next to the existing TTS English-only lock.
Empty (default) keeps the multilingual "reply in the user's language"
behaviour, so upstream is unaffected.
For the Korean-only deployment this also suppresses the occasional trailing
CJK/Hanja fragment qwen2.5:3b leaks on free-form chit-chat.
- system_prompt.py: language-agnostic output_language_directive() helper
- engine.py: read OUTPUT_LANGUAGE, append directive in _build_initial_system_message
- docker-compose.yml + .env.example: document/pass the new var
- docs/llm_contexts.md: note the new gating on the main reply context
- tests: cover unset/set/agnostic/whitespace cases
Completes the STREAM_BROWSER=true behaviour:
- handleJoin auto-starts the broadcast on voice join and wires the session to
the guild streamer; each turn reports the live state to the brain so search
routes Chrome (live) vs Gemini (off).
- New setBroadcast tool lets the user toggle the broadcast by voice ("방송
켜줘/꺼줘") via the LLM (no hardcoded phrases); it refuses when
STREAM_BROWSER=false. The directive flows brain -> bridge (broadcast_action)
-> bot streamer.start/stop, guarded by isActive() so it's idempotent.
- Per-turn IPC uses a thread-local (reply/turn_state.py) instead of threading
params through the whole engine chain: bridge sets broadcasting in, tool
records the directive out; Tool.execute exposes broadcasting on ToolContext.
Bot typecheck clean; brain covered by tests/test_set_broadcast.py (+ existing
routing tests). Specs + docs updated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
STREAM_BROWSER becomes the broadcast *capability* master flag; the live
screen-share state (new ToolContext.broadcasting, passed per turn by the bot)
decides the backend:
- master off -> broadcast disabled, always Gemini
- master on + live on -> on-screen Chrome (visible on the stream)
- master on + live off -> Gemini
context.broadcasting is None outside the voice path (evals, text entry, older
bot) and falls back to the master flag, so current behaviour is unchanged.
This is the brain-side foundation; bot-side wiring (bridge passes broadcast
state, auto-broadcast on voice join, voice on/off toggle) follows.
Specs + docs/llm_contexts.md updated. Covered by
tests/test_web_search_broadcast_routing.py.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
yolo auto-approves every tool call, so a real-time search query could in
principle trigger write/shell tools. Default approval mode still auto-runs the
CLI's read-only web search in headless mode but never silently approves
destructive tools. Verified end-to-end: a grounded query returns a current
answer in ~23s with the account OAuth login. Test asserts yolo is absent;
specs and docs/llm_contexts.md updated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a GEMINI_AUTH=oauth (default) sub-mode that shells out to the Gemini CLI
using the user's Google-account login instead of an API key. gemini_cli_search()
runs `gemini -p <query> -o json --skip-trust --approval-mode yolo`, strips
GEMINI_API_KEY/GOOGLE_API_KEY and sets GOOGLE_GENAI_USE_GCA=true so the CLI
selects the account OAuth method and fails fast when no login exists. Bounded by
a 30s timeout and fail-open to the DDG/Brave/Wikipedia cascade on any failure
(CLI missing, login expired, quota 429, timeout). GEMINI_AUTH=apikey keeps the
legacy REST path. Specs and docs/llm_contexts.md updated; behaviour covered by
tests/test_realtime_gemini_cli.py.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the two info modes in the Python brain:
- config.py: read STREAM_BROWSER / GEMINI_API_KEY / GEMINI_MODEL from env into
Settings (stream_browser, gemini_api_key, gemini_model). Verified load_settings
reads both modes.
- realtime_search.py: two fail-open backends returning the same fenced
UNTRUSTED-WEB-EXTRACT envelope: browser_search() shells the Node CDP helper to
drive the on-screen Chrome (visible on the broadcast); gemini_search() calls
the Gemini REST API with google_search grounding.
- web_search.run(): routes by mode before the DDG cascade (true->browser,
false->Gemini), falling through to DDG/Brave/Wikipedia on any miss.
- browse_and_play tool: plays a YouTube video on the shared screen (true mode
only); registered in the tool registry.
- specs + docs/llm_contexts.md updated (new Gemini LLM context); CLAUDE.md spec
registry updated.
Verified live against the running Chrome: true-mode webSearch returned real
Google results for "오늘 서울 날씨", browseAndPlay played the IU 밤편지 MV, and
false-mode degrades gracefully on a bad/absent key. A valid GEMINI_API_KEY is
still needed to confirm the real Gemini grounding output.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First increment of the STREAM_BROWSER real-time-info modes (true = browser,
false = Gemini):
- browse-search.mjs: drives the on-screen Chrome via CDP so the action shows on
the broadcast. `search` returns the top Google results (title/url/snippet);
`youtube` plays the first result. Verified live: real-time Seoul weather
results, and IU 'Good Day' MV playback.
- .env.example: GEMINI_API_KEY / GEMINI_MODEL for the false-mode Gemini account.
- docs/stream_browser_modes.md: architecture + integration map (brain config,
the two mode-gated tools, registry, design decisions) for the remaining wiring.
The Python brain wiring (config.py mode/gemini fields, browseAndSearch +
geminiSearch tools, registry, specs, llm_contexts) lands next - it needs a
running brain and a Gemini key to verify, rather than committing untested edits
into the 39k-line engine.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>