OAuth cannot be done interactively in the headless container, so the login
must be seeded into the mounted ~/.gemini. Three problems are fixed:
- Mount fragility on the Windows Docker Desktop target: the creds mount
defaulted to ${HOME}/.config/javis/gemini, but ${HOME} is often unset when
compose runs outside a WSL shell, silently mounting the wrong dir. Default is
now the project-local ./docker/gemini-oauth (cross-platform), GEMINI_OAUTH_DIR
still overrides.
- No visibility: when oauth is selected but no login is seeded, the path
silently degraded to DDG/Brave. Added gemini_oauth_ready() + a one-time debug
hint and a startup entrypoint warning (skipped on the browser role, fail-open).
- Seeding guidance: oauth_creds.json is the essential credential (refresh token;
GOOGLE_GENAI_USE_GCA=true forces OAuth), which is what the readiness check and
warning verify; docs recommend copying the whole ~/.gemini for convenience.
Adds docker/gemini-oauth/ seed dir (.gitkeep) with the login files gitignored,
GEMINI_OAUTH_DIR in .env.example, and updates DEPLOY.md, stream_browser_modes.md
and llm_contexts.md. Covered by 3 new tests (10 passed total).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds /settings (served by the bridge) to change the LLM model (from installed
Ollama models), Whisper model, TTS engine + MeloTTS speed, output language,
agentic max-turns, thinking mode, and free-form LLM instructions — live, with a
'apply' that restarts the bridge + TTS worker. Settings persist to the runtime
config JSON; engine reads output_language + llm_instructions and the TTS worker
reads melo_speed from it. Bridge port publishable for access.
Base compose is GPU-agnostic; GPU is added by a per-OS override selected via
COMPOSE_FILE in .env (docker-compose.gpu-linux.yml for Ubuntu/CDI,
docker-compose.gpu-windows.yml for Windows 11 Docker Desktop). Adds .env.example
split-deployment section + docs/DEPLOY.md covering all-in-one and browser+bot
layouts on both OSes.
Confirmed root cause of "broadcast doesn't appear in Discord" in userbot mode:
the conversation and the Go-Live broadcaster were the SAME Discord account on
two sessions. Discord allows one voice presence per account, so the broadcaster's
voice connection never connects (state voiceReady:false). Proven by isolating the
broadcaster: alone it connects ("Go-Live WebRTC connected"); alongside the
conversation it times out.
The broadcaster now logs in with DISCORD_STREAM_TOKEN when set (a second burner
account dedicated to streaming), falling back to DISCORD_SELFBOT_TOKEN (correct
for normal-bot mode). When userbot mode shares one account it warns loudly with
the fix. Documents the var in .env.example.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The config template never set intent_judge_model, so it fell back to the code
default gemma4:e2b. That model is not pulled by this stack (Ollama only has
qwen2.5:3b, qwen3:8b, nomic-embed-text), so every auxiliary small-model call —
intent judge, tool router, weather place extraction, query decomposition —
targeted a non-existent model, silently failed, and fell open. This crippled
tool routing and argument extraction on the 3B brain.
Render intent_judge_model from a new OLLAMA_INTENT_MODEL env var that defaults
to OLLAMA_CHAT_MODEL, so the auxiliary calls reuse the already-warm chat model
(one resident model, no extra load). tool_router_model="" then resolves through
the chain to the same model.
Verified: rendered jarvis.json now has intent_judge_model=qwen2.5:3b, and the
weather place extractor returns "서울" / "Tokyo" (it returned None for
everything while pointed at the missing gemma4:e2b).
Add an optional OUTPUT_LANGUAGE env var that forces every reply into a
single language. When set, output_language_directive() injects a "respond
only in <language>" instruction (also forbidding other scripts) into the
chat loop's system prompt, next to the existing TTS English-only lock.
Empty (default) keeps the multilingual "reply in the user's language"
behaviour, so upstream is unaffected.
For the Korean-only deployment this also suppresses the occasional trailing
CJK/Hanja fragment qwen2.5:3b leaks on free-form chit-chat.
- system_prompt.py: language-agnostic output_language_directive() helper
- engine.py: read OUTPUT_LANGUAGE, append directive in _build_initial_system_message
- docker-compose.yml + .env.example: document/pass the new var
- docs/llm_contexts.md: note the new gating on the main reply context
- tests: cover unset/set/agnostic/whitespace cases
Replace the blunt global OLLAMA_KEEP_ALIVE=-1 (which kept every model,
including nomic-embed, resident in VRAM forever) with per-request residency:
- llm.py: all three /api/chat payloads send keep_alive=30m so the actively
used chat model stays resident and voice turns never pay a cold reload.
- embeddings.py: /api/embeddings sends keep_alive=0 so nomic-embed unloads
right after each call instead of squatting in VRAM next to the chat model.
- docker-compose.yml: drop the global OLLAMA_KEEP_ALIVE=-1; document the
per-request scheme on the ollama service.
Switch the default chat model qwen3:8b -> qwen2.5:3b. Verified live on the
RTX 5050 (8GB):
- ollama ps: qwen2.5:3b 2.4GB, 100% GPU (8B was 92% GPU / 8% CPU), UNTIL ~30m
(the 30m pin, not "Forever"); nomic-embed absent after several enriched turns.
- nvidia-smi: ~3.2GB VRAM used total (qwen 2.4GB + whisper 0.7GB) vs ~6.6GB.
- Korean /text turns: warm 1.7-4s (cold first load ~52s), vs ~5-7s on 8B;
time/weather/places tool calls fire and reply in Korean.
Known limitation: qwen2.5:3b can occasionally leak a trailing CJK phrase on
free-form chit-chat (factual/tool replies stay clean).
Adds a dedicated MeloTTS Korean voice (speed 1.5) as the primary TTS engine,
served by a long-lived in-container worker so each Discord turn pays only
inference cost, not model-load cost.
- bridge/melo_worker.py: tiny HTTP service in its own /opt/melo py3.11 venv,
keeps the KR model warm, returns PCM16 WAV on POST /synth.
- bridge/server.py: synthesize() routes to the melo worker first; Piper stays
as an opt-in fallback (MELO_FALLBACK_PIPER, default off so Korean is never
mangled through the English voice). /health reports tts_engine.
- docker/setup-melo.sh: builds the isolated venv (pinned torch 2.12.0 /
torchaudio 2.11.0 CPU, MeloTTS pinned to a commit for reproducible rebuilds),
pre-fetches mecab-ko, and warms a dedicated HF cache (/opt/melo-cache) with a
real KR synth so all BERT + KR checkpoint assets are baked into the image.
- docker/supervisord.conf: runs melo-worker before the bridge with
HF_HOME=/opt/melo-cache (the whisper_cache volume shadows the default HF
cache) plus HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE so it reads the baked cache
and never retries the network on load.
- Dockerfile/.env.example: wire the melo build layer and config knobs.
Verified: offline synth passes with --network none and the prod volume mounted;
prod container recreated, all supervisord services up, bot logged in, and an
end-to-end /tts call returns a 44.1kHz mono PCM16 WAV.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a GEMINI_AUTH=oauth (default) sub-mode that shells out to the Gemini CLI
using the user's Google-account login instead of an API key. gemini_cli_search()
runs `gemini -p <query> -o json --skip-trust --approval-mode yolo`, strips
GEMINI_API_KEY/GOOGLE_API_KEY and sets GOOGLE_GENAI_USE_GCA=true so the CLI
selects the account OAuth method and fails fast when no login exists. Bounded by
a 30s timeout and fail-open to the DDG/Brave/Wikipedia cascade on any failure
(CLI missing, login expired, quota 429, timeout). GEMINI_AUTH=apikey keeps the
legacy REST path. Specs and docs/llm_contexts.md updated; behaviour covered by
tests/test_realtime_gemini_cli.py.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First increment of the STREAM_BROWSER real-time-info modes (true = browser,
false = Gemini):
- browse-search.mjs: drives the on-screen Chrome via CDP so the action shows on
the broadcast. `search` returns the top Google results (title/url/snippet);
`youtube` plays the first result. Verified live: real-time Seoul weather
results, and IU 'Good Day' MV playback.
- .env.example: GEMINI_API_KEY / GEMINI_MODEL for the false-mode Gemini account.
- docs/stream_browser_modes.md: architecture + integration map (brain config,
the two mode-gated tools, registry, design decisions) for the remaining wiring.
The Python brain wiring (config.py mode/gemini fields, browseAndSearch +
geminiSearch tools, registry, specs, llm_contexts) lands next - it needs a
running brain and a Gemini key to verify, rather than committing untested edits
into the 39k-line engine.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add STREAM_BROWSER (.env) gating screen-share/browser mode. false => the
/자비스 stream command stays voice + API/MCP only (no Go-Live); true (default)
=> screen share as before. (Browser-driven info retrieval in true mode is a
follow-up build; the bot has no browser-control tools yet.)
- Make the two test-time fixes broadcast-wide defaults via broadcast-helper.mjs:
it now also watches every tab for HTML5 fullscreen and toggles Chrome window
fullscreen so the address bar is hidden for ANY video (xfwm4 won't hide it on
'f' alone), restoring on exit. Subtitles were already enforced per video.
scenario.mjs drops its own fullscreen toggle and relies on the helper.
- Revert the test-settings env vars from .env.example (not wanted).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Addresses review of the ad/subtitle work (the ad-skip.mjs -> broadcast-helper.mjs
rename's other half; the prior commit only recorded the deletion):
- ad mute leak: the ad-skipper muted during an ad but never un-muted, so the
main video stayed silent after the first ad. Save the pre-ad muted/playbackRate
and restore them when the ad ends (verified: muted false -> true -> false).
- captions were only applied once when scenario.mjs ran, not for the whole
broadcast. The persistent helper now applies the rule (OFF by default, Korean
ON if offered) per video and ENFORCES it every tick - one-shot did not hold
because YouTube silently re-enabled captions (verified it stays off across 8s).
- ad-skip + captions merged into broadcast-helper.mjs (one CDP process).
- the 60fps MV test now lives in the repo: scenario.mjs gains MV_QUERY (search +
auto-pick the first >=60fps result) and WATCH_SECONDS, plus the
fullscreen-toolbar-hide fix. The broadcast runs via the committed
stream-hold.ts (audio + keepalive), not an out-of-repo copy.
- document the test env vars (CDP_PORT, HOLD_MS, TEST_*, MV_QUERY, WATCH_SECONDS).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two broadcast-experience improvements:
- Audio: the Go-Live stream was video-only. Capture the desktop sound (the
default PipeWire/Pulse sink monitor, @DEFAULT_MONITOR@) as a second ffmpeg
input and mux AAC into the mpegts; the library re-encodes it to Opus for
Discord. Controlled by STREAM_AUDIO / STREAM_AUDIO_SOURCE (default on). ffmpeg
inherits XDG_RUNTIME_DIR to reach the pulse socket. Verified: the streamer now
reports "Found audio stream" and the monitor carries Chrome audio (~-11 dB).
- Subtitles: in the browse scenario, default captions OFF, but auto-enable a
Korean track when the video offers one (getOption captions tracklist ->
setOption / unloadModule).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Go-Live broadcast looked badly choppy: video and scrolling stuttered while
the cursor stayed smooth. Root cause is TigerVNC: it only refreshes its
framebuffer while a VNC client is attached, but the broadcast reads that
framebuffer with x11grab (not as a VNC client). With no viewer attached the
captured screen idled at ~1.5 fps (measured 3/30 distinct frames); the cursor
looked smooth only because x11grab overlays the live cursor on every frame.
- Add a headless RFB keepalive (vnc-keepalive.ts) that stays connected for the
life of the stream and requests incremental framebuffer updates at the stream
framerate. SelfbotStreamer starts it on broadcast start and tears it down on
stop/self-end. Measured 3/30 -> 57/60 distinct frames at 60 fps. Fail-open;
authenticates with VNC_PASSWORD or the ~/.config/tigervnc/passwd file.
- Fix a resource leak: when the Go-Live ended on its own, only the active flag
was cleared, leaving the x11grab->nvenc ffmpeg running forever (pinning a CPU
core while no media was transmitted, with only the gateway TCP left and no UDP
media). The self-end path now tears down capture, keepalive and voice like
stop() does.
- Tests for both paths (self-end teardown; keepalive DES auth, port mapping,
password resolution). Add @types/bun so bun:test typechecks; document the
keepalive and recommended Chrome flags in README and .env.example.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bump the default broadcast to 1080p 60fps at 8 Mbps and route both encode
stages through the GPU (RTX 5050, h264_nvenc) so 60fps stays smooth without
loading the 4-core host.
- selfbot.ts: capture ffmpeg uses h264_nvenc when streamHw is on (falls back
to software x264 otherwise), and prepareStream now passes Encoders.nvenc()
so the library's transcode runs on the GPU too. Guard loadLib for Encoders.
- config.ts: VNC_FRAMERATE default 30 -> 60, VNC_BITRATE_KBPS 4000 -> 8000.
- .env.example: document the new 1080p60/8 Mbps defaults and STREAM_HW.
Verified locally: h264_nvenc x11grab holds a steady 60fps with headroom,
Encoders.nvenc() returns valid h264_nvenc settings, and tsc --noEmit passes.
Live Discord voice-channel verification pending a host reboot.
GPU acceleration is now on by default and verified end-to-end on the
Blackwell RTX 5050 (sm_120):
- Ollama offloads 100% to GPU (log: library=CUDA compute=12.0,
BLACKWELL_NATIVE_FP4=1). compose passes GPU via CDI
(devices: nvidia.com/gpu=all) to both ollama and javis.
- Whisper STT on GPU: faster-whisper>=1.1.0 + nvidia-cublas/cudnn cu12,
LD_LIBRARY_PATH baked into the image. Verified float16 transcribe on
sm_120; bridge auto-falls back to CPU when no GPU is present.
- Model: default chat model -> qwen3:8b (best 8GB-VRAM tool-calling,
~5GB Q4). Embed stays nomic-embed-text.
- README documents the host one-time setup (nvidia-container-toolkit +
`nvidia-ctk cdi generate`) and GPU on/off.
Verified: image builds; GPU visible in both containers via compose;
ollama ps = 100% GPU; faster-whisper cuda OK + CPU fallback OK;
bridge /health 200.
`docker compose up -d --build` now brings up the whole thing automatically —
no host setup needed:
- All-in-one javis image: TigerVNC+XFCE desktop, Chrome, Python brain bridge,
Node/bun bot, managed by supervisord (verified: all 6 programs RUNNING).
- ollama service + one-shot ollama-init that auto-pulls chat+embed models
(verified end-to-end; `ollama list` shows pulled models).
- Discord token deferred: without DISCORD_BOT_TOKEN the desktop, bridge,
Ollama and models all run; only the bot waits (no crash loop).
- Slim container deps (bridge/requirements-bridge.txt) drop the unused
PyQt6/torch/chatterbox/sounddevice stack. Piper voice + Whisper models
auto-download into named volumes.
- Configurable host ports (VNC_PORT/NOVNC_PORT/BRIDGE_PORT) to avoid clashing
with a host VNC already on 5901. Bridge binds 0.0.0.0 in-container.
Verified: image builds; brain imports; bridge /health 200; noVNC 200;
X display :1 @1920x1080; auto-pull completes; supervisorctl status all RUNNING.
