Intent judging, tool routing and arg extraction are classification/JSON calls,
not the spoken answer, yet the stack aliased OLLAMA_INTENT_MODEL back to the big
chat model — so each command paid the big model's cost 2-3 times for routing
before the reply even ran. With the GPU on, that round-trip stacking is the main
remaining per-turn latency. Default OLLAMA_INTENT_MODEL to qwen2.5:3b (the
project's reference small model, clean Korean on classification) and have
ollama-init pull it. The reply engine already routes these calls through
intent_judge_model, so answer quality is untouched; set OLLAMA_INTENT_MODEL =
OLLAMA_CHAT_MODEL to fold back onto one resident model.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Spoken (TTS) replies are 1-2 sentences, so an unbounded num_predict only
exposes the worst case where the chat model rambles or loops. Add an
ollama_num_predict config (default 512, 0 disables) wired into the reply
loop's chat call on both the native- and text-tool paths. The 512-token
headroom stays well above this app's short tool-call JSON, so capping never
truncates a tool call. This keeps the user's quality model instead of
downgrading it. Configurable in the container via OLLAMA_NUM_PREDICT.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- run-chrome.sh: render web content (YouTube/Google/Naver) in Korean. --lang
only sets Chrome's UI; the Accept-Language sent to sites comes from the
profile's intl.accept_languages, which a persisted user-data-dir kept at
en-US. Seed the profile to ko-KR and add --accept-lang=ko-KR,ko.
- agents/llm.md: runtime instructions for the reply LLM (loaded by the agents
feature) — name "자비스", concise 1-2 sentence TTS replies (no emojis/markdown),
nuance-based intent, and YouTube voice controls (open/search/play Nth/pause/
back) via the on-screen browser tool.
- CLAUDE.md: drop the "use emojis in CLI output" rule — this assistant replies by
Discord voice, not CLI, so output should be plain.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
load_settings() coerced any tts_engine outside {piper, chatterbox} to piper, so
with TTS_ENGINE=edge the reply engine saw "piper" and treated the voice as
English-only in reply_language_directive() (only the OUTPUT_LANGUAGE lock kept
replies Korean). Add "edge" (and "melo") to the accepted set so the engine is
labelled multilingual correctly.
Also: a stale tts_engine in the persistent /data/jarvis-settings.json (melo/xtts
from an earlier voice, no longer built) would override the configured engine via
the entrypoint merge and leave the bot silent. Reset those to the env engine
during the merge.
Verified: load_settings() with tts_engine=edge now returns "edge"; the merge
maps melo/xtts -> edge; reply_language_directive("edge") is multilingual; 27
tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The user chose Microsoft Edge TTS, voice ko-KR-HyunsuMultilingualNeural at rate
+45% (~1.45x), as the natural Korean voice. Wire it into the bridge and make it
the default engine.
- bridge/server.py: _edge_synthesize() calls edge-tts and transcodes the MP3 to
PCM16 mono WAV with the system ffmpeg (temp file for a correct header);
TTS_ENGINE default -> edge; EDGE_TTS_VOICE / EDGE_TTS_RATE env-driven
- requirements-bridge.txt: add edge-tts (lightweight; httpx)
- compose/.env.example/README: TTS_ENGINE=edge + EDGE_TTS_* knobs; note the
online/privacy trade-off (reply text is sent to Microsoft, needs internet)
- drop the now-unused MeloTTS build layer (Dockerfile) and melo-worker
(supervisord) — edge synthesises in-process, no model/worker baked, slimmer
and faster image; settings UI engine list -> edge/piper, restart only bridge
Verified on host: edge-tts -> ffmpeg yields a valid 16-bit mono 24kHz WAV;
envsubst renders tts_engine=edge; docker build --check + 26 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
docker/jarvis-config.template.json hardcoded "tts_engine": "piper". entrypoint
renders it into /app/config/jarvis.json, and bridge _tts_engine_setting() reads
that JSON BEFORE the env — so TTS_ENGINE=melo in .env was ignored and the bot
synthesised Korean with the English Piper voice (the "foreign accent" the user
heard); the warm melo-worker sat unused.
Template now carries ${TTS_ENGINE}; compose sets TTS_ENGINE=${TTS_ENGINE:-melo}
so envsubst renders the real engine. Verified: envsubst with TTS_ENGINE=melo
yields "tts_engine": "melo", and `docker compose config` passes TTS_ENGINE=melo.
Added a regression test that the template stays env-driven and renders the
configured engine.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
MeloTTS's single Korean speaker sounded non-native ("foreign accent"). Swap it
for Coqui XTTS-v2 with the built-in female studio speaker "Ana Florence"
(language ko), the natural voice used in earlier local runs.
- bridge/xtts_worker.py: new warm HTTP worker (own /opt/xtts venv), same
/synth + /health contract and PCM16 output as the old melo worker
- docker/setup-xtts.sh: builds the venv with cu128 torch (Blackwell) + Coqui
TTS and bakes the XTTS-v2 model offline. Pins transformers>=4.57,<5 (5.x
removed isin_mps_friendly, breaking XTTS) and installs the [codec] extra
(torch>=2.9 needs torchcodec) — both verified by a real host synth
- Dockerfile: replace the melo build layer with the xtts layer
- supervisord.conf: melo-worker -> xtts-worker, env passthrough for
XTTS_DEVICE/SPEAKER/LANGUAGE (always set via compose defaults)
- bridge/server.py: default TTS_ENGINE=xtts, route to the xtts worker, generic
worker-synth helper, neural-only fallback flag (XTTS_FALLBACK_PIPER)
- settings UI: engine dropdown xtts/piper, drop the dead melo_speed field, fix
the supervisorctl restart target to xtts-worker
- compose/.env.example/README: XTTS_* vars, speaker/language knobs, remove melo
- remove bridge/melo_worker.py and docker/setup-melo.sh
- tests: xtts treated as multilingual (not English-only)
Verified on host: coqui-tts loads XTTS-v2 and synthesises Korean as
"Ana Florence" to a 16-bit mono 24kHz WAV.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
supervisord.conf passed MELO_DEVICE through as %(ENV_MELO_DEVICE)s but pinned
MELO_SPEED="1.5", so lowering MELO_SPEED in .env had no effect — the worker
always got 1.5. Pass MELO_SPEED through with %(ENV_MELO_SPEED)s and set a
compose default (MELO_SPEED=${MELO_SPEED:-1.5}, same pattern as MELO_DEVICE) so
the supervisord expansion always resolves and an .env value actually changes
the speaking rate. Default rate is unchanged (1.5). melo_worker logs the
resolved speed at startup, so the env->worker path is verifiable.
Verified: _resolve_speed() returns 1.1 for MELO_SPEED=1.1 (1.5 otherwise), and
`MELO_SPEED=1.1 docker compose config` renders MELO_SPEED: "1.1" into the env.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
OAuth cannot be done interactively in the headless container, so the login
must be seeded into the mounted ~/.gemini. Three problems are fixed:
- Mount fragility on the Windows Docker Desktop target: the creds mount
defaulted to ${HOME}/.config/javis/gemini, but ${HOME} is often unset when
compose runs outside a WSL shell, silently mounting the wrong dir. Default is
now the project-local ./docker/gemini-oauth (cross-platform), GEMINI_OAUTH_DIR
still overrides.
- No visibility: when oauth is selected but no login is seeded, the path
silently degraded to DDG/Brave. Added gemini_oauth_ready() + a one-time debug
hint and a startup entrypoint warning (skipped on the browser role, fail-open).
- Seeding guidance: oauth_creds.json is the essential credential (refresh token;
GOOGLE_GENAI_USE_GCA=true forces OAuth), which is what the readiness check and
warning verify; docs recommend copying the whole ~/.gemini for convenience.
Adds docker/gemini-oauth/ seed dir (.gitkeep) with the login files gitignored,
GEMINI_OAUTH_DIR in .env.example, and updates DEPLOY.md, stream_browser_modes.md
and llm_contexts.md. Covered by 3 new tests (10 passed total).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per the flag hypothesis: remove the automation-signaling flags that can trigger
Google's /sorry/ challenge. Suppress the --no-sandbox warning bar with a Chrome
managed policy (CommandLineFlagSecurityWarningsEnabled=false) instead of
--test-type, so the infobar stays hidden without the automation signal.
- Chrome: --disable-blink-features=AutomationControlled (+ ko-KR) so Google
shows results, not the /sorry/ automation block.
- Settings persist to /data/jarvis-settings.json (survives recreate; entrypoint
re-merges it) AND the runtime config; apply restarts via a DETACHED process so
the HTTP response isn't dropped when the bridge restarts.
- Bridge reads tts_engine from the settings config so the TTS-engine choice
actually applies.
- control-server.mjs runs chrome-control.mjs LOCALLY on the browser host, so a
remote bot's controlBrowser (BROWSER_CONTROL_URL) drives real xdotool input
on THIS screen instead of the bot machine. Published on the LAN.
- controlBrowser tool posts to BROWSER_CONTROL_URL when set, else runs locally.
- Drop hard depends_on ollama so a browser-host doesn't start Ollama.
- gitignore .env.bak*/*.bak (a backup with tokens had been left untracked).
Add JARVIS_ROLE (full|browser|bot) via a run-if-role.sh supervisord guard so
one image serves three layouts. Make Chrome CDP bind configurable (CDP_BIND)
and publishable on the LAN (CDP_PUBLISH_BIND) so a bot on another PC can drive
this host's on-screen Chrome over the internal network (no auth, as requested).
Root cause of "weather/search do nothing": the engine forced TEXT tool-calling
for all <=7B models, but qwen2.5:3b emits clean NATIVE tool calls and fails at
the text format — so it just narrated ("부산 날씨는 맑습니다") and never called
getWeather/webSearch/controlBrowser. Use native tool-calling for tool-capable
small families (qwen2.5/qwen3/llama3.x/mistral); native still auto-falls back
to text on HTTP 400, so non-tool models (gemma) are unaffected.
Also launch Chrome with --test-type (removes the "--no-sandbox unsupported
flag" infobar) and disable the Translate feature/popup.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
supervisord hardcoded MELO_DEVICE=cpu, overriding the compose MELO_DEVICE=cuda
so MeloTTS stayed on CPU even after the GPU torch swap. Interpolate the
container env instead.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CPU MeloTTS serialised under concurrent load (whisper STT + bot) and blew
voice-reply TTS to 7-8s. Install the Blackwell-verified cu128 torch in the
melo venv, select the GPU via MELO_DEVICE=cuda, and do a throwaway synth at
worker startup so the one-off CUDA kernel-init (~5s) doesn't land on the
user's first reply. Measured: ~0.3s/sentence on GPU vs ~1.2-2.6s on CPU.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The config template never set intent_judge_model, so it fell back to the code
default gemma4:e2b. That model is not pulled by this stack (Ollama only has
qwen2.5:3b, qwen3:8b, nomic-embed-text), so every auxiliary small-model call —
intent judge, tool router, weather place extraction, query decomposition —
targeted a non-existent model, silently failed, and fell open. This crippled
tool routing and argument extraction on the 3B brain.
Render intent_judge_model from a new OLLAMA_INTENT_MODEL env var that defaults
to OLLAMA_CHAT_MODEL, so the auxiliary calls reuse the already-warm chat model
(one resident model, no extra load). tool_router_model="" then resolves through
the chain to the same model.
Verified: rendered jarvis.json now has intent_judge_model=qwen2.5:3b, and the
weather place extractor returns "서울" / "Tokyo" (it returned None for
everything while pointed at the missing gemma4:e2b).
Adds a dedicated MeloTTS Korean voice (speed 1.5) as the primary TTS engine,
served by a long-lived in-container worker so each Discord turn pays only
inference cost, not model-load cost.
- bridge/melo_worker.py: tiny HTTP service in its own /opt/melo py3.11 venv,
keeps the KR model warm, returns PCM16 WAV on POST /synth.
- bridge/server.py: synthesize() routes to the melo worker first; Piper stays
as an opt-in fallback (MELO_FALLBACK_PIPER, default off so Korean is never
mangled through the English voice). /health reports tts_engine.
- docker/setup-melo.sh: builds the isolated venv (pinned torch 2.12.0 /
torchaudio 2.11.0 CPU, MeloTTS pinned to a commit for reproducible rebuilds),
pre-fetches mecab-ko, and warms a dedicated HF cache (/opt/melo-cache) with a
real KR synth so all BERT + KR checkpoint assets are baked into the image.
- docker/supervisord.conf: runs melo-worker before the bridge with
HF_HOME=/opt/melo-cache (the whisper_cache volume shadows the default HF
cache) plus HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE so it reads the baked cache
and never retries the network on load.
- Dockerfile/.env.example: wire the melo build layer and config knobs.
Verified: offline synth passes with --network none and the prod volume mounted;
prod container recreated, all supervisord services up, bot logged in, and an
end-to-end /tts call returns a 44.1kHz mono PCM16 WAV.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
A user account is the only kind Discord lets Go Live, so add a userbot run mode:
when DISCORD_BOT_TOKEN is absent but DISCORD_SELFBOT_TOKEN is set, the app runs
as a userbot instead of the normal slash-command bot.
Stage 1 (conversation): log in with discord.js-selfbot-v13, auto-join
DISCORD_VOICE_CHANNEL_ID on startup (or "!자비스 join" when unset), listen via the
selfbot VoiceReceiver (pcm), transcribe+reply through the bridge, and speak the
reply back with playAudio. Shared PCM/WAV helpers extracted to audio.ts.
run-bot.sh now starts in either userbot or normal mode. Go-Live broadcast +
broadcast-coupled routing land in stage 2 on the same voice connection.
ToS note: selfbots violate Discord ToS; burner account only.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Addresses review findings on the dockerized stack:
- Container Chrome search was dead: add --remote-debugging-port + a non-default
--user-data-dir (Chrome 136+ refuses CDP on the default profile), add the
playwright dep (browse-search.mjs connectOverCDP) with browser download
skipped, and connect to 127.0.0.1 not "localhost" (container localhost -> ::1
while Chrome binds IPv4). Verified: browse-search returns real results.
- Broadcast toggle reliability: always offer setBroadcast in screen-share mode
(the embedding/keyword router dropped it for non-English utterances) and make
its description force a tool call. "방송 꺼줘"->stop now 5/5; no false triggers.
- Stop the broadcast on voice leave (no orphaned stream).
- Security: bind VNC/noVNC to loopback by default (VNC_BIND override) and the
bridge to the container loopback (BRIDGE_HOST=127.0.0.1), not published.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Makes the all-in-one image actually run the new real-time-search features and
closes review gaps:
- Gemini OAuth path: install Node 22 + @google/gemini-cli (pinned 0.46.0) in the
image; mount a DEDICATED host dir (~/.config/javis/gemini) holding only the
OAuth creds to /root/.gemini (not the whole ~/.gemini). Verified in-container:
`gemini -p ... -o json` returns a grounded answer with no API key.
- Broadcast audio: add PulseAudio + a headless null-sink (run-pulse.sh, new
supervisor program); export XDG_RUNTIME_DIR/PULSE_SERVER so Chrome playback
and the selfbot `ffmpeg -f pulse -i @DEFAULT_MONITOR@` share one daemon.
Verified: default sink virtual_speaker, monitor present, ffmpeg capture OK.
- Bind the brain bridge to 127.0.0.1 only (internal, unauthenticated API).
- VNC host port is overridable; this server pins VNC_PORT=5902 (.env) since the
host already runs Xvnc on 5901.
Verified in-container with CDI GPU passthrough: RTX 5050 visible, NVENC
encoders (h264/hevc/av1) available.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Code review of the bridge/bot/docker work found:
- TTS bug: bridge called PiperVoice.synthesize(text, wav) but that method
returns AudioChunks and takes a SynthesisConfig as its 2nd arg, not a wav
file -> TTS would fail. Switched to synthesize_wav(text, wav_file).
Verified: produces a valid 22050Hz mono WAV.
- run-bot.sh now waits if ANY of DISCORD_BOT_TOKEN/APP_ID/GUILD_ID is missing
(config.ts throws on a missing one), preventing a supervisor crash-loop.
Verified clean: discord.js Events.ClientReady == 'clientReady' (existing
handler correct); image rebuilds.
GPU acceleration is now on by default and verified end-to-end on the
Blackwell RTX 5050 (sm_120):
- Ollama offloads 100% to GPU (log: library=CUDA compute=12.0,
BLACKWELL_NATIVE_FP4=1). compose passes GPU via CDI
(devices: nvidia.com/gpu=all) to both ollama and javis.
- Whisper STT on GPU: faster-whisper>=1.1.0 + nvidia-cublas/cudnn cu12,
LD_LIBRARY_PATH baked into the image. Verified float16 transcribe on
sm_120; bridge auto-falls back to CPU when no GPU is present.
- Model: default chat model -> qwen3:8b (best 8GB-VRAM tool-calling,
~5GB Q4). Embed stays nomic-embed-text.
- README documents the host one-time setup (nvidia-container-toolkit +
`nvidia-ctk cdi generate`) and GPU on/off.
Verified: image builds; GPU visible in both containers via compose;
ollama ps = 100% GPU; faster-whisper cuda OK + CPU fallback OK;
bridge /health 200.
`docker compose up -d --build` now brings up the whole thing automatically —
no host setup needed:
- All-in-one javis image: TigerVNC+XFCE desktop, Chrome, Python brain bridge,
Node/bun bot, managed by supervisord (verified: all 6 programs RUNNING).
- ollama service + one-shot ollama-init that auto-pulls chat+embed models
(verified end-to-end; `ollama list` shows pulled models).
- Discord token deferred: without DISCORD_BOT_TOKEN the desktop, bridge,
Ollama and models all run; only the bot waits (no crash loop).
- Slim container deps (bridge/requirements-bridge.txt) drop the unused
PyQt6/torch/chatterbox/sounddevice stack. Piper voice + Whisper models
auto-download into named volumes.
- Configurable host ports (VNC_PORT/NOVNC_PORT/BRIDGE_PORT) to avoid clashing
with a host VNC already on 5901. Bridge binds 0.0.0.0 in-container.
Verified: image builds; brain imports; bridge /health 200; noVNC 200;
X display :1 @1920x1080; auto-pull completes; supervisorctl status all RUNNING.
