docker/jarvis-config.template.json hardcoded "tts_engine": "piper". entrypoint
renders it into /app/config/jarvis.json, and bridge _tts_engine_setting() reads
that JSON BEFORE the env — so TTS_ENGINE=melo in .env was ignored and the bot
synthesised Korean with the English Piper voice (the "foreign accent" the user
heard); the warm melo-worker sat unused.
Template now carries ${TTS_ENGINE}; compose sets TTS_ENGINE=${TTS_ENGINE:-melo}
so envsubst renders the real engine. Verified: envsubst with TTS_ENGINE=melo
yields "tts_engine": "melo", and `docker compose config` passes TTS_ENGINE=melo.
Added a regression test that the template stays env-driven and renders the
configured engine.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
MeloTTS's single Korean speaker sounded non-native ("foreign accent"). Swap it
for Coqui XTTS-v2 with the built-in female studio speaker "Ana Florence"
(language ko), the natural voice used in earlier local runs.
- bridge/xtts_worker.py: new warm HTTP worker (own /opt/xtts venv), same
/synth + /health contract and PCM16 output as the old melo worker
- docker/setup-xtts.sh: builds the venv with cu128 torch (Blackwell) + Coqui
TTS and bakes the XTTS-v2 model offline. Pins transformers>=4.57,<5 (5.x
removed isin_mps_friendly, breaking XTTS) and installs the [codec] extra
(torch>=2.9 needs torchcodec) — both verified by a real host synth
- Dockerfile: replace the melo build layer with the xtts layer
- supervisord.conf: melo-worker -> xtts-worker, env passthrough for
XTTS_DEVICE/SPEAKER/LANGUAGE (always set via compose defaults)
- bridge/server.py: default TTS_ENGINE=xtts, route to the xtts worker, generic
worker-synth helper, neural-only fallback flag (XTTS_FALLBACK_PIPER)
- settings UI: engine dropdown xtts/piper, drop the dead melo_speed field, fix
the supervisorctl restart target to xtts-worker
- compose/.env.example/README: XTTS_* vars, speaker/language knobs, remove melo
- remove bridge/melo_worker.py and docker/setup-melo.sh
- tests: xtts treated as multilingual (not English-only)
Verified on host: coqui-tts loads XTTS-v2 and synthesises Korean as
"Ana Florence" to a 16-bit mono 24kHz WAV.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
supervisord.conf passed MELO_DEVICE through as %(ENV_MELO_DEVICE)s but pinned
MELO_SPEED="1.5", so lowering MELO_SPEED in .env had no effect — the worker
always got 1.5. Pass MELO_SPEED through with %(ENV_MELO_SPEED)s and set a
compose default (MELO_SPEED=${MELO_SPEED:-1.5}, same pattern as MELO_DEVICE) so
the supervisord expansion always resolves and an .env value actually changes
the speaking rate. Default rate is unchanged (1.5). melo_worker logs the
resolved speed at startup, so the env->worker path is verifiable.
Verified: _resolve_speed() returns 1.1 for MELO_SPEED=1.1 (1.5 otherwise), and
`MELO_SPEED=1.1 docker compose config` renders MELO_SPEED: "1.1" into the env.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drop Markdown files into an agents/ folder and their contents are appended to
the main reply LLM's system prompt, so an operator can extend the assistant's
rules/tone without code changes. Files are concatenated in filename order
(use 00-, 10- prefixes to control ordering) and re-read once per turn, so edits
apply on the next reply with no rebuild/restart. Fail-open: a missing, empty,
or unreadable folder yields no instructions and never breaks a reply.
- load_agent_instructions() in system_prompt.py (AGENTS_DIR env, default
/app/agents); reads *.md only, skips blanks, ignores non-dir paths
- engine.py appends it alongside the existing settings-UI llm_instructions,
under the same "Additional instructions from the operator:" framing
- docker-compose.yml bind-mounts ./agents:/app/agents:ro and sets AGENTS_DIR
- agents/example.md.sample starter template (.sample is not loaded)
- tests cover ordering, md-only filtering, blank-skip, env/arg resolution,
and fail-open paths
- README, .env.example, docs/llm_contexts.md updated
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The bridge only logged Whisper's device on the CPU-fallback path, so a
successful GPU (or silent CPU) load was invisible. Print the CTranslate2-
resolved device on success and on the fallback load, so it is verifiable that
STT is actually running on cuda alongside ollama (GPU) and MeloTTS (cuda).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Go-Live broadcast encodes with h264_nvenc, but the image only requested
NVIDIA_DRIVER_CAPABILITIES=compute,utility. The NVIDIA Container Toolkit gates
which driver libraries it injects by capability, and the NVENC/NVDEC libs
(libnvidia-encode.so.1 / libnvidia-decode.so.1) come with the `video`
capability. Without it the broadcast ffmpeg dies with
"Cannot load libnvidia-encode.so.1", the capture produces no packets, and
Go-Live never connects, while CUDA workloads (ollama/whisper/melo) and
nvidia-smi keep working because compute+utility are present.
Add `video` so hardware encode is available. Applies to both Linux (CDI) and
Windows Docker Desktop (WSL2).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
.gitattributes pins *.sh to LF, but that only helps after a full working-tree
renormalise, which a Windows build box may not have done. The image build kept
failing at `RUN bash setup-melo.sh` because the checked-out file still had CRLF,
so bash read line 18 as `set -euxo pipefail\r` and aborted with
"set: pipefail: invalid option name".
Strip CR from setup-melo.sh before running it, and normalise all docker/scripts
shell scripts to LF after the app COPY so their shebangs (entrypoint, run-*.sh)
also survive a CRLF checkout. Makes the build EOL-agnostic regardless of host
autocrlf settings.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Windows checkouts with autocrlf=true inject CR into docker/*.sh, so inside
the Linux container `set -euxo pipefail` is read as `pipefail\r` and bash
aborts with "set: pipefail: invalid option name", failing setup-melo.sh and
the whole image build. .gitattributes already pinned .bat/.cmd/.ps1 to CRLF
but never pinned .sh, leaving all nine container scripts exposed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Windows users following the docs hit "The system cannot find the file
specified" because COMPOSE_FILE's separator is OS-specific (':' collides
with the C: drive letter). Fix every Windows example to use ';', add an
explicit OS-separator warning in .env.example, README, DEPLOY.md and the
gpu-windows compose comment, and point users at the explicit `-f` form as
a separator-agnostic alternative.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Make the browser search helper search the way a person does: load the
site home page, type the query into the search box one key at a time, and
press Enter — for both Google `search` and `youtube` — instead of jumping
straight to a results URL. Supports the goal of a human-like assistant.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add CHROME_USER_DATA_DIR so the browser search fallback can open Chrome
against a dedicated, Google-signed-in profile instead of a fresh anonymous
session. A returning signed-in profile is what actually avoids Google's
/sorry bot-detection page, so this is the reliable way to get browser
Google search in plain text turns. Fallback order is now CDP (broadcast
Chrome) -> persistent profile (when configured) -> ephemeral headless,
all still fail-open to the DDG/Brave/Wikipedia cascade.
Document the profile in .env.example and web_search.spec.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
browse-search.mjs only connected to the on-screen broadcast Chrome over
CDP, so browser-based Google search worked only during a live broadcast;
plain text turns fell through to the DDG cascade. Add a headless fallback
(system Chrome via channel:'chrome', else Playwright's bundled chromium)
for `search` mode so general conversation can use Google at no API cost.
`youtube` still requires the visible broadcast Chrome.
Detect Google's /sorry bot-detection interstitial structurally by URL and
fail fast so the caller fail-opens to DDG/Brave/Wikipedia instead of
treating the challenge page as empty results.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Google now rejects personal Google accounts on the Gemini CLI OAuth login
("This client is no longer supported for Gemini Code Assist for individuals").
The setup docs previously sent every user down "Sign in with Google" with no
warning. Note the block, recommend GEMINI_AUTH=apikey for personal accounts,
and clarify that real-time search fail-opens to DDG/Brave/Wikipedia regardless.
Docs only; no runtime default change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
OAuth cannot be done interactively in the headless container, so the login
must be seeded into the mounted ~/.gemini. Three problems are fixed:
- Mount fragility on the Windows Docker Desktop target: the creds mount
defaulted to ${HOME}/.config/javis/gemini, but ${HOME} is often unset when
compose runs outside a WSL shell, silently mounting the wrong dir. Default is
now the project-local ./docker/gemini-oauth (cross-platform), GEMINI_OAUTH_DIR
still overrides.
- No visibility: when oauth is selected but no login is seeded, the path
silently degraded to DDG/Brave. Added gemini_oauth_ready() + a one-time debug
hint and a startup entrypoint warning (skipped on the browser role, fail-open).
- Seeding guidance: oauth_creds.json is the essential credential (refresh token;
GOOGLE_GENAI_USE_GCA=true forces OAuth), which is what the readiness check and
warning verify; docs recommend copying the whole ~/.gemini for convenience.
Adds docker/gemini-oauth/ seed dir (.gitkeep) with the login files gitignored,
GEMINI_OAUTH_DIR in .env.example, and updates DEPLOY.md, stream_browser_modes.md
and llm_contexts.md. Covered by 3 new tests (10 passed total).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Document that the base compose has no GPU and the GPU is enabled via an
OS-specific override (docker-compose.gpu-linux.yml CDI vs
docker-compose.gpu-windows.yml deploy-reservations), with per-OS host prep,
COMPOSE_FILE shortcut, CPU-only fallback, and Windows manual-run differences
(venv activation, ffmpeg, no .sh scripts / WSL2). Fix stale lines (GPU moved
out of base compose; default model qwen2.5:3b) and add MELO_DEVICE /
output_language to the env list.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Integration test driving the real bridge _save() and engine
_resolve_output_language(): a language chosen in the settings UI is written to
both the persistent volume and the runtime config, applies immediately (config
wins over the OUTPUT_LANGUAGE env), and survives a simulated container recreate
(entrypoint re-renders the config then merges the persistent override). Also
asserts the persona and reply directive both follow the persisted language.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The persona prompt was built from the raw OUTPUT_LANGUAGE env while the
reply-language directive read the settings-web UI value (config JSON), so
changing the language in the settings page was honoured by the directive
but ignored by the persona, leaving them contradicting each other.
Add _resolve_output_language() as the single source of truth (config wins
over env) and feed the same resolved value to both build_system_prompt()
and reply_language_directive(). Update docs/llm_contexts.md to match.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The settings-UI output_language was ignored because the OUTPUT_LANGUAGE env took
precedence. Read the config value first, fall back to env, so changing the
language in /settings actually applies.
Per the flag hypothesis: remove the automation-signaling flags that can trigger
Google's /sorry/ challenge. Suppress the --no-sandbox warning bar with a Chrome
managed policy (CommandLineFlagSecurityWarningsEnabled=false) instead of
--test-type, so the infobar stays hidden without the automation signal.
- Chrome: --disable-blink-features=AutomationControlled (+ ko-KR) so Google
shows results, not the /sorry/ automation block.
- Settings persist to /data/jarvis-settings.json (survives recreate; entrypoint
re-merges it) AND the runtime config; apply restarts via a DETACHED process so
the HTTP response isn't dropped when the bridge restarts.
- Bridge reads tts_engine from the settings config so the TTS-engine choice
actually applies.
Adds /settings (served by the bridge) to change the LLM model (from installed
Ollama models), Whisper model, TTS engine + MeloTTS speed, output language,
agentic max-turns, thinking mode, and free-form LLM instructions — live, with a
'apply' that restarts the bridge + TTS worker. Settings persist to the runtime
config JSON; engine reads output_language + llm_instructions and the TTS worker
reads melo_speed from it. Bridge port publishable for access.
Base compose is GPU-agnostic; GPU is added by a per-OS override selected via
COMPOSE_FILE in .env (docker-compose.gpu-linux.yml for Ubuntu/CDI,
docker-compose.gpu-windows.yml for Windows 11 Docker Desktop). Adds .env.example
split-deployment section + docs/DEPLOY.md covering all-in-one and browser+bot
layouts on both OSes.
controlBrowser search opened the results URL directly (search.naver.com?query).
Now it navigates to the homepage (www.naver.com / google.com), clicks the
on-page search box, types the query char-by-char and presses Enter — real
human-style search, visible on screen.
- control-server.mjs runs chrome-control.mjs LOCALLY on the browser host, so a
remote bot's controlBrowser (BROWSER_CONTROL_URL) drives real xdotool input
on THIS screen instead of the bot machine. Published on the LAN.
- controlBrowser tool posts to BROWSER_CONTROL_URL when set, else runs locally.
- Drop hard depends_on ollama so a browser-host doesn't start Ollama.
- gitignore .env.bak*/*.bak (a backup with tokens had been left untracked).
Add JARVIS_ROLE (full|browser|bot) via a run-if-role.sh supervisord guard so
one image serves three layouts. Make Chrome CDP bind configurable (CDP_BIND)
and publishable on the LAN (CDP_PUBLISH_BIND) so a bot on another PC can drive
this host's on-screen Chrome over the internal network (no auth, as requested).
GeoIP auto-detect is unavailable in the container, so '부산 날씨' failed
(no location). Extract the city from the utterance and pass it to getWeather.
Also report navigate by site name instead of the mid-load about:blank url.
The 7B narrated '구글 메인으로 돌아갑니다' without acting, so the screen stayed on
Naver. Split site intent into SEARCH vs NAV: nav words (돌아가/이동/열어/메인/go
back) now drive controlBrowser.navigate to the site homepage directly, search
words run controlBrowser.search — both deterministically, no LLM.
getWeather now returns only the Korean sentence (지금 <곳> 날씨는 <상태>, 기온 N도
(체감 M도)입니다) with no English/°C source. A deterministic weather path in the
engine returns it verbatim, bypassing the 7B which was rephrasing into multiple
sentences and leaking 'Celsius'.
qwen2.5:7b leaked Chinese/Cyrillic mid-reply despite the OUTPUT_LANGUAGE
lock, which was buried mid-prompt. Repeat the lock at the END of the system
prompt (recency) and ban specific foreign scripts explicitly.
getWeather returned a verbose multi-section English forecast that the 3B
re-synthesised into long, CJK/°F-leaking answers. Hand it a ready-to-speak
Korean one-liner (지금 <곳> 날씨는 <상태>, 기온 N도(체감 M도)입니다) and drop the
hourly/7-day firehose from the default voice reply.
Log the captured speech-clip duration (녹음/음성) separately from the Whisper
transcription time (STT처리) so it's clear whether a slow turn is the
listening/recording or the transcription, per the user's request.
- Site-specified search ("네이버에서 X 검색해줘") now runs controlBrowser.search
directly in the engine when broadcasting, instead of relying on the 3B model
to emit the tool call (it kept narrating "검색하겠습니다" without acting).
- Set OUTPUT_LANGUAGE=ko so replies are Korean-only — stops the small model
leaking CJK/Hanja and English fragments (每, 朗, "feels like") into weather
and other answers, and keeps them concise.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Greetings/small-talk routed no data tool yet still ran the episodic memory
enrichment (LLM keyword extract + diary/graph search, ~1s) every turn. Skip it
when the router picked no external-data tool — the always-injected warm profile
still personalises the reply. Also drop the voice silence-detection wait
800ms -> 600ms for snappier turn-taking. Warm "안녕" now lands well under the
3-4s target.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of "weather/search do nothing": the engine forced TEXT tool-calling
for all <=7B models, but qwen2.5:3b emits clean NATIVE tool calls and fails at
the text format — so it just narrated ("부산 날씨는 맑습니다") and never called
getWeather/webSearch/controlBrowser. Use native tool-calling for tool-capable
small families (qwen2.5/qwen3/llama3.x/mistral); native still auto-falls back
to text on HTTP 400, so non-tool models (gemma) are unaffected.
Also launch Chrome with --test-type (removes the "--no-sandbox unsupported
flag" infobar) and disable the Translate feature/popup.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The 3B model kept choosing webSearch over controlBrowser even when offered, so
'네이버에서 X 검색' still used the invisible web path. When broadcasting and the
user explicitly names a site, remove webSearch from the allow-list so the
on-screen browser is the only search route.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The small router reflexively routed every "search/open" intent to webSearch
and never surfaced controlBrowser, so "네이버에서 X 검색해줘" did nothing on the
broadcast. Union controlBrowser (+browseAndPlay) into the allow-list every
turn in screen-share mode (like setBroadcast), and steer the model in the
system prompt to prefer the on-screen browser over webSearch when available.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add a one-shot `search` action (site=naver/google/daum/youtube/bing) that
navigates the on-screen browser straight to the results page, so a small
model can satisfy "search X on Naver" in a single tool call instead of a
fragile navigate->type->enter chain.
- Sharpen the tool description to steer the router to controlBrowser (not
webSearch) for anything that should happen IN the visible browser.
- System prompt: answer in one short sentence (voice assistant) — also cuts
TTS time.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Install xdotool + wmctrl (late Docker layer, preserves the melo cache) so
the on-screen Chrome gets real X input (visible cursor, char-by-char typing)
instead of synthetic events; falls back to the Playwright API if absent.
- Fix active-tab detection (probe document.visibilityState instead of assuming
tab 0) so sequential ops target the right tab.
- Add back / forward / refresh; new/switch/close tabs via real keyboard
(Ctrl+T / Ctrl+<n> / Ctrl+W) when xdotool is present.
- Auto-dismiss native JS dialogs; closePopups clears blank/popup tabs.
- Report broadcast (Go-Live) state in status from the turn context.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
supervisord hardcoded MELO_DEVICE=cpu, overriding the compose MELO_DEVICE=cuda
so MeloTTS stayed on CPU even after the GPU torch swap. Interpolate the
container env instead.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a general browser-control tool (navigate to any site, list/open/close/
switch tabs, close popups, click, type, scroll, screenshot) for the Go-Live
Chrome, on top of the existing CDP + xdotool human-input layer (visible
cursor, char-by-char typing). Closes the gap where "open Naver" had no tool
and the model confabulated success. Also adds a system-prompt rule against
claiming actions no tool actually performed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CPU MeloTTS serialised under concurrent load (whisper STT + bot) and blew
voice-reply TTS to 7-8s. Install the Blackwell-verified cu128 torch in the
melo venv, select the GPU via MELO_DEVICE=cuda, and do a throwaway synth at
worker startup so the one-off CUDA kernel-init (~5s) doesn't land on the
user's first reply. Measured: ~0.3s/sentence on GPU vs ~1.2-2.6s on CPU.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- llm.py: log each Ollama call's caller + total/load/prompt/gen durations
so a slow voice turn is attributable to a specific internal call
(router/enrichment/digest/main); a RELOAD marker flags cold reloads.
- voice.ts: track in-flight Opus captures and abort them on session
destroy(); drop any utterance that finishes after the user left, so no
trailing post-leave VAD turns are reported.
- userbot.ts: show the speaker's Discord user ID on each transcript line
(answered and dropped) so it's clear whose audio produced the turn.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The startup warm-up loaded qwen at the default context, but the reply
engine chats at num_ctx=8192 — a different Ollama instance — so the first
real turn still cold-reloaded. Warm at OLLAMA_NUM_CTX via /api/chat.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Ollama keeps a separate loaded model instance per (model, num_ctx). The
main agentic chat used num_ctx=8192 while the router/enrichment/digest
passes used 4096, so every voice turn forced at least one cold reload
(~3.4s) when switching context sizes — the dominant per-turn latency
(measured: resident chat call 0.27s vs cold 3.4s).
Introduce a single OLLAMA_NUM_CTX (default 8192, env-tunable for tight
VRAM) used by call_llm_direct, chat_with_messages, call_llm_streaming and
the planner, collapsing a turn to one resident instance.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The first spoken turn paid a ~10s cold start because Whisper (default
"medium") and the Ollama chat model loaded lazily on the first request.
Warm them (and ping the TTS worker) in a background thread at startup so
the server accepts requests immediately while models load, and the first
real utterance is fast.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The transcript channel only showed STT and LLM seconds. Add wall-clock
start/end times and durations for listening, LLM and TTS so it's obvious
what takes long; STT surfaces as the gap between listening end and LLM start.
- bridge: emit llm_start_ms/llm_end_ms on meta and tts_*_ms on the end event
- bot: capture the listening window, assemble full timing after the stream,
and render a per-stage breakdown in the transcript message
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Proven approach: the conversation (hear+speak) runs on @discordjs/voice; the
Go-Live broadcast is a SEPARATE stream connection created on the SAME selfbot
session (exactly like a real Discord client), so ONE account hears, speaks, AND
broadcasts — no second login, no self_deaf, no voice conflict.
- voice.ts captures its own voice session_id (adapter wrap) and exposes
getSharedSession() {client, guildId, channelId, sessionId, botId}.
- broadcast.ts threads it into the StreamContext.
- selfbot.ts: when a shared session is present, build the Streamer on the
conversation client and create the stream on its session_id (no login/joinVoice/
humanPause); teardown only stops the stream (never leaves voice/destroys the
shared client). Falls back to the dedicated-account path otherwise.
Verified live: Go-Live connected in ~7s while the conversation voice stayed
ready, and the broadcast was visible in Discord — all on one account.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The transcript channel only showed successful transcripts, so dropped utterances
(the 47/50 misses) were invisible. Now every captured utterance is mirrored with
its outcome and per-stage timing:
- too-short blip (<300ms), VAD "음성 아님(VAD 차단)", "인식 실패", "답변 없음", or "ok"
- transcript + reply (or "(무응답)")
- ⏱️ stt/llm seconds
The bridge meta now carries note + stt_sec + think_sec; voice.ts fires onTurn for
every turn (not only non-empty transcripts) and for the too-short drop; userbot
formats the diagnostic line.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
