7 Commits

Author SHA1 Message Date
javis-bot
140fc56f18 feat: play the Nth YouTube result in browseAndPlay via an index arg
Some checks failed
Release / semantic-release (push) Successful in 22s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m16s
Release / build-linux (push) Failing after 7m10s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
agents/llm.md promises "play the Nth video from the top", but browseAndPlay
only ever clicked the first result. Add an optional 1-based index argument
(default 1, backward-compatible) threaded to the Node helper, which now clicks
the Nth a#video-title and clamps to the number of results returned so asking
beyond the list plays the last available video instead of failing.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-23 15:33:45 +09:00
javis-bot
5ee47827f3 perf: cap chat output tokens via ollama_num_predict to bound reply latency
Spoken (TTS) replies are 1-2 sentences, so an unbounded num_predict only
exposes the worst case where the chat model rambles or loops. Add an
ollama_num_predict config (default 512, 0 disables) wired into the reply
loop's chat call on both the native- and text-tool paths. The 512-token
headroom stays well above this app's short tool-call JSON, so capping never
truncates a tool call. This keeps the user's quality model instead of
downgrading it. Configurable in the container via OLLAMA_NUM_PREDICT.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-23 15:33:45 +09:00
javis-bot
c189ce2e65 feat: Korean Chrome locale, agents/llm.md voice instructions, drop emoji rule
Some checks failed
Release / semantic-release (push) Successful in 26s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m17s
Release / build-linux (push) Failing after 7m7s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
- run-chrome.sh: render web content (YouTube/Google/Naver) in Korean. --lang
  only sets Chrome's UI; the Accept-Language sent to sites comes from the
  profile's intl.accept_languages, which a persisted user-data-dir kept at
  en-US. Seed the profile to ko-KR and add --accept-lang=ko-KR,ko.
- agents/llm.md: runtime instructions for the reply LLM (loaded by the agents
  feature) — name "자비스", concise 1-2 sentence TTS replies (no emojis/markdown),
  nuance-based intent, and YouTube voice controls (open/search/play Nth/pause/
  back) via the on-screen browser tool.
- CLAUDE.md: drop the "use emojis in CLI output" rule — this assistant replies by
  Discord voice, not CLI, so output should be plain.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-23 04:34:17 +09:00
javis-bot
086dd5cde7 fix: accept edge as a valid tts_engine and migrate stale persisted engines
Some checks failed
Release / semantic-release (push) Successful in 34s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m17s
Release / build-linux (push) Failing after 7m9s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
load_settings() coerced any tts_engine outside {piper, chatterbox} to piper, so
with TTS_ENGINE=edge the reply engine saw "piper" and treated the voice as
English-only in reply_language_directive() (only the OUTPUT_LANGUAGE lock kept
replies Korean). Add "edge" (and "melo") to the accepted set so the engine is
labelled multilingual correctly.

Also: a stale tts_engine in the persistent /data/jarvis-settings.json (melo/xtts
from an earlier voice, no longer built) would override the configured engine via
the entrypoint merge and leave the bot silent. Reset those to the env engine
during the merge.

Verified: load_settings() with tts_engine=edge now returns "edge"; the merge
maps melo/xtts -> edge; reply_language_directive("edge") is multilingual; 27
tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-23 03:49:53 +09:00
javis-bot
f64d76e737 feat: use Edge TTS (Korean Hyunsu voice @ +45%) as the default voice
Some checks failed
Release / semantic-release (push) Successful in 31s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / build-linux (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
tests / Unit tests (Linux, Python 3.11) (push) Has been cancelled
The user chose Microsoft Edge TTS, voice ko-KR-HyunsuMultilingualNeural at rate
+45% (~1.45x), as the natural Korean voice. Wire it into the bridge and make it
the default engine.

- bridge/server.py: _edge_synthesize() calls edge-tts and transcodes the MP3 to
  PCM16 mono WAV with the system ffmpeg (temp file for a correct header);
  TTS_ENGINE default -> edge; EDGE_TTS_VOICE / EDGE_TTS_RATE env-driven
- requirements-bridge.txt: add edge-tts (lightweight; httpx)
- compose/.env.example/README: TTS_ENGINE=edge + EDGE_TTS_* knobs; note the
  online/privacy trade-off (reply text is sent to Microsoft, needs internet)
- drop the now-unused MeloTTS build layer (Dockerfile) and melo-worker
  (supervisord) — edge synthesises in-process, no model/worker baked, slimmer
  and faster image; settings UI engine list -> edge/piper, restart only bridge

Verified on host: edge-tts -> ffmpeg yields a valid 16-bit mono 24kHz WAV;
envsubst renders tts_engine=edge; docker build --check + 26 tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-23 03:44:15 +09:00
javis-bot
11c3621093 fix: make container TTS engine env-driven so melo isn't overridden by piper
Some checks failed
Release / semantic-release (push) Successful in 30s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m17s
Release / build-linux (push) Failing after 7m3s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
docker/jarvis-config.template.json hardcoded "tts_engine": "piper". entrypoint
renders it into /app/config/jarvis.json, and bridge _tts_engine_setting() reads
that JSON BEFORE the env — so TTS_ENGINE=melo in .env was ignored and the bot
synthesised Korean with the English Piper voice (the "foreign accent" the user
heard); the warm melo-worker sat unused.

Template now carries ${TTS_ENGINE}; compose sets TTS_ENGINE=${TTS_ENGINE:-melo}
so envsubst renders the real engine. Verified: envsubst with TTS_ENGINE=melo
yields "tts_engine": "melo", and `docker compose config` passes TTS_ENGINE=melo.
Added a regression test that the template stays env-driven and renders the
configured engine.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-23 03:27:33 +09:00
javis-bot
7ad5d99380 Revert "feat: replace MeloTTS with Coqui XTTS-v2 natural Korean voice"
Some checks failed
Release / semantic-release (push) Successful in 35s
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m16s
Release / build-windows (push) Has been cancelled
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Release / release-main (push) Has been cancelled
Release / release-develop (push) Has been cancelled
Release / build-linux (push) Has been cancelled
This reverts commit 39a0944105.
2026-06-23 03:15:54 +09:00
27 changed files with 646 additions and 285 deletions

View File

@@ -34,23 +34,18 @@ WHISPER_DEVICE=cuda
WHISPER_COMPUTE_TYPE=float16 WHISPER_COMPUTE_TYPE=float16
# Optional explicit Piper voice model (.onnx). If empty, the jarvis default is used. # Optional explicit Piper voice model (.onnx). If empty, the jarvis default is used.
TTS_PIPER_MODEL_PATH= TTS_PIPER_MODEL_PATH=
# TTS engine: "xtts" (default) uses the Coqui XTTS-v2 natural Korean voice # TTS engine: "edge" (default) uses Microsoft Edge TTS, a natural Korean neural
# served by the warm xtts-worker. Set to "piper" to use the English Piper voice # voice. Set to "piper" for the offline English voice. NOTE: edge is ONLINE —
# directly. (MeloTTS was removed; "melo" only works with an out-of-band worker.) # reply text is sent to Microsoft's servers and needs internet.
TTS_ENGINE=xtts TTS_ENGINE=edge
# XTTS-v2 voice settings. Speaker is any built-in studio voice; "Ana Florence" # Edge voice + speaking rate. Rate is a percentage (+45% ≈ 1.45×). Korean voices:
# is a natural female voice. Language is the synthesis language (ko = Korean). # ko-KR-HyunsuMultilingualNeural (M), ko-KR-InJoonNeural (M), ko-KR-SunHiNeural (F).
XTTS_SPEAKER=Ana Florence EDGE_TTS_VOICE=ko-KR-HyunsuMultilingualNeural
XTTS_LANGUAGE=ko EDGE_TTS_RATE=+45%
XTTS_DEVICE=cuda # Neural-only by default: if synthesis fails the bridge returns no audio rather
# Where the bridge reaches the in-container XTTS worker, and how long it waits # than speaking Korean through the English Piper voice. Set to 1 to allow the
# for a synthesis (XTTS is slower than Melo: ~1-2s/sentence on GPU). # Piper fallback.
XTTS_WORKER_URL=http://127.0.0.1:8771 MELO_FALLBACK_PIPER=0
XTTS_TIMEOUT=30
# Neural-only by default: if XTTS synthesis fails the bridge returns no audio
# rather than speaking Korean through the English Piper voice (which mangles it).
# Set to 1 only if you explicitly want the Piper fallback.
XTTS_FALLBACK_PIPER=0
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Jarvis brain (Ollama-backed). In Docker these populate the rendered # Jarvis brain (Ollama-backed). In Docker these populate the rendered
@@ -231,7 +226,7 @@ COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-linux.yml
# OLLAMA_CHAT_MODEL=qwen2.5:7b # quality (needs ~5GB VRAM + whisper small) # OLLAMA_CHAT_MODEL=qwen2.5:7b # quality (needs ~5GB VRAM + whisper small)
# OLLAMA_CHAT_MODEL=qwen2.5:3b # speed (fits easily, faster on 8GB GPUs) # OLLAMA_CHAT_MODEL=qwen2.5:3b # speed (fits easily, faster on 8GB GPUs)
# WHISPER_MODEL=small # small frees VRAM for a bigger LLM; medium=more accurate # WHISPER_MODEL=small # small frees VRAM for a bigger LLM; medium=more accurate
# XTTS_DEVICE=cuda # cpu if no GPU on the bot host (XTTS is slow on CPU) # MELO_DEVICE=cuda # cpu if no GPU on the bot host
# --- Settings web UI (http://localhost:8765/settings on the bot host) --- # --- Settings web UI (http://localhost:8765/settings on the bot host) ---
# To reach it, expose the bridge to the host loopback: # To reach it, expose the bridge to the host loopback:

View File

@@ -1,6 +1,6 @@
Data privacy comes first, always. Data privacy comes first, always.
All user-facing command line output should make use of emojis. Especially an initial emoji to start off the lines that depict what the line is about. Output should make use of indentation spacing to establish a visual hierarchy and aim to make output as easy to sift through as possible. Exception: Windows .bat scripts cannot use emojis (cmd.exe doesn't render Unicode properly). This assistant is used through a Discord bot with voice (TTS) replies, not a CLI. Do not add emojis to user-facing assistant output. Keep output plain and readable. (Runtime assistant behaviour lives in `agents/*.md`, which is injected into the reply LLM's prompt.)
Any important point in our logical flows should have debug logs using the `debug_log` method from `src/jarvis/debug.py`. Avoid excessive logging to keep the logs easily readable and actionable. Any important point in our logical flows should have debug logs using the `debug_log` method from `src/jarvis/debug.py`. Avoid excessive logging to keep the logs easily readable and actionable.

View File

@@ -65,21 +65,14 @@ RUN ls -d /opt/venv/lib/python*/site-packages/nvidia/cublas/lib \
> /etc/ld.so.conf.d/nvidia-cu12.conf 2>/dev/null \ > /etc/ld.so.conf.d/nvidia-cu12.conf 2>/dev/null \
&& /sbin/ldconfig || true && /sbin/ldconfig || true
# --- Korean voice: Coqui XTTS-v2 (separate /opt/xtts py3.11 venv; see # --- Korean voice: Microsoft Edge TTS (online neural). No model is baked — the
# setup-xtts.sh). Natural female Korean ("Ana Florence"); replaces MeloTTS. # `edge-tts` pip package (in requirements-bridge.txt) calls the MS service at
# Heavy layer (torch cu128 + Coqui TTS + the baked XTTS-v2 model); placed # runtime and the bridge transcodes the MP3 to PCM16 with ffmpeg. No heavy
# before the app COPY so it stays cached across source-only changes. --- # TTS build layer is needed. ---
COPY docker/setup-xtts.sh /app/docker/setup-xtts.sh
# Strip CR before running: a Windows checkout (autocrlf) yields CRLF, which makes
# bash read `set -euxo pipefail\r` and abort with "set: pipefail: invalid option
# name". .gitattributes pins *.sh to LF, but this keeps the build working even on
# a not-yet-renormalised working tree.
RUN sed -i 's/\r$//' /app/docker/setup-xtts.sh && bash /app/docker/setup-xtts.sh
# --- Human input + window management for the on-screen Chrome control tool. # --- Human input + window management for the on-screen Chrome control tool.
# Placed AFTER the heavy TTS layer so it doesn't bust that cache. xdotool # xdotool injects real X pointer/keyboard events (visible cursor,
# injects real X pointer/keyboard events (visible cursor, char-by-char # char-by-char typing) into the broadcast; wmctrl lists/moves windows. ---
# typing) into the broadcast; wmctrl lists/moves windows. ---
RUN apt-get update && apt-get install -y --no-install-recommends \ RUN apt-get update && apt-get install -y --no-install-recommends \
xdotool wmctrl \ xdotool wmctrl \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*

View File

@@ -69,7 +69,7 @@ docker compose -f docker-compose.yml -f docker-compose.gpu-linux.yml up -d --bui
docker compose -f docker-compose.yml -f docker-compose.gpu-windows.yml up -d --build docker compose -f docker-compose.yml -f docker-compose.gpu-windows.yml up -d --build
# ── GPU 없이 (CPU 전용 호스트) ── # ── GPU 없이 (CPU 전용 호스트) ──
# .env 에 WHISPER_DEVICE=cpu, XTTS_DEVICE=cpu 를 넣고 베이스만 사용 # .env 에 WHISPER_DEVICE=cpu 를 넣고 베이스만 사용
docker compose up -d --build docker compose up -d --build
``` ```
@@ -113,7 +113,7 @@ docker compose up -d # 유저봇이 로그인해 지정 음성채널에
### GPU 가속 (OS별) ### GPU 가속 (OS별)
LLM(Ollama), Whisper STT, XTTS-v2 TTS가 GPU에서 돕니다(env 기본 `WHISPER_DEVICE=cuda`, `XTTS_DEVICE=cuda`). NVIDIA Blackwell(sm_120, 예: RTX 5050/5070Ti)에서 검증: 컨테이너 내 torch cu128 CUDA 동작, Ollama GPU 오프로드, faster-whisper float16, XTTS-v2 GPU 합성 모두 확인. LLM(Ollama) Whisper STT가 GPU에서 돕니다(env 기본 `WHISPER_DEVICE=cuda`). TTS는 기본값이 Edge TTS(온라인 한국어 음성)라 GPU를 쓰지 않습니다. NVIDIA Blackwell(sm_120, 예: RTX 5050/5070Ti)에서 검증: 컨테이너 내 torch cu128 CUDA 동작, Ollama GPU 오프로드, faster-whisper float16 모두 확인.
GPU는 위 "실행 — Docker"의 OS별 override 파일로 켜집니다. 호스트 사전 준비는 OS마다 다릅니다: GPU는 위 "실행 — Docker"의 OS별 override 파일로 켜집니다. 호스트 사전 준비는 OS마다 다릅니다:
@@ -137,7 +137,7 @@ docker run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi -L # GPU 보이
**공통:** **공통:**
- 모델: 베이스 compose 기본은 `qwen2.5:3b`(8GB VRAM에서 도구호출 안정적). 더 무겁게(`qwen2.5:7b`, `qwen3:8b` 등) 쓰려면 `.env``OLLAMA_CHAT_MODEL` 변경. - 모델: 베이스 compose 기본은 `qwen2.5:3b`(8GB VRAM에서 도구호출 안정적). 더 무겁게(`qwen2.5:7b`, `qwen3:8b` 등) 쓰려면 `.env``OLLAMA_CHAT_MODEL` 변경.
- **GPU가 없거나 인식 실패 시 자동으로 CPU 폴백**(Whisper)하므로 안전합니다. 명시적으로 CPU만 쓰려면 override 파일 없이 베이스만 올리고 `.env``WHISPER_DEVICE=cpu`, `XTTS_DEVICE=cpu`를 두세요. - **GPU가 없거나 인식 실패 시 자동으로 CPU 폴백**(Whisper)하므로 안전합니다. 명시적으로 CPU만 쓰려면 override 파일 없이 베이스만 올리고 `.env``WHISPER_DEVICE=cpu`를 두세요.
- 데이터(메모리 DB), Whisper 캐시, Piper 음성은 named volume에 영속됩니다. - 데이터(메모리 DB), Whisper 캐시, Piper 음성은 named volume에 영속됩니다.
- 셀프봇 영상 송출 의존성은 이미지에 기본 포함하지 않습니다. 쓰려면 컨테이너에서 `cd /app/bot && bun add discord.js-selfbot-v13 @dank074/discord-video-stream` 후 재시작(또는 Dockerfile에 추가). - 셀프봇 영상 송출 의존성은 이미지에 기본 포함하지 않습니다. 쓰려면 컨테이너에서 `cd /app/bot && bun add discord.js-selfbot-v13 @dank074/discord-video-stream` 후 재시작(또는 Dockerfile에 추가).
@@ -243,7 +243,7 @@ cd bot; bun run register; bun run start # 창 2: (일반 봇이면) 슬래시
- `BRIDGE_URL` — 봇이 호출할 브릿지 주소 (기본 `http://127.0.0.1:8765`) - `BRIDGE_URL` — 봇이 호출할 브릿지 주소 (기본 `http://127.0.0.1:8765`)
- `STREAM_BACKEND`, `DISCORD_SELFBOT_TOKEN`, `NOVNC_URL` — 화면 송출 - `STREAM_BACKEND`, `DISCORD_SELFBOT_TOKEN`, `NOVNC_URL` — 화면 송출
- `VNC_DISPLAY=:1`, `VNC_RESOLUTION`, `VNC_FRAMERATE`, `VNC_BITRATE_KBPS` — 캡처 - `VNC_DISPLAY=:1`, `VNC_RESOLUTION`, `VNC_FRAMERATE`, `VNC_BITRATE_KBPS` — 캡처
- `WHISPER_DEVICE/COMPUTE_TYPE`, `XTTS_DEVICE` — GPU 호스트면 `cuda`/`float16`, CPU 전용이면 `cpu`(GPU 자체는 OS별 override compose 파일로 켬) - `WHISPER_DEVICE/COMPUTE_TYPE` — GPU 호스트면 `cuda`/`float16`, CPU 전용이면 `cpu`(GPU 자체는 OS별 override compose 파일로 켬)
- `OLLAMA_CHAT_MODEL` — 두뇌 LLM (기본 `qwen2.5:3b`) - `OLLAMA_CHAT_MODEL` — 두뇌 LLM (기본 `qwen2.5:3b`)
- `COMPOSE_FILE` — OS별 GPU override를 매번 `-f`로 안 치고 자동 적용 (위 "실행 — Docker" 참고) - `COMPOSE_FILE` — OS별 GPU override를 매번 `-f`로 안 치고 자동 적용 (위 "실행 — Docker" 참고)
- `output_language` — 출력 언어 고정(비우면 사용자 언어). 설정 웹 UI(`/settings`)에서 바꾸면 env 기본값보다 우선하며 컨테이너 재생성 후에도 유지됩니다. - `output_language` — 출력 언어 고정(비우면 사용자 언어). 설정 웹 UI(`/settings`)에서 바꾸면 env 기본값보다 우선하며 컨테이너 재생성 후에도 유지됩니다.

13
agents/llm.md Normal file
View File

@@ -0,0 +1,13 @@
# 자비스 운영자 지시
- 너의 이름은 자비스다.
- 모든 답변은 음성(TTS)으로 읽혀 나간다. 그러니 최대한 간결하게, 한두 문장으로 답한다. 목록, 마크다운, 이모지, 그리고 소리 내어 읽기 어려운 특수문자는 쓰지 않는다.
- 정해진 문구에만 반응하지 말고, 실제 사람처럼 말의 뉘앙스와 맥락으로 의도를 알아듣고 처리한다.
화면 속 크롬(방송 화면)에서 유튜브를 다룰 때 (화면에 보여야 하므로 반드시 on-screen 브라우저 제어 도구로 수행한다):
- "유튜브 켜줘" → 방송 크롬에서 유튜브를 연다.
- "유튜브에서 OO 검색해줘" → 유튜브로 가서 검색창에 OO를 사람이 직접 타이핑하듯 입력하고 검색한다.
- "위에서 N번째 영상 재생해줘" 또는 "왼쪽에서 N번째 영상 재생해줘" → 검색 결과 목록에서 그 위치의 영상을 재생한다.
- "일시정지해줘" → 현재 영상을 일시정지한다. "다시 재생해줘" → 이어서 재생한다.
- "영상 종료" 또는 "그만 보여줘" → 뒤로 가서 직전 화면으로 돌아간다.

View File

@@ -2,10 +2,11 @@
// 9222) so the action is visible on the Go-Live broadcast, and prints a JSON // 9222) so the action is visible on the Go-Live broadcast, and prints a JSON
// result on stdout for the Python `browseAndSearch` tool to wrap. // result on stdout for the Python `browseAndSearch` tool to wrap.
// //
// node browse-search.mjs "<query>" [search|youtube] // node browse-search.mjs "<query>" [search|youtube] [index]
// //
// - search : Google-search the query, return the top organic results. // - search : Google-search the query, return the top organic results.
// - youtube : search YouTube and play the first result. // - youtube : search YouTube and play a result. `index` is the 1-based position
// from the top of the result list (default 1 = first result).
// //
// Backend selection for `search`: // Backend selection for `search`:
// 1. The broadcast Chrome over CDP (visible on the Go-Live stream). // 1. The broadcast Chrome over CDP (visible on the Go-Live stream).
@@ -29,6 +30,9 @@ const UA =
'(KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36'; '(KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36';
const query = process.argv[2] || ''; const query = process.argv[2] || '';
const mode = (process.argv[3] || 'search').toLowerCase(); const mode = (process.argv[3] || 'search').toLowerCase();
// 1-based position of the YouTube result to play, counted from the top of the
// list. Defaults to 1 (first result). Anything <1 or non-numeric falls back to 1.
const playIndex = Math.max(1, parseInt(process.argv[4], 10) || 1);
const out = (o) => { process.stdout.write(JSON.stringify(o)); }; const out = (o) => { process.stdout.write(JSON.stringify(o)); };
if (!query) { out({ ok: false, error: 'no query' }); process.exit(1); } if (!query) { out({ ok: false, error: 'no query' }); process.exit(1); }
@@ -105,15 +109,21 @@ try {
await page.bringToFront().catch(() => {}); await page.bringToFront().catch(() => {});
if (mode === 'youtube') { if (mode === 'youtube') {
// Type into YouTube's search box like a person, then play the first result. // Type into YouTube's search box like a person, then play the requested
// result (the Nth from the top of the list; default the first).
await typeSearch('https://www.youtube.com/?hl=ko', 'input#search, input[name="search_query"]', query); await typeSearch('https://www.youtube.com/?hl=ko', 'input#search, input[name="search_query"]', query);
await page.waitForSelector('ytd-video-renderer a#video-title, a#video-title', { timeout: 20000 }); await page.waitForSelector('ytd-video-renderer a#video-title, a#video-title', { timeout: 20000 });
const first = page.locator('ytd-video-renderer a#video-title, a#video-title').first(); const results = page.locator('ytd-video-renderer a#video-title, a#video-title');
const title = (await first.getAttribute('title').catch(() => '')) || (await first.innerText().catch(() => '')); // Clamp to what's actually on the page so "play the 5th" still plays the
await first.click(); // last available result rather than failing when fewer were returned.
const available = await results.count();
const targetIdx = Math.min(playIndex, Math.max(available, 1)) - 1;
const target = results.nth(targetIdx);
const title = (await target.getAttribute('title').catch(() => '')) || (await target.innerText().catch(() => ''));
await target.click();
await page.waitForSelector('#movie_player', { timeout: 20000 }); await page.waitForSelector('#movie_player', { timeout: 20000 });
await page.evaluate(() => { const v = document.querySelector('video'); if (v && v.paused) v.play(); }); await page.evaluate(() => { const v = document.querySelector('video'); if (v && v.paused) v.play(); });
out({ ok: true, mode, title: (title || '').trim(), url: page.url() }); out({ ok: true, mode, index: targetIdx + 1, title: (title || '').trim(), url: page.url() });
} else { } else {
// Type into Google's search box like a person, then read the results. // Type into Google's search box like a person, then read the results.
await typeSearch('https://www.google.com/?hl=ko', 'textarea[name="q"], input[name="q"]', query); await typeSearch('https://www.google.com/?hl=ko', 'textarea[name="q"], input[name="q"]', query);

View File

@@ -1,30 +1,25 @@
""" """
XTTS worker MeloTTS worker
=========== ==============
A tiny HTTP service that keeps a Coqui XTTS-v2 voice warm and synthesises A tiny, dependency-light HTTP service that keeps a MeloTTS voice warm and
speech on demand. It mirrors ``melo_worker.py`` (same ``/synth`` + ``/health`` synthesises speech on demand. It runs in its OWN Python venv (``/opt/melo`` in
contract, same PCM16 WAV output) so the bridge can talk to either worker the the container) so the heavy MeloTTS/torch/transformers stack stays isolated
same way. from the slim brain-bridge venv (which pins ``numpy<2`` for faster-whisper).
XTTS-v2 is a natural, multilingual neural voice. The default speaker is the The bridge's ``synthesize()`` POSTs ``{"text": "..."}`` here and gets back a
built-in female studio voice "Ana Florence" speaking Korean the voice this 16-bit PCM WAV. The MeloTTS model is loaded once at startup and reused, so each
deployment uses in place of MeloTTS. No reference WAV is needed for the request only pays inference cost, not model-load cost.
built-in studio speakers.
It runs in its OWN Python venv (``/opt/xtts`` in the container) so the heavy
Coqui TTS / torch stack stays isolated from the slim brain-bridge venv.
Config (env): Config (env):
XTTS_WORKER_HOST bind host (default 127.0.0.1) MELO_WORKER_HOST bind host (default 127.0.0.1)
XTTS_WORKER_PORT bind port (default 8771) MELO_WORKER_PORT bind port (default 8770)
XTTS_MODEL Coqui model id (default tts_models/multilingual/multi-dataset/xtts_v2) MELO_LANGUAGE MeloTTS language (default KR)
XTTS_SPEAKER built-in speaker (default "Ana Florence") MELO_SPEED speaking rate (default 1.5 -> the approved "150")
XTTS_LANGUAGE synthesis language (default ko) MELO_DEVICE torch device (default cpu)
XTTS_DEVICE torch device (default cpu; compose sets cuda)
Run: Run:
/opt/xtts/bin/python -m bridge.xtts_worker /opt/melo/bin/python -m bridge.melo_worker
""" """
from __future__ import annotations from __future__ import annotations
@@ -38,72 +33,94 @@ import threading
import wave import wave
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
# XTTS-v2 is gated behind a one-time license prompt; agreeing here keeps the HOST = os.environ.get("MELO_WORKER_HOST", "127.0.0.1")
# load non-interactive in a container. XTTS-v2 is non-commercial (CPML). PORT = int(os.environ.get("MELO_WORKER_PORT", "8770"))
os.environ.setdefault("COQUI_TOS_AGREED", "1") LANGUAGE = os.environ.get("MELO_LANGUAGE", "KR")
HOST = os.environ.get("XTTS_WORKER_HOST", "127.0.0.1")
PORT = int(os.environ.get("XTTS_WORKER_PORT", "8771"))
MODEL = os.environ.get("XTTS_MODEL", "tts_models/multilingual/multi-dataset/xtts_v2")
SPEAKER = os.environ.get("XTTS_SPEAKER", "Ana Florence")
LANGUAGE = os.environ.get("XTTS_LANGUAGE", "ko")
DEVICE = os.environ.get("XTTS_DEVICE", "cpu")
# Model is loaded once, guarded by a lock because TTS inference is not def _resolve_speed() -> float:
# guaranteed thread-safe. """Speaking rate: the settings-UI value (runtime config JSON) wins, else the
MELO_SPEED env, else 1.5. Read at startup; the settings UI restarts this
worker on apply so a new value takes effect."""
try:
cp = os.environ.get("JARVIS_CONFIG_PATH", "/app/config/jarvis.json")
v = json.loads(open(cp, encoding="utf-8").read()).get("melo_speed")
if v is not None:
return float(v)
except Exception:
pass
try:
return float(os.environ.get("MELO_SPEED", "1.5"))
except ValueError:
return 1.5
SPEED = _resolve_speed()
DEVICE = os.environ.get("MELO_DEVICE", "cpu")
# Model + speaker id are loaded once, guarded by a lock because MeloTTS
# inference is not guaranteed thread-safe.
_model = None _model = None
_speaker_id = None
_model_lock = threading.Lock() _model_lock = threading.Lock()
_load_error: str | None = None _load_error: str | None = None
def _ensure_model() -> None: def _ensure_model() -> None:
global _model, _load_error global _model, _speaker_id, _load_error
if _model is not None or _load_error is not None: if _model is not None or _load_error is not None:
return return
with _model_lock: with _model_lock:
if _model is not None or _load_error is not None: if _model is not None or _load_error is not None:
return return
try: try:
from TTS.api import TTS # type: ignore from melo.api import TTS # type: ignore
model = TTS(MODEL).to(DEVICE) model = TTS(language=LANGUAGE, device=DEVICE)
# spk2id is a melo HParams object (dict-like, supports __getitem__,
# __contains__, keys) but NOT .get(). The KR model exposes a single
# 'KR' speaker; fall back to the first id for other languages.
spk_map = model.hps.data.spk2id
keys = list(spk_map.keys())
speaker_id = spk_map[LANGUAGE] if LANGUAGE in spk_map else spk_map[keys[0]]
_model = model _model = model
# Warm once: the first GPU synth pays a one-off kernel-init cost _speaker_id = speaker_id
# that would otherwise land on the user's first reply. # Warm the GPU once at load: the first CUDA synth pays a one-off
# kernel-init cost (~5s) that would otherwise land on the user's
# first reply. A throwaway synth here moves it to startup. No-op
# cost on CPU.
try: try:
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as _wt: with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as _wt:
_wp = _wt.name _wp = _wt.name
model.tts_to_file( model.tts_to_file("워밍업", speaker_id, _wp, speed=SPEED)
text="워밍업", speaker=SPEAKER, language=LANGUAGE, file_path=_wp
)
try: try:
os.unlink(_wp) os.unlink(_wp)
except OSError: except OSError:
pass pass
except Exception as _we: # pragma: no cover except Exception as _we: # pragma: no cover
print(f"[xtts-worker] warmup synth skipped: {_we}", flush=True) print(f"[melo-worker] warmup synth skipped: {_we}", flush=True)
print( print(
f"[xtts-worker] ready (model={MODEL} speaker={SPEAKER!r} " f"[melo-worker] ready (lang={LANGUAGE} speed={SPEED} "
f"language={LANGUAGE} device={DEVICE})", f"device={DEVICE} speakers={list(spk_map.keys())})",
flush=True, flush=True,
) )
except Exception as e: # pragma: no cover - depends on local model files except Exception as e: # pragma: no cover - depends on local model files
_load_error = f"{type(e).__name__}: {e}" _load_error = f"{type(e).__name__}: {e}"
print(f"[xtts-worker] model load FAILED: {_load_error}", flush=True) print(f"[melo-worker] model load FAILED: {_load_error}", flush=True)
def _synthesize(text: str) -> bytes: def _synthesize(text: str) -> bytes:
"""Synthesise ``text`` to a 16-bit PCM WAV (bytes).""" """Synthesise ``text`` to a 16-bit PCM WAV (bytes)."""
_ensure_model() _ensure_model()
if _model is None: if _model is None:
raise RuntimeError(_load_error or "xtts model unavailable") raise RuntimeError(_load_error or "melo model unavailable")
# MeloTTS writes to a file via soundfile; render to a container-disk temp
# file (NOT tmpfs), read it back, then drop it.
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp: with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
tmp_path = tmp.name tmp_path = tmp.name
try: try:
with _model_lock: with _model_lock:
_model.tts_to_file( _model.tts_to_file(text, _speaker_id, tmp_path, speed=SPEED)
text=text, speaker=SPEAKER, language=LANGUAGE, file_path=tmp_path
)
with open(tmp_path, "rb") as f: with open(tmp_path, "rb") as f:
raw = f.read() raw = f.read()
finally: finally:
@@ -115,15 +132,16 @@ def _synthesize(text: str) -> bytes:
def _ensure_pcm16_wav(raw: bytes) -> bytes: def _ensure_pcm16_wav(raw: bytes) -> bytes:
"""Guarantee a 16-bit PCM WAV. Coqui writes float/other WAVs; the Discord """Guarantee a 16-bit PCM WAV. MeloTTS/soundfile usually emit float WAVs;
playback path tolerates both, but we normalise to PCM16 so the contract the Discord playback path (ffmpeg) tolerates both, but we normalise to
matches the previous Melo/Piper output (mono, file's own sample rate).""" PCM16 so the contract matches the previous Piper output."""
try: try:
with wave.open(io.BytesIO(raw), "rb") as wf: with wave.open(io.BytesIO(raw), "rb") as wf:
if wf.getsampwidth() == 2: if wf.getsampwidth() == 2:
return raw # already PCM16 return raw # already PCM16
except wave.Error: except wave.Error:
pass pass
# Non-PCM16 (e.g. float) — convert with soundfile if available.
try: try:
import numpy as np import numpy as np
import soundfile as sf import soundfile as sf
@@ -141,7 +159,7 @@ def _ensure_pcm16_wav(raw: bytes) -> bytes:
wf.writeframes(pcm) wf.writeframes(pcm)
return buf.getvalue() return buf.getvalue()
except Exception: except Exception:
return raw # last resort: hand back whatever XTTS produced return raw # last resort: hand back whatever MeloTTS produced
class _Handler(BaseHTTPRequestHandler): class _Handler(BaseHTTPRequestHandler):
@@ -194,7 +212,7 @@ def main() -> int:
# Warm the model at startup so the first Discord turn isn't slow. # Warm the model at startup so the first Discord turn isn't slow.
_ensure_model() _ensure_model()
server = ThreadingHTTPServer((HOST, PORT), _Handler) server = ThreadingHTTPServer((HOST, PORT), _Handler)
print(f"[xtts-worker] listening on http://{HOST}:{PORT}", flush=True) print(f"[melo-worker] listening on http://{HOST}:{PORT}", flush=True)
try: try:
server.serve_forever() server.serve_forever()
except KeyboardInterrupt: except KeyboardInterrupt:

View File

@@ -21,7 +21,11 @@ nvidia-cudnn-cu12
# --- Bridge HTTP service --- # --- Bridge HTTP service ---
flask>=3.0.0 flask>=3.0.0
# --- Text-to-speech (Piper) --- # --- Text-to-speech ---
# Edge TTS: the primary Korean voice (online MS neural). Lightweight (httpx);
# emits MP3, transcoded to PCM16 by the system ffmpeg in the bridge.
edge-tts>=6.1.0
# Piper: offline English fallback.
piper-tts>=1.3.0 piper-tts>=1.3.0
# --- Built-in tools (lazily imported; needed for full functionality) --- # --- Built-in tools (lazily imported; needed for full functionality) ---

View File

@@ -87,13 +87,11 @@ VAD_MIN_SPEECH_MS = int(os.environ.get("VAD_MIN_SPEECH_MS", "200"))
# Korean phrase decoded as Chinese) and shaves a little latency. Empty = auto. # Korean phrase decoded as Chinese) and shaves a little latency. Empty = auto.
STT_LANGUAGE = os.environ.get("STT_LANGUAGE", "ko").strip() or None STT_LANGUAGE = os.environ.get("STT_LANGUAGE", "ko").strip() or None
# TTS engine: "xtts" (Coqui XTTS-v2 natural Korean voice, the warm worker) is # TTS engine: "edge" (Microsoft Edge TTS, natural Korean neural voice) is the
# the primary voice; Piper is kept as a fallback only if explicitly enabled. Set # primary voice. "melo" (a warm MeloTTS worker) and "piper" remain selectable.
# TTS_ENGINE=piper to disable the neural Korean voice entirely. "melo" is still
# accepted for backward compatibility but is no longer built into the image.
def _tts_engine_setting() -> str: def _tts_engine_setting() -> str:
"""TTS engine: settings-UI value (runtime config JSON) wins, else env, else """TTS engine: settings-UI value (runtime config JSON) wins, else env, else
xtts. Read at startup; the settings UI restarts the bridge on apply.""" edge. Read at startup; the settings UI restarts the bridge on apply."""
try: try:
_cp = os.environ.get("JARVIS_CONFIG_PATH", "/app/config/jarvis.json") _cp = os.environ.get("JARVIS_CONFIG_PATH", "/app/config/jarvis.json")
_v = json.loads(open(_cp, encoding="utf-8").read()).get("tts_engine") _v = json.loads(open(_cp, encoding="utf-8").read()).get("tts_engine")
@@ -101,29 +99,23 @@ def _tts_engine_setting() -> str:
return str(_v).strip().lower() return str(_v).strip().lower()
except Exception: except Exception:
pass pass
return os.environ.get("TTS_ENGINE", "xtts").strip().lower() return os.environ.get("TTS_ENGINE", "edge").strip().lower()
TTS_ENGINE = _tts_engine_setting() TTS_ENGINE = _tts_engine_setting()
# Coqui XTTS-v2 worker (the natural Korean voice). # Edge TTS (online MS neural voice). Voice + rate are env-driven so they can be
XTTS_WORKER_URL = os.environ.get("XTTS_WORKER_URL", "http://127.0.0.1:8771") # changed without code. Default: Korean "Hyunsu" multilingual voice at +45%
XTTS_TIMEOUT = float(os.environ.get("XTTS_TIMEOUT", "30")) # (≈1.45×), the chosen settings. NOTE: edge synthesis sends the reply TEXT to
# Legacy MeloTTS worker (no longer built into the image; kept for back-compat # Microsoft's servers and needs internet — an intentional privacy trade-off for
# if someone runs an old worker out-of-band). # the more natural voice.
EDGE_TTS_VOICE = os.environ.get("EDGE_TTS_VOICE", "ko-KR-HyunsuMultilingualNeural").strip()
EDGE_TTS_RATE = os.environ.get("EDGE_TTS_RATE", "+45%").strip()
MELO_WORKER_URL = os.environ.get("MELO_WORKER_URL", "http://127.0.0.1:8770") MELO_WORKER_URL = os.environ.get("MELO_WORKER_URL", "http://127.0.0.1:8770")
MELO_TIMEOUT = float(os.environ.get("MELO_TIMEOUT", "30")) MELO_TIMEOUT = float(os.environ.get("MELO_TIMEOUT", "30"))
# Do NOT silently fall back to the English Piper voice on a neural-voice failure: # Do NOT silently fall back to the English Piper voice on a neural-voice failure:
# speaking Korean text through an English voice produces mangled audio. Default # speaking Korean through an English voice produces mangled audio. Default is
# is neural-only (return no audio on failure); set XTTS_FALLBACK_PIPER=1 (or the # neural-only (return no audio on failure); set MELO_FALLBACK_PIPER=1 to opt in.
# legacy MELO_FALLBACK_PIPER=1) to opt into the Piper fallback. MELO_FALLBACK_PIPER = os.environ.get("MELO_FALLBACK_PIPER", "0") in ("1", "true", "True", "yes", "on")
def _truthy_env(*names: str) -> bool:
for _n in names:
if os.environ.get(_n, "").strip().lower() in ("1", "true", "yes", "on"):
return True
return False
NEURAL_FALLBACK_PIPER = _truthy_env("XTTS_FALLBACK_PIPER", "MELO_FALLBACK_PIPER")
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Lazy singletons. The first request pays the model-load cost; afterwards the # Lazy singletons. The first request pays the model-load cost; afterwards the
@@ -315,38 +307,75 @@ def _coerce_bool(value) -> Optional[bool]:
return str(value).strip().lower() in ("1", "true", "yes", "on") return str(value).strip().lower() in ("1", "true", "yes", "on")
def _worker_synthesize(name: str, url: str, timeout: float, text: str) -> Optional[bytes]: def _edge_synthesize(text: str) -> Optional[bytes]:
"""POST text to a warm TTS worker's /synth and return its WAV bytes, or None """Synthesise via Microsoft Edge TTS (online neural voice) and return a
on any failure so the caller can decide whether to fall back.""" 16-bit PCM WAV, or None on any failure. Edge emits MP3; we transcode to
PCM16 mono with the system ffmpeg, writing to a temp file (seekable) so the
WAV header carries a correct length. Needs internet."""
import asyncio
import subprocess
import tempfile
try:
import edge_tts # type: ignore
async def _gen() -> bytes:
comm = edge_tts.Communicate(text, EDGE_TTS_VOICE, rate=EDGE_TTS_RATE)
buf = bytearray()
async for chunk in comm.stream():
if chunk.get("type") == "audio":
buf.extend(chunk["data"])
return bytes(buf)
mp3 = asyncio.run(_gen())
if not mp3:
print("[bridge] edge TTS returned no audio", flush=True)
return None
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as t:
out_path = t.name
try:
proc = subprocess.run(
["ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
"-i", "pipe:0", "-ac", "1", "-ar", "24000",
"-acodec", "pcm_s16le", out_path],
input=mp3, capture_output=True,
)
if proc.returncode != 0:
print(f"[bridge] edge ffmpeg transcode failed: {proc.stderr.decode('utf-8','ignore')[:200]}", flush=True)
return None
with open(out_path, "rb") as f:
return f.read()
finally:
try:
os.unlink(out_path)
except OSError:
pass
except Exception as e: # pragma: no cover - network / dep dependent
print(f"[bridge] edge synth failed: {e}", flush=True)
return None
def _melo_synthesize(text: str) -> Optional[bytes]:
"""Synthesise via the warm MeloTTS worker (separate /opt/melo venv, Korean
speaker @ speed 1.5). Returns a 16-bit PCM WAV, or None on any failure so
the caller can fall back to Piper."""
import urllib.request import urllib.request
try: try:
req = urllib.request.Request( req = urllib.request.Request(
f"{url}/synth", f"{MELO_WORKER_URL}/synth",
data=json.dumps({"text": text}).encode("utf-8"), data=json.dumps({"text": text}).encode("utf-8"),
headers={"Content-Type": "application/json"}, headers={"Content-Type": "application/json"},
) )
with urllib.request.urlopen(req, timeout=timeout) as resp: with urllib.request.urlopen(req, timeout=MELO_TIMEOUT) as resp:
if resp.status == 200: if resp.status == 200:
return resp.read() return resp.read()
print(f"[bridge] {name} worker HTTP {resp.status}", flush=True) print(f"[bridge] melo worker HTTP {resp.status}", flush=True)
except Exception as e: # pragma: no cover - worker may be down except Exception as e: # pragma: no cover - worker may be down
print(f"[bridge] {name} worker unreachable: {e}", flush=True) print(f"[bridge] melo worker unreachable: {e}", flush=True)
return None return None
def _xtts_synthesize(text: str) -> Optional[bytes]:
"""Synthesise via the warm Coqui XTTS-v2 worker (separate /opt/xtts venv,
natural female Korean). Returns a 16-bit PCM WAV, or None on failure."""
return _worker_synthesize("xtts", XTTS_WORKER_URL, XTTS_TIMEOUT, text)
def _melo_synthesize(text: str) -> Optional[bytes]:
"""Legacy: synthesise via a MeloTTS worker if one is running out-of-band.
Returns a 16-bit PCM WAV, or None on any failure."""
return _worker_synthesize("melo", MELO_WORKER_URL, MELO_TIMEOUT, text)
def _piper_synthesize(text: str) -> Optional[bytes]: def _piper_synthesize(text: str) -> Optional[bytes]:
"""Fallback: synthesise with Piper (English voice). Returns WAV bytes.""" """Fallback: synthesise with Piper (English voice). Returns WAV bytes."""
_ensure_piper() _ensure_piper()
@@ -373,12 +402,11 @@ def _tts_ready() -> bool:
""" """
if not TTS_ENABLED: if not TTS_ENABLED:
return True return True
_worker_health = {"xtts": XTTS_WORKER_URL, "melo": MELO_WORKER_URL}.get(TTS_ENGINE) if TTS_ENGINE == "melo":
if _worker_health:
import urllib.request import urllib.request
try: try:
with urllib.request.urlopen(f"{_worker_health}/health", timeout=2) as resp: with urllib.request.urlopen(f"{MELO_WORKER_URL}/health", timeout=2) as resp:
return resp.status == 200 return resp.status == 200
except Exception: except Exception:
return False return False
@@ -386,22 +414,20 @@ def _tts_ready() -> bool:
def synthesize(text: str) -> Optional[bytes]: def synthesize(text: str) -> Optional[bytes]:
"""Synthesize text to a 16-bit PCM WAV. The primary voice is Coqui XTTS-v2 """Synthesize text to a 16-bit PCM WAV. The primary voice is Edge TTS (a
(natural female Korean) served by the warm xtts worker; Piper is used only natural Korean neural voice); "melo" uses the warm MeloTTS worker. For a
when explicitly enabled as a fallback. Returns None if TTS is off.""" neural engine, Piper (English) is only used if explicitly enabled, since
speaking Korean through an English voice mangles it. Returns None if off."""
if not TTS_ENABLED or not text.strip(): if not TTS_ENABLED or not text.strip():
return None return None
_neural = {"xtts": _xtts_synthesize, "melo": _melo_synthesize}.get(TTS_ENGINE) _neural = {"edge": _edge_synthesize, "melo": _melo_synthesize}.get(TTS_ENGINE)
if _neural is not None: if _neural is not None:
audio = _neural(text) audio = _neural(text)
if audio: if audio:
return audio return audio
if not NEURAL_FALLBACK_PIPER: if not MELO_FALLBACK_PIPER:
# Neural-only: better silent than mangled English for Korean text. # Neural-only: better silent than mangled English for Korean text.
print( print(f"[bridge] {TTS_ENGINE} synth failed; no audio (Piper fallback disabled)", flush=True)
f"[bridge] {TTS_ENGINE} synth failed; no audio (Piper fallback disabled)",
flush=True,
)
return None return None
print(f"[bridge] {TTS_ENGINE} synth failed; falling back to Piper", flush=True) print(f"[bridge] {TTS_ENGINE} synth failed; falling back to Piper", flush=True)
return _piper_synthesize(text) return _piper_synthesize(text)

View File

@@ -22,7 +22,7 @@ from typing import Any, Dict
FIELDS = [ FIELDS = [
("ollama_chat_model", "LLM 모델", "model"), ("ollama_chat_model", "LLM 모델", "model"),
("whisper_model", "STT(Whisper) 모델", "select:tiny,base,small,medium,large,large-v3"), ("whisper_model", "STT(Whisper) 모델", "select:tiny,base,small,medium,large,large-v3"),
("tts_engine", "TTS 엔진", "select:xtts,piper"), ("tts_engine", "TTS 엔진", "select:edge,piper"),
("output_language", "출력 언어 (비우면 사용자 언어)", "text"), ("output_language", "출력 언어 (비우면 사용자 언어)", "text"),
("llm_thinking_enabled", "LLM 사고(thinking) 모드", "bool"), ("llm_thinking_enabled", "LLM 사고(thinking) 모드", "bool"),
("agentic_max_turns", "에이전트 최대 반복", "number:1:12:1"), ("agentic_max_turns", "에이전트 최대 반복", "number:1:12:1"),
@@ -106,15 +106,15 @@ def _save(updates: Dict[str, Any]) -> None:
def _apply() -> str: def _apply() -> str:
# Restart the TTS worker + bridge AFTER this response is sent. Detached (new # Restart the bridge AFTER this response is sent. Detached (new session) so
# session) so the bridge being killed mid-restart doesn't drop the restart # the bridge being killed mid-restart doesn't drop the restart itself, and
# itself, and the HTTP client still receives this response. # the HTTP client still receives this response. (Edge TTS has no worker.)
try: try:
subprocess.Popen( subprocess.Popen(
["sh", "-c", "sleep 1; supervisorctl restart xtts-worker bridge"], ["sh", "-c", "sleep 1; supervisorctl restart bridge"],
start_new_session=True, start_new_session=True,
) )
return "1초 후 브리지/TTS 워커가 재시작되어 반영됩니다." return "1초 후 브리지가 재시작되어 반영됩니다."
except Exception as e: # pragma: no cover except Exception as e: # pragma: no cover
return str(e) return str(e)

View File

@@ -66,15 +66,15 @@ services:
WHISPER_MODEL: ${WHISPER_MODEL:-medium} WHISPER_MODEL: ${WHISPER_MODEL:-medium}
WHISPER_DEVICE: ${WHISPER_DEVICE:-cuda} WHISPER_DEVICE: ${WHISPER_DEVICE:-cuda}
WHISPER_COMPUTE_TYPE: ${WHISPER_COMPUTE_TYPE:-float16} WHISPER_COMPUTE_TYPE: ${WHISPER_COMPUTE_TYPE:-float16}
# Coqui XTTS-v2 (natural female Korean voice, replaces MeloTTS) on the GPU # TTS engine. Rendered into /app/config/jarvis.json via envsubst (the
# (cu128 torch baked by docker/setup-xtts.sh). Set here WITH DEFAULTS so # bridge reads that JSON BEFORE the env, so it must carry the real engine,
# supervisord's %(ENV_XTTS_*)s passthrough always resolves and an .env # not a hardcoded one — otherwise Korean text is read by the English Piper
# override actually reaches the xtts-worker. # voice). Default edge; .env can override (e.g. piper for offline).
XTTS_DEVICE: ${XTTS_DEVICE:-cuda} TTS_ENGINE: ${TTS_ENGINE:-edge}
# Built-in studio speaker (female). Other options include "Daisy Studious", # Edge TTS voice + rate (the chosen natural Korean voice). NOTE: edge is an
# "Sofia Hellen", "Alma María", etc. — any XTTS-v2 studio speaker name. # ONLINE engine — reply text is sent to Microsoft and needs internet.
XTTS_SPEAKER: ${XTTS_SPEAKER:-Ana Florence} EDGE_TTS_VOICE: ${EDGE_TTS_VOICE:-ko-KR-HyunsuMultilingualNeural}
XTTS_LANGUAGE: ${XTTS_LANGUAGE:-ko} EDGE_TTS_RATE: ${EDGE_TTS_RATE:-+45%}
# Optional single-language lock for replies (empty = user's own language). # Optional single-language lock for replies (empty = user's own language).
OUTPUT_LANGUAGE: ${OUTPUT_LANGUAGE:-ko} OUTPUT_LANGUAGE: ${OUTPUT_LANGUAGE:-ko}
# Drop the pre-loop planner LLM call to cut voice-reply latency on small # Drop the pre-loop planner LLM call to cut voice-reply latency on small

View File

@@ -16,6 +16,9 @@ set -euo pipefail
# by default so everything runs on one resident model; override if you pull a # by default so everything runs on one resident model; override if you pull a
# dedicated small model. # dedicated small model.
: "${OLLAMA_INTENT_MODEL:=${OLLAMA_CHAT_MODEL}}" : "${OLLAMA_INTENT_MODEL:=${OLLAMA_CHAT_MODEL}}"
# Cap chat-model output tokens per turn (worst-case latency guard). Spoken
# answers are 1-2 sentences; 512 is safe headroom above tool-call JSON. 0 = off.
: "${OLLAMA_NUM_PREDICT:=512}"
: "${OLLAMA_EMBED_MODEL:=nomic-embed-text}" : "${OLLAMA_EMBED_MODEL:=nomic-embed-text}"
: "${WHISPER_MODEL:=small}" : "${WHISPER_MODEL:=small}"
: "${WHISPER_DEVICE:=cuda}" : "${WHISPER_DEVICE:=cuda}"
@@ -32,7 +35,7 @@ set -euo pipefail
: "${XDG_RUNTIME_DIR:=/run/user/0}" : "${XDG_RUNTIME_DIR:=/run/user/0}"
: "${PULSE_SERVER:=unix:${XDG_RUNTIME_DIR}/pulse/native}" : "${PULSE_SERVER:=unix:${XDG_RUNTIME_DIR}/pulse/native}"
export VNC_RESOLUTION OLLAMA_BASE_URL OLLAMA_CHAT_MODEL OLLAMA_INTENT_MODEL OLLAMA_EMBED_MODEL \ export VNC_RESOLUTION OLLAMA_BASE_URL OLLAMA_CHAT_MODEL OLLAMA_NUM_PREDICT OLLAMA_INTENT_MODEL OLLAMA_EMBED_MODEL \
WHISPER_MODEL WHISPER_DEVICE WHISPER_COMPUTE_TYPE JARVIS_DB_PATH \ WHISPER_MODEL WHISPER_DEVICE WHISPER_COMPUTE_TYPE JARVIS_DB_PATH \
PIPER_VOICE PIPER_VOICE_DIR TTS_PIPER_MODEL_PATH BRIDGE_HOST BRIDGE_PORT \ PIPER_VOICE PIPER_VOICE_DIR TTS_PIPER_MODEL_PATH BRIDGE_HOST BRIDGE_PORT \
XDG_RUNTIME_DIR PULSE_SERVER XDG_RUNTIME_DIR PULSE_SERVER
@@ -51,12 +54,18 @@ export JARVIS_CONFIG_PATH=/app/config/jarvis.json
# the env-rendered config, so changes survive container recreate. # the env-rendered config, so changes survive container recreate.
if [ -f /data/jarvis-settings.json ]; then if [ -f /data/jarvis-settings.json ]; then
python3 - <<'PY' || true python3 - <<'PY' || true
import json import json, os
try: try:
base = json.load(open("/app/config/jarvis.json")) base = json.load(open("/app/config/jarvis.json"))
ov = json.load(open("/data/jarvis-settings.json")) ov = json.load(open("/data/jarvis-settings.json"))
if isinstance(base, dict) and isinstance(ov, dict): if isinstance(base, dict) and isinstance(ov, dict):
base.update(ov) base.update(ov)
# A stale persisted tts_engine from an earlier voice (melo/xtts, no
# longer built into the image) would override the configured engine and
# leave the bot silent. Reset those to the env-configured engine.
if base.get("tts_engine") in ("melo", "xtts"):
base["tts_engine"] = os.environ.get("TTS_ENGINE", "edge")
print(f"[entrypoint] reset stale tts_engine -> {base['tts_engine']}")
json.dump(base, open("/app/config/jarvis.json", "w"), ensure_ascii=False, indent=2) json.dump(base, open("/app/config/jarvis.json", "w"), ensure_ascii=False, indent=2)
print("[entrypoint] merged persistent settings overrides") print("[entrypoint] merged persistent settings overrides")
except Exception as e: except Exception as e:

View File

@@ -4,9 +4,10 @@
"ollama_base_url": "${OLLAMA_BASE_URL}", "ollama_base_url": "${OLLAMA_BASE_URL}",
"ollama_embed_model": "${OLLAMA_EMBED_MODEL}", "ollama_embed_model": "${OLLAMA_EMBED_MODEL}",
"ollama_chat_model": "${OLLAMA_CHAT_MODEL}", "ollama_chat_model": "${OLLAMA_CHAT_MODEL}",
"ollama_num_predict": "${OLLAMA_NUM_PREDICT}",
"intent_judge_model": "${OLLAMA_INTENT_MODEL}", "intent_judge_model": "${OLLAMA_INTENT_MODEL}",
"tts_enabled": true, "tts_enabled": true,
"tts_engine": "piper", "tts_engine": "${TTS_ENGINE}",
"tts_piper_model_path": "${TTS_PIPER_MODEL_PATH}", "tts_piper_model_path": "${TTS_PIPER_MODEL_PATH}",
"whisper_model": "${WHISPER_MODEL}", "whisper_model": "${WHISPER_MODEL}",
"whisper_backend": "faster-whisper", "whisper_backend": "faster-whisper",

View File

@@ -18,6 +18,27 @@ cat > /etc/opt/chrome/policies/managed/jarvis.json <<'JSON'
{ "CommandLineFlagSecurityWarningsEnabled": false } { "CommandLineFlagSecurityWarningsEnabled": false }
JSON JSON
# Seed the profile's web-content language to Korean so sites (YouTube, Google,
# Naver) render in Korean. --lang sets Chrome's own UI, but the Accept-Language
# sent to sites comes from the profile's intl.accept_languages, which a persisted
# user-data-dir would otherwise keep at en-US regardless of --accept-lang.
PREFS_DIR="${CHROME_PROFILE_DIR:-/root/chrome-profile}/Default"
PREFS="${PREFS_DIR}/Preferences"
mkdir -p "$PREFS_DIR"
if [ -f "$PREFS" ]; then
python3 - "$PREFS" <<'PY' 2>/dev/null || true
import json, sys
p = sys.argv[1]
d = json.load(open(p))
d.setdefault("intl", {})
d["intl"]["accept_languages"] = "ko-KR,ko"
d["intl"]["selected_languages"] = "ko-KR,ko"
json.dump(d, open(p, "w"), ensure_ascii=False)
PY
else
printf '%s' '{"intl":{"accept_languages":"ko-KR,ko","selected_languages":"ko-KR,ko"}}' > "$PREFS"
fi
# Minimal, non-automation flags. --remote-debugging exposes CDP so the brain can # Minimal, non-automation flags. --remote-debugging exposes CDP so the brain can
# drive this on-screen Chrome (Google/YouTube/Naver), --disable-features=Translate # drive this on-screen Chrome (Google/YouTube/Naver), --disable-features=Translate
# hides the translate popup. NO --test-type / --disable-blink-features. # hides the translate popup. NO --test-type / --disable-blink-features.
@@ -26,6 +47,7 @@ exec google-chrome \
--no-default-browser-check \ --no-default-browser-check \
--disable-features=Translate,TranslateUI \ --disable-features=Translate,TranslateUI \
--lang=ko-KR \ --lang=ko-KR \
--accept-lang=ko-KR,ko \
--remote-debugging-port="${CDP_PORT:-9222}" \ --remote-debugging-port="${CDP_PORT:-9222}" \
--remote-debugging-address="${CDP_BIND:-127.0.0.1}" \ --remote-debugging-address="${CDP_BIND:-127.0.0.1}" \
--user-data-dir="${CHROME_PROFILE_DIR:-/root/chrome-profile}" \ --user-data-dir="${CHROME_PROFILE_DIR:-/root/chrome-profile}" \

80
docker/setup-melo.sh Executable file
View File

@@ -0,0 +1,80 @@
#!/usr/bin/env bash
# ============================================================================
# Install a dedicated MeloTTS (Korean voice) venv at /opt/melo.
#
# Why a SEPARATE venv (not the brain-bridge /opt/venv):
# - MeloTTS pins old deps (transformers 4.27.4 / tokenizers 0.13.3 / fugashi)
# whose binary wheels exist only for cp311, so we use python3.11 here even
# though the image's default interpreter is 3.12.
# - It isolates the heavy torch/transformers stack from the slim bridge env,
# which pins numpy<2 for faster-whisper.
#
# torch is the CUDA (cu128) build so MeloTTS runs on the GPU alongside Ollama +
# Whisper. CPU synth serialised under concurrent load (whisper STT + bot) and
# blew TTS up to 7-8s per reply; on the GPU a sentence synthesises in ~0.3s.
# cu128 is the Blackwell (sm_120) wheel verified on this host's RTX 5050.
# The worker selects the device via MELO_DEVICE=cuda (compose).
# ============================================================================
set -euxo pipefail
export DEBIAN_FRONTEND=noninteractive
apt-get update
# Build deps for fugashi / mecab-python3 + a system MeCab dict, plus python3.11.
apt-get install -y --no-install-recommends \
software-properties-common build-essential pkg-config swig \
libmecab-dev mecab mecab-ipadic-utf8
add-apt-repository -y ppa:deadsnakes/ppa
apt-get update
apt-get install -y --no-install-recommends python3.11 python3.11-venv python3.11-dev
rm -rf /var/lib/apt/lists/*
python3.11 -m venv /opt/melo
/opt/melo/bin/pip install --no-cache-dir --upgrade pip wheel setuptools
# CUDA (cu128) torch first, so MeloTTS's unpinned `torch` dep is already
# satisfied with the GPU build. Pinned to the Blackwell-verified versions
# (2.11.0+cu128) for reproducible rebuilds.
/opt/melo/bin/pip install --no-cache-dir torch==2.11.0+cu128 torchaudio==2.11.0+cu128 \
--index-url https://download.pytorch.org/whl/cu128
# MeloTTS from GitHub. The PyPI sdist is broken (its setup.py reads a
# requirements.txt that is not shipped in the sdist), so install from the repo.
# Pinned to a commit (not refs/heads/main) so rebuilds are reproducible.
/opt/melo/bin/pip install --no-cache-dir \
"https://github.com/myshell-ai/MeloTTS/archive/209145371cff8fc3bd60d7be902ea69cbdb7965a.tar.gz"
# Korean g2p backend. MeloTTS otherwise tries to pip-install this on the first
# Korean request, which fails in a network-isolated container at runtime.
/opt/melo/bin/pip install --no-cache-dir python-mecab-ko python-mecab-ko-dic
# Remove the full `unidic` package (its dictionary is never downloaded, only a
# stub) so mecab-python3 falls back to the bundled `unidic_lite` dict. Without
# this, importing melo's Japanese module fails with a missing-mecabrc error.
/opt/melo/bin/pip uninstall -y unidic || true
# Pre-cache every model asset MeloTTS pulls at runtime, so the worker starts
# offline and the first Discord turn pays no download cost. Importing melo.api
# fetches the Japanese (tohoku-nlp/bert-base-japanese-v3) and Korean
# (kykim/bert-kor-base) BERT tokenizers plus nltk g2p data; loading the KR voice
# downloads the OpenVoice KR config+checkpoint, and a real synth pulls the
# Korean BERT weights. All of these go through huggingface_hub.
#
# CRITICAL: at runtime docker-compose mounts the `whisper_cache` named volume
# over /root/.cache/huggingface (for faster-whisper). That volume would SHADOW
# anything baked into the default HF cache, so we pin the melo worker to a
# DEDICATED, non-volume cache dir (/opt/melo-cache) here AND in supervisord, and
# warm it once. nltk_data (/root/nltk_data) is not volume-mounted so it stays.
export HF_HOME=/opt/melo-cache
mkdir -p "$HF_HOME"
MELO_LANGUAGE=KR HF_HOME=/opt/melo-cache /opt/melo/bin/python - <<'PY'
import tempfile
from melo.api import TTS
model = TTS(language="KR", device="cpu")
out = tempfile.mktemp(suffix=".wav")
model.tts_to_file("초기화 워밍업입니다.", model.hps.data.spk2id["KR"], out, speed=1.5)
print("[setup-melo] warm-up KR synth OK ->", out)
PY
echo "[setup-melo] MeloTTS venv ready at /opt/melo"

View File

@@ -1,72 +0,0 @@
#!/usr/bin/env bash
# ============================================================================
# Install a dedicated Coqui XTTS-v2 (natural Korean voice) venv at /opt/xtts.
#
# Why a SEPARATE venv (not the brain-bridge /opt/venv or /opt/melo):
# - Coqui TTS pulls its own heavy torch/transformers stack; isolating it keeps
# the slim bridge env (numpy<2 for faster-whisper) untouched.
# - We use python3.11 (installed for the melo layer) because Coqui ships cp311
# wheels and torch cu128 is available for it.
#
# torch is the CUDA (cu128) build so XTTS runs on the GPU alongside Ollama +
# Whisper. cu128 is the Blackwell (sm_120) wheel verified on this host.
# The worker selects the device via XTTS_DEVICE=cuda (compose).
#
# XTTS-v2 is non-commercial (Coqui Public Model License). COQUI_TOS_AGREED=1
# accepts it non-interactively so the model can load in a headless container.
# ============================================================================
set -euxo pipefail
export DEBIAN_FRONTEND=noninteractive
export COQUI_TOS_AGREED=1
# Install python3.11 if not already present, so this layer is self-contained.
if ! command -v python3.11 >/dev/null 2>&1; then
apt-get update
apt-get install -y --no-install-recommends software-properties-common
add-apt-repository -y ppa:deadsnakes/ppa
apt-get update
apt-get install -y --no-install-recommends python3.11 python3.11-venv python3.11-dev
rm -rf /var/lib/apt/lists/*
fi
python3.11 -m venv /opt/xtts
/opt/xtts/bin/pip install --no-cache-dir --upgrade pip wheel setuptools
# CUDA (cu128) torch first so Coqui's `torch` dep is satisfied with the GPU
# build. Pinned to the Blackwell-verified versions for reproducible rebuilds.
/opt/xtts/bin/pip install --no-cache-dir torch==2.11.0+cu128 torchaudio==2.11.0+cu128 \
--index-url https://download.pytorch.org/whl/cu128
# Coqui TTS (maintained fork; provides the `TTS` package and XTTS-v2). The
# [codec] extra pulls torchcodec, which torch >=2.9 requires for audio IO
# (without it the import fails with TORCHCODEC_IMPORT_ERROR). torchcodec also
# needs the system FFmpeg shared libs, which are present (ffmpeg apt package).
/opt/xtts/bin/pip install --no-cache-dir "coqui-tts[codec]"
# Pin transformers to the 4.57+ / <5 range. coqui-tts requires >=4.57 but does
# NOT cap the upper bound, and transformers 5.x removed `isin_mps_friendly`
# (used by XTTS's tortoise layer), so an unpinned install pulls 5.x and the
# model import fails with "cannot import name 'isin_mps_friendly'". Pin <5.
/opt/xtts/bin/pip install --no-cache-dir "transformers>=4.57,<5"
# Pre-bake the XTTS-v2 model so the worker starts offline and the first Discord
# turn pays no download cost. The model is cached under TTS_HOME; we pin a
# DEDICATED, non-volume dir (/opt/xtts-cache) AND set it in supervisord, because
# runtime volume mounts (whisper_cache over /root/.cache) must not shadow it.
export TTS_HOME=/opt/xtts-cache
mkdir -p "$TTS_HOME"
COQUI_TOS_AGREED=1 TTS_HOME=/opt/xtts-cache XTTS_SPEAKER="Ana Florence" \
/opt/xtts/bin/python - <<'PY'
import os
os.environ["COQUI_TOS_AGREED"] = "1"
from TTS.api import TTS
speaker = os.environ.get("XTTS_SPEAKER", "Ana Florence")
model = TTS("tts_models/multilingual/multi-dataset/xtts_v2") # downloads to TTS_HOME
out = "/tmp/xtts_warm.wav"
model.tts_to_file(text="초기화 워밍업입니다.", speaker=speaker, language="ko", file_path=out)
print("[setup-xtts] warm-up KR synth OK ->", out, "speaker:", speaker)
PY
echo "[setup-xtts] Coqui XTTS-v2 venv ready at /opt/xtts (cache /opt/xtts-cache)"

View File

@@ -49,28 +49,8 @@ stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0 stderr_logfile_maxbytes=0
[program:xtts-worker] # (No TTS worker program: the default Edge TTS engine synthesises in-process in
; Warm Coqui XTTS-v2 Korean voice (natural female "Ana Florence") in its own # the bridge via the `edge-tts` package — no warm model/worker is needed.)
; py3.11 venv. The bridge's synthesize() POSTs here; if this is down the bridge
; falls back to Piper (English) only when XTTS_FALLBACK_PIPER=1.
command=/app/docker/run-if-role.sh full,bot /opt/xtts/bin/python /app/bridge/xtts_worker.py
directory=/app
; TTS_HOME points at the dedicated, image-baked XTTS cache (warmed in
; setup-xtts.sh). The brain's whisper_cache volume is mounted over
; /root/.cache, so a dedicated non-volume cache dir avoids the baked model being
; shadowed and re-downloaded (which would fail if the host is offline).
; XTTS_DEVICE / XTTS_SPEAKER / XTTS_LANGUAGE inherit from the container env
; (compose sets them with defaults: cuda / "Ana Florence" / ko). supervisord
; interpolates %(ENV_x)s from its own environment, which is the container's — so
; these must always be set in the env (compose guarantees it) or this expansion
; fails at startup. COQUI_TOS_AGREED accepts the non-commercial XTTS license.
environment=XTTS_DEVICE="%(ENV_XTTS_DEVICE)s",XTTS_SPEAKER="%(ENV_XTTS_SPEAKER)s",XTTS_LANGUAGE="%(ENV_XTTS_LANGUAGE)s",XTTS_WORKER_HOST="127.0.0.1",XTTS_WORKER_PORT="8771",TTS_HOME="/opt/xtts-cache",COQUI_TOS_AGREED="1"
priority=280
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:bridge] [program:bridge]
command=/app/docker/run-if-role.sh full,bot /opt/venv/bin/python -m bridge.server command=/app/docker/run-if-role.sh full,bot /opt/venv/bin/python -m bridge.server

View File

@@ -20,7 +20,7 @@ Every distinct LLM call in Jarvis, what feeds it, what consumes it, and how it i
- Tool schema: native via `generate_tools_json_schema()` ([src/jarvis/tools/registry.py](src/jarvis/tools/registry.py)) or text fallback via `_text_tool_call_guidance()` ([engine.py:68](src/jarvis/reply/engine.py:68)) - Tool schema: native via `generate_tools_json_schema()` ([src/jarvis/tools/registry.py](src/jarvis/tools/registry.py)) or text fallback via `_text_tool_call_guidance()` ([engine.py:68](src/jarvis/reply/engine.py:68))
- Tool results from prior turns (raw or digested — see #5) - Tool results from prior turns (raw or digested — see #5)
- **Output**: OpenAI-style `{content, tool_calls, thinking}`. Consumed by the tool orchestrator and TTS pipeline. Natural-language content is delivered immediately; no post-turn evaluator runs. - **Output**: OpenAI-style `{content, tool_calls, thinking}`. Consumed by the tool orchestrator and TTS pipeline. Natural-language content is delivered immediately; no post-turn evaluator runs.
- **Limits**: `num_ctx: 8192` (explicit). Timeout `llm_chat_timeout_sec` (45s). Auto-fallback from native to text tool-calls on HTTP 400 (`ToolsNotSupportedError`), sticky for the session. Risk: `fetch_web_page` truncates at 50,000 chars (~37k tokens) — mitigated for SMALL models by tool-result digest (#5) which compresses the payload before it enters the messages history. LARGE models receive the raw payload and may silently see a truncated context. - **Limits**: `num_ctx: 8192` (explicit). Output `num_predict: cfg.ollama_num_predict` (default 512, `0`/negative disables) caps generated tokens per turn — a worst-case latency guard for short spoken answers; the headroom stays above tool-call JSON so it does not truncate tool calls (both native and text tool-call paths). Timeout `llm_chat_timeout_sec` (45s). Auto-fallback from native to text tool-calls on HTTP 400 (`ToolsNotSupportedError`), sticky for the session. Risk: `fetch_web_page` truncates at 50,000 chars (~37k tokens) — mitigated for SMALL models by tool-result digest (#5) which compresses the payload before it enters the messages history. LARGE models receive the raw payload and may silently see a truncated context.
## 2. Intent Judge ## 2. Intent Judge

View File

@@ -85,6 +85,12 @@ class Settings:
llm_digest_timeout_sec: float llm_digest_timeout_sec: float
llm_embedding_timeout_sec: float llm_embedding_timeout_sec: float
llm_profile_select_timeout_sec: float llm_profile_select_timeout_sec: float
# Upper bound on tokens the chat model may generate per reply turn. Spoken
# (TTS) answers are 1-2 sentences, so a cap bounds the worst-case latency of
# a model that occasionally rambles or loops without changing normal answers.
# The headroom (default 512) sits well above this app's short tool-call JSON,
# so capping never truncates a tool call. 0 (or negative) disables the cap.
ollama_num_predict: int
# Profiles & Behavior # Profiles & Behavior
active_profiles: list[str] active_profiles: list[str]
@@ -394,6 +400,9 @@ def get_default_config() -> Dict[str, Any]:
"llm_digest_timeout_sec": 8.0, "llm_digest_timeout_sec": 8.0,
"llm_embedding_timeout_sec": 60.0, "llm_embedding_timeout_sec": 60.0,
"llm_profile_select_timeout_sec": 30.0, "llm_profile_select_timeout_sec": 30.0,
# Cap on chat-model output tokens per turn (worst-case latency guard).
# 512 is safe headroom above short TTS answers and tool-call JSON; 0 off.
"ollama_num_predict": 512,
# Profiles & Behavior # Profiles & Behavior
"active_profiles": ["developer", "business", "life"], "active_profiles": ["developer", "business", "life"],
@@ -608,7 +617,11 @@ def load_settings() -> Settings:
active_profiles = _ensure_list(merged.get("active_profiles")) active_profiles = _ensure_list(merged.get("active_profiles"))
tts_enabled = bool(merged.get("tts_enabled", True)) tts_enabled = bool(merged.get("tts_enabled", True))
tts_engine = str(merged.get("tts_engine", "piper")).lower() tts_engine = str(merged.get("tts_engine", "piper")).lower()
if tts_engine not in ("piper", "chatterbox"): # "edge" (Microsoft Edge TTS) is the containerized bridge's Korean voice;
# "melo" is the legacy warm-worker voice. Both are multilingual, so they must
# be preserved here — coercing them to "piper" would mislabel the engine as
# English-only in reply_language_directive().
if tts_engine not in ("piper", "chatterbox", "edge", "melo"):
tts_engine = "piper" # Default to piper if invalid value tts_engine = "piper" # Default to piper if invalid value
tts_voice_val = merged.get("tts_voice") tts_voice_val = merged.get("tts_voice")
tts_voice = None if tts_voice_val in (None, "", "null") else str(tts_voice_val) tts_voice = None if tts_voice_val in (None, "", "null") else str(tts_voice_val)
@@ -759,6 +772,10 @@ def load_settings() -> Settings:
llm_digest_timeout_sec = float(merged.get("llm_digest_timeout_sec", 8.0)) llm_digest_timeout_sec = float(merged.get("llm_digest_timeout_sec", 8.0))
llm_embedding_timeout_sec = float(merged.get("llm_embedding_timeout_sec", 60.0)) llm_embedding_timeout_sec = float(merged.get("llm_embedding_timeout_sec", 60.0))
llm_profile_select_timeout_sec = float(merged.get("llm_profile_select_timeout_sec", 30.0)) llm_profile_select_timeout_sec = float(merged.get("llm_profile_select_timeout_sec", 30.0))
try:
ollama_num_predict = int(merged.get("ollama_num_predict", 512))
except (TypeError, ValueError):
ollama_num_predict = 512
return Settings( return Settings(
# Database & Storage # Database & Storage
@@ -774,6 +791,7 @@ def load_settings() -> Settings:
llm_digest_timeout_sec=llm_digest_timeout_sec, llm_digest_timeout_sec=llm_digest_timeout_sec,
llm_embedding_timeout_sec=llm_embedding_timeout_sec, llm_embedding_timeout_sec=llm_embedding_timeout_sec,
llm_profile_select_timeout_sec=llm_profile_select_timeout_sec, llm_profile_select_timeout_sec=llm_profile_select_timeout_sec,
ollama_num_predict=ollama_num_predict,
# Profiles & Behavior # Profiles & Behavior
active_profiles=active_profiles, active_profiles=active_profiles,

View File

@@ -2233,6 +2233,16 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
has_tool_calls = " (has tool_calls)" if msg.get("tool_calls") else "" has_tool_calls = " (has tool_calls)" if msg.get("tool_calls") else ""
debug_log(f" [{i}] {role}: {content}{has_tool_calls}", "planning") debug_log(f" [{i}] {role}: {content}{has_tool_calls}", "planning")
# Bound worst-case generation latency: spoken answers are 1-2 sentences,
# so cap the chat model's output tokens. The default headroom sits well
# above this app's tool-call JSON, so capping never truncates a tool
# call. 0/negative disables it. See config.ollama_num_predict.
try:
_num_predict = int(getattr(cfg, 'ollama_num_predict', 0) or 0)
except (TypeError, ValueError):
_num_predict = 0
_chat_extra_options = {"num_predict": _num_predict} if _num_predict > 0 else None
# Send messages to Ollama — try native tool calling first, fall back to text-based # Send messages to Ollama — try native tool calling first, fall back to text-based
# if the model returns HTTP 400 (native tools API not supported). # if the model returns HTTP 400 (native tools API not supported).
_dump_tools_schema = None if use_text_tools else tools_json_schema _dump_tools_schema = None if use_text_tools else tools_json_schema
@@ -2242,7 +2252,7 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
chat_model=cfg.ollama_chat_model, chat_model=cfg.ollama_chat_model,
messages=messages, messages=messages,
timeout_sec=float(getattr(cfg, 'llm_chat_timeout_sec', 45.0)), timeout_sec=float(getattr(cfg, 'llm_chat_timeout_sec', 45.0)),
extra_options=None, extra_options=_chat_extra_options,
tools=_dump_tools_schema, tools=_dump_tools_schema,
thinking=getattr(cfg, 'llm_thinking_enabled', False), thinking=getattr(cfg, 'llm_thinking_enabled', False),
) )
@@ -2273,7 +2283,7 @@ def run_reply_engine(db: "Database", cfg, tts: Optional[Any],
chat_model=cfg.ollama_chat_model, chat_model=cfg.ollama_chat_model,
messages=messages, messages=messages,
timeout_sec=float(getattr(cfg, 'llm_chat_timeout_sec', 45.0)), timeout_sec=float(getattr(cfg, 'llm_chat_timeout_sec', 45.0)),
extra_options=None, extra_options=_chat_extra_options,
tools=None, tools=None,
thinking=getattr(cfg, 'llm_thinking_enabled', False), thinking=getattr(cfg, 'llm_thinking_enabled', False),
) )

View File

@@ -287,6 +287,8 @@ Turn 4: LLM → {content: "Here's a comprehensive comparison of the iPhone 15 mo
- `llm_tools_timeout_sec` (enrichment extraction) - `llm_tools_timeout_sec` (enrichment extraction)
- `llm_embed_timeout_sec` (vector search) - `llm_embed_timeout_sec` (vector search)
- `llm_chat_timeout_sec` (messages loop turn) - `llm_chat_timeout_sec` (messages loop turn)
- Output bound:
- `ollama_num_predict` (default `512`, `0`/negative disables) caps the chat model's generated tokens per turn via the Ollama `num_predict` option on the messages-loop call. Spoken (TTS) answers are 1-2 sentences, so this never clips a normal answer; it bounds the worst-case latency of a model that occasionally rambles or loops. The default headroom sits well above this app's short tool-call JSON, so it does not truncate tool calls. Applied uniformly to the reply loop's chat call (both native-tools and text-tools paths); the small classification passes (intent judge, digests) keep their own caps. Note: this is a worst-case guard, not the primary latency lever, which is model size and GPU residency.
- Memory enrichment: - Memory enrichment:
- `memory_enrichment_max_results` limits recalled snippets. - `memory_enrichment_max_results` limits recalled snippets.
- `memory_digest_enabled` (default `null` = auto-on for SMALL models ≤7B, off for LARGE) distils the combined diary + graph dump into a short relevance-filtered note via a cheap LLM pass before injecting into the system prompt. See **Memory Digest for Small Models** below. - `memory_digest_enabled` (default `null` = auto-on for SMALL models ≤7B, off for LARGE) distils the combined diary + graph dump into a short relevance-filtered note via a cheap LLM pass before injecting into the system prompt. See **Memory Digest for Small Models** below.

View File

@@ -30,8 +30,10 @@ class BrowseAndPlayTool(Tool):
def description(self) -> str: def description(self) -> str:
return ( return (
"Play a song / music video / clip on the shared screen by searching YouTube " "Play a song / music video / clip on the shared screen by searching YouTube "
"and playing the first result. Use when the user asks you to play or watch " "and playing a result. Use when the user asks you to play or watch "
"something. Only available in screen-share mode." "something. Plays the first result by default; pass 'index' to play the "
"Nth result from the top of the search list (e.g. 'play the 3rd video' -> "
"index=3). Only available in screen-share mode."
) )
@property @property
@@ -42,7 +44,16 @@ class BrowseAndPlayTool(Tool):
"query": { "query": {
"type": "string", "type": "string",
"description": "What to play, e.g. 'IU Good Day' or 'lofi hip hop'.", "description": "What to play, e.g. 'IU Good Day' or 'lofi hip hop'.",
} },
"index": {
"type": "integer",
"description": (
"1-based position of the video to play in the search results, "
"counted from the top of the list. Defaults to 1 (first result). "
"Use for 'play the Nth video' / 'play the second one'."
),
"minimum": 1,
},
}, },
"required": ["query"], "required": ["query"],
} }
@@ -55,18 +66,25 @@ class BrowseAndPlayTool(Tool):
reply_text="화면 공유 모드(STREAM_BROWSER=true)에서만 영상을 재생할 수 있습니다.", reply_text="화면 공유 모드(STREAM_BROWSER=true)에서만 영상을 재생할 수 있습니다.",
) )
query = "" query = ""
index = 1
if args and isinstance(args, dict): if args and isinstance(args, dict):
query = str(args.get("query", "")).strip() query = str(args.get("query", "")).strip()
try:
index = int(args.get("index", 1) or 1)
except (TypeError, ValueError):
index = 1
if index < 1:
index = 1
if not query: if not query:
return ToolExecutionResult(success=False, reply_text="재생할 내용을 알려주세요.") return ToolExecutionResult(success=False, reply_text="재생할 내용을 알려주세요.")
if not _NODE_SCRIPT.exists(): if not _NODE_SCRIPT.exists():
return ToolExecutionResult(success=False, reply_text="브라우저 재생 도구를 찾을 수 없습니다.") return ToolExecutionResult(success=False, reply_text="브라우저 재생 도구를 찾을 수 없습니다.")
context.user_print(f"▶️ 화면에서 '{query}' 재생 중…") context.user_print(f"▶️ 화면에서 '{query}' 재생 중… (#{index})")
debug_log(f" ▶️ browseAndPlay '{query}'", "tools") debug_log(f" ▶️ browseAndPlay '{query}' index={index}", "tools")
try: try:
proc = subprocess.run( proc = subprocess.run(
["node", str(_NODE_SCRIPT), query, "youtube"], ["node", str(_NODE_SCRIPT), query, "youtube", str(index)],
capture_output=True, capture_output=True,
text=True, text=True,
timeout=40, timeout=40,

View File

@@ -6,16 +6,24 @@ video, or clip.
### Behaviour ### Behaviour
- Public schema is a single required `query` string (what to play). - Public schema is a required `query` string (what to play) plus an optional
`index` integer (1-based position in the search results, counted from the top
of the list). `index` defaults to `1` (first result), so existing callers and
"play X" requests are unchanged; "play the 3rd video" / "play the second one"
map to `index=3` / `index=2`.
- **Mode-gated**: only acts when `STREAM_BROWSER` is true (`cfg.stream_browser`). - **Mode-gated**: only acts when `STREAM_BROWSER` is true (`cfg.stream_browser`).
In voice-only mode (false) there is no screen to show, so it returns a short In voice-only mode (false) there is no screen to show, so it returns a short
message and does nothing. message and does nothing.
- Drives the on-screen Chrome by subprocessing the Node CDP helper - Drives the on-screen Chrome by subprocessing the Node CDP helper
`bot/scripts/stream-test/browse-search.mjs <query> youtube`, which searches `bot/scripts/stream-test/browse-search.mjs <query> youtube <index>`, which
YouTube and plays the first result on display `:1`. The broadcast captures searches YouTube and plays the chosen result on display `:1`. The broadcast
that display, so the playback is what viewers see. captures that display, so the playback is what viewers see.
- Returns `success` with the played video's title, or a failure message if the - The helper clicks the `index`-th `a#video-title` in the results list. The
helper/Chrome is unavailable. It does NOT make an LLM call. index is clamped to the number of results actually returned, so asking for a
position beyond the list plays the last available result rather than failing.
- Returns `success` with the played video's title (and the resolved `index`), or
a failure message if the helper/Chrome is unavailable. It does NOT make an LLM
call.
### Principles ### Principles

View File

@@ -0,0 +1,79 @@
"""Tests for browseAndPlay's ``index`` argument (play the Nth search result).
Behaviour verified:
- default plays the first result (index 1) and stays backward-compatible,
- an explicit index is forwarded to the Node helper as the 4th argv,
- bad / sub-1 index values clamp to 1,
- the index is advertised in the tool schema.
"""
import json
from unittest.mock import Mock, patch
import pytest
from src.jarvis.tools.builtin.browse_and_play import BrowseAndPlayTool, _NODE_SCRIPT
def _ctx():
cfg = Mock()
cfg.stream_browser = True
return Mock(cfg=cfg, user_print=Mock())
def _run(args):
tool = BrowseAndPlayTool()
with patch("src.jarvis.tools.builtin.browse_and_play.subprocess.run") as mock_run:
mock_run.return_value = Mock(
stdout=json.dumps({"ok": True, "title": "Some Video"}),
stderr="",
)
result = tool.run(args, _ctx())
return mock_run, result
def _argv(mock_run):
return list(mock_run.call_args[0][0])
@pytest.mark.unit
def test_schema_exposes_index():
schema = BrowseAndPlayTool().inputSchema
assert "index" in schema["properties"]
assert schema["properties"]["index"]["type"] == "integer"
assert "query" in schema["required"]
assert "index" not in schema["required"] # optional
@pytest.mark.unit
def test_default_index_is_first():
mock_run, result = _run({"query": "IU Good Day"})
argv = _argv(mock_run)
assert argv[:4] == ["node", str(_NODE_SCRIPT), "IU Good Day", "youtube"]
assert argv[4] == "1"
assert result.success is True
@pytest.mark.unit
def test_explicit_index_forwarded():
mock_run, _ = _run({"query": "lofi", "index": 3})
assert _argv(mock_run)[4] == "3"
@pytest.mark.unit
@pytest.mark.parametrize("bad", [0, -2, "nope", None])
def test_bad_index_clamps_to_one(bad):
mock_run, _ = _run({"query": "lofi", "index": bad})
assert _argv(mock_run)[4] == "1"
@pytest.mark.unit
def test_voice_only_mode_does_not_play():
tool = BrowseAndPlayTool()
cfg = Mock()
cfg.stream_browser = False
ctx = Mock(cfg=cfg, user_print=Mock())
with patch("src.jarvis.tools.builtin.browse_and_play.subprocess.run") as mock_run:
result = tool.run({"query": "x", "index": 2}, ctx)
assert result.success is False
mock_run.assert_not_called()

View File

@@ -0,0 +1,112 @@
"""Tests for the ``ollama_num_predict`` chat-output cap.
The cap bounds worst-case reply latency by limiting how many tokens the chat
model may generate per turn. Spoken (TTS) answers are 1-2 sentences, so the
default headroom never clips a normal answer and stays above tool-call JSON.
These tests verify behaviour:
- the config default is present,
- the value is threaded into the Ollama request as the ``num_predict`` option,
- the reply loop forwards it to the chat call (and disables it at 0).
"""
from unittest.mock import Mock, patch
import pytest
from src.jarvis.config import get_default_config
from src.jarvis.memory.conversation import DialogueMemory
from src.jarvis.reply.engine import run_reply_engine
# ---------------------------------------------------------------------------
# Config default
# ---------------------------------------------------------------------------
def test_default_config_has_num_predict_cap():
config = get_default_config()
assert config["ollama_num_predict"] == 512
# ---------------------------------------------------------------------------
# Transport: extra_options.num_predict reaches the Ollama payload options
# ---------------------------------------------------------------------------
@patch("jarvis.llm.requests.post")
def test_chat_with_messages_forwards_num_predict(mock_post):
from jarvis.llm import chat_with_messages
mock_resp = Mock()
mock_resp.status_code = 200
mock_resp.json.return_value = {"message": {"content": "ok"}}
mock_resp.raise_for_status = Mock()
mock_post.return_value = mock_resp
chat_with_messages(
"http://localhost:11434",
"test-large",
[{"role": "user", "content": "hi"}],
extra_options={"num_predict": 512},
)
_, kwargs = mock_post.call_args
options = (kwargs.get("json") or {}).get("options") or {}
assert options.get("num_predict") == 512
# ---------------------------------------------------------------------------
# Reply loop wiring
# ---------------------------------------------------------------------------
def _mock_cfg(num_predict):
cfg = Mock()
cfg.ollama_base_url = "http://localhost:11434"
cfg.ollama_chat_model = "test-large" # avoid SMALL-model text-tool path
cfg.ollama_num_predict = num_predict
cfg.voice_debug = False
cfg.llm_tools_timeout_sec = 8.0
cfg.llm_embed_timeout_sec = 10.0
cfg.llm_chat_timeout_sec = 45.0
cfg.llm_digest_timeout_sec = 8.0
cfg.memory_enrichment_max_results = 5
cfg.memory_enrichment_source = "diary"
cfg.memory_digest_enabled = False
cfg.tool_result_digest_enabled = False
cfg.location_ip_address = None
cfg.location_auto_detect = False
cfg.location_enabled = False
cfg.agentic_max_turns = 8
cfg.tool_search_max_calls = 3
cfg.tool_selection_strategy = "all"
cfg.tool_carryover_max_turns = 2
cfg.tool_carryover_per_entry_chars = 1200
cfg.mcps = {}
cfg.llm_thinking_enabled = False
cfg.tts_engine = "none"
cfg.ollama_embed_model = "test-embed"
return cfg
def _run_single_turn(cfg):
"""Drive one reply turn that answers in plain text and capture the
chat call's extra_options."""
with patch("src.jarvis.reply.engine.plan_query", return_value=[]), \
patch("src.jarvis.reply.engine.extract_search_params_for_memory", return_value={}), \
patch("src.jarvis.reply.engine.extract_text_from_response", return_value="Hello."), \
patch("src.jarvis.reply.engine.chat_with_messages") as mock_chat:
mock_chat.return_value = {"message": {"content": "Hello."}}
run_reply_engine(db=Mock(), cfg=cfg, tts=None,
text="hi", dialogue_memory=DialogueMemory())
assert mock_chat.called
return mock_chat.call_args.kwargs.get("extra_options")
@pytest.mark.unit
def test_reply_loop_caps_output_when_enabled():
extra = _run_single_turn(_mock_cfg(512))
assert extra == {"num_predict": 512}
@pytest.mark.unit
def test_reply_loop_no_cap_when_zero():
extra = _run_single_turn(_mock_cfg(0))
assert extra is None

View File

@@ -101,14 +101,6 @@ class TestReplyLanguageDirective:
# user's own language, so no directive. # user's own language, so no directive.
assert reply_language_directive(None, "melo") is None assert reply_language_directive(None, "melo") is None
def test_xtts_is_multilingual(self):
# XTTS-v2 (the Korean voice) is not English-only: no lock → free, and a
# lock is honoured (not overridden to English).
assert reply_language_directive(None, "xtts") is None
directive = reply_language_directive("Korean", "xtts")
assert directive is not None and "Korean" in directive
assert directive != ENGLISH_ONLY_DIRECTIVE
def test_unknown_tts_defaults_to_english_only(self): def test_unknown_tts_defaults_to_english_only(self):
# Preserves the original getattr(cfg, 'tts_engine', 'piper') default: # Preserves the original getattr(cfg, 'tts_engine', 'piper') default:
# an unknown/missing engine is treated conservatively as English-only. # an unknown/missing engine is treated conservatively as English-only.
@@ -118,6 +110,14 @@ class TestReplyLanguageDirective:
directive = reply_language_directive("Korean", "melo") directive = reply_language_directive("Korean", "melo")
assert directive is not None and "Korean" in directive assert directive is not None and "Korean" in directive
def test_edge_is_multilingual(self):
# Edge TTS (the default Korean voice) is not English-only: no lock → the
# user's own language, and a lock is honoured (not forced to English).
assert reply_language_directive(None, "edge") is None
directive = reply_language_directive("Korean", "edge")
assert directive is not None and "Korean" in directive
assert directive != ENGLISH_ONLY_DIRECTIVE
class TestLoadAgentInstructions: class TestLoadAgentInstructions:
"""Operator can extend the reply LLM's system prompt by dropping *.md files """Operator can extend the reply LLM's system prompt by dropping *.md files

View File

@@ -0,0 +1,35 @@
"""The container's TTS engine must be env-driven, not hardcoded.
Regression for a bug where docker/jarvis-config.template.json hardcoded
`"tts_engine": "piper"`. The bridge reads the rendered /app/config/jarvis.json
*before* the environment, so a hardcoded "piper" overrode `TTS_ENGINE=melo` in
.env and the bot read Korean text with the English Piper voice ("foreign
accent"). The template must carry `${TTS_ENGINE}` so envsubst (entrypoint.sh)
renders whatever engine the deployment configured.
"""
import json
import string
from pathlib import Path
TEMPLATE = Path(__file__).resolve().parent.parent / "docker" / "jarvis-config.template.json"
def _render(**env) -> dict:
"""Mimic entrypoint.sh `envsubst < template`: substitute env vars, leaving
any unset ones as literal text (valid JSON string values)."""
raw = TEMPLATE.read_text(encoding="utf-8")
return json.loads(string.Template(raw).safe_substitute(**env))
def test_template_does_not_hardcode_an_engine():
raw = TEMPLATE.read_text(encoding="utf-8")
assert '"tts_engine": "${TTS_ENGINE}"' in raw
assert '"tts_engine": "piper"' not in raw
assert '"tts_engine": "melo"' not in raw
def test_rendered_engine_follows_env():
assert _render(TTS_ENGINE="melo")["tts_engine"] == "melo"
assert _render(TTS_ENGINE="piper")["tts_engine"] == "piper"
assert _render(TTS_ENGINE="xtts")["tts_engine"] == "xtts"