feat: replace MeloTTS with Coqui XTTS-v2 natural Korean voice

MeloTTS's single Korean speaker sounded non-native ("foreign accent"). Swap it
for Coqui XTTS-v2 with the built-in female studio speaker "Ana Florence"
(language ko), the natural voice used in earlier local runs.

- bridge/xtts_worker.py: new warm HTTP worker (own /opt/xtts venv), same
  /synth + /health contract and PCM16 output as the old melo worker
- docker/setup-xtts.sh: builds the venv with cu128 torch (Blackwell) + Coqui
  TTS and bakes the XTTS-v2 model offline. Pins transformers>=4.57,<5 (5.x
  removed isin_mps_friendly, breaking XTTS) and installs the [codec] extra
  (torch>=2.9 needs torchcodec) — both verified by a real host synth
- Dockerfile: replace the melo build layer with the xtts layer
- supervisord.conf: melo-worker -> xtts-worker, env passthrough for
  XTTS_DEVICE/SPEAKER/LANGUAGE (always set via compose defaults)
- bridge/server.py: default TTS_ENGINE=xtts, route to the xtts worker, generic
  worker-synth helper, neural-only fallback flag (XTTS_FALLBACK_PIPER)
- settings UI: engine dropdown xtts/piper, drop the dead melo_speed field, fix
  the supervisorctl restart target to xtts-worker
- compose/.env.example/README: XTTS_* vars, speaker/language knobs, remove melo
- remove bridge/melo_worker.py and docker/setup-melo.sh
- tests: xtts treated as multilingual (not English-only)

Verified on host: coqui-tts loads XTTS-v2 and synthesises Korean as
"Ana Florence" to a 16-bit mono 24kHz WAV.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

.env.example

@@ -34,18 +34,23 @@ WHISPER_DEVICE=cuda
 WHISPER_COMPUTE_TYPE=float16
 # Optional explicit Piper voice model (.onnx). If empty, the jarvis default is used.
 TTS_PIPER_MODEL_PATH=
-# TTS engine: "melo" (default) uses the MeloTTS Korean voice served by the warm
+# TTS engine: "xtts" (default) uses the Coqui XTTS-v2 natural Korean voice
-# melo-worker (Korean speaker, speed 1.5). Set to "piper" to use Piper directly.
+# served by the warm xtts-worker. Set to "piper" to use the English Piper voice
-TTS_ENGINE=melo
+# directly. (MeloTTS was removed; "melo" only works with an out-of-band worker.)
-# Melo-only by default: if MeloTTS synthesis fails the bridge returns no audio
+TTS_ENGINE=xtts
+# XTTS-v2 voice settings. Speaker is any built-in studio voice; "Ana Florence"
+# is a natural female voice. Language is the synthesis language (ko = Korean).
+XTTS_SPEAKER=Ana Florence
+XTTS_LANGUAGE=ko
+XTTS_DEVICE=cuda
+# Where the bridge reaches the in-container XTTS worker, and how long it waits
+# for a synthesis (XTTS is slower than Melo: ~1-2s/sentence on GPU).
+XTTS_WORKER_URL=http://127.0.0.1:8771
+XTTS_TIMEOUT=30
+# Neural-only by default: if XTTS synthesis fails the bridge returns no audio
 # rather than speaking Korean through the English Piper voice (which mangles it).
 # Set to 1 only if you explicitly want the Piper fallback.
-MELO_FALLBACK_PIPER=0
+XTTS_FALLBACK_PIPER=0
-# Where the bridge reaches the in-container MeloTTS worker, and how long it
-# waits for a synthesis. Speaking rate is set on the worker via MELO_SPEED.
-MELO_WORKER_URL=http://127.0.0.1:8770
-MELO_TIMEOUT=30
-MELO_SPEED=1.5
 
 # ---------------------------------------------------------------------------
 # Jarvis brain (Ollama-backed). In Docker these populate the rendered
@@ -226,7 +231,7 @@ COMPOSE_FILE=docker-compose.yml:docker-compose.gpu-linux.yml
 # OLLAMA_CHAT_MODEL=qwen2.5:7b   # quality (needs ~5GB VRAM + whisper small)
 # OLLAMA_CHAT_MODEL=qwen2.5:3b   # speed (fits easily, faster on 8GB GPUs)
 # WHISPER_MODEL=small            # small frees VRAM for a bigger LLM; medium=more accurate
-# MELO_DEVICE=cuda               # cpu if no GPU on the bot host
+# XTTS_DEVICE=cuda               # cpu if no GPU on the bot host (XTTS is slow on CPU)
 
 # --- Settings web UI (http://localhost:8765/settings on the bot host) ---
 # To reach it, expose the bridge to the host loopback:

Dockerfile

@@ -65,18 +65,19 @@ RUN ls -d /opt/venv/lib/python*/site-packages/nvidia/cublas/lib \
         > /etc/ld.so.conf.d/nvidia-cu12.conf 2>/dev/null \
     && /sbin/ldconfig || true
 
-# --- MeloTTS Korean voice (separate /opt/melo py3.11 venv; see setup-melo.sh).
+# --- Korean voice: Coqui XTTS-v2 (separate /opt/xtts py3.11 venv; see
-#     Heavy layer (torch CPU + transformers + MeCab); placed before the app
+#     setup-xtts.sh). Natural female Korean ("Ana Florence"); replaces MeloTTS.
-#     COPY so it stays cached across source-only changes. ---
+#     Heavy layer (torch cu128 + Coqui TTS + the baked XTTS-v2 model); placed
-COPY docker/setup-melo.sh /app/docker/setup-melo.sh
+#     before the app COPY so it stays cached across source-only changes. ---
+COPY docker/setup-xtts.sh /app/docker/setup-xtts.sh
 # Strip CR before running: a Windows checkout (autocrlf) yields CRLF, which makes
-# bash read line 18 as `set -euxo pipefail\r` and abort with
+# bash read `set -euxo pipefail\r` and abort with "set: pipefail: invalid option
-# "set: pipefail: invalid option name". .gitattributes pins *.sh to LF, but this
+# name". .gitattributes pins *.sh to LF, but this keeps the build working even on
-# keeps the build working even on a not-yet-renormalised working tree.
+# a not-yet-renormalised working tree.
-RUN sed -i 's/\r$//' /app/docker/setup-melo.sh && bash /app/docker/setup-melo.sh
+RUN sed -i 's/\r$//' /app/docker/setup-xtts.sh && bash /app/docker/setup-xtts.sh
 
 # --- Human input + window management for the on-screen Chrome control tool.
-#     Placed AFTER the heavy melo layer so it doesn't bust that cache. xdotool
+#     Placed AFTER the heavy TTS layer so it doesn't bust that cache. xdotool
 #     injects real X pointer/keyboard events (visible cursor, char-by-char
 #     typing) into the broadcast; wmctrl lists/moves windows. ---
 RUN apt-get update && apt-get install -y --no-install-recommends \

README.md

@@ -69,7 +69,7 @@ docker compose -f docker-compose.yml -f docker-compose.gpu-linux.yml up -d --bui
 docker compose -f docker-compose.yml -f docker-compose.gpu-windows.yml up -d --build
 
 # ── GPU 없이 (CPU 전용 호스트) ──
-#   .env 에 WHISPER_DEVICE=cpu, MELO_DEVICE=cpu 를 넣고 베이스만 사용
+#   .env 에 WHISPER_DEVICE=cpu, XTTS_DEVICE=cpu 를 넣고 베이스만 사용
 docker compose up -d --build
 ```
 
@@ -113,7 +113,7 @@ docker compose up -d          # 유저봇이 로그인해 지정 음성채널에
 
 ### GPU 가속 (OS별)
 
-LLM(Ollama), Whisper STT, MeloTTS가 GPU에서 돕니다(env 기본 `WHISPER_DEVICE=cuda`, `MELO_DEVICE=cuda`). NVIDIA Blackwell(sm_120, 예: RTX 5050/5070Ti)에서 검증: 컨테이너 내 torch cu128 CUDA 동작, Ollama GPU 오프로드, faster-whisper float16, MeloTTS GPU 합성 모두 확인.
+LLM(Ollama), Whisper STT, XTTS-v2 TTS가 GPU에서 돕니다(env 기본 `WHISPER_DEVICE=cuda`, `XTTS_DEVICE=cuda`). NVIDIA Blackwell(sm_120, 예: RTX 5050/5070Ti)에서 검증: 컨테이너 내 torch cu128 CUDA 동작, Ollama GPU 오프로드, faster-whisper float16, XTTS-v2 GPU 합성 모두 확인.
 
 GPU는 위 "실행 — Docker"의 OS별 override 파일로 켜집니다. 호스트 사전 준비는 OS마다 다릅니다:
 
@@ -137,7 +137,7 @@ docker run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi -L   # GPU 보이
 **공통:**
 
 - 모델: 베이스 compose 기본은 `qwen2.5:3b`(8GB VRAM에서 도구호출 안정적). 더 무겁게(`qwen2.5:7b`, `qwen3:8b` 등) 쓰려면 `.env`의 `OLLAMA_CHAT_MODEL` 변경.
-- **GPU가 없거나 인식 실패 시 자동으로 CPU 폴백**(Whisper)하므로 안전합니다. 명시적으로 CPU만 쓰려면 override 파일 없이 베이스만 올리고 `.env`에 `WHISPER_DEVICE=cpu`, `MELO_DEVICE=cpu`를 두세요.
+- **GPU가 없거나 인식 실패 시 자동으로 CPU 폴백**(Whisper)하므로 안전합니다. 명시적으로 CPU만 쓰려면 override 파일 없이 베이스만 올리고 `.env`에 `WHISPER_DEVICE=cpu`, `XTTS_DEVICE=cpu`를 두세요.
 
 - 데이터(메모리 DB), Whisper 캐시, Piper 음성은 named volume에 영속됩니다.
 - 셀프봇 영상 송출 의존성은 이미지에 기본 포함하지 않습니다. 쓰려면 컨테이너에서 `cd /app/bot && bun add discord.js-selfbot-v13 @dank074/discord-video-stream` 후 재시작(또는 Dockerfile에 추가).
@@ -243,7 +243,7 @@ cd bot; bun run register; bun run start   # 창 2: (일반 봇이면) 슬래시
 - `BRIDGE_URL` — 봇이 호출할 브릿지 주소 (기본 `http://127.0.0.1:8765`)
 - `STREAM_BACKEND`, `DISCORD_SELFBOT_TOKEN`, `NOVNC_URL` — 화면 송출
 - `VNC_DISPLAY=:1`, `VNC_RESOLUTION`, `VNC_FRAMERATE`, `VNC_BITRATE_KBPS` — 캡처
-- `WHISPER_DEVICE/COMPUTE_TYPE`, `MELO_DEVICE` — GPU 호스트면 `cuda`/`float16`, CPU 전용이면 `cpu`(GPU 자체는 OS별 override compose 파일로 켬)
+- `WHISPER_DEVICE/COMPUTE_TYPE`, `XTTS_DEVICE` — GPU 호스트면 `cuda`/`float16`, CPU 전용이면 `cpu`(GPU 자체는 OS별 override compose 파일로 켬)
 - `OLLAMA_CHAT_MODEL` — 두뇌 LLM (기본 `qwen2.5:3b`)
 - `COMPOSE_FILE` — OS별 GPU override를 매번 `-f`로 안 치고 자동 적용 (위 "실행 — Docker" 참고)
 - `output_language` — 출력 언어 고정(비우면 사용자 언어). 설정 웹 UI(`/settings`)에서 바꾸면 env 기본값보다 우선하며 컨테이너 재생성 후에도 유지됩니다.

bridge/server.py

@@ -87,12 +87,13 @@ VAD_MIN_SPEECH_MS = int(os.environ.get("VAD_MIN_SPEECH_MS", "200"))
 # Korean phrase decoded as Chinese) and shaves a little latency. Empty = auto.
 STT_LANGUAGE = os.environ.get("STT_LANGUAGE", "ko").strip() or None
 
-# TTS engine: "melo" (MeloTTS Korean speaker, the warm worker) is the primary
+# TTS engine: "xtts" (Coqui XTTS-v2 natural Korean voice, the warm worker) is
-# voice; Piper is kept as a fallback if the worker is unreachable. Set
+# the primary voice; Piper is kept as a fallback only if explicitly enabled. Set
-# TTS_ENGINE=piper to disable MeloTTS entirely.
+# TTS_ENGINE=piper to disable the neural Korean voice entirely. "melo" is still
+# accepted for backward compatibility but is no longer built into the image.
 def _tts_engine_setting() -> str:
     """TTS engine: settings-UI value (runtime config JSON) wins, else env, else
-    melo. Read at startup; the settings UI restarts the bridge on apply."""
+    xtts. Read at startup; the settings UI restarts the bridge on apply."""
     try:
         _cp = os.environ.get("JARVIS_CONFIG_PATH", "/app/config/jarvis.json")
         _v = json.loads(open(_cp, encoding="utf-8").read()).get("tts_engine")
@@ -100,17 +101,29 @@ def _tts_engine_setting() -> str:
             return str(_v).strip().lower()
     except Exception:
         pass
-    return os.environ.get("TTS_ENGINE", "melo").strip().lower()
+    return os.environ.get("TTS_ENGINE", "xtts").strip().lower()
 
 
 TTS_ENGINE = _tts_engine_setting()
+# Coqui XTTS-v2 worker (the natural Korean voice).
+XTTS_WORKER_URL = os.environ.get("XTTS_WORKER_URL", "http://127.0.0.1:8771")
+XTTS_TIMEOUT = float(os.environ.get("XTTS_TIMEOUT", "30"))
+# Legacy MeloTTS worker (no longer built into the image; kept for back-compat
+# if someone runs an old worker out-of-band).
 MELO_WORKER_URL = os.environ.get("MELO_WORKER_URL", "http://127.0.0.1:8770")
 MELO_TIMEOUT = float(os.environ.get("MELO_TIMEOUT", "30"))
-# When MeloTTS is the engine, do NOT silently fall back to the English Piper
+# Do NOT silently fall back to the English Piper voice on a neural-voice failure:
-# voice on failure: speaking Korean text through an English voice produces
+# speaking Korean text through an English voice produces mangled audio. Default
-# mangled audio. Default is melo-only (return no audio on failure); set
+# is neural-only (return no audio on failure); set XTTS_FALLBACK_PIPER=1 (or the
-# MELO_FALLBACK_PIPER=1 to opt into the Piper fallback.
+# legacy MELO_FALLBACK_PIPER=1) to opt into the Piper fallback.
-MELO_FALLBACK_PIPER = os.environ.get("MELO_FALLBACK_PIPER", "0") in ("1", "true", "True", "yes", "on")
+def _truthy_env(*names: str) -> bool:
+    for _n in names:
+        if os.environ.get(_n, "").strip().lower() in ("1", "true", "yes", "on"):
+            return True
+    return False
+
+
+NEURAL_FALLBACK_PIPER = _truthy_env("XTTS_FALLBACK_PIPER", "MELO_FALLBACK_PIPER")
 
 # ---------------------------------------------------------------------------
 # Lazy singletons. The first request pays the model-load cost; afterwards the
@@ -302,27 +315,38 @@ def _coerce_bool(value) -> Optional[bool]:
     return str(value).strip().lower() in ("1", "true", "yes", "on")
 
 
-def _melo_synthesize(text: str) -> Optional[bytes]:
+def _worker_synthesize(name: str, url: str, timeout: float, text: str) -> Optional[bytes]:
-    """Synthesise via the warm MeloTTS worker (separate /opt/melo venv, Korean
+    """POST text to a warm TTS worker's /synth and return its WAV bytes, or None
-    speaker @ speed 1.5). Returns a 16-bit PCM WAV, or None on any failure so
+    on any failure so the caller can decide whether to fall back."""
-    the caller can fall back to Piper."""
     import urllib.request
 
     try:
         req = urllib.request.Request(
-            f"{MELO_WORKER_URL}/synth",
+            f"{url}/synth",
             data=json.dumps({"text": text}).encode("utf-8"),
             headers={"Content-Type": "application/json"},
         )
-        with urllib.request.urlopen(req, timeout=MELO_TIMEOUT) as resp:
+        with urllib.request.urlopen(req, timeout=timeout) as resp:
             if resp.status == 200:
                 return resp.read()
-            print(f"[bridge] melo worker HTTP {resp.status}", flush=True)
+            print(f"[bridge] {name} worker HTTP {resp.status}", flush=True)
     except Exception as e:  # pragma: no cover - worker may be down
-        print(f"[bridge] melo worker unreachable: {e}", flush=True)
+        print(f"[bridge] {name} worker unreachable: {e}", flush=True)
     return None
 
 
+def _xtts_synthesize(text: str) -> Optional[bytes]:
+    """Synthesise via the warm Coqui XTTS-v2 worker (separate /opt/xtts venv,
+    natural female Korean). Returns a 16-bit PCM WAV, or None on failure."""
+    return _worker_synthesize("xtts", XTTS_WORKER_URL, XTTS_TIMEOUT, text)
+
+
+def _melo_synthesize(text: str) -> Optional[bytes]:
+    """Legacy: synthesise via a MeloTTS worker if one is running out-of-band.
+    Returns a 16-bit PCM WAV, or None on any failure."""
+    return _worker_synthesize("melo", MELO_WORKER_URL, MELO_TIMEOUT, text)
+
+
 def _piper_synthesize(text: str) -> Optional[bytes]:
     """Fallback: synthesise with Piper (English voice). Returns WAV bytes."""
     _ensure_piper()
@@ -349,11 +373,12 @@ def _tts_ready() -> bool:
     """
     if not TTS_ENABLED:
         return True
-    if TTS_ENGINE == "melo":
+    _worker_health = {"xtts": XTTS_WORKER_URL, "melo": MELO_WORKER_URL}.get(TTS_ENGINE)
+    if _worker_health:
         import urllib.request
 
         try:
-            with urllib.request.urlopen(f"{MELO_WORKER_URL}/health", timeout=2) as resp:
+            with urllib.request.urlopen(f"{_worker_health}/health", timeout=2) as resp:
                 return resp.status == 200
         except Exception:
             return False
@@ -361,20 +386,24 @@ def _tts_ready() -> bool:
 
 
 def synthesize(text: str) -> Optional[bytes]:
-    """Synthesize text to a 16-bit PCM WAV. The primary voice is MeloTTS
+    """Synthesize text to a 16-bit PCM WAV. The primary voice is Coqui XTTS-v2
-    (Korean speaker, speed 1.5) served by the warm melo worker; Piper is a
+    (natural female Korean) served by the warm xtts worker; Piper is used only
-    fallback if the worker is unavailable. Returns None if TTS is off."""
+    when explicitly enabled as a fallback. Returns None if TTS is off."""
     if not TTS_ENABLED or not text.strip():
         return None
-    if TTS_ENGINE == "melo":
+    _neural = {"xtts": _xtts_synthesize, "melo": _melo_synthesize}.get(TTS_ENGINE)
-        audio = _melo_synthesize(text)
+    if _neural is not None:
+        audio = _neural(text)
         if audio:
             return audio
-        if not MELO_FALLBACK_PIPER:
+        if not NEURAL_FALLBACK_PIPER:
-            # Melo-only: better silent than mangled English for Korean text.
+            # Neural-only: better silent than mangled English for Korean text.
-            print("[bridge] melo synth failed; no audio (Piper fallback disabled)", flush=True)
+            print(
+                f"[bridge] {TTS_ENGINE} synth failed; no audio (Piper fallback disabled)",
+                flush=True,
+            )
             return None
-        print("[bridge] melo synth failed; falling back to Piper", flush=True)
+        print(f"[bridge] {TTS_ENGINE} synth failed; falling back to Piper", flush=True)
     return _piper_synthesize(text)
 
 

bridge/settings_web.py

@@ -22,8 +22,7 @@ from typing import Any, Dict
 FIELDS = [
     ("ollama_chat_model", "LLM 모델", "model"),
     ("whisper_model", "STT(Whisper) 모델", "select:tiny,base,small,medium,large,large-v3"),
-    ("tts_engine", "TTS 엔진", "select:melo,piper"),
+    ("tts_engine", "TTS 엔진", "select:xtts,piper"),
-    ("melo_speed", "TTS 속도 (MeloTTS)", "number:0.5:2.5:0.1"),
     ("output_language", "출력 언어 (비우면 사용자 언어)", "text"),
     ("llm_thinking_enabled", "LLM 사고(thinking) 모드", "bool"),
     ("agentic_max_turns", "에이전트 최대 반복", "number:1:12:1"),
@@ -54,9 +53,7 @@ def _current() -> Dict[str, Any]:
     cfg = _read_config()
     out: Dict[str, Any] = {}
     for k in _KEYS:
-        if k == "melo_speed":
+        if k == "output_language":
-            out[k] = cfg.get("melo_speed", os.environ.get("MELO_SPEED", "1.5"))
-        elif k == "output_language":
             out[k] = cfg.get("output_language", os.environ.get("OUTPUT_LANGUAGE", ""))
         else:
             out[k] = cfg.get(k, "")
@@ -78,12 +75,7 @@ def _coerce(updates: Dict[str, Any]) -> Dict[str, Any]:
     for k, v in updates.items():
         if k not in _KEYS:
             continue
-        if k == "melo_speed":
+        if k == "agentic_max_turns":
-            try:
-                v = float(v)
-            except (TypeError, ValueError):
-                continue
-        elif k == "agentic_max_turns":
             try:
                 v = int(v)
             except (TypeError, ValueError):
@@ -114,12 +106,12 @@ def _save(updates: Dict[str, Any]) -> None:
 
 
 def _apply() -> str:
-    # Restart melo + bridge AFTER this response is sent. Detached (new session)
+    # Restart the TTS worker + bridge AFTER this response is sent. Detached (new
-    # so the bridge being killed mid-restart doesn't drop the restart itself,
+    # session) so the bridge being killed mid-restart doesn't drop the restart
-    # and the HTTP client still receives this response.
+    # itself, and the HTTP client still receives this response.
     try:
         subprocess.Popen(
-            ["sh", "-c", "sleep 1; supervisorctl restart melo-worker bridge"],
+            ["sh", "-c", "sleep 1; supervisorctl restart xtts-worker bridge"],
             start_new_session=True,
         )
         return "1초 후 브리지/TTS 워커가 재시작되어 반영됩니다."

bridge/xtts_worker.py

@@ -1,25 +1,30 @@
 """
-MeloTTS worker
+XTTS worker
-==============
+===========
 
-A tiny, dependency-light HTTP service that keeps a MeloTTS voice warm and
+A tiny HTTP service that keeps a Coqui XTTS-v2 voice warm and synthesises
-synthesises speech on demand. It runs in its OWN Python venv (``/opt/melo`` in
+speech on demand. It mirrors ``melo_worker.py`` (same ``/synth`` + ``/health``
-the container) so the heavy MeloTTS/torch/transformers stack stays isolated
+contract, same PCM16 WAV output) so the bridge can talk to either worker the
-from the slim brain-bridge venv (which pins ``numpy<2`` for faster-whisper).
+same way.
 
-The bridge's ``synthesize()`` POSTs ``{"text": "..."}`` here and gets back a
+XTTS-v2 is a natural, multilingual neural voice. The default speaker is the
-16-bit PCM WAV. The MeloTTS model is loaded once at startup and reused, so each
+built-in female studio voice "Ana Florence" speaking Korean — the voice this
-request only pays inference cost, not model-load cost.
+deployment uses in place of MeloTTS. No reference WAV is needed for the
+built-in studio speakers.
+
+It runs in its OWN Python venv (``/opt/xtts`` in the container) so the heavy
+Coqui TTS / torch stack stays isolated from the slim brain-bridge venv.
 
 Config (env):
-    MELO_WORKER_HOST   bind host          (default 127.0.0.1)
+    XTTS_WORKER_HOST   bind host            (default 127.0.0.1)
-    MELO_WORKER_PORT   bind port          (default 8770)
+    XTTS_WORKER_PORT   bind port            (default 8771)
-    MELO_LANGUAGE      MeloTTS language   (default KR)
+    XTTS_MODEL         Coqui model id       (default tts_models/multilingual/multi-dataset/xtts_v2)
-    MELO_SPEED         speaking rate      (default 1.5  -> the approved "150")
+    XTTS_SPEAKER       built-in speaker     (default "Ana Florence")
-    MELO_DEVICE        torch device       (default cpu)
+    XTTS_LANGUAGE      synthesis language   (default ko)
+    XTTS_DEVICE        torch device         (default cpu; compose sets cuda)
 
 Run:
-    /opt/melo/bin/python -m bridge.melo_worker
+    /opt/xtts/bin/python -m bridge.xtts_worker
 """
 
 from __future__ import annotations
@@ -33,94 +38,72 @@ import threading
 import wave
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 
-HOST = os.environ.get("MELO_WORKER_HOST", "127.0.0.1")
+# XTTS-v2 is gated behind a one-time license prompt; agreeing here keeps the
-PORT = int(os.environ.get("MELO_WORKER_PORT", "8770"))
+# load non-interactive in a container. XTTS-v2 is non-commercial (CPML).
-LANGUAGE = os.environ.get("MELO_LANGUAGE", "KR")
+os.environ.setdefault("COQUI_TOS_AGREED", "1")
 
+HOST = os.environ.get("XTTS_WORKER_HOST", "127.0.0.1")
+PORT = int(os.environ.get("XTTS_WORKER_PORT", "8771"))
+MODEL = os.environ.get("XTTS_MODEL", "tts_models/multilingual/multi-dataset/xtts_v2")
+SPEAKER = os.environ.get("XTTS_SPEAKER", "Ana Florence")
+LANGUAGE = os.environ.get("XTTS_LANGUAGE", "ko")
+DEVICE = os.environ.get("XTTS_DEVICE", "cpu")
 
-def _resolve_speed() -> float:
+# Model is loaded once, guarded by a lock because TTS inference is not
-    """Speaking rate: the settings-UI value (runtime config JSON) wins, else the
+# guaranteed thread-safe.
-    MELO_SPEED env, else 1.5. Read at startup; the settings UI restarts this
-    worker on apply so a new value takes effect."""
-    try:
-        cp = os.environ.get("JARVIS_CONFIG_PATH", "/app/config/jarvis.json")
-        v = json.loads(open(cp, encoding="utf-8").read()).get("melo_speed")
-        if v is not None:
-            return float(v)
-    except Exception:
-        pass
-    try:
-        return float(os.environ.get("MELO_SPEED", "1.5"))
-    except ValueError:
-        return 1.5
-
-
-SPEED = _resolve_speed()
-DEVICE = os.environ.get("MELO_DEVICE", "cpu")
-
-# Model + speaker id are loaded once, guarded by a lock because MeloTTS
-# inference is not guaranteed thread-safe.
 _model = None
-_speaker_id = None
 _model_lock = threading.Lock()
 _load_error: str | None = None
 
 
 def _ensure_model() -> None:
-    global _model, _speaker_id, _load_error
+    global _model, _load_error
     if _model is not None or _load_error is not None:
         return
     with _model_lock:
         if _model is not None or _load_error is not None:
             return
         try:
-            from melo.api import TTS  # type: ignore
+            from TTS.api import TTS  # type: ignore
 
-            model = TTS(language=LANGUAGE, device=DEVICE)
+            model = TTS(MODEL).to(DEVICE)
-            # spk2id is a melo HParams object (dict-like, supports __getitem__,
-            # __contains__, keys) but NOT .get(). The KR model exposes a single
-            # 'KR' speaker; fall back to the first id for other languages.
-            spk_map = model.hps.data.spk2id
-            keys = list(spk_map.keys())
-            speaker_id = spk_map[LANGUAGE] if LANGUAGE in spk_map else spk_map[keys[0]]
             _model = model
-            _speaker_id = speaker_id
+            # Warm once: the first GPU synth pays a one-off kernel-init cost
-            # Warm the GPU once at load: the first CUDA synth pays a one-off
+            # that would otherwise land on the user's first reply.
-            # kernel-init cost (~5s) that would otherwise land on the user's
-            # first reply. A throwaway synth here moves it to startup. No-op
-            # cost on CPU.
             try:
                 with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as _wt:
                     _wp = _wt.name
-                model.tts_to_file("워밍업", speaker_id, _wp, speed=SPEED)
+                model.tts_to_file(
+                    text="워밍업", speaker=SPEAKER, language=LANGUAGE, file_path=_wp
+                )
                 try:
                     os.unlink(_wp)
                 except OSError:
                     pass
             except Exception as _we:  # pragma: no cover
-                print(f"[melo-worker] warmup synth skipped: {_we}", flush=True)
+                print(f"[xtts-worker] warmup synth skipped: {_we}", flush=True)
             print(
-                f"[melo-worker] ready (lang={LANGUAGE} speed={SPEED} "
+                f"[xtts-worker] ready (model={MODEL} speaker={SPEAKER!r} "
-                f"device={DEVICE} speakers={list(spk_map.keys())})",
+                f"language={LANGUAGE} device={DEVICE})",
                 flush=True,
             )
         except Exception as e:  # pragma: no cover - depends on local model files
             _load_error = f"{type(e).__name__}: {e}"
-            print(f"[melo-worker] model load FAILED: {_load_error}", flush=True)
+            print(f"[xtts-worker] model load FAILED: {_load_error}", flush=True)
 
 
 def _synthesize(text: str) -> bytes:
     """Synthesise ``text`` to a 16-bit PCM WAV (bytes)."""
     _ensure_model()
     if _model is None:
-        raise RuntimeError(_load_error or "melo model unavailable")
+        raise RuntimeError(_load_error or "xtts model unavailable")
-    # MeloTTS writes to a file via soundfile; render to a container-disk temp
-    # file (NOT tmpfs), read it back, then drop it.
     with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
         tmp_path = tmp.name
     try:
         with _model_lock:
-            _model.tts_to_file(text, _speaker_id, tmp_path, speed=SPEED)
+            _model.tts_to_file(
+                text=text, speaker=SPEAKER, language=LANGUAGE, file_path=tmp_path
+            )
         with open(tmp_path, "rb") as f:
             raw = f.read()
     finally:
@@ -132,16 +115,15 @@ def _synthesize(text: str) -> bytes:
 
 
 def _ensure_pcm16_wav(raw: bytes) -> bytes:
-    """Guarantee a 16-bit PCM WAV. MeloTTS/soundfile usually emit float WAVs;
+    """Guarantee a 16-bit PCM WAV. Coqui writes float/other WAVs; the Discord
-    the Discord playback path (ffmpeg) tolerates both, but we normalise to
+    playback path tolerates both, but we normalise to PCM16 so the contract
-    PCM16 so the contract matches the previous Piper output."""
+    matches the previous Melo/Piper output (mono, file's own sample rate)."""
     try:
         with wave.open(io.BytesIO(raw), "rb") as wf:
             if wf.getsampwidth() == 2:
                 return raw  # already PCM16
     except wave.Error:
         pass
-    # Non-PCM16 (e.g. float) — convert with soundfile if available.
     try:
         import numpy as np
         import soundfile as sf
@@ -159,7 +141,7 @@ def _ensure_pcm16_wav(raw: bytes) -> bytes:
             wf.writeframes(pcm)
         return buf.getvalue()
     except Exception:
-        return raw  # last resort: hand back whatever MeloTTS produced
+        return raw  # last resort: hand back whatever XTTS produced
 
 
 class _Handler(BaseHTTPRequestHandler):
@@ -212,7 +194,7 @@ def main() -> int:
     # Warm the model at startup so the first Discord turn isn't slow.
     _ensure_model()
     server = ThreadingHTTPServer((HOST, PORT), _Handler)
-    print(f"[melo-worker] listening on http://{HOST}:{PORT}", flush=True)
+    print(f"[xtts-worker] listening on http://{HOST}:{PORT}", flush=True)
     try:
         server.serve_forever()
     except KeyboardInterrupt:

docker-compose.yml

@@ -66,9 +66,15 @@ services:
       WHISPER_MODEL: ${WHISPER_MODEL:-medium}
       WHISPER_DEVICE: ${WHISPER_DEVICE:-cuda}
       WHISPER_COMPUTE_TYPE: ${WHISPER_COMPUTE_TYPE:-float16}
-      # MeloTTS on the GPU (cu128 torch baked by docker/setup-melo.sh). CPU synth
+      # Coqui XTTS-v2 (natural female Korean voice, replaces MeloTTS) on the GPU
-      # serialised under load and pushed TTS to 7-8s; GPU does ~0.3s/sentence.
+      # (cu128 torch baked by docker/setup-xtts.sh). Set here WITH DEFAULTS so
-      MELO_DEVICE: ${MELO_DEVICE:-cuda}
+      # supervisord's %(ENV_XTTS_*)s passthrough always resolves and an .env
+      # override actually reaches the xtts-worker.
+      XTTS_DEVICE: ${XTTS_DEVICE:-cuda}
+      # Built-in studio speaker (female). Other options include "Daisy Studious",
+      # "Sofia Hellen", "Alma María", etc. — any XTTS-v2 studio speaker name.
+      XTTS_SPEAKER: ${XTTS_SPEAKER:-Ana Florence}
+      XTTS_LANGUAGE: ${XTTS_LANGUAGE:-ko}
       # Optional single-language lock for replies (empty = user's own language).
       OUTPUT_LANGUAGE: ${OUTPUT_LANGUAGE:-ko}
       # Drop the pre-loop planner LLM call to cut voice-reply latency on small

docker/setup-melo.sh

@@ -1,80 +0,0 @@
-#!/usr/bin/env bash
-# ============================================================================
-# Install a dedicated MeloTTS (Korean voice) venv at /opt/melo.
-#
-# Why a SEPARATE venv (not the brain-bridge /opt/venv):
-#   - MeloTTS pins old deps (transformers 4.27.4 / tokenizers 0.13.3 / fugashi)
-#     whose binary wheels exist only for cp311, so we use python3.11 here even
-#     though the image's default interpreter is 3.12.
-#   - It isolates the heavy torch/transformers stack from the slim bridge env,
-#     which pins numpy<2 for faster-whisper.
-#
-# torch is the CUDA (cu128) build so MeloTTS runs on the GPU alongside Ollama +
-# Whisper. CPU synth serialised under concurrent load (whisper STT + bot) and
-# blew TTS up to 7-8s per reply; on the GPU a sentence synthesises in ~0.3s.
-# cu128 is the Blackwell (sm_120) wheel verified on this host's RTX 5050.
-# The worker selects the device via MELO_DEVICE=cuda (compose).
-# ============================================================================
-set -euxo pipefail
-
-export DEBIAN_FRONTEND=noninteractive
-
-apt-get update
-# Build deps for fugashi / mecab-python3 + a system MeCab dict, plus python3.11.
-apt-get install -y --no-install-recommends \
-    software-properties-common build-essential pkg-config swig \
-    libmecab-dev mecab mecab-ipadic-utf8
-add-apt-repository -y ppa:deadsnakes/ppa
-apt-get update
-apt-get install -y --no-install-recommends python3.11 python3.11-venv python3.11-dev
-rm -rf /var/lib/apt/lists/*
-
-python3.11 -m venv /opt/melo
-/opt/melo/bin/pip install --no-cache-dir --upgrade pip wheel setuptools
-
-# CUDA (cu128) torch first, so MeloTTS's unpinned `torch` dep is already
-# satisfied with the GPU build. Pinned to the Blackwell-verified versions
-# (2.11.0+cu128) for reproducible rebuilds.
-/opt/melo/bin/pip install --no-cache-dir torch==2.11.0+cu128 torchaudio==2.11.0+cu128 \
-    --index-url https://download.pytorch.org/whl/cu128
-
-# MeloTTS from GitHub. The PyPI sdist is broken (its setup.py reads a
-# requirements.txt that is not shipped in the sdist), so install from the repo.
-# Pinned to a commit (not refs/heads/main) so rebuilds are reproducible.
-/opt/melo/bin/pip install --no-cache-dir \
-    "https://github.com/myshell-ai/MeloTTS/archive/209145371cff8fc3bd60d7be902ea69cbdb7965a.tar.gz"
-
-# Korean g2p backend. MeloTTS otherwise tries to pip-install this on the first
-# Korean request, which fails in a network-isolated container at runtime.
-/opt/melo/bin/pip install --no-cache-dir python-mecab-ko python-mecab-ko-dic
-
-# Remove the full `unidic` package (its dictionary is never downloaded, only a
-# stub) so mecab-python3 falls back to the bundled `unidic_lite` dict. Without
-# this, importing melo's Japanese module fails with a missing-mecabrc error.
-/opt/melo/bin/pip uninstall -y unidic || true
-
-# Pre-cache every model asset MeloTTS pulls at runtime, so the worker starts
-# offline and the first Discord turn pays no download cost. Importing melo.api
-# fetches the Japanese (tohoku-nlp/bert-base-japanese-v3) and Korean
-# (kykim/bert-kor-base) BERT tokenizers plus nltk g2p data; loading the KR voice
-# downloads the OpenVoice KR config+checkpoint, and a real synth pulls the
-# Korean BERT weights. All of these go through huggingface_hub.
-#
-# CRITICAL: at runtime docker-compose mounts the `whisper_cache` named volume
-# over /root/.cache/huggingface (for faster-whisper). That volume would SHADOW
-# anything baked into the default HF cache, so we pin the melo worker to a
-# DEDICATED, non-volume cache dir (/opt/melo-cache) here AND in supervisord, and
-# warm it once. nltk_data (/root/nltk_data) is not volume-mounted so it stays.
-export HF_HOME=/opt/melo-cache
-mkdir -p "$HF_HOME"
-MELO_LANGUAGE=KR HF_HOME=/opt/melo-cache /opt/melo/bin/python - <<'PY'
-import tempfile
-from melo.api import TTS
-
-model = TTS(language="KR", device="cpu")
-out = tempfile.mktemp(suffix=".wav")
-model.tts_to_file("초기화 워밍업입니다.", model.hps.data.spk2id["KR"], out, speed=1.5)
-print("[setup-melo] warm-up KR synth OK ->", out)
-PY
-
-echo "[setup-melo] MeloTTS venv ready at /opt/melo"

docker/setup-xtts.sh

@@ -0,0 +1,72 @@
+#!/usr/bin/env bash
+# ============================================================================
+# Install a dedicated Coqui XTTS-v2 (natural Korean voice) venv at /opt/xtts.
+#
+# Why a SEPARATE venv (not the brain-bridge /opt/venv or /opt/melo):
+#   - Coqui TTS pulls its own heavy torch/transformers stack; isolating it keeps
+#     the slim bridge env (numpy<2 for faster-whisper) untouched.
+#   - We use python3.11 (installed for the melo layer) because Coqui ships cp311
+#     wheels and torch cu128 is available for it.
+#
+# torch is the CUDA (cu128) build so XTTS runs on the GPU alongside Ollama +
+# Whisper. cu128 is the Blackwell (sm_120) wheel verified on this host.
+# The worker selects the device via XTTS_DEVICE=cuda (compose).
+#
+# XTTS-v2 is non-commercial (Coqui Public Model License). COQUI_TOS_AGREED=1
+# accepts it non-interactively so the model can load in a headless container.
+# ============================================================================
+set -euxo pipefail
+
+export DEBIAN_FRONTEND=noninteractive
+export COQUI_TOS_AGREED=1
+
+# Install python3.11 if not already present, so this layer is self-contained.
+if ! command -v python3.11 >/dev/null 2>&1; then
+    apt-get update
+    apt-get install -y --no-install-recommends software-properties-common
+    add-apt-repository -y ppa:deadsnakes/ppa
+    apt-get update
+    apt-get install -y --no-install-recommends python3.11 python3.11-venv python3.11-dev
+    rm -rf /var/lib/apt/lists/*
+fi
+
+python3.11 -m venv /opt/xtts
+/opt/xtts/bin/pip install --no-cache-dir --upgrade pip wheel setuptools
+
+# CUDA (cu128) torch first so Coqui's `torch` dep is satisfied with the GPU
+# build. Pinned to the Blackwell-verified versions for reproducible rebuilds.
+/opt/xtts/bin/pip install --no-cache-dir torch==2.11.0+cu128 torchaudio==2.11.0+cu128 \
+    --index-url https://download.pytorch.org/whl/cu128
+
+# Coqui TTS (maintained fork; provides the `TTS` package and XTTS-v2). The
+# [codec] extra pulls torchcodec, which torch >=2.9 requires for audio IO
+# (without it the import fails with TORCHCODEC_IMPORT_ERROR). torchcodec also
+# needs the system FFmpeg shared libs, which are present (ffmpeg apt package).
+/opt/xtts/bin/pip install --no-cache-dir "coqui-tts[codec]"
+
+# Pin transformers to the 4.57+ / <5 range. coqui-tts requires >=4.57 but does
+# NOT cap the upper bound, and transformers 5.x removed `isin_mps_friendly`
+# (used by XTTS's tortoise layer), so an unpinned install pulls 5.x and the
+# model import fails with "cannot import name 'isin_mps_friendly'". Pin <5.
+/opt/xtts/bin/pip install --no-cache-dir "transformers>=4.57,<5"
+
+# Pre-bake the XTTS-v2 model so the worker starts offline and the first Discord
+# turn pays no download cost. The model is cached under TTS_HOME; we pin a
+# DEDICATED, non-volume dir (/opt/xtts-cache) AND set it in supervisord, because
+# runtime volume mounts (whisper_cache over /root/.cache) must not shadow it.
+export TTS_HOME=/opt/xtts-cache
+mkdir -p "$TTS_HOME"
+COQUI_TOS_AGREED=1 TTS_HOME=/opt/xtts-cache XTTS_SPEAKER="Ana Florence" \
+    /opt/xtts/bin/python - <<'PY'
+import os
+os.environ["COQUI_TOS_AGREED"] = "1"
+from TTS.api import TTS
+
+speaker = os.environ.get("XTTS_SPEAKER", "Ana Florence")
+model = TTS("tts_models/multilingual/multi-dataset/xtts_v2")  # downloads to TTS_HOME
+out = "/tmp/xtts_warm.wav"
+model.tts_to_file(text="초기화 워밍업입니다.", speaker=speaker, language="ko", file_path=out)
+print("[setup-xtts] warm-up KR synth OK ->", out, "speaker:", speaker)
+PY
+
+echo "[setup-xtts] Coqui XTTS-v2 venv ready at /opt/xtts (cache /opt/xtts-cache)"

docker/supervisord.conf

@@ -49,22 +49,22 @@ stdout_logfile_maxbytes=0
 stderr_logfile=/dev/stderr
 stderr_logfile_maxbytes=0
 
-[program:melo-worker]
+[program:xtts-worker]
-; Warm MeloTTS Korean voice (speed 1.5) in its own py3.11 venv. The bridge's
+; Warm Coqui XTTS-v2 Korean voice (natural female "Ana Florence") in its own
-; synthesize() POSTs here; if this is down the bridge falls back to Piper.
+; py3.11 venv. The bridge's synthesize() POSTs here; if this is down the bridge
-command=/app/docker/run-if-role.sh full,bot /opt/melo/bin/python /app/bridge/melo_worker.py
+; falls back to Piper (English) only when XTTS_FALLBACK_PIPER=1.
+command=/app/docker/run-if-role.sh full,bot /opt/xtts/bin/python /app/bridge/xtts_worker.py
 directory=/app
-; HF_HOME points at the dedicated, image-baked melo cache (warmed in
+; TTS_HOME points at the dedicated, image-baked XTTS cache (warmed in
-; setup-melo.sh). The brain's whisper_cache volume is mounted over
+; setup-xtts.sh). The brain's whisper_cache volume is mounted over
-; /root/.cache/huggingface, so without this the pre-cached BERT + KR checkpoint
+; /root/.cache, so a dedicated non-volume cache dir avoids the baked model being
-; would be shadowed and re-downloaded (and would fail if the host is offline).
+; shadowed and re-downloaded (which would fail if the host is offline).
-; HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE force pure-cache reads: the pinned old
+; XTTS_DEVICE / XTTS_SPEAKER / XTTS_LANGUAGE inherit from the container env
-; transformers/huggingface_hub otherwise retry the network on every load and
+; (compose sets them with defaults: cuda / "Ana Florence" / ko). supervisord
-; error out instead of falling back to the (complete) baked cache.
+; interpolates %(ENV_x)s from its own environment, which is the container's — so
-; MELO_DEVICE inherits from the container env (compose sets it; default cuda)
+; these must always be set in the env (compose guarantees it) or this expansion
-; so the worker runs MeloTTS on the GPU. supervisord interpolates %(ENV_x)s
+; fails at startup. COQUI_TOS_AGREED accepts the non-commercial XTTS license.
-; from its own environment, which is the container's.
+environment=XTTS_DEVICE="%(ENV_XTTS_DEVICE)s",XTTS_SPEAKER="%(ENV_XTTS_SPEAKER)s",XTTS_LANGUAGE="%(ENV_XTTS_LANGUAGE)s",XTTS_WORKER_HOST="127.0.0.1",XTTS_WORKER_PORT="8771",TTS_HOME="/opt/xtts-cache",COQUI_TOS_AGREED="1"
-environment=MELO_LANGUAGE="KR",MELO_SPEED="1.5",MELO_DEVICE="%(ENV_MELO_DEVICE)s",MELO_WORKER_HOST="127.0.0.1",MELO_WORKER_PORT="8770",HF_HOME="/opt/melo-cache",HF_HUB_OFFLINE="1",TRANSFORMERS_OFFLINE="1"
 priority=280
 autorestart=true
 stdout_logfile=/dev/stdout

tests/test_system_prompt.py

@@ -101,6 +101,14 @@ class TestReplyLanguageDirective:
         # user's own language, so no directive.
         assert reply_language_directive(None, "melo") is None
 
+    def test_xtts_is_multilingual(self):
+        # XTTS-v2 (the Korean voice) is not English-only: no lock → free, and a
+        # lock is honoured (not overridden to English).
+        assert reply_language_directive(None, "xtts") is None
+        directive = reply_language_directive("Korean", "xtts")
+        assert directive is not None and "Korean" in directive
+        assert directive != ENGLISH_ONLY_DIRECTIVE
+
     def test_unknown_tts_defaults_to_english_only(self):
         # Preserves the original getattr(cfg, 'tts_engine', 'piper') default:
         # an unknown/missing engine is treated conservatively as English-only.