tkrmagid/javis_bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

feat: replace MeloTTS with Coqui XTTS-v2 natural Korean voice
Some checks failed
Release / semantic-release (push) Successful in 30s
Details
tests / Unit tests (Linux, Python 3.11) (push) Failing after 5m17s
Details
Release / build-windows (push) Has been cancelled
Details
Release / build-macos (arm64, macos-latest) (push) Has been cancelled
Details
Release / build-macos (x64, macos-15-intel) (push) Has been cancelled
Details
Release / release-main (push) Has been cancelled
Details
Release / release-develop (push) Has been cancelled
Details
Release / build-linux (push) Has been cancelled
Details

Browse Source 
MeloTTS's single Korean speaker sounded non-native ("foreign accent"). Swap it
for Coqui XTTS-v2 with the built-in female studio speaker "Ana Florence"
(language ko), the natural voice used in earlier local runs.

- bridge/xtts_worker.py: new warm HTTP worker (own /opt/xtts venv), same
  /synth + /health contract and PCM16 output as the old melo worker
- docker/setup-xtts.sh: builds the venv with cu128 torch (Blackwell) + Coqui
  TTS and bakes the XTTS-v2 model offline. Pins transformers>=4.57,<5 (5.x
  removed isin_mps_friendly, breaking XTTS) and installs the [codec] extra
  (torch>=2.9 needs torchcodec) — both verified by a real host synth
- Dockerfile: replace the melo build layer with the xtts layer
- supervisord.conf: melo-worker -> xtts-worker, env passthrough for
  XTTS_DEVICE/SPEAKER/LANGUAGE (always set via compose defaults)
- bridge/server.py: default TTS_ENGINE=xtts, route to the xtts worker, generic
  worker-synth helper, neural-only fallback flag (XTTS_FALLBACK_PIPER)
- settings UI: engine dropdown xtts/piper, drop the dead melo_speed field, fix
  the supervisorctl restart target to xtts-worker
- compose/.env.example/README: XTTS_* vars, speaker/language knobs, remove melo
- remove bridge/melo_worker.py and docker/setup-melo.sh
- tests: xtts treated as multilingual (not English-only)

Verified on host: coqui-tts loads XTTS-v2 and synthesises Korean as
"Ana Florence" to a 16-bit mono 24kHz WAV.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-06-23 03:08:01 +09:00
parent b9f637faa4
commit 39a0944105
11 changed files with 251 additions and 243 deletions
Download Patch File Download Diff File Expand all files Collapse all files

87
bridge/server.py
View File

@@ -87,12 +87,13 @@ VAD_MIN_SPEECH_MS = int(os.environ.get("VAD_MIN_SPEECH_MS", "200"))
# Korean phrase decoded as Chinese) and shaves a little latency. Empty = auto.
STT_LANGUAGE = os.environ.get("STT_LANGUAGE", "ko").strip() or None
# TTS engine: "melo" (MeloTTS Korean speaker, the warm worker) is the primary
# voice; Piper is kept as a fallback if the worker is unreachable. Set
# TTS_ENGINE=piper to disable MeloTTS entirely.
# TTS engine: "xtts" (Coqui XTTS-v2 natural Korean voice, the warm worker) is
# the primary voice; Piper is kept as a fallback only if explicitly enabled. Set
# TTS_ENGINE=piper to disable the neural Korean voice entirely. "melo" is still
# accepted for backward compatibility but is no longer built into the image.
def _tts_engine_setting() -> str:
"""TTS engine: settings-UI value (runtime config JSON) wins, else env, else
melo. Read at startup; the settings UI restarts the bridge on apply."""
xtts. Read at startup; the settings UI restarts the bridge on apply."""
try:
_cp = os.environ.get("JARVIS_CONFIG_PATH", "/app/config/jarvis.json")
_v = json.loads(open(_cp, encoding="utf-8").read()).get("tts_engine")
@@ -100,17 +101,29 @@ def _tts_engine_setting() -> str:
return str(_v).strip().lower()
except Exception:
pass
return os.environ.get("TTS_ENGINE", "melo").strip().lower()
return os.environ.get("TTS_ENGINE", "xtts").strip().lower()
TTS_ENGINE = _tts_engine_setting()
# Coqui XTTS-v2 worker (the natural Korean voice).
XTTS_WORKER_URL = os.environ.get("XTTS_WORKER_URL", "http://127.0.0.1:8771")
XTTS_TIMEOUT = float(os.environ.get("XTTS_TIMEOUT", "30"))
# Legacy MeloTTS worker (no longer built into the image; kept for back-compat
# if someone runs an old worker out-of-band).
MELO_WORKER_URL = os.environ.get("MELO_WORKER_URL", "http://127.0.0.1:8770")
MELO_TIMEOUT = float(os.environ.get("MELO_TIMEOUT", "30"))
# When MeloTTS is the engine, do NOT silently fall back to the English Piper
# voice on failure: speaking Korean text through an English voice produces
# mangled audio. Default is melo-only (return no audio on failure); set
# MELO_FALLBACK_PIPER=1 to opt into the Piper fallback.
MELO_FALLBACK_PIPER = os.environ.get("MELO_FALLBACK_PIPER", "0") in ("1", "true", "True", "yes", "on")
# Do NOT silently fall back to the English Piper voice on a neural-voice failure:
# speaking Korean text through an English voice produces mangled audio. Default
# is neural-only (return no audio on failure); set XTTS_FALLBACK_PIPER=1 (or the
# legacy MELO_FALLBACK_PIPER=1) to opt into the Piper fallback.
def _truthy_env(*names: str) -> bool:
for _n in names:
if os.environ.get(_n, "").strip().lower() in ("1", "true", "yes", "on"):
return True
return False
NEURAL_FALLBACK_PIPER = _truthy_env("XTTS_FALLBACK_PIPER", "MELO_FALLBACK_PIPER")
# ---------------------------------------------------------------------------
# Lazy singletons. The first request pays the model-load cost; afterwards the
@@ -302,27 +315,38 @@ def _coerce_bool(value) -> Optional[bool]:
return str(value).strip().lower() in ("1", "true", "yes", "on")
def _melo_synthesize(text: str) -> Optional[bytes]:
"""Synthesise via the warm MeloTTS worker (separate /opt/melo venv, Korean
speaker @ speed 1.5). Returns a 16-bit PCM WAV, or None on any failure so
the caller can fall back to Piper."""
def _worker_synthesize(name: str, url: str, timeout: float, text: str) -> Optional[bytes]:
"""POST text to a warm TTS worker's /synth and return its WAV bytes, or None
on any failure so the caller can decide whether to fall back."""
import urllib.request
try:
req = urllib.request.Request(
f"{MELO_WORKER_URL}/synth",
f"{url}/synth",
data=json.dumps({"text": text}).encode("utf-8"),
headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(req, timeout=MELO_TIMEOUT) as resp:
with urllib.request.urlopen(req, timeout=timeout) as resp:
if resp.status == 200:
return resp.read()
print(f"[bridge] melo worker HTTP {resp.status}", flush=True)
print(f"[bridge] {name} worker HTTP {resp.status}", flush=True)
except Exception as e: # pragma: no cover - worker may be down
print(f"[bridge] melo worker unreachable: {e}", flush=True)
print(f"[bridge] {name} worker unreachable: {e}", flush=True)
return None
def _xtts_synthesize(text: str) -> Optional[bytes]:
"""Synthesise via the warm Coqui XTTS-v2 worker (separate /opt/xtts venv,
natural female Korean). Returns a 16-bit PCM WAV, or None on failure."""
return _worker_synthesize("xtts", XTTS_WORKER_URL, XTTS_TIMEOUT, text)
def _melo_synthesize(text: str) -> Optional[bytes]:
"""Legacy: synthesise via a MeloTTS worker if one is running out-of-band.
Returns a 16-bit PCM WAV, or None on any failure."""
return _worker_synthesize("melo", MELO_WORKER_URL, MELO_TIMEOUT, text)
def _piper_synthesize(text: str) -> Optional[bytes]:
"""Fallback: synthesise with Piper (English voice). Returns WAV bytes."""
_ensure_piper()
@@ -349,11 +373,12 @@ def _tts_ready() -> bool:
"""
if not TTS_ENABLED:
return True
if TTS_ENGINE == "melo":
_worker_health = {"xtts": XTTS_WORKER_URL, "melo": MELO_WORKER_URL}.get(TTS_ENGINE)
if _worker_health:
import urllib.request
try:
with urllib.request.urlopen(f"{MELO_WORKER_URL}/health", timeout=2) as resp:
with urllib.request.urlopen(f"{_worker_health}/health", timeout=2) as resp:
return resp.status == 200
except Exception:
return False
@@ -361,20 +386,24 @@ def _tts_ready() -> bool:
def synthesize(text: str) -> Optional[bytes]:
"""Synthesize text to a 16-bit PCM WAV. The primary voice is MeloTTS
(Korean speaker, speed 1.5) served by the warm melo worker; Piper is a
fallback if the worker is unavailable. Returns None if TTS is off."""
"""Synthesize text to a 16-bit PCM WAV. The primary voice is Coqui XTTS-v2
(natural female Korean) served by the warm xtts worker; Piper is used only
when explicitly enabled as a fallback. Returns None if TTS is off."""
if not TTS_ENABLED or not text.strip():
return None
if TTS_ENGINE == "melo":
audio = _melo_synthesize(text)
_neural = {"xtts": _xtts_synthesize, "melo": _melo_synthesize}.get(TTS_ENGINE)
if _neural is not None:
audio = _neural(text)
if audio:
return audio
if not MELO_FALLBACK_PIPER:
# Melo-only: better silent than mangled English for Korean text.
print("[bridge] melo synth failed; no audio (Piper fallback disabled)", flush=True)
if not NEURAL_FALLBACK_PIPER:
# Neural-only: better silent than mangled English for Korean text.
print(
f"[bridge] {TTS_ENGINE} synth failed; no audio (Piper fallback disabled)",
flush=True,
)
return None
print("[bridge] melo synth failed; falling back to Piper", flush=True)
print(f"[bridge] {TTS_ENGINE} synth failed; falling back to Piper", flush=True)
return _piper_synthesize(text)

22
bridge/settings_web.py
View File

@@ -22,8 +22,7 @@ from typing import Any, Dict
FIELDS = [
("ollama_chat_model", "LLM 모델", "model"),
("whisper_model", "STT(Whisper) 모델", "select:tiny,base,small,medium,large,large-v3"),
("tts_engine", "TTS 엔진", "select:melo,piper"),
("melo_speed", "TTS 속도 (MeloTTS)", "number:0.5:2.5:0.1"),
("tts_engine", "TTS 엔진", "select:xtts,piper"),
("output_language", "출력 언어 (비우면 사용자 언어)", "text"),
("llm_thinking_enabled", "LLM 사고(thinking) 모드", "bool"),
("agentic_max_turns", "에이전트 최대 반복", "number:1:12:1"),
@@ -54,9 +53,7 @@ def _current() -> Dict[str, Any]:
cfg = _read_config()
out: Dict[str, Any] = {}
for k in _KEYS:
if k == "melo_speed":
out[k] = cfg.get("melo_speed", os.environ.get("MELO_SPEED", "1.5"))
elif k == "output_language":
if k == "output_language":
out[k] = cfg.get("output_language", os.environ.get("OUTPUT_LANGUAGE", ""))
else:
out[k] = cfg.get(k, "")
@@ -78,12 +75,7 @@ def _coerce(updates: Dict[str, Any]) -> Dict[str, Any]:
for k, v in updates.items():
if k not in _KEYS:
continue
if k == "melo_speed":
try:
v = float(v)
except (TypeError, ValueError):
continue
elif k == "agentic_max_turns":
if k == "agentic_max_turns":
try:
v = int(v)
except (TypeError, ValueError):
@@ -114,12 +106,12 @@ def _save(updates: Dict[str, Any]) -> None:
def _apply() -> str:
# Restart melo + bridge AFTER this response is sent. Detached (new session)
# so the bridge being killed mid-restart doesn't drop the restart itself,
# and the HTTP client still receives this response.
# Restart the TTS worker + bridge AFTER this response is sent. Detached (new
# session) so the bridge being killed mid-restart doesn't drop the restart
# itself, and the HTTP client still receives this response.
try:
subprocess.Popen(
["sh", "-c", "sleep 1; supervisorctl restart melo-worker bridge"],
["sh", "-c", "sleep 1; supervisorctl restart xtts-worker bridge"],
start_new_session=True,
)
return "1초 후 브리지/TTS 워커가 재시작되어 반영됩니다."

122
bridge/melo_worker.py → bridge/xtts_worker.py
View File

@@ -1,25 +1,30 @@
"""
MeloTTS worker
==============
XTTS worker
===========
A tiny, dependency-light HTTP service that keeps a MeloTTS voice warm and
synthesises speech on demand. It runs in its OWN Python venv (``/opt/melo`` in
the container) so the heavy MeloTTS/torch/transformers stack stays isolated
from the slim brain-bridge venv (which pins ``numpy<2`` for faster-whisper).
A tiny HTTP service that keeps a Coqui XTTS-v2 voice warm and synthesises
speech on demand. It mirrors ``melo_worker.py`` (same ``/synth`` + ``/health``
contract, same PCM16 WAV output) so the bridge can talk to either worker the
same way.
The bridge's ``synthesize()`` POSTs ``{"text": "..."}`` here and gets back a
16-bit PCM WAV. The MeloTTS model is loaded once at startup and reused, so each
request only pays inference cost, not model-load cost.
XTTS-v2 is a natural, multilingual neural voice. The default speaker is the
built-in female studio voice "Ana Florence" speaking Korean — the voice this
deployment uses in place of MeloTTS. No reference WAV is needed for the
built-in studio speakers.
It runs in its OWN Python venv (``/opt/xtts`` in the container) so the heavy
Coqui TTS / torch stack stays isolated from the slim brain-bridge venv.
Config (env):
MELO_WORKER_HOST bind host (default 127.0.0.1)
MELO_WORKER_PORT bind port (default 8770)
MELO_LANGUAGE MeloTTS language (default KR)
MELO_SPEED speaking rate (default 1.5 -> the approved "150")
MELO_DEVICE torch device (default cpu)
XTTS_WORKER_HOST bind host (default 127.0.0.1)
XTTS_WORKER_PORT bind port (default 8771)
XTTS_MODEL Coqui model id (default tts_models/multilingual/multi-dataset/xtts_v2)
XTTS_SPEAKER built-in speaker (default "Ana Florence")
XTTS_LANGUAGE synthesis language (default ko)
XTTS_DEVICE torch device (default cpu; compose sets cuda)
Run:
/opt/melo/bin/python -m bridge.melo_worker
/opt/xtts/bin/python -m bridge.xtts_worker
"""
from __future__ import annotations
@@ -33,94 +38,72 @@ import threading
import wave
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
HOST = os.environ.get("MELO_WORKER_HOST", "127.0.0.1")
PORT = int(os.environ.get("MELO_WORKER_PORT", "8770"))
LANGUAGE = os.environ.get("MELO_LANGUAGE", "KR")
# XTTS-v2 is gated behind a one-time license prompt; agreeing here keeps the
# load non-interactive in a container. XTTS-v2 is non-commercial (CPML).
os.environ.setdefault("COQUI_TOS_AGREED", "1")
HOST = os.environ.get("XTTS_WORKER_HOST", "127.0.0.1")
PORT = int(os.environ.get("XTTS_WORKER_PORT", "8771"))
MODEL = os.environ.get("XTTS_MODEL", "tts_models/multilingual/multi-dataset/xtts_v2")
SPEAKER = os.environ.get("XTTS_SPEAKER", "Ana Florence")
LANGUAGE = os.environ.get("XTTS_LANGUAGE", "ko")
DEVICE = os.environ.get("XTTS_DEVICE", "cpu")
def _resolve_speed() -> float:
"""Speaking rate: the settings-UI value (runtime config JSON) wins, else the
MELO_SPEED env, else 1.5. Read at startup; the settings UI restarts this
worker on apply so a new value takes effect."""
try:
cp = os.environ.get("JARVIS_CONFIG_PATH", "/app/config/jarvis.json")
v = json.loads(open(cp, encoding="utf-8").read()).get("melo_speed")
if v is not None:
return float(v)
except Exception:
pass
try:
return float(os.environ.get("MELO_SPEED", "1.5"))
except ValueError:
return 1.5
SPEED = _resolve_speed()
DEVICE = os.environ.get("MELO_DEVICE", "cpu")
# Model + speaker id are loaded once, guarded by a lock because MeloTTS
# inference is not guaranteed thread-safe.
# Model is loaded once, guarded by a lock because TTS inference is not
# guaranteed thread-safe.
_model = None
_speaker_id = None
_model_lock = threading.Lock()
_load_error: str | None = None
def _ensure_model() -> None:
global _model, _speaker_id, _load_error
global _model, _load_error
if _model is not None or _load_error is not None:
return
with _model_lock:
if _model is not None or _load_error is not None:
return
try:
from melo.api import TTS # type: ignore
from TTS.api import TTS # type: ignore
model = TTS(language=LANGUAGE, device=DEVICE)
# spk2id is a melo HParams object (dict-like, supports __getitem__,
# __contains__, keys) but NOT .get(). The KR model exposes a single
# 'KR' speaker; fall back to the first id for other languages.
spk_map = model.hps.data.spk2id
keys = list(spk_map.keys())
speaker_id = spk_map[LANGUAGE] if LANGUAGE in spk_map else spk_map[keys[0]]
model = TTS(MODEL).to(DEVICE)
_model = model
_speaker_id = speaker_id
# Warm the GPU once at load: the first CUDA synth pays a one-off
# kernel-init cost (~5s) that would otherwise land on the user's
# first reply. A throwaway synth here moves it to startup. No-op
# cost on CPU.
# Warm once: the first GPU synth pays a one-off kernel-init cost
# that would otherwise land on the user's first reply.
try:
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as _wt:
_wp = _wt.name
model.tts_to_file("워밍업", speaker_id, _wp, speed=SPEED)
model.tts_to_file(
text="워밍업", speaker=SPEAKER, language=LANGUAGE, file_path=_wp
)
try:
os.unlink(_wp)
except OSError:
pass
except Exception as _we: # pragma: no cover
print(f"[melo-worker] warmup synth skipped: {_we}", flush=True)
print(f"[xtts-worker] warmup synth skipped: {_we}", flush=True)
print(
f"[melo-worker] ready (lang={LANGUAGE} speed={SPEED} "
f"device={DEVICE} speakers={list(spk_map.keys())})",
f"[xtts-worker] ready (model={MODEL} speaker={SPEAKER!r} "
f"language={LANGUAGE} device={DEVICE})",
flush=True,
)
except Exception as e: # pragma: no cover - depends on local model files
_load_error = f"{type(e).__name__}: {e}"
print(f"[melo-worker] model load FAILED: {_load_error}", flush=True)
print(f"[xtts-worker] model load FAILED: {_load_error}", flush=True)
def _synthesize(text: str) -> bytes:
"""Synthesise ``text`` to a 16-bit PCM WAV (bytes)."""
_ensure_model()
if _model is None:
raise RuntimeError(_load_error or "melo model unavailable")
# MeloTTS writes to a file via soundfile; render to a container-disk temp
# file (NOT tmpfs), read it back, then drop it.
raise RuntimeError(_load_error or "xtts model unavailable")
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
tmp_path = tmp.name
try:
with _model_lock:
_model.tts_to_file(text, _speaker_id, tmp_path, speed=SPEED)
_model.tts_to_file(
text=text, speaker=SPEAKER, language=LANGUAGE, file_path=tmp_path
)
with open(tmp_path, "rb") as f:
raw = f.read()
finally:
@@ -132,16 +115,15 @@ def _synthesize(text: str) -> bytes:
def _ensure_pcm16_wav(raw: bytes) -> bytes:
"""Guarantee a 16-bit PCM WAV. MeloTTS/soundfile usually emit float WAVs;
the Discord playback path (ffmpeg) tolerates both, but we normalise to
PCM16 so the contract matches the previous Piper output."""
"""Guarantee a 16-bit PCM WAV. Coqui writes float/other WAVs; the Discord
playback path tolerates both, but we normalise to PCM16 so the contract
matches the previous Melo/Piper output (mono, file's own sample rate)."""
try:
with wave.open(io.BytesIO(raw), "rb") as wf:
if wf.getsampwidth() == 2:
return raw # already PCM16
except wave.Error:
pass
# Non-PCM16 (e.g. float) — convert with soundfile if available.
try:
import numpy as np
import soundfile as sf
@@ -159,7 +141,7 @@ def _ensure_pcm16_wav(raw: bytes) -> bytes:
wf.writeframes(pcm)
return buf.getvalue()
except Exception:
return raw # last resort: hand back whatever MeloTTS produced
return raw # last resort: hand back whatever XTTS produced
class _Handler(BaseHTTPRequestHandler):
@@ -212,7 +194,7 @@ def main() -> int:
# Warm the model at startup so the first Discord turn isn't slow.
_ensure_model()
server = ThreadingHTTPServer((HOST, PORT), _Handler)
print(f"[melo-worker] listening on http://{HOST}:{PORT}", flush=True)
print(f"[xtts-worker] listening on http://{HOST}:{PORT}", flush=True)
try:
server.serve_forever()
except KeyboardInterrupt: