tkrmagid/javis_bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 20 Wiki Activity

perf: run MeloTTS on the GPU (cu128 torch) + warm CUDA at startup

Browse Source 
CPU MeloTTS serialised under concurrent load (whisper STT + bot) and blew
voice-reply TTS to 7-8s. Install the Blackwell-verified cu128 torch in the
melo venv, select the GPU via MELO_DEVICE=cuda, and do a throwaway synth at
worker startup so the one-off CUDA kernel-init (~5s) doesn't land on the
user's first reply. Measured: ~0.3s/sentence on GPU vs ~1.2-2.6s on CPU.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-06-14 02:22:36 +09:00
parent 44ebfeafa8
commit 927d59f805
3 changed files with 27 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
docker/setup-melo.sh
View File

@@ -9,8 +9,11 @@
# - It isolates the heavy torch/transformers stack from the slim bridge env,
# which pins numpy<2 for faster-whisper.
#
# torch is pinned to the CPU build: TTS runs on CPU so the GPU stays reserved
# for Ollama + Whisper, and we avoid pulling multi-GB CUDA wheels.
# torch is the CUDA (cu128) build so MeloTTS runs on the GPU alongside Ollama +
# Whisper. CPU synth serialised under concurrent load (whisper STT + bot) and
# blew TTS up to 7-8s per reply; on the GPU a sentence synthesises in ~0.3s.
# cu128 is the Blackwell (sm_120) wheel verified on this host's RTX 5050.
# The worker selects the device via MELO_DEVICE=cuda (compose).
# ============================================================================
set -euxo pipefail
@@ -29,11 +32,11 @@ rm -rf /var/lib/apt/lists/*
python3.11 -m venv /opt/melo
/opt/melo/bin/pip install --no-cache-dir --upgrade pip wheel setuptools
# CPU-only torch first, so MeloTTS's unpinned `torch` dep is already satisfied
# and pip does not pull the CUDA build. Pinned for reproducible rebuilds (these
# are the versions the CPU index resolved when this layer was verified).
/opt/melo/bin/pip install --no-cache-dir torch==2.12.0 torchaudio==2.11.0 \
--index-url https://download.pytorch.org/whl/cpu
# CUDA (cu128) torch first, so MeloTTS's unpinned `torch` dep is already
# satisfied with the GPU build. Pinned to the Blackwell-verified versions
# (2.11.0+cu128) for reproducible rebuilds.
/opt/melo/bin/pip install --no-cache-dir torch==2.11.0+cu128 torchaudio==2.11.0+cu128 \
--index-url https://download.pytorch.org/whl/cu128
# MeloTTS from GitHub. The PyPI sdist is broken (its setup.py reads a
# requirements.txt that is not shipped in the sdist), so install from the repo.