fix(tts): MeloTTS speaks up to 1.25 itself so fast speech sounds natural

Stretching all of the speed-up from 1.0 kept every sound but sounded
awkward (UTMOS 3.13 at 1.5, against 3.73 for MeloTTS's own 1.5), while
MeloTTS's own speed past 1.25 drops sounds (sentence-final "요" and first
syllables, 4.6% character errors at 1.5). MeloTTS now speaks up to 1.25
and the WSOLA stretch does only the rest, with a 5 ms search: 2.6% and
UTMOS 3.56 at 1.5 through the real child. Version 0.1.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-16 00:06:52 +09:00
parent 8a37144cd8
commit 27eb1315b9
8 changed files with 24 additions and 19 deletions

2
app/Cargo.lock generated
View File

@@ -1484,7 +1484,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
[[package]]
name = "jarvis-app"
version = "0.1.5"
version = "0.1.6"
dependencies = [
"async-trait",
"base64 0.22.1",

View File

@@ -1,6 +1,6 @@
[package]
name = "jarvis-app"
version = "0.1.5"
version = "0.1.6"
edition = "2021"
rust-version = "1.85"
description = "Jarvis v2 Windows app core: Discord voice, voice worker supervisor, chat channels"

4
app/shell/Cargo.lock generated
View File

@@ -2380,7 +2380,7 @@ checksum = "3bd1574e617a9a6c1f8f8181bd7e841e1f20e4277607ef28945765188d6284d4"
[[package]]
name = "jarvis-app"
version = "0.1.5"
version = "0.1.6"
dependencies = [
"async-trait",
"base64 0.22.1",
@@ -2408,7 +2408,7 @@ dependencies = [
[[package]]
name = "jarvis-shell"
version = "0.1.5"
version = "0.1.6"
dependencies = [
"jarvis-app",
"serde_json",

View File

@@ -1,6 +1,6 @@
[package]
name = "jarvis-shell"
version = "0.1.5"
version = "0.1.6"
edition = "2021"
rust-version = "1.85"
description = "Jarvis v2 management app window (Tauri 2 over the app core's AdminApi)"

View File

@@ -70,7 +70,7 @@ strings (numbers would lose precision).
| `voice.wakeWords` | `["자비스"]` | Sent to the worker as STT `hotwords`, and the voice wake words of call detection. |
| `voice.stt` | `{}` | Other worker STT settings (`partials`, `end_silence_ms`, `final_beam`, `language`, ...); the worker validates them. |
| `chatChannels` | `[]` | `{id, name, description, read (true), send (true), edit (true), react (true), delete (false), pin (false), answer ("wakeWord"), syncDiscord (true)}` per channel; `answer` is in `conversation.spec.md`, `syncDiscord` (the channel's Discord name and topic follow `name` and `description`) in `admin.spec.md`. |
| `voice.tts.speed` | `1.5` | How fast the voice speaks, 0.5-2.0 (1.0 is MeloTTS's natural pace; above it the worker stretches the speech and keeps its pitch, `voice_worker.spec.md`, "TTS"). Passed to the worker as `--melo-speed` when it speaks (`--melo-python`) and its arguments set no speed. A change saved while the services run goes to the worker at once (`tts_speed`) and applies from the next sentence, without a restart (saved while they start, it goes once they run, since the start may have read the settings before); it is sent again whenever the worker connects again. MeloTTS's Korean model has one voice, so there is no voice choice. |
| `voice.tts.speed` | `1.5` | How fast the voice speaks, 0.5-2.0 (1.0 is MeloTTS's natural pace; MeloTTS speaks up to 1.25 itself and above that the worker stretches the speech and keeps its pitch, `voice_worker.spec.md`, "TTS"). Passed to the worker as `--melo-speed` when it speaks (`--melo-python`) and its arguments set no speed. A change saved while the services run goes to the worker at once (`tts_speed`) and applies from the next sentence, without a restart (saved while they start, it goes once they run, since the start may have read the settings before); it is sent again whenever the worker connects again. MeloTTS's Korean model has one voice, so there is no voice choice. |
| `voiceWorker.python` | `python3` (`python.exe` on Windows) | Python of the worker venv. |
| `voiceWorker.cwd` | `python-src` next to the app's executable when it is there (the Windows build), else the repo's `src/` | Folder holding the `voice_worker` package. |
| `voiceWorker.args` | `[]` | Extra worker arguments, e.g. `["--melo-python", "<MeloTTS venv python>"]`. |

View File

@@ -9,11 +9,13 @@ stdout: frames of u32 LE header length | header JSON | payload. The first frame
int16 LE mono samples at the requested rate, or {"ok": false, "error": "..."} with no payload.
Library output that would land on stdout is sent to stderr so it cannot corrupt the frames.
Speed: MeloTTS's own speed shortens every sound and slurs faster speech (Whisper large-v3 on five
Korean sentences, three renders each: character errors 5.5% at 1.5 and 20% at 2.0, against 1.4% at
1.0). So the model speaks at most at MODEL_SPEED_MAX (its natural pace) and a time stretch that keeps
the pitch (WSOLA) makes it faster: 1.9% at 1.5 and 2.2% at 2.0. Letting the model go to 1.25 first
measured slightly worse (2.2% and 3.1%).
Speed: MeloTTS's own speed past about 1.25 drops sounds (five Korean sentences, three renders each,
read back by Whisper large-v3: at 1.5 sentence-final "요" and first syllables went missing, 4.6%
character errors, 9.6% at 1.75, against 1.2% at 1.0 and none added at 1.25). A time stretch that keeps
the pitch (WSOLA) keeps every sound but sounds less natural the more it stretches (UTMOS 3.13 at 1.5
stretched from 1.0, against 3.73 for the model's own 1.5). So the model speaks up to MODEL_SPEED_MAX
and the stretch does only the rest: 2.6% and UTMOS 3.56 at 1.5, 1.9% and 3.09 at 1.75 (through this
child). A 5 ms search scored higher than 10 ms (UTMOS 3.42 against 3.27 for a 1.2 stretch).
"""
from __future__ import annotations
@@ -26,7 +28,7 @@ import struct
import sys
import types
MODEL_SPEED_MAX = 1.0
MODEL_SPEED_MAX = 1.25
def split_speed(speed: float) -> tuple[float, float]:
@@ -35,7 +37,7 @@ def split_speed(speed: float) -> tuple[float, float]:
return model, speed / model
def stretch(audio, factor: float, rate: int, frame_ms: float = 30.0, search_ms: float = 10.0):
def stretch(audio, factor: float, rate: int, frame_ms: float = 30.0, search_ms: float = 5.0):
"""Shorten ``audio`` by ``factor`` and keep its pitch (WSOLA): half-overlapping Hann frames are
taken ``factor`` times further apart than they are laid down, each shifted within ``search_ms``
to where it best continues the frame before it, so the pieces join without dips or clicks."""

View File

@@ -181,11 +181,14 @@ synthesise is reported with `error` (`code: "tts_failed"`) and the request ends
Speed starts at `--melo-speed` and `tts_speed` changes it for every sentence made after it (a speed
outside 0.5-2.0 or not a number is `bad_message` and the last good one stays). MeloTTS itself speaks
at most at its natural pace (1.0): its own speed shortens every sound and slurs faster speech
(Whisper large-v3 on five Korean sentences, three renders each: 5.5% character errors at 1.5 and 20%
at 2.0, against 1.4% at 1.0). Faster speech is made at 1.0 and shortened with WSOLA, a time stretch
that keeps the pitch (30 ms half-overlapping Hann frames, each shifted within 10 ms to where it best
continues the one before): 1.9% at 1.5 and 2.2% at 2.0, about 15 ms per 4 s of speech.
up to 1.25: its own speed shortens every sound, and past that it drops them (five Korean sentences,
three renders each, read back by Whisper large-v3: at 1.5 sentence-final "요" and first syllables went
missing, 4.6% character errors, 9.6% at 1.75, against 1.2% at 1.0). Faster speech is made at 1.25 and
shortened with WSOLA, a time stretch that keeps the pitch (30 ms half-overlapping Hann frames, each
shifted within 5 ms to where it best continues the one before): 2.6% at 1.5 and 1.9% at 1.75. The
stretch costs naturalness the more it does (UTMOS at 1.5: 3.56, against 3.13 when all of it is
stretched from 1.0), so it does only the part the model cannot; up to 1.25 nothing is stretched. About 15 ms per 4 s
of speech.
### History and brain (M3)

View File

@@ -1,5 +1,5 @@
"""How the MeloTTS child speeds speech up: the model never speaks faster than MODEL_SPEED_MAX (faster
slurs), and the rest is a time stretch that keeps the pitch. And the engine sends its current speed
drops sounds), and the rest is a time stretch that keeps the pitch. And the engine sends its current speed
with every sentence."""
from __future__ import annotations